Weld key point identification method, device, equipment, medium and program product

Abstract

The present invention discloses a method, device, equipment, medium and program product for identifying key points of welds. The method comprises: generating a data set based on training image data and annotation data, constructing an original recognition model, improving the original recognition model, training and optimizing the improved original recognition model based on the data set, and obtaining a target recognition model; acquiring a depth image and an RGB image of a welded part by an image acquisition device on a welding robot; inputting the RGB image into the target recognition model for identifying key points of welds, determining the coordinates of key points of welds of the welded part based on the recognition result and the depth image, and realizing accurate identification of key points of welds, thereby guiding the welding robot to accurately reach the coordinate position of key points of welds for welding, improving the autonomy and adaptability of key point identification, effectively adapting to welding tasks with different requirements, ensuring welding quality, greatly improving welding efficiency and reducing time cost.

Description

Technical Field

[0001] The present invention relates to the field of welding, and in particular to a method, device, equipment, medium and program product for identifying key points of a weld. Background Art

[0002] Metal welding is a common processing method for connecting two or more metal parts into an integral part. Metal processing technology composed of metal machining, casting and heat treatment has become a basic means of production in the automotive, shipbuilding, aerospace, petrochemical and electronics industries. Therefore, robot welding technology plays an increasingly important role in the field of welding. Existing welding robots are mainly teaching and reproducing type and offline programming type. The teaching and reproducing type requires a lot of manual intervention during the welding process, and the accuracy depends entirely on the visual inspection and operating experience of the instructor; the offline programming type robot relies on the 3D model of the workpiece. If there is an error in the model, it will directly affect the accuracy of the welding path and the welding quality, and the operator's programming skills are required to be high. Both cannot adapt to environmental changes and cope with the welding tasks of complex components and non-standard components.

[0003] Accurate identification of key points of welds (starting point, end point, spot welding fixed point and broken weld point, etc.) is a key link in robot intelligent welding. During the welding process, the recognition accuracy of key points will directly affect the accurate positioning of the welding gun and the welding path planning, which in turn has an important impact on the welding quality and production efficiency. Existing key point detection technology relies on manually set scanning paths for key point recognition, which reduces the autonomy and adaptability of recognition, resulting in the inability to efficiently respond to different welding requirements and low welding efficiency. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, equipment, medium and program product for identifying key points of welds, aiming to solve the technical problems that the key point detection technology in the prior art relies on manual assistance, resulting in low recognition autonomy and adaptability, and cannot efficiently respond to different welding requirements, resulting in poor welding quality and low welding efficiency.

[0005] To achieve the above object, the present invention provides a method for identifying key points of a weld, the method comprising the following steps:

[0006] Acquire training image data and annotation data corresponding to the training image data;

[0007] generating a data set based on the training image data and the annotated data;

[0008] Constructing an original recognition model, and improving the original recognition model to obtain an improved original recognition model;

[0009] Based on the data set, the improved original recognition model is trained and optimized to obtain a target recognition model;

[0010] The image acquisition device on the welding robot acquires a depth image and an RGB image of the weldment, wherein the image acquisition device includes a depth camera and an RGB camera;

[0011] The RGB image is input into the target recognition model for weld key point recognition, and the weld key point coordinates of the welded part are determined based on the recognition result and the depth image. The weld key point coordinates are used to guide the welding robot to locate the weld key points on the welded part.

[0012] Optionally, the training image data includes RGB image data; and the acquiring the training image data and the annotation data corresponding to the training image data includes:

[0013] Performing hand-eye calibration on the image acquisition device of the welding robot to obtain a hand-eye calibration matrix, wherein the hand-eye calibration matrix is ​​a coordinate transformation relationship matrix between a camera coordinate system of the image acquisition device and a tool coordinate system of the welding robot;

[0014] Acquire a posture transformation matrix based on the control parameters of the welding robot, wherein the posture transformation matrix is ​​a coordinate transformation relationship matrix between a base coordinate system of the welding robot and the tool coordinate system;

[0015] Based on Zhang Zhengyou's chessboard calibration method, the intrinsic parameter matrix, rotation matrix and translation matrix of each camera in the image acquisition device are obtained to complete the calibration of the image acquisition device, wherein the intrinsic parameter matrix is ​​a coordinate transformation matrix between the camera coordinate system and the pixel coordinate system of the image acquisition device, and the rotation matrix and the translation matrix are coordinate transformation matrices between the camera coordinate system and the world coordinate system of the image acquisition device;

[0016] The welded parts are imaged from multiple angles using the calibrated image acquisition device to obtain RGB image data;

[0017] The key points of the weld are annotated on the RGB image data to obtain annotated data.

[0018] Optionally, the data set includes a training set, a validation set and a test set; and generating the data set based on the training image data and the annotated data includes:

[0019] generating an initial set based on the training image data and the annotated data;

[0020] generating a rotation matrix according to the initial set, and performing a rotation process on the initial set based on the rotation matrix to obtain a first candidate set;

[0021] Determine a flip type, and perform flip processing on the initial set based on the flip type to obtain a second candidate set;

[0022] generating noise information, and adding the noise information to the initial set to obtain a third candidate set;

[0023] adjusting the image contrast of the initial set based on a preset contrast factor to obtain a fourth candidate set;

[0024] Merging the initial set with the first candidate set, the second candidate set, the third candidate set and the fourth candidate set to obtain a data set;

[0025] The data set is divided based on a preset data ratio to obtain a training set, a validation set, and a test set.

[0026] Optionally, inputting the RGB image into the target recognition model to perform weld key point recognition, and determining the weld key point coordinates of the welded part based on the recognition result and the depth image, comprises:

[0027] Inputting the RGB image into the target recognition model to perform weld key point recognition, and obtaining RGB pixel key point coordinates in a pixel coordinate system in the RGB image;

[0028] Obtain a target transformation matrix required for coordinate transformation between the depth camera and the RGB camera, wherein the target transformation matrix includes a target rotation matrix and a target translation matrix;

[0029] Obtain the RGB intrinsic parameter matrix of the RGB camera and the depth intrinsic parameter matrix of the depth camera;

[0030] Acquire a first conversion relationship between a camera coordinate system and a pixel coordinate system under the RGB camera based on the RGB intrinsic parameter matrix;

[0031] Acquire a second conversion relationship between a camera coordinate system and a pixel coordinate system under the depth camera according to the depth intrinsic parameter matrix;

[0032] Acquire depth information corresponding to the RGB pixel key point coordinates based on the target rotation matrix, the first conversion relationship, and the second conversion relationship;

[0033] The key point coordinates of the weld of the welded part are determined according to the depth information, the key point coordinates of the RGB pixel, the hand-eye calibration matrix and the posture transformation matrix.

[0034] Optionally, obtaining a target transformation matrix required for coordinate transformation between the depth camera and the RGB camera includes:

[0035] Obtain a depth conversion matrix between a camera coordinate system and a world coordinate system under the depth camera, wherein the depth conversion matrix includes a depth rotation matrix and a depth translation matrix;

[0036] Obtain an RGB conversion matrix between a camera coordinate system and a world coordinate system under the RGB camera, wherein the RGB conversion matrix includes an RGB rotation matrix and an RGB translation matrix;

[0037] Determine a target rotation matrix required for coordinate conversion between the depth camera and the RGB camera based on the depth rotation matrix and the RGB rotation matrix;

[0038] A target translation matrix required for coordinate conversion between the depth camera and the RGB camera is determined according to the target rotation matrix, the depth translation matrix and the RGB translation matrix.

[0039] Optionally, the original recognition model includes a CBS module, a LAE module, a C2f layer and a plurality of detection head modules; and the improving the original recognition model includes at least one of the following:

[0040] Adding a maximum pooling layer to the CBS module;

[0041] Adding an adaptive hybrid pooling module to the LAE module, wherein the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters;

[0042] The Bottleneck module in the C2f layer is replaced by an EXconv module, wherein the EXconv module is composed of a local feature fusion module, a depth convolution module, a depth separable convolution module, a batch normalization layer module and a residual connection layer;

[0043] A multi-scale spatial attention module is added after the EXconv module of the C2f layer before the multiple detection head modules, wherein the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window dilation attention at different dilation rates, and aggregate semantic information;

[0044] Each detection head module is replaced with a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes a weld key point identification device, the weld key point identification device comprising:

[0046] A data acquisition module, used to obtain training image data and annotation data corresponding to the training image data;

[0047] A data processing module, used to generate a data set based on the training image data and the annotated data;

[0048] A model building module, used to build an original recognition model and improve the original recognition model to obtain an improved original recognition model;

[0049] A training optimization module, used for training and optimizing the improved original recognition model based on the data set to obtain a target recognition model;

[0050] An image acquisition module, used to acquire a depth image and an RGB image of the weldment through an image acquisition device on the welding robot, wherein the image acquisition device includes a depth camera and an RGB camera;

[0051] A key point recognition module is used to input the RGB image into the target recognition model to perform weld key point recognition, and determine the weld key point coordinates of the welded part based on the recognition result and the depth image, and the weld key point coordinates are used to guide the welding robot to locate the weld key points on the welded part.

[0052] In addition, to achieve the above-mentioned purpose, the present application also proposes a weld key point identification device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the weld key point identification method described above.

[0053] In addition, to achieve the above-mentioned purpose, the present application also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the weld key point identification method described above are implemented.

[0054] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the weld key point identification method described above are implemented.

[0055] The present invention obtains training image data and annotation data corresponding to the training image data, generates a data set based on the training image data and the annotation data, constructs an original recognition model, improves the original recognition model, obtains an improved original recognition model, trains and optimizes the improved original recognition model based on the data set, obtains a target recognition model, collects a depth image and an RGB image of the weldment through an image acquisition device on a welding robot, the image acquisition device includes a depth camera and an RGB camera, inputs the RGB image into the target recognition model for weld key point recognition, and determines the weld key point coordinates of the weldment based on the recognition result and the depth image; because the present invention improves the original recognition model, trains and optimizes the improved original recognition model, obtains the target recognition model, inputs the RGB image of the weldment into the target recognition model for weld key point recognition, and then combines the depth image to obtain the coordinates with depth information, realizes accurate recognition of the weld key points, thereby guiding the welding robot to accurately reach the weld key point coordinate position for welding, improves the autonomy and adaptability of key point recognition, effectively adapts to welding tasks with different requirements, ensures welding quality, and greatly improves welding efficiency and reduces time cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 It is a structural schematic diagram of a weld key point identification device in a hardware operating environment involved in an embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of the flow chart of the first embodiment of the method for identifying key points of a weld according to the present invention;

[0060] Figure 3 It is a schematic diagram of the network structure of the improved original recognition model in the first embodiment of the weld key point recognition method of the present invention;

[0061] Figure 4 It is a schematic diagram of the composition structure of each module in the improved original recognition model in the first embodiment of the weld key point recognition method of the present invention;

[0062] Figure 5It is a flow chart of the second embodiment of the method for identifying key points of a weld according to the present invention;

[0063] Figure 6 This is a structural block diagram of the first embodiment of the weld key point identification device of the present invention.

[0064] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0066] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a weld key point identification device in the hardware operating environment involved in the embodiment of the present invention.

[0067] like Figure 1 As shown, the weld key point identification device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (RandomAccess Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0068] Those skilled in the art will understand that Figure 1 The structure shown in does not constitute a limitation on the weld critical point identification device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0069] like Figure 1 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a weld key point recognition program.

[0070] exist Figure 1In the weld key point identification device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the weld key point identification device of the present invention can be set in the weld key point identification device, and the weld key point identification device calls the weld key point identification program stored in the memory 1005 through the processor 1001, and executes the weld key point identification method provided by the embodiment of the present invention.

[0071] The embodiment of the present invention provides a method for identifying key points of a weld. Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the method for identifying key points of a weld according to the present invention.

[0072] In this embodiment, the weld key point identification method comprises the following steps:

[0073] Step S10: Acquire training image data and annotation data corresponding to the training image data.

[0074] It should be understood that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a terminal electronic device capable of realizing the above functions, etc. The following takes a weld key point recognition device (referred to as recognition device) as an example to illustrate this embodiment and the following embodiments.

[0075] It should be noted that the training image data may be image data of welded parts acquired by an image acquisition device on a welding robot, and the training image data may include RGB images. The above-mentioned annotation data may be data obtained after the recognition device annotates the key points of the weld on the training image data.

[0076] It is understandable that the recognition device can acquire images of the weldment from multiple angles through the image acquisition device on the welding robot to obtain training image data, and then annotate the key points of the weld on the training image data to obtain annotated data.

[0077] In some embodiments, before collecting image data, the recognition device may perform hand-eye calibration on the image acquisition device on the welding robot, and obtain training image data through the calibrated image acquisition device.

[0078] In some embodiments, the welding robot may be a welding robot arm or other types of robots for welding. The above-mentioned image acquisition device may be an RGB-D camera (i.e., an RGB-Depth camera), and the image acquisition device may include an RGB camera and a depth camera, and the image acquisition device may acquire RGB image data and depth image data.

[0079] Furthermore, in order to improve the key point recognition and positioning accuracy, the above step S10 may include:

[0080] Step S101: Perform hand-eye calibration on the image acquisition device of the welding robot to obtain a hand-eye calibration matrix.

[0081] It should be noted that the hand-eye calibration matrix is ​​a coordinate transformation relationship matrix between the camera coordinate system of the image acquisition device and the tool coordinate system of the welding robot.

[0082] It can be understood that the recognition device performs hand-eye calibration on the RGB-D camera fixed on the robot arm to obtain the coordinate transformation relationship between the camera coordinate system (CCS) and the tool coordinate system (TCS), that is, the hand-eye calibration matrix. By accurately aligning the visual information obtained by the image acquisition device with the coordinate system of the welding robot, the positioning accuracy of the robot arm on the weld or welding point can be greatly improved.

[0083] Step S102: Acquire a posture transformation matrix based on the control parameters of the welding robot.

[0084] It should be noted that the posture transformation matrix is ​​a coordinate transformation relationship matrix between the base coordinate system of the welding robot and the tool coordinate system.

[0085] In the specific implementation, the recognition device reads the posture transformation matrix in the robot controller parameters in real time , to represent the coordinate transformation relationship between the tool coordinate system and the robot base coordinate system (BCS).

[0086] Step S103: Based on Zhang Zhengyou's chessboard calibration method, the intrinsic parameter matrix, rotation matrix and translation matrix of each camera in the image acquisition device are obtained to complete the calibration of the image acquisition device.

[0087] It should be noted that the intrinsic parameter matrix is ​​the coordinate transformation matrix between the camera coordinate system and the pixel coordinate system of the image acquisition device, and the rotation matrix and the translation matrix are the coordinate transformation matrix between the camera coordinate system and the world coordinate system of the image acquisition device.

[0088] It should be noted that in this embodiment, the Zhang Zhengyou chessboard calibration method can be used to calibrate the image acquisition device of the multi-camera system. The Zhang Zhengyou chessboard calibration method is used to determine the internal parameters (i.e., the internal parameter matrix) of the image acquisition device and the external parameters (i.e., the external parameter matrix) when shooting at different positions and angles by using a chessboard pattern of known geometric shapes.

[0089] In a specific implementation, the recognition device can use Zhang Zhengyou's checkerboard calibration method to capture multiple images of the calibration plate from different angles and distances so as to cover the entire field of view of the camera, and obtain the intrinsic parameter matrix K of the RGB camera and depth camera, the camera extrinsic parameter rotation matrix R and translation matrix T based on the captured images.

[0090] Step S104: Capture images of the weldment from multiple angles using the calibrated image acquisition device to obtain RGB image data.

[0091] It should be noted that the recognition is carried out by taking RGB images and depth images of self-made complex three-dimensional welded parts from multiple different angles through calibrated image acquisition equipment, using RGB images to train the model, and using depth images to assign depth values ​​to the coordinates output by the model, thereby obtaining the three-dimensional coordinates of the key points of the weld.

[0092] Step S105: annotating the key points of the weld on the RGB image data to obtain annotated data.

[0093] In some embodiments, the recognition device can use a labeling tool to label the key points of the weld on the RGB image data, for example, use Labelme to label the key points of the RGB image. Specifically, the RGB image folder is input into the Labelme graphic image annotation tool. First, the actual welding key point position is marked with a matrix box to mark the approximate range, and the label name is "fillet joint"; then the key point in the box is marked with a target point for the specific position, and the label name is "1".

[0094] Step S20: Generate a data set based on the training image data and the labeled data.

[0095] It should be noted that a dataset can be a collection of data used to train, validate, test, and optimize a model.

[0096] In some embodiments, the recognition device may generate an initial set based on the training image data and the annotated data, and then preprocess the initial set to eliminate invalid data, enhance the data, improve training efficiency, and obtain a data set.

[0097] Furthermore, in order to improve data quality and ensure model performance, the above step S20 may include:

[0098] Step S201: generating an initial set based on the training image data and the annotated data;

[0099] Step S202: generating a rotation matrix according to the initial set, and performing a rotation process on the initial set based on the rotation matrix to obtain a first candidate set;

[0100] Step S203: determining a flip type, and performing flip processing on the initial set based on the flip type to obtain a second candidate set;

[0101] Step S204: generating noise information, and adding the noise information to the initial set to obtain a third candidate set;

[0102] Step S205: adjusting the image contrast of the initial set based on a preset contrast factor to obtain a fourth candidate set;

[0103] Step S206: fusing the initial set with the first candidate set, the second candidate set, the third candidate set and the fourth candidate set to obtain a data set;

[0104] Step S207: Divide the data set based on a preset data ratio to obtain a training set, a validation set, and a test set.

[0105] In some embodiments, in order to improve the quality of training data, improve training efficiency, and expand the data set, data enhancement technology is used to perform enhancement operations such as rotation, flipping, and adding noise on the original data set, and all the images and corresponding labels finally obtained are divided into training set, verification set, and test set in a ratio of 7:2:1.

[0106] For example, the recognition device may perform contrast adjustment, image flipping, adding noise information, image rotation and other processing operations on the original training image data to enhance the training image data. The above-mentioned image flipping may include horizontal flipping, vertical flipping, etc. The above-mentioned image rotation may be to rotate the image around its center point by a certain angle to generate a new image, thereby helping the model learn features from different perspectives. The above-mentioned noise information may be salt and pepper noise and Gaussian noise, etc.

[0107] Step S30: constructing an original recognition model, and improving the original recognition model to obtain an improved original recognition model.

[0108] In some embodiments, the original recognition model may be a Yolo model (eg, a Yolo V8 model) or another type of target detection model.

[0109] In some embodiments, the recognition device may improve the original recognition model by, but is not limited to:

[0110] Improve the Yolov8 feature fusion module C2f, use the EXconv module to replace the Bottleneck module in the C2f layer to obtain a lightweight C2f_EXconv module, which greatly reduces the number of module parameters without reducing model performance;

[0111] Improve the attention mechanism of the LAE module, introduce a mixed pooling layer, fuse adaptive parameters, and dynamically balance the attention weights generated by average pooling and maximum pooling. This enables the model to automatically learn how to optimally fuse the attention information generated by these two pooling methods during training, thereby obtaining more accurate feature representation.

[0112] Furthermore, in order to significantly reduce the amount of module parameters without reducing model performance, the improvement of the original recognition model includes at least one of the following:

[0113] Step S31: adding a maximum pooling layer to the CBS module.

[0114] It should be noted that, refer to Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of the network structure of the improved original recognition model. Figure 4 This is a schematic diagram of the composition structure of each module in the improved original recognition model. The recognition device inputs the image data into the improved original recognition model, and the image data passes through the input layer, CBSM module, LAE-A module, C2f_EXconv module and Pose_LSCD detection head.

[0115] Among them, the CBSM module is the feature extraction layer, which includes the CBS module and the maximum pooling layer (MaxPool). The CBS module includes the convolution layer (Conv), batch normalization layer (BN) and activation function layer (SiLu). The CBSM module extracts feature maps through convolution, batch normalization, activation function and spatial maximum pooling.

[0116] The LAE-A module can be a lightweight feature extraction layer with adaptive hybrid pooling. The LAE-A module generates enhanced feature maps through multi-pooling branches, weight learning and weighted fusion.

[0117] The C2f_EXconv module can be a lightweight feature fusion layer. The C2f_EXconv module enhances the expressiveness and receptive field of the feature map through the C2f module and the extended convolution. The C2f_EXconv module can achieve feature enhancement through the CBS module, the deep convolution layer (DepthConv), the CBS module, the DBR module, the convolution layer (Conv), the batch normalization layer (BN) and the channel concatenation (Concat). The DBR module includes the deep convolution layer (DepthConv), the batch normalization layer (BN) and the activation function layer (ReLU6).

[0118] The Pose_LSCD detection head can be a lightweight shared convolution detection head. The Pose_LSCD detection head extracts features related to human key points from the multi-scale feature map, generates the precise coordinates of each key point, and generates a confidence score for each key point.

[0119] It is understandable that after the recognition device inputs the image into the initial convolutional module CBS layer, it adds a maximum pooling layer (MaxPooling) to form a CBSM (CBS+MaxPooling) module. The maximum pooling layer increases the translation invariance of the model by extracting significant features.

[0120] Step S32: Add an adaptive hybrid pooling module to the LAE module, wherein the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters.

[0121] It is understandable that, referring to Figure 4 ,The recognition device improves the LAE module in the original recognition model, introduces an adaptive hybrid pooling module, and adds a learnable parameter x to form a lightweight feature extraction module LAE-A based on adaptive hybrid pooling, so that the model can obtain more contextual information and details from the multi-scale feature map, and extract more detailed features while reducing the influence of feature map noise.

[0122] In the specific implementation, in order to lightweight the model and enhance the model feature extraction capability, the maximum pooling and average pooling operations with a kernel of 3x3, a stride of 1, and a padding of 1 are introduced in the upstream branch of the LAE-A block to combine the smoothing characteristics of average pooling and the significant feature enhancement capability of maximum pooling. The obtained feature map matrix is ​​fused with features through a CBS module with a kernel size of 1x1, so that the network can learn more complex feature combinations. Then, the softmax function is used to calculate the attention weights in each channel dimension to ensure that each feature point receives corresponding attention according to its importance. Subsequently, the fifth dimension s is added to the obtained feature matrix through the rearrange function, and the original dimensions (batch size (Batchsize, bs), number of channels (Channel, ch), height (Height, h), width (Weight, w)) are rearranged to (bs, ch, h / 2, w / 2, s), that is, the arrangement is rearranged from (bs, ch, h, w) to (bs, ch, h / 2, w / 2, 4); then a learnable parameter x is introduced to dynamically balance the attention weights generated by average pooling and maximum pooling during training, so that the model can automatically learn how to optimally fuse the attention information of the two pooling methods to obtain more accurate feature representation.

[0123] In the downstream branch, a convolutional layer (Conv) module with a kernel size of 1x1 is used to expand the number of channels to 4 times the original number to increase the representation capability of the model. Then, it is input into a depth convolution (DepthConv) module with a kernel size of 3x3 and a step size of 2 to further reduce the spatial dimension of the feature map and extract deeper features. Finally, the dimension of the feature map is reconstructed into a five-dimensional structure that matches the upstream branch through the rearrange function. The results are multiplied and added in the fifth dimension to restore the original dimensional representation. This improvement improves the model performance while maintaining low computational complexity, so that it can better meet the stringent requirements of intelligent welding for rapid positioning of key points of welds, thereby showing better real-time performance and accuracy in practical applications.

[0124] It can be understood that this embodiment improves the attention mechanism of the LAE module, introduces a hybrid pooling layer, fuses adaptive parameters, and dynamically balances the attention weights generated by average pooling and maximum pooling. This enables the model to automatically learn how to optimally fuse the attention information generated by the two pooling methods during training, thereby obtaining a more accurate feature representation.

[0125] Step S33: The Bottleneck layer is replaced by an EXconv module, where the EXconv module consists of a local feature fusion module, a depth convolution module, a depth separable convolution module, a batch normalization layer module and a residual connection layer.

[0126] It should be noted that, refer to Figure 4 , the EXconv module may be a C2f_EXconv module. In this embodiment, an improved convolutional feature extraction module EXconv is introduced to replace the original Bottleneck layer, and a lightweight C2f_EXconv module is constructed by combining the cross-stage local feature fusion module C2f;

[0127] In the specific implementation, the EXconv module combines the advantages of the deep convolution (DepthConv) and the deep separable convolution modules, and cleverly adds a residual connection mechanism. Before entering the deep separable convolution, a deep convolution layer is used to perform preliminary feature extraction on the input data. Since deep convolution extracts features channel by channel, it can provide richer feature input for subsequent deep separable convolution. Next, a deep separable convolution module is used to significantly reduce the amount of computation and model size while maintaining good performance. The process includes using a CBS module with a kernel size of 1x1 to expand the number of channels, and then passing the result through a DBR module consisting of deep convolution, batch normalization, and ReLU6 activation function to extract features at the channel dimension level. Finally, the features are fused through a Conv with a kernel size of 1x1 and a batch normalization layer module to reduce the number of channels to the original size.

[0128] At the same time, a residual connection mechanism is added to the EXconv layer. In the backbone stage of the model, this mechanism achieves feature reuse and enhancement by directly adding the input feature map to the output of the depthwise separable convolution module. This design not only helps to reduce the risk of gradient explosion and gradient vanishing, but also improves the training stability and final performance of the model. In the Neck part, the EXconv module directly outputs the result without residual connection. Because the main task of the Neck stage is to integrate and refine the features from the backbone to provide accurate feature representation for the final classification or regression task. At this time, residual connection is not necessary, because too much feature reuse may introduce redundant information, thereby affecting the generalization ability of the model.

[0129] Step S34: Add a multi-scale spatial attention module after the EXconv module of the C2f layer before the multiple detection head modules, and the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window dilation attention with different dilation rates, and aggregate semantic information.

[0130] It should be noted that, refer to Figure 4 , after the output of the large object detection layer, that is, the output of the last C2f_EXconv module, a multi-scale dilated attention mechanism (MSDA) is added, which simultaneously generates the corresponding query Q, key K and value V branches of the feature map X through a single convolutional layer. Subsequently, the channels of the input feature map are divided into 8 different heads, and multi-scale sliding window dilated attention (SWDA) is performed in different heads with different dilation rates to improve the processing efficiency and detection accuracy of the model, and aggregate semantic information at various scales within the attended receptive field;

[0131] In the specific implementation, an MSDA module is added after the last large-scale C2f_EXconv layer. Compared with the expansion rate set in the original module [1, 2, 3], this model chooses to apply an MSDA to the large feature map detection layer, so that it can capture the features of the image at a larger scale. Therefore, the expansion rate is set to [1, 2, 3, 4], and the corresponding receptive field sizes are 3x3, 5x5, 7x7 and 9x9 respectively. The MSDA module generates the corresponding Q, K and V from the input feature map through a convolutional layer at the same time, and extracts different slices Qi, Ki and Vi according to the defined 4 expansion rates. Since 8 heads and 4 expansion rates are defined, each expansion rate corresponds to 2 heads. Then, the SWDA mechanism with a specific expansion rate is applied to each slice, and the attention weights are calculated for the corresponding heads to generate the corresponding output hi. Finally, all outputs are spliced, and feature fusion and refinement are performed through an additional linear layer to form the final aggregated feature representation. The designed MSDA mechanism not only significantly improves the model's ability to parse complex scenes, but also further enhances its performance in multi-scale object detection tasks.

[0132] Step S35: Replace each detection head module with a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer.

[0133] It should be noted that, refer to Figure 4 , replace the original Pose detection head with the shared convolution detection head Pose_LSCD to build a lightweight detection head, and coordinate the use of network resources to avoid resource overload or idleness.

[0134] Specifically, in the detection head module, two shared convolutional layers are used to efficiently extract and utilize the shared features of the input feature maps. This helps to improve the feature representation ability of the model, thereby improving the detection accuracy. First, all feature maps input to the Pose_LSCD detection head are passed through a convolutional layer, and the number of channels is increased to 256 using a 1x1 convolution kernel; then they are input into two shared convolutional layers. Since the same convolutional layer is used, the model parameters can be reduced while information can be fused between different feature maps. Since each feature map may contain features of different levels (for example, low-level edge and texture information, and high-level semantic information), through shared convolutional layers, the model can learn how to combine these different levels of features to produce a richer representation; then, regression output and category output are generated through their respective convolutional layers; finally, the regression output is scaled by an adaptive scaling factor and concatenated with the category output result to return the result.

[0135] In some embodiments, considering the requirements of intelligent robot welding, aiming at the problem of key point identification and positioning of welds in complex three-dimensional structures, in order to facilitate real-time identification and algorithm model deployment, the recognition device improves the YoloV8 model to obtain a lightweight YoloV8 model, reducing the number of parameters by 50.5%, with a weight file of only 3.3MB, and the mAP value remains basically unchanged. On the basis of ensuring the lightweight of the network model and the real-time performance of the algorithm, the one-time identification of the key points of the weld is achieved, and the improved YoloV8 model is compared with the original YoloV8 model. Refer to Table 1, which is a parameter table comparing the various indicators of the original recognition model before improvement and the improved original recognition model:

[0136] Table 1 Index comparison parameter table

[0137]

[0138] Step S40: training and optimizing the improved original recognition model based on the data set to obtain a target recognition model.

[0139] It should be noted that the recognition device can divide the data set according to a preset ratio, for example, all the images and corresponding labels finally obtained can be divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0140] It can be understood that the recognition device trains the improved original recognition model through the training set, and then uses the verification set to tune the hyperparameters and select the best model data set during the training process, and uses the test set to evaluate the performance of the trained and verified model, and optimizes the model based on the performance evaluation results to ensure that the model output results meet the requirements.

[0141] Step S50: Capture a depth image and an RGB image of the weldment by using an image acquisition device on the welding robot.

[0142] It should be noted that the image acquisition device includes a depth camera and an RGB camera.

[0143] In some embodiments, the welding robot may be a welding robot arm or other types of robots for welding. The above-mentioned image acquisition device may be an RGB-D camera (i.e., an RGB-Depth camera), and the image acquisition device may include an RGB camera and a depth camera, and the image acquisition device may acquire RGB image data and depth image data.

[0144] In some embodiments, corresponding to the same key point, the coordinates of the key point under the depth camera are different from the coordinates of the key point under the RGB camera. Therefore, the two are connected based on the relationship conversion matrix between the depth camera and the RGB camera, so as to assign depth information to the pixel points in the RGB image.

[0145] Step S60: inputting the RGB image into the target recognition model to perform weld key point recognition, and determining the weld key point coordinates of the welded part based on the recognition result and the depth image.

[0146] It should be noted that the weld key point coordinates are used to guide the welding robot to locate the weld key points on the welded part. The weld key points may include a starting point, an end point, a spot welding fixed point, and a broken weld point.

[0147] It can be understood that this embodiment accurately identifies the starting point and the end point of the weld, thereby accurately planning the welding path to ensure that the welding process ends at the predetermined position, avoiding welding defects caused by too long or too short welding, such as incomplete penetration and weld bumps, thereby improving the overall quality of the welded joint and achieving precise control of automated welding. The precise identification of spot welds and broken welds provides more reliable guidance for the precise planning of the welding path.

[0148] It should be noted that the key point coordinates of the weld may be the key point coordinates under the base coordinate system (BCS) of the welding robot.

[0149] In some embodiments, the output of the target recognition model can be the pixel coordinates of the weld key points in the RGB image in the pixel coordinate system. In order to accurately guide and control the welding robot to move to the position of the weld key points, the recognition device can configure depth information for the pixel coordinates of the weld key points in combination with the depth image, and perform a coordinate system conversion to convert the coordinates in the pixel coordinate system into the coordinates of the weld key points in the base coordinate system of the welding robot, thereby realizing the conversion between two-dimensional coordinates and three-dimensional coordinates and accurately locating the weld key points.

[0150] In some embodiments, the recognition device converts the optimal weight file corresponding to the trained target recognition model into ONNX format and deploys it on the welding robot. It controls the image acquisition device on the welding robot to take pictures of the welded parts in real time, and passes the corresponding RGB image and depth image into the model. After the pixel coordinates are obtained by real-time reasoning, the three-dimensional space point coordinates in the base coordinate system are obtained according to the conversion relationship. Finally, the welding robot is guided to the key point position according to the obtained three-dimensional coordinates.

[0151] In this embodiment, training image data and annotation data corresponding to the training image data are acquired, a data set is generated based on the training image data and the annotation data, an original recognition model is constructed, and the original recognition model is improved to obtain an improved original recognition model. The improved original recognition model is trained and optimized based on the data set to obtain a target recognition model. The depth image and RGB image of the weldment are acquired by an image acquisition device on the welding robot. The image acquisition device includes a depth camera and an RGB camera. The RGB image is input into the target recognition model for weld key point recognition, and the weld key point coordinates of the weldment are determined based on the recognition result and the depth image. Since the present embodiment improves the original recognition model, trains and optimizes the improved original recognition model to obtain a target recognition model, the RGB image of the weldment is input into the target recognition model for weld key point recognition, and the coordinates with depth information are obtained in combination with the depth image, so as to realize accurate recognition of weld key points, thereby guiding the welding robot to accurately reach the weld key point coordinate position for welding, thereby improving the autonomy and adaptability of key point recognition, effectively adapting to welding tasks with different requirements, ensuring welding quality, and greatly improving welding efficiency and reducing time cost.

[0152] refer to Figure 5 , Figure 5 Schematic diagram of the flow chart of the second embodiment of the method for identifying key points of a weld according to the present invention.

[0153] Based on the above first embodiment, in this embodiment, step S60 further includes:

[0154] Step S601: input the RGB image into the target recognition model to perform weld key point recognition, and obtain the RGB pixel key point coordinates in the pixel coordinate system of the RGB image.

[0155] It should be noted that the RGB pixel key point coordinates may be the two-dimensional coordinates of the weld key points marked in the RGB image in the pixel coordinate system.

[0156] Step S602: Obtain a target transformation matrix required for coordinate transformation between the depth camera and the RGB camera.

[0157] It should be noted that the target transformation matrix includes a target rotation matrix and a target translation matrix.

[0158] It can be understood that the recognition device can rotate and translate the coordinates of the CCS under the RGB camera based on the target rotation matrix and the target translation matrix, thereby converting the coordinates of the CCS under the depth camera.

[0159] It should be noted that, since the coordinates of CCS under the depth camera are different from those under the RGB camera, the rotation matrix and translation matrix between the depth camera and the RGB camera can be used to connect the two.

[0160] Further, in order to accurately obtain the conversion matrix between the depth camera and the RGB camera, the above step S602 may include:

[0161] Step S6021: Obtain a depth conversion matrix between the camera coordinate system and the world coordinate system under the depth camera;

[0162] Step S6022: Obtain an RGB conversion matrix between the camera coordinate system and the world coordinate system under the RGB camera;

[0163] Step S6023: Determine a target rotation matrix required for coordinate conversion between the depth camera and the RGB camera based on the depth rotation matrix and the RGB rotation matrix;

[0164] Step S6024: Determine a target translation matrix required for coordinate transformation between the depth camera and the RGB camera according to the target rotation matrix, the depth translation matrix and the RGB translation matrix.

[0165] It should be noted that the depth conversion matrix includes a depth rotation matrix and a depth translation matrix, and the RGB conversion matrix includes an RGB rotation matrix and an RGB translation matrix.

[0166] It is understandable that for the same point, the coordinates of the point in the world coordinate system (World Coordinate System, WCS) are unchanged. According to the principle of camera imaging, it can be known that the point of the image in CCS ( or ) and the WCS point There is a conversion relationship, refer to the following formula:

[0167]

[0168]

[0169] in and The RGB rotation matrix and RGB translation matrix required to convert the CCS coordinates under the RGB camera to the WCS; and For the depth rotation matrix and depth translation matrix required to convert the CCS coordinates under the depth camera to the WCS coordinate system, refer to the following formula, from which the target rotation matrix R and target translation matrix T can be obtained:

[0170]

[0171]

[0172]

[0173] Step S603: Obtain the RGB intrinsic parameter matrix of the RGB camera and the depth intrinsic parameter matrix of the depth camera.

[0174] It should be noted that the RGB intrinsic parameter matrix may be a matrix describing the internal geometric characteristics of the RGB camera, and may include parameters such as the focal length, principal point offset, and pixel scale factor of the camera. The above-mentioned depth intrinsic parameter matrix may be the intrinsic parameter matrix of the depth camera, and the intrinsic parameter matrix may include the focal length of the camera and the coordinates of the origin of the image coordinate system in the pixel coordinate system, which is generally located at the center of the image.

[0175] Step S604: acquiring a first conversion relationship between a camera coordinate system and a pixel coordinate system under the RGB camera based on the RGB intrinsic parameter matrix.

[0176] It should be noted that the first conversion relationship can be a conversion relationship for converting the same key point in the RGB image from the pixel coordinate system to the camera coordinate system. It is the spatial coordinate of the same point under the RGB camera under CCS. is the projection coordinate of the corresponding pixel plane, The first conversion relationship can be obtained by referring to the following formula for the internal parameter matrix of the RGB camera:

[0177]

[0178] Step S605: obtaining a second conversion relationship between a camera coordinate system and a pixel coordinate system under the depth camera according to the depth intrinsic parameter matrix.

[0179] It should be noted that the second transformation relationship can be a transformation relationship for transforming the same key point in the depth image from the pixel coordinate system to the camera coordinate system. is the spatial coordinate of a point in the depth map under CCS, is the projection coordinate of the point on the pixel plane, is the internal parameter matrix of the depth camera, which is obtained by referring to the following formula based on the pinhole imaging principle:

[0180]

[0181]

[0182] in, and is the focal length of the camera in pixels, and It is the coordinate of the origin of the image coordinate system in the pixel coordinate system, which is usually located at the center of the image.

[0183] Step S606: acquiring depth information corresponding to the RGB pixel key point coordinates based on the target rotation matrix, the first conversion relationship, and the second conversion relationship.

[0184] It should be noted that the depth information may be a depth value assigned to the coordinates of the RGB pixel key points.

[0185] It is understandable that due to the coordinates of CCS under the depth camera And the coordinates of CCS under RGB camera Different, but the rotation matrix between the two cameras can be used and translation matrix The two can be connected; then the internal reference matrix under the RGB camera can be used Can be realized as each pixel of the color image Assign depth value according to the following formula:

[0186]

[0187] Step S607: determining the coordinates of the weld key points of the welded part according to the depth information, the RGB pixel key point coordinates, the hand-eye calibration matrix and the posture transformation matrix.

[0188] It can be understood that this embodiment assigns depth information to the RGB pixel key point coordinates based on the target transformation matrix, thereby realizing the registration of the depth map and the color map captured by the image acquisition device, that is, attaching corresponding depth information to each pixel on the color map. and the pose transformation matrix , realize the pixel coordinates of the key points of the weld to the three-dimensional coordinates of the robot base coordinate system, refer to the following formula to obtain the coordinates of the key points of the weld :

[0189]

[0190] In this embodiment, the RGB image is input into the target recognition model for weld key point recognition, and the RGB pixel key point coordinates in the pixel coordinate system in the RGB image are obtained; the target conversion matrix required for coordinate conversion between the depth camera and the RGB camera is obtained, and the target conversion matrix includes a target rotation matrix and a target translation matrix; the RGB intrinsic reference matrix of the RGB camera and the depth intrinsic reference matrix of the depth camera are obtained; the first conversion relationship between the camera coordinate system and the pixel coordinate system under the RGB camera is obtained based on the RGB intrinsic reference matrix; the second conversion relationship between the camera coordinate system and the pixel coordinate system under the depth camera is obtained according to the depth intrinsic reference matrix; the depth information corresponding to the RGB pixel key point coordinates is obtained based on the target rotation matrix, the first conversion relationship and the second conversion relationship; the weld key point coordinates of the welded part are determined according to the depth information, the RGB pixel key point coordinates, the hand-eye calibration matrix and the posture transformation matrix, so as to realize the conversion of the two-dimensional coordinates output by the target recognition model into three-dimensional coordinates, provide accurate coordinate positions for the movement and welding of the welding robot, effectively improve the accuracy of welding path planning, and improve welding precision and welding efficiency.

[0191] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a weld key point recognition program is stored. When the weld key point recognition program is executed by a processor, the steps of the weld key point recognition method described above are implemented.

[0192] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0193] The computer-readable storage medium may be included in the weld key point identification device; or may exist independently without being assembled into the weld key point identification device.

[0194] In addition, an embodiment of the present invention further provides a computer program product, including a weld key point recognition program, which implements the steps of the weld key point recognition method described above when executed by a processor.

[0195] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the above-mentioned weld key point identification method, and will not be repeated here.

[0196] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the weld key point identification device of the present invention.

[0197] like Figure 6 As shown, the weld key point identification device proposed in the embodiment of the present invention includes:

[0198] A data acquisition module 10 is used to obtain training image data and annotation data corresponding to the training image data;

[0199] A data processing module 20, configured to generate a data set based on the training image data and the annotated data;

[0200] The model building module 30 is used to build an original recognition model and improve the original recognition model to obtain an improved original recognition model;

[0201] A training optimization module 40, configured to train and optimize the improved original recognition model based on the data set to obtain a target recognition model;

[0202] An image acquisition module 50 is used to acquire a depth image and an RGB image of the weldment through an image acquisition device on the welding robot, wherein the image acquisition device includes a depth camera and an RGB camera;

[0203] The key point recognition module 60 is used to input the RGB image into the target recognition model to perform weld key point recognition, and determine the weld key point coordinates of the welded part based on the recognition result and the depth image, and the weld key point coordinates are used to guide the welding robot to locate the weld key points on the welded part.

[0204] Furthermore, the training image data includes RGB image data; the data acquisition module 10 is also used to perform hand-eye calibration on the image acquisition device of the welding robot to obtain a hand-eye calibration matrix, wherein the hand-eye calibration matrix is ​​a coordinate transformation relationship matrix between a camera coordinate system of the image acquisition device and a tool coordinate system of the welding robot; a posture transformation matrix is ​​obtained based on control parameters of the welding robot, wherein the posture transformation matrix is ​​a coordinate transformation relationship matrix between a base coordinate system of the welding robot and the tool coordinate system; the image acquisition device is calibrated according to the hand-eye calibration matrix and the posture transformation matrix; images of the weldment are acquired from multiple angles by the calibrated image acquisition device to obtain RGB image data; key points of the weld are annotated on the RGB image data to obtain annotated data.

[0205] Furthermore, the data set includes a training set, a validation set and a test set; the data processing module 20 is also used to generate an initial set based on the training image data and the annotated data; generate a rotation matrix according to the initial set, and rotate the initial set based on the rotation matrix; determine a flipping type, and flip the rotated initial set based on the flipping type to obtain a candidate set; generate noise information, and add the noise information to the candidate set to obtain a data set; divide the data set based on a preset data ratio to obtain a training set, a validation set and a test set.

[0206] Furthermore, the key point recognition module 60 is also used to input the RGB image into the target recognition model for weld key point recognition, and obtain the RGB pixel key point coordinates in the pixel coordinate system in the RGB image; obtain the target transformation matrix required for the coordinate transformation between the depth camera and the RGB camera, and the target transformation matrix includes a target rotation matrix and a target translation matrix; obtain the RGB intrinsic parameter matrix of the RGB camera and the depth intrinsic parameter matrix of the depth camera; based on the RGB intrinsic parameter matrix, obtain the first transformation relationship between the camera coordinate system and the pixel coordinate system under the RGB camera; according to the depth intrinsic parameter matrix, obtain the second transformation relationship between the camera coordinate system and the pixel coordinate system under the depth camera; based on the target rotation matrix, the first transformation relationship and the second transformation relationship, obtain the depth information corresponding to the RGB pixel key point coordinates; determine the weld key point coordinates of the welded part according to the depth information, the RGB pixel key point coordinates, the hand-eye calibration matrix and the posture transformation matrix.

[0207] Furthermore, the key point recognition module 60 is also used to obtain a depth conversion matrix between the camera coordinate system of the depth camera and the world coordinate system, wherein the depth conversion matrix includes a depth rotation matrix and a depth translation matrix; obtain an RGB conversion matrix between the camera coordinate system of the RGB camera and the world coordinate system, wherein the RGB conversion matrix includes an RGB rotation matrix and an RGB translation matrix; determine a target rotation matrix required for coordinate conversion between the depth camera and the RGB camera based on the depth rotation matrix and the RGB rotation matrix; determine a target translation matrix required for coordinate conversion between the depth camera and the RGB camera based on the target rotation matrix, the depth translation matrix and the RGB translation matrix.

[0208] Furthermore, the original recognition model includes a CBS module, a LAE module, a Bottleneck layer and a plurality of detection head modules; and the improvement of the original recognition model includes at least one of the following:

[0209] Adding a maximum pooling layer to the CBS module;

[0210] Adding an adaptive hybrid pooling module to the LAE module, wherein the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters;

[0211] The Bottleneck layer is replaced by an EXconv module, wherein the EXconv module is composed of a local feature fusion module, a depth convolution module, a depth separable convolution module, a batch normalization layer module and a residual connection layer;

[0212] A multi-scale spatial attention module is added after the EXconv module, wherein the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window dilation attention at different dilation rates, and aggregate semantic information;

[0213] Each detection head module is replaced with a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer.

[0214] In this embodiment, training image data and annotation data corresponding to the training image data are acquired, a data set is generated based on the training image data and the annotation data, an original recognition model is constructed, and the original recognition model is improved to obtain an improved original recognition model. The improved original recognition model is trained and optimized based on the data set to obtain a target recognition model. The depth image and RGB image of the weldment are acquired by an image acquisition device on the welding robot. The image acquisition device includes a depth camera and an RGB camera. The RGB image is input into the target recognition model for weld key point recognition, and the weld key point coordinates of the weldment are determined based on the recognition result and the depth image. Since the present embodiment improves the original recognition model, trains and optimizes the improved original recognition model to obtain a target recognition model, the RGB image of the weldment is input into the target recognition model for weld key point recognition, and the coordinates with depth information are obtained in combination with the depth image, so as to realize accurate recognition of weld key points, thereby guiding the welding robot to accurately reach the weld key point coordinate position for welding, thereby improving the autonomy and adaptability of key point recognition, effectively adapting to welding tasks with different requirements, ensuring welding quality, and greatly improving welding efficiency and reducing time cost.

[0215] The weld key point identification device provided by the present application adopts the weld key point identification method in the above embodiment, which can solve the technical problem of weld key point identification. Compared with the prior art, the beneficial effects of the weld key point identification device provided by the present application are the same as the beneficial effects of the weld key point identification method provided by the above embodiment, and other technical features in the weld key point identification device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0216] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0217] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0218] In addition, for technical details not fully described in this embodiment, reference can be made to the weld key point identification method provided in any embodiment of the present invention, and will not be repeated here.

[0219] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0220] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0221] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0222] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying key points of a weld, characterized in that: The weld key point identification method comprises: Acquire training image data and annotation data corresponding to the training image data; generating a data set based on the training image data and the annotated data; Constructing an original recognition model, and improving the original recognition model to obtain an improved original recognition model; Based on the data set, the improved original recognition model is trained and optimized to obtain a target recognition model; The image acquisition device on the welding robot acquires a depth image and an RGB image of the weldment, wherein the image acquisition device includes a depth camera and an RGB camera; Inputting the RGB image into the target recognition model to perform weld key point recognition, and determining the weld key point coordinates of the welded part based on the recognition result and the depth image, wherein the weld key point coordinates are used to guide the welding robot to locate the weld key points on the welded part; The original recognition model includes a CBS module, a LAE module, a C2f layer and a plurality of detection head modules; the improvement of the original recognition model includes at least one of the following: Adding a maximum pooling layer to the CBS module; Adding an adaptive hybrid pooling module to the LAE module, wherein the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters; The Bottleneck module in the C2f layer is replaced by an EXconv module, wherein the EXconv module is composed of a local feature fusion module, a depth convolution module, a depth separable convolution module, a batch normalization layer module and a residual connection layer; A multi-scale spatial attention module is added after the EXconv module of the C2f layer before the multiple detection head modules, wherein the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window dilation attention at different dilation rates, and aggregate semantic information; Each detection head module is replaced with a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer. The original recognition model includes a CBS module, a LAE module, a C2f layer and multiple detection head modules; the improvement of the original recognition model includes at least one of the following: Adding a maximum pooling layer to the CBS module; Adding an adaptive hybrid pooling module to the LAE module, wherein the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters; The Bottleneck module in the C2f layer is replaced by an EXconv module, wherein the EXconv module is composed of a local feature fusion module, a depth convolution module, a depth separable convolution module, a batch normalization layer module and a residual connection layer; A multi-scale spatial attention module is added after the EXconv module of the C2f layer before the multiple detection head modules, wherein the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window dilation attention at different dilation rates, and aggregate semantic information; Each detection head module is replaced with a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer.

2. The weld key point identification method according to claim 1, characterized in that: The training image data includes RGB image data; the acquiring the training image data and the annotation data corresponding to the training image data includes: Performing hand-eye calibration on the image acquisition device of the welding robot to obtain a hand-eye calibration matrix, wherein the hand-eye calibration matrix is ​​a coordinate transformation relationship matrix between a camera coordinate system of the image acquisition device and a tool coordinate system of the welding robot; Acquire a posture transformation matrix based on the control parameters of the welding robot, wherein the posture transformation matrix is ​​a coordinate transformation relationship matrix between a base coordinate system of the welding robot and the tool coordinate system; Based on Zhang Zhengyou's chessboard calibration method, the intrinsic parameter matrix, rotation matrix and translation matrix of each camera in the image acquisition device are obtained to complete the calibration of the image acquisition device, wherein the intrinsic parameter matrix is ​​a coordinate transformation matrix between the camera coordinate system and the pixel coordinate system of the image acquisition device, and the rotation matrix and the translation matrix are coordinate transformation matrices between the camera coordinate system and the world coordinate system of the image acquisition device; The welded parts are imaged from multiple angles using the calibrated image acquisition device to obtain RGB image data; The key points of the weld are annotated on the RGB image data to obtain annotated data.

3. The weld key point identification method according to claim 2, characterized in that: The data set includes a training set, a validation set and a test set; generating the data set based on the training image data and the annotated data includes: generating an initial set based on the training image data and the annotated data; generating a rotation matrix according to the initial set, and performing a rotation process on the initial set based on the rotation matrix to obtain a first candidate set; Determine a flip type, and perform flip processing on the initial set based on the flip type to obtain a second candidate set; generating noise information, and adding the noise information to the initial set to obtain a third candidate set; adjusting the image contrast of the initial set based on a preset contrast factor to obtain a fourth candidate set; Merging the initial set with the first candidate set, the second candidate set, the third candidate set and the fourth candidate set to obtain a data set; The data set is divided based on a preset data ratio to obtain a training set, a validation set, and a test set.

4. The weld key point identification method according to claim 3, characterized in that: The step of inputting the RGB image into the target recognition model to perform weld key point recognition, and determining the weld key point coordinates of the welded part based on the recognition result and the depth image, comprises: Inputting the RGB image into the target recognition model to perform weld key point recognition, and obtaining RGB pixel key point coordinates in a pixel coordinate system in the RGB image; Obtain a target transformation matrix required for coordinate transformation between the depth camera and the RGB camera, wherein the target transformation matrix includes a target rotation matrix and a target translation matrix; Obtain the RGB intrinsic parameter matrix of the RGB camera and the depth intrinsic parameter matrix of the depth camera; Acquire a first conversion relationship between a camera coordinate system and a pixel coordinate system under the RGB camera based on the RGB intrinsic parameter matrix; Acquire a second conversion relationship between a camera coordinate system and a pixel coordinate system under the depth camera according to the depth intrinsic parameter matrix; Acquire depth information corresponding to the RGB pixel key point coordinates based on the target rotation matrix, the first conversion relationship, and the second conversion relationship; The key point coordinates of the weld of the welded part are determined according to the depth information, the key point coordinates of the RGB pixel, the hand-eye calibration matrix and the posture transformation matrix.

5. The weld key point identification method according to claim 4, characterized in that: The obtaining of the target transformation matrix required for coordinate transformation between the depth camera and the RGB camera includes: Obtain a depth conversion matrix between a camera coordinate system and a world coordinate system under the depth camera, wherein the depth conversion matrix includes a depth rotation matrix and a depth translation matrix; Obtain an RGB conversion matrix between a camera coordinate system and a world coordinate system under the RGB camera, wherein the RGB conversion matrix includes an RGB rotation matrix and an RGB translation matrix; Determine a target rotation matrix required for coordinate conversion between the depth camera and the RGB camera based on the depth rotation matrix and the RGB rotation matrix; A target translation matrix required for coordinate conversion between the depth camera and the RGB camera is determined according to the target rotation matrix, the depth translation matrix and the RGB translation matrix.

6. A device for identifying key points of a weld, characterized in that: The weld key point identification device comprises: A data acquisition module, used to obtain training image data and annotation data corresponding to the training image data; A data processing module, used to generate a data set based on the training image data and the annotated data; A model building module, used to build an original recognition model and improve the original recognition model to obtain an improved original recognition model; A training optimization module, used for training and optimizing the improved original recognition model based on the data set to obtain a target recognition model; An image acquisition module, used to acquire a depth image and an RGB image of the weldment through an image acquisition device on the welding robot, wherein the image acquisition device includes a depth camera and an RGB camera; A key point recognition module, used for inputting the RGB image into the target recognition model to perform weld key point recognition, and determining the weld key point coordinates of the welded part based on the recognition result and the depth image, wherein the weld key point coordinates are used to guide the welding robot to locate the weld key points on the welded part; The original recognition model includes a CBS module, a LAE module, a C2f layer and a plurality of detection head modules; The model building module is also used to add a maximum pooling layer to the CBS module; add an adaptive hybrid pooling module to the LAE module, the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters; use an EXconv module to replace the Bottleneck module in the C2f layer, the EXconv module is composed of a local feature fusion module, a deep convolution module, a deep separable convolution module, a batch normalization layer module and a residual connection layer; add a multi-scale spatial attention module after the EXconv module of the C2f layer before the multiple detection head modules, the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window expansion attention with different expansion rates, and aggregate semantic information; replace each detection head module with a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer. The original recognition model includes a CBS module, an LAE module, a C2f layer and multiple detection head modules; the improvement of the original recognition model includes at least one of the following: adding a maximum pooling layer to the CBS module; adding an adaptive hybrid pooling module to the LAE module, the adaptive hybrid pooling module includes a maximum pooling layer and an average pooling layer, and the attention weights of the maximum pooling layer and the average pooling layer are configured based on preset learning parameters; using an EXconv module to replace the Bottleneck module in the C2f layer, the EXconv module consists of a local feature fusion module, a deep convolution module, a deep separable convolution module, a batch normalization layer module and a residual connection layer; adding a multi-scale spatial attention module after the EXconv module of the C2f layer before the multiple detection head modules, the multi-scale spatial attention module is used to divide the channels of the input feature map into multiple heads, perform multi-scale sliding window expansion attention with different expansion rates, and aggregate semantic information; each detection head module is replaced by a shared convolution detection head module, and multiple shared convolution detection head modules share the same convolution layer.

7. A weld key point identification device, characterized in that: The weld key point identification device includes: a memory, a processor, and a weld key point identification program stored in the memory and executable on the processor, wherein the weld key point identification program is configured to implement the weld key point identification method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a weld key point recognition program, and when the weld key point recognition program is executed by the processor, the weld key point recognition method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The computer program product comprises a weld key point identification program, and when the weld key point identification program is executed by a processor, the steps of the weld key point identification method according to any one of claims 1 to 5 are implemented.

Images