A method, device, system and storage medium for automatically planning a weld photographing point

Abstract

This invention discloses an automatic planning method, apparatus, system, storage medium, and computer program for weld seam imaging points. The method includes: acquiring a global depth image of the workpiece to be welded; extracting and analyzing the workpiece contour from the image to identify the set of corner points to be photographed corresponding to the weld seam; subsequently, constructing a local reference coordinate system based on the geometric features of the workpiece's outer contour; projecting the corner points to be photographed onto this coordinate system and then clustering and sorting them to generate a continuously scanned, ordered sequence of imaging points; finally, calculating the camera's 3D shooting pose according to the ordered sequence of imaging points, combining the geometric relationships of adjacent contour edges corresponding to each corner point to be photographed, converting it into an executable pose in the robot's base coordinate system, and outputting it. This solution effectively adapts to operational scenarios where workpieces exhibit dimensional deformation and positional deviations, improves the automation level and operational efficiency of weld seam imaging point planning, and ensures the stability of weld seam imaging quality.

Description

Technical Field

[0001] This invention belongs to the field of industrial automation welding and machine vision technology, specifically relating to an automatic planning method, device, system, storage medium and computer program product for weld seam imaging points, and particularly to an automatic planning method, device, system, storage medium and computer program product for weld seam imaging points of industrial pallet steel components based on depth maps. Background Technology

[0002] With the rapid improvement of automation in the logistics and warehousing industry, large-scale grid-like steel industrial pallets, as core equipment for cargo carrying and transfer, are facing increasing demands for production capacity and welding quality. These components are complex in structure, with numerous welds distributed over a wide area. Visual-guided robotic automated welding is the mainstream technology for ensuring welding accuracy and improving production efficiency. Among these, the planning of weld seam imaging points is a crucial preliminary step in the visual-guided process. The accuracy of point recognition, the rationality of the path, and the adaptability of the pose directly determine the quality of weld seam image acquisition, as well as the accuracy and efficiency of subsequent welding operations.

[0003] Currently, visual acquisition solutions for weld seams of large steel components mainly fall into two categories. One type is a fixed imaging solution based on manual teaching or offline programming: the imaging points and camera poses are pre-set according to the CAD model of the standard workpiece, and the robot's motion trajectory and imaging parameters are solidified through manual teaching. The other type uses a small-field-of-view camera mounted on the end effector of a robotic arm to complete weld seam image acquisition by traversing and searching area by area.

[0004] In actual industrial production scenarios, large pallet steel components generally suffer from dimensional errors, structural deformations, and positional deviations during processing, transportation, and placement. The existing solutions mentioned above all have obvious technical limitations: solutions based on CAD models and manual teaching rely heavily on the preset parameters of standard workpieces and cannot adapt to the actual deformation and positional deviations of the workpieces. After the workpiece shifts, the camera's perspective and the shooting point become mismatched, directly leading to the failure of weld feature extraction and unstable welding quality. Moreover, the manual teaching cycle is long and the cost of changing workpieces is high, making it difficult to meet the needs of flexible production. Solutions relying solely on a small "eye in hand" camera lack global coarse positioning capabilities, have extremely low global search efficiency, and have a limited field of view that easily leads to the loss of weld targets. At the same time, existing path planning does not incorporate global optimization based on the geometric features of the workpiece, resulting in long idle strokes and frequent reversals during the movement of the robotic arm, which not only lengthens the overall welding cycle but also increases the risk of equipment collisions.

[0005] In summary, existing technologies cannot simultaneously achieve automatic identification of global weld seam imaging points, global optimization of motion paths, and adaptive calculation of imaging poses, making it difficult to meet the needs of efficient and stable automated welding in scenarios where workpieces have dimensional deformation and positional deviations.

[0006] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0007] The purpose of this invention is to provide an automatic planning method, device, system, storage medium, and computer program for weld seam imaging points. This addresses the problem that existing weld seam imaging point planning technologies for large pallet steel components cannot adaptively complete global imaging point recognition, path optimization, and imaging pose calculation, making it difficult to meet the needs of efficient and stable automated welding in scenarios where workpieces exhibit dimensional deformation and positional deviations. The invention effectively adapts to operational scenarios with workpiece dimensional deformation and positional deviations, improving the automation level and operational efficiency of weld seam imaging point planning, and enhancing the stability and scenario adaptability of automated welding operations.

[0008] This invention provides an automatic planning method for weld seam photography points, comprising: acquiring a global depth image of the workpiece to be welded; performing workpiece contour extraction and feature analysis on the global depth image to identify a set of corner points to be photographed corresponding to the weld seam; constructing a local reference coordinate system based on the geometric features of the outer contour of the workpiece to be welded, projecting the set of corner points to be photographed onto the local reference coordinate system and then performing clustering and sorting to generate a continuously scanned ordered sequence of photography points; calculating the camera 3D shooting pose corresponding to each photography point according to the order of the ordered sequence of photography points and based on the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed, converting the camera 3D shooting pose into an executable pose in the robot base coordinate system and outputting it.

[0009] In some implementations, the global depth image is subjected to workpiece contour extraction and feature analysis to identify the set of corner points to be photographed corresponding to the weld. This includes: preprocessing the global depth image to extract and distinguish the outer contour and at least one inner contour of the workpiece to be welded; extracting polygon features from each inner contour and filtering out candidate corner points corresponding to the inner contour; for candidate corner points that meet the preset distance constraints, making perpendicular feet to the outer contour edge to fill in the missing corner points, and adding the perpendicular feet fill points that meet the preset direction constraints to the set of corner points to be photographed.

[0010] In some implementations, after projecting the set of corner points to be photographed onto the local reference coordinate system, clustering and sorting are performed to generate a continuously scanned ordered sequence of photographed points. This includes: after projecting each corner point to be photographed onto the local reference coordinate system, performing gap clustering along the first coordinate axis to obtain multiple sets of corner point columns; sorting each set of corner point columns along the second coordinate axis, with adjacent corner point columns using opposite sorting directions along the second coordinate axis, and connecting each set of corner point columns end-to-end along the first coordinate axis to obtain a continuously scanned ordered sequence of photographed points.

[0011] In some implementations, the camera 3D shooting pose corresponding to each shooting point is calculated based on the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed. This includes: calculating the angle bisector direction vector based on the direction vectors of the two adjacent contour edges of the corner point to be photographed; offsetting the camera by a preset distance along the angle bisector direction in the workpiece plane, and then offsetting it by a preset height along the direction perpendicular to the workpiece plane to determine the spatial position of the camera; using the direction of the camera pointing to the corner point to be photographed as the optical axis direction, and combining the angle bisector direction to calculate the three-axis rotation attitude of the camera, thereby obtaining the camera 3D shooting pose.

[0012] In some implementations, when calculating the camera 3D shooting pose corresponding to each shooting point, for each corner point to be photographed, the effective depth values ​​in the preset area around the corner point are extracted, and the minimum depth value among them is selected as the depth reference value of the corner point, which is used to calculate the 3D spatial coordinates of the corner point.

[0013] In some implementations, the global depth image is acquired by a fixed first 3D camera; the 3D shooting pose of the camera corresponds to a second 3D camera mounted on the end effector of the robot; the first 3D camera and the second 3D camera are respectively pre-calibrated for coordinate system transformation.

[0014] In conjunction with the above method, another aspect of the present invention provides an automatic planning device for weld seam photography points, comprising: an acquisition unit configured to acquire a global depth image of the workpiece to be welded; a point planning unit configured to extract the workpiece contour and perform feature analysis on the global depth image to identify a set of corner points to be photographed corresponding to the weld; the point planning unit is further configured to construct a local reference coordinate system based on the geometric features of the outer contour of the workpiece to be welded, project the set of corner points to be photographed onto the local reference coordinate system, and then perform clustering and sorting to generate a continuously scanned ordered sequence of photography points; and a pose calculation unit configured to calculate the camera 3D shooting pose corresponding to each photography point according to the order of the ordered sequence of photography points and based on the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed, convert the camera 3D shooting pose into an executable pose in the robot base coordinate system, and output it.

[0015] In conjunction with the above-mentioned device, the present invention further provides an automatic planning system for weld seam photographing points, comprising: the automatic planning device for weld seam photographing points described above.

[0016] In conjunction with the above method, the present invention further provides a storage medium comprising a stored program, wherein, when the program is executed, the device on which the storage medium is located executes the above-described automatic planning method for weld seam photographing points.

[0017] In conjunction with the above method, the present invention further provides a computer program product, the computer program product comprising a computer program that, when processed and executed, implements the steps of the above-described automatic planning method for weld seam photographing points.

[0018] The present invention first acquires a global depth image of the workpiece to be welded, performs workpiece contour extraction and feature analysis on the image, and identifies the set of corner points to be photographed corresponding to the weld. Then, a local reference coordinate system is constructed based on the geometric features of the workpiece's outer contour. The corner points to be photographed are projected onto this coordinate system and clustered and sorted to generate a continuously scanned, ordered sequence of photographing points. Finally, according to the order of the ordered photographing point sequence, the 3D shooting pose of the camera is calculated based on the geometric relationship of the adjacent contour edges corresponding to each corner point, and converted into an executable pose in the robot base coordinate system and output. This method can adapt to workpiece size deformation and placement deviations, automatically completing global photographing point recognition, path optimization, and imaging pose calculation. It effectively improves the automation level and operational efficiency of weld photographing point planning, ensures the stability of weld imaging quality, and can fully adapt to the automated welding needs in scenarios where workpiece deviations exist.

[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an embodiment of the automatic planning method for weld seam photographing points according to the present invention.

[0022] Figure 2 This is a schematic diagram of a structure of an embodiment of the automatic planning device for weld seam photography points of the present invention;

[0023] Figure 3 This is a schematic diagram of a welding scenario for steel components on an industrial pallet.

[0024] Figure 4 Flowchart of an automatic planning method for weld seam photography points of steel components on industrial pallets based on depth maps;

[0025] Figure 5 Flowchart for processing depth map of pallet workpiece;

[0026] Figure 6 A before-and-after comparison diagram of the 2D corner point sorting and planning for the photo to be taken;

[0027] Figure 7 A schematic diagram for calculating the pose of a 3D camera point.

[0028] Referring to the accompanying drawings, the reference numerals in the embodiments of the present invention are as follows:

[0029] 1-First 3D camera; 2-Second 3D camera; 3-Industrial pallet; 4-Robotic arm; 101-Acquisition unit; 102-Position planning unit; 103-Pose calculation unit. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0031] According to embodiments of the present invention, an automatic planning method for weld seam photography points is provided, such as... Figure 1 The flowchart of an embodiment of the method of the present invention is shown. The automatic planning method for weld seam photography points may include steps S110 to S140.

[0032] In step S110, a global depth image of the workpiece to be welded is obtained.

[0033] A global depth image is a two-dimensional depth mapping image that can fully reflect the overall shape and spatial distance information of the workpiece to be welded. The value of each pixel in the image corresponds to the straight-line spatial distance between the corresponding position on the workpiece surface and the acquisition device, which can directly reflect the three-dimensional shape characteristics of the workpiece.

[0034] In some implementations, the global depth image is acquired by a fixed first 3D camera; the 3D shooting pose of the camera corresponds to a second 3D camera mounted on the end effector of the robot; the first 3D camera and the second 3D camera are respectively pre-calibrated for coordinate system transformation.

[0035] The first 3D camera is fixedly mounted above the workpiece to be welded using a gantry bracket and a fixed frame, maintaining a constant spatial position and orientation during operation. The first 3D camera performs a single 3D imaging acquisition of the workpiece within the workstation, outputting a global depth image covering the entire shape of the workpiece. The calculated 3D shooting pose of the camera is entirely targeted at the second 3D camera mounted at the robot's end effector, with the pose data corresponding to the optical center position and optical axis orientation of the second 3D camera. When the robot moves the end effector to the designated pose, the second 3D camera is positioned in the preset shooting position and orientation, completing the acquisition of local weld seam images. This dual-camera collaborative architecture balances the dual requirements of global workpiece coverage and high-precision acquisition of local weld seams, resolving the technical contradiction of the inability to simultaneously achieve both the field of view and accuracy of a single camera.

[0036] Before the formal operation, a hand-eye calibration process was performed on both cameras: for the fixed first 3D camera, the transformation matrix between the first 3D camera coordinate system and the robot base coordinate system was obtained; for the second 3D camera installed at the robot's end effector, the transformation matrix between the second 3D camera coordinate system and the robot end effector coordinate system was obtained. The two sets of calibration matrices together constitute a complete coordinate transformation link, used to convert the point and pose data of global visual recognition into a base coordinate system pose that the robot can recognize and execute. Through the mechanism of hand-eye calibration of the two cameras, a complete coordinate transformation link was established, breaking down the coordinate barrier between visual perception and robot execution, and ensuring the accuracy of the planned pose execution.

[0037] Figure 3 This is a schematic diagram of a welding scene for steel components on an industrial pallet. The first 3D camera 1 is fixedly mounted above the workstation via a gantry frame. Its position and orientation remain constant throughout the operation. It is a large-field-of-view 3D acquisition device with an "eye in the hand" configuration, used to acquire global depth images of the complete workpiece. The robotic arm 4 is a multi-axis industrial robot, serving as the motion execution carrier for visual acquisition and subsequent welding operations. The second 3D camera 2 is fixedly mounted on the end flange of the robotic arm 4. It can adjust its spatial pose synchronously with the robotic arm. It is a high-precision 3D acquisition device with an "eye in the hand" configuration, used to sequentially reach the target position according to the planned image points, acquiring high-definition depth images of local welds to achieve precise weld positioning. The industrial pallet 3 is a large steel component workpiece with a grid structure, placed horizontally in the central work area of ​​the workstation. It is the target object for weld image acquisition and welding processing.

[0038] In step S120, the workpiece contour is extracted and feature analysis is performed on the global depth image to identify the set of corner points to be photographed corresponding to the weld.

[0039] The set of corner points to be photographed refers to the collection of coordinates of all contour feature corner points on the workpiece that need to be acquired for high-definition weld images. These corner points correspond to the weld positions at the junctions of steel component profiles and are the core target points for weld image acquisition.

[0040] Welds in mesh-like steel components are typically located at the corner points of the profile intersections, and there is a strong correspondence between the workpiece contour corner points and the weld acquisition points. By extracting contours and analyzing features, these points can be located directly from the actual acquired depth images, eliminating the need for pre-importing a standard CAD model of the workpiece or manually marking the points. This approach can adapt to actual dimensional deviations and structural deformations of the workpiece. Specifically, the global depth image is used to distinguish between the workpiece area and the background area, extracting the complete edge contour information of the workpiece. Combining the correspondence between the welds and contour corner points of the mesh-like steel components, feature points of all contours are analyzed and filtered to extract all valid feature corner points corresponding to the weld positions. Finally, the coordinates of the corner points to be photographed are compiled.

[0041] In some implementations, step S120 involves extracting the workpiece contour and performing feature analysis on the global depth image to identify the set of corner points to be photographed corresponding to the weld, including steps S210 to S230.

[0042] Step S210: Preprocess the global depth image to extract and distinguish the outer contour and at least one inner contour of the workpiece to be welded.

[0043] The purpose of preprocessing is to remove irrelevant and interfering information, retain the effective workpiece area, and improve the accuracy of subsequent feature extraction. The outer contour refers to the outermost closed edge contour of the workpiece to be welded; this contour constitutes the overall outer boundary of the workpiece and is not surrounded by other contours of the workpiece itself. The inner contour refers to the closed edge contour inside the workpiece, surrounded by the outer contour. This type of contour corresponds to the internal boundaries formed by internal holes and profile partitions in a mesh-like workpiece, and its corner points correspond to the weld positions inside the workpiece. For example... Figure 6 As shown in Figure a, the outer contour refers to the outermost contour, which includes the contour containing points 25, 26, 27, and 28; the rest of the contours are inner contours.

[0044] The original depth image contains a large amount of background information about the workstation environment and supporting structures, as well as tiny noise generated during the imaging process. Directly extracting the contours will result in a large number of invalid contours, interfering with corner point recognition. Mesh-shaped workpieces have multiple layers of closed contours, and different contours at different locations correspond to different weld seam distribution patterns. Distinguishing between inner and outer contours allows for the application of different corner point extraction strategies for the two types of contours, improving the accuracy and completeness of the extraction. Specifically, firstly, background areas with depth values ​​exceeding the workpiece's range are removed by depth threshold filtering. Then, the remaining workpiece area is binarized and connected component calculations are performed, deleting tiny interfering connected components with areas smaller than a set threshold. Subsequently, edge detection is performed on the processed workpiece area to extract all closed contour lines. Finally, contour inclusion relationships are determined to distinguish the outermost outer contour from all inner contours enclosed by it.

[0045] Step S220: Extract polygon features from each inner contour and filter to obtain candidate corner points corresponding to the inner contour.

[0046] Polygon feature extraction specifically involves fitting continuous contour pixels into a polygonal structure composed of several line segments, and extracting the vertices of the polygon as contour feature points. Candidate corner points refer to contour vertices with weld seam corner features obtained from the initial screening of the inner contour, and are the basis for subsequent completion and screening.

[0047] The original contour consists of a large number of continuous pixels. Directly using all contour points would result in numerous redundant points and would fail to locate the true weld corner positions. The internal welds of the grid-like steel components are distributed at the corner positions where the profiles intersect, corresponding one-to-one with the polygon vertices of the inner contour. Through polygon fitting and feature filtering, redundant contour points can be eliminated, and the corner points corresponding to the internal welds can be accurately located. Specifically, for each inner contour, the continuous sequence of contour points is fitted into a polygon structure composed of straight line segments, and all vertices of the polygon are extracted. Then, feature filtering is performed on the polygon vertices, retaining vertices with obvious corner features as candidate corner points for the inner contour.

[0048] Step S230: For candidate corner points that meet the preset distance constraints, make perpendicular feet to the outer contour edge to fill in the missing corner points, and add the perpendicular feet fill points that meet the preset direction constraints to the set of corner points to be photographed.

[0049] Preset distance constraints refer to pre-defined distance thresholds used to determine whether other identified corner points exist around the candidate corner point, thus identifying any missing points. Perpendicular foot completion refers to drawing a perpendicular line from the candidate corner point to the contour edge, using the intersection of the perpendicular and the contour edge as the point to fill in any missing weld points on the contour edge. Preset direction constraints refer to pre-defined angle rules used to filter perpendicular foot completion points, retaining only those whose directions conform to the weld distribution pattern and filtering out invalid points.

[0050] Extracting only inner contour corner points will miss weld points on the outer contour edges, resulting in incomplete weld acquisition coverage. Triggering completion logic through distance constraints can accurately locate areas with missing points, avoiding redundant points generated by indiscriminate completion. Filtering perpendicular points through direction constraints can filter out invalid perpendicular points whose directions do not conform to the weld distribution, avoiding the mistaken addition of points in non-weld locations. Specifically, all candidate corner points are traversed to determine if there are other identified corner points within a preset distance range around the candidate corner point. If no other corner points are found within the range, it is determined that there is a missing weld point on the corresponding outer contour edge. A perpendicular line is drawn from the candidate corner point to the nearest outer contour edge to obtain the perpendicular point. Further, it is determined whether the direction of the line connecting the perpendicular point and the candidate corner point conforms to preset direction rules. Perpendicular points that conform to the rules are added to the set of corner points to be photographed as completion corner points.

[0051] In step S130, a local reference coordinate system is constructed based on the geometric features of the outer contour of the workpiece to be welded. The set of corner points to be photographed is then projected onto the local reference coordinate system and clustered and sorted to generate an ordered sequence of photographing points for continuous scanning.

[0052] The local reference coordinate system is a two-dimensional coordinate system established based on the geometric features of the workpiece's outer contour. This coordinate system closely matches the workpiece's shape and is used to eliminate the interference of workpiece placement angle deviations on path planning, ensuring that the sorting logic adapts to the actual shape of the workpiece. Cluster sorting refers to a processing method that first groups points according to their distribution interval characteristics on the coordinate axes, and then sorts the points within each group. This is used to organize scattered points into a continuous scanning path.

[0053] The placement angle of the workpiece at the workstation exhibits random deviations. Directly using the world coordinate system for point sorting leads to a mismatch between the planned path and the actual shape of the workpiece, resulting in a large amount of redundant idle travel. Establishing a local coordinate system based on the workpiece's own outer contour can completely eliminate the interference of placement angles. Furthermore, clustering and sorting can organize scattered and disordered points into a continuous scanning path, reducing the robot's idle travel and the number of reversals. Specifically, the geometric features of the reference edge of the workpiece's outer contour are extracted, and a local reference coordinate system conforming to the workpiece's shape is established using this reference edge as the coordinate axis reference. Through coordinate transformation operations, the original coordinates of all corner points to be photographed are transformed and projected into this local reference coordinate system. Clustering and grouping are performed according to the distribution interval characteristics of the points on the coordinate axes, and finally, an ordered sequence of photographing points corresponding to a continuous scanning path is formed.

[0054] In some implementations, step S130 involves projecting the set of corner points to be photographed onto the local reference coordinate system and then performing clustering and sorting to generate an ordered sequence of photographed points for continuous scanning, including steps S310 and S320.

[0055] Step S310: After projecting each corner point to be photographed onto the local reference coordinate system, perform gap clustering along the first coordinate axis to obtain multiple sets of corner point columns.

[0056] Coordinate projection specifically involves transforming the coordinates of points in the original coordinate system into coordinates in a local reference coordinate system. After transformation, the relative positions of the points remain consistent with the shape of the workpiece. Gap clustering refers to an automatic grouping method that uses the coordinate intervals of points on a specified coordinate axis as the criterion, grouping consecutive points with coordinate differences less than a set threshold into the same group and points with coordinate differences exceeding the threshold into different groups, without the need to pre-set the number of groups. Corner point columns refer to each set of corner points obtained after clustering along the first coordinate axis. Corner points within the same group are close in position on the first coordinate axis and arranged along the second coordinate axis. The first coordinate axis corresponds to the main extension direction of the workpiece and is used to determine the arrangement direction of the scanning column; the second coordinate axis corresponds to the sorting direction of the corner points within the column and is used to determine the scanning order within a single column. The two coordinate axes are perpendicular to each other.

[0057] The weld corner points of a grid-like workpiece naturally exhibit a column-like distribution. However, when the workpiece's placement angle deviates, directly grouping them in the original coordinate system can lead to column misalignment and grouping chaos. Projecting to a local coordinate system that fits the workpiece ensures that the grouping logic matches the actual shape of the workpiece. Automatic grouping using gap clustering eliminates the need to pre-set the number of columns and column spacing, adapting to workpieces of different specifications and sizes and avoiding the limitations of manually preset parameters. Specifically, through coordinate transformation, all corner points to be photographed are converted from the original image coordinate system to the local reference coordinate system, obtaining the coordinates of each corner point in the new coordinate system. All corner points are then sorted in ascending order along the first coordinate axis. The sorted corner points are traversed, and the coordinate difference between the current corner point and the last corner point in the current group on the first coordinate axis is calculated sequentially. If the coordinate difference is greater than a preset gap threshold, the current corner point is assigned to a new group; if the coordinate difference is less than or equal to the threshold, the current corner point is assigned to the current group. After traversal, all groups consist of multiple columns of corner points arranged along the first coordinate axis.

[0058] Step S320: Sort each group of corner points along the second coordinate axis. Adjacent corner point groups are sorted in opposite directions along the second coordinate axis. The corner point groups are then connected end to end along the first coordinate axis to obtain a continuously scanned, ordered sequence of photographed points.

[0059] The opposite sorting direction refers to the alternating ascending and descending sorting rules used for two adjacent sets of corner point columns, so that the last point of the previous column is spatially adjacent to the starting point of the next column. End-to-end connection refers to sequentially piecing together the sorting results of all corner point columns along the first coordinate axis to form a complete and continuous sequence of points.

[0060] If all corner point columns use the same sorting direction, after scanning each column, the robot needs to return from the end of the column to the beginning of the column before moving to the beginning of the next column, resulting in a large amount of invalid empty travel and increasing the number of robot reversals. Using reverse sorting of adjacent columns ensures that the end point of the previous column is spatially adjacent to the beginning point of the next column. After scanning the previous column, the robot can directly move to the next column, eliminating invalid return travel between columns, reducing robot reversals, improving motion continuity, and lowering the risk of equipment collisions. Specifically, for all corner points within each group of corner point columns, coordinate sorting is performed along the second coordinate axis. Along the arrangement order of the first coordinate axis, adjacent groups of corner point columns use opposite sorting directions; that is, when the previous group is arranged in ascending order, the next group is arranged in descending order, alternating in this manner. Finally, according to the arrangement order from left to right on the first coordinate axis, the sorting results of all corner point columns are sequentially spliced ​​together to form a complete and ordered sequence of image points. Figure 6 As shown in b, the ascending order of the point numbers is the order in which the robotic arm drives the camera to perform the acquisition operation. All corner points to be photographed are clustered into multiple vertical corner point columns along the long side of the workpiece. Adjacent corner point columns are connected end to end in opposite sorting directions to form a continuous scanning path that goes back and forth along the short side of the workpiece. Among them, 0, 5, 23, and 28 are corner points of the outer contour of the pallet, and each corner point of the internal grid corresponds to the acquisition position of the weld seam at the intersection of the profiles.

[0061] In step S140, according to the order of the ordered photo point sequence, based on the geometric relationship of the adjacent contour edges corresponding to each photo point, the camera 3D shooting pose corresponding to each photo point is calculated, and the camera 3D shooting pose is converted into an executable pose in the robot base coordinate system and output.

[0062] The camera 3D shooting pose refers to the complete pose data containing the camera's three-dimensional spatial position coordinates and three-axis rotation attitude. It is used to determine the shooting position and optical axis orientation of the second 3D camera 2, directly determining the acquisition angle and quality of the weld seam image. The robot base coordinate system refers to the three-dimensional coordinate system established with the fixed origin of the robot base as the reference. It is the reference coordinate system for the industrial robot to execute motion commands, and all robot-executable pose data must be transformed to this coordinate system.

[0063] The quality of weld seam image acquisition is directly related to the camera's shooting angle. Calculating the pose based on the geometric relationship of the contour edges ensures that the camera's optical axis matches the weld seam area, resulting in clear and complete weld seam features. Calculating and outputting the pose sequentially in an ordered order ensures that the robot executes the acquisition task along the optimal path, avoiding efficiency losses caused by disordered movement. Specifically, according to the ordered sequence of shooting points, for each corner point to be photographed, the 3D spatial position and rotational attitude of the camera at the corresponding shooting location are calculated by combining the geometric directions of the adjacent contour edges on both sides of the corner point. Through a pre-defined coordinate system transformation relationship, the pose data in the camera coordinate system is converted into an executable motion pose in the robot's base coordinate system, ultimately outputting the ordered pose data for all points.

[0064] In some implementations, step S140 involves calculating the camera 3D shooting pose corresponding to each shooting point based on the geometric relationship between the adjacent contour edges corresponding to each corner point to be photographed, including steps S410 to S430.

[0065] Step S410: Calculate the angle bisector direction vector based on the direction vectors of the two adjacent contour edges of the corner point to be photographed.

[0066] Adjacent contour edges refer to the two contour line segments on either side of the corner to be photographed, which together form the corner, corresponding to the two profile edges on the steel component that intersect at the weld point. Angle bisector direction vector refers to the unit direction vector corresponding to the angle bisector of the angle between the two adjacent contour edges. This direction is located at the center of symmetry of the two contour edges and is the reference direction for the horizontal offset of the camera.

[0067] Specifically, two adjacent contour edges on both sides of the corner point to be photographed are extracted. The direction vectors corresponding to the two edges are calculated by using the coordinates of the two endpoints of each edge. The two direction vectors are then normalized. The two unit direction vectors are vector-added and then normalized again to finally obtain the angle bisector direction vector of the angle between the two contour edges.

[0068] Step S420: Offset the camera by a preset distance along the angle bisector direction within the workpiece plane, and then offset it by a preset height along a direction perpendicular to the workpiece plane to determine the spatial position of the camera.

[0069] The workpiece plane refers to the reference plane on which the main surface of the workpiece to be welded lies. The spatial position of the camera refers to the coordinate point of the camera's optical center in three-dimensional space, used to determine the camera's shooting position, and is the translation component of the camera's 3D pose.

[0070] Horizontal offset along the angle bisector allows the camera to be positioned away from directly above the weld, preventing interference between the camera tip and the workpiece structure while ensuring complete coverage of the weld area on both sides of the corner point. Vertical elevation by a preset height places the camera within its depth-of-field range, ensuring clear weld imaging and further mitigating the risk of collision between the camera and the workpiece. Specifically, starting from the corner point to be photographed, a preset horizontal distance is translated within the workpiece plane along the angle bisector to obtain the midpoint after horizontal offset. Then, starting from this midpoint, a preset vertical height is translated along a direction perpendicular to the workpiece plane and away from the workpiece to finally obtain the three-dimensional spatial coordinates of the camera's optical center.

[0071] Step S430: Using the direction in which the camera points to the corner point to be photographed as the optical axis direction, and combining the direction of the angle bisector, calculate the three-axis rotation attitude of the camera to obtain the 3D shooting pose of the camera.

[0072] The optical axis direction refers to the direction of the normal to the center of the camera lens, which is the direction of the central line of sight for the camera image. It corresponds to the z-axis of the camera coordinate system and directly determines the central area of ​​the shot and the orientation of the viewpoint. The three-axis rotational pose is a rotation matrix composed of three pairwise perpendicular unit vectors (corresponding to the camera's x, y, and z-axis directions, respectively). It is used to uniquely determine the camera's shooting orientation in three-dimensional space and is the rotational component of the camera's 3D pose.

[0073] Pointing the optical axis towards the corner point to be photographed ensures that the weld corner point is always centered in the camera's field of view, guaranteeing a clear and complete core acquisition area. Constructing two additional coordinate axes through vector cross products ensures both the orthogonality of the camera coordinate system and alignment of the camera's imaging direction with the weld contour direction. Specifically, the vector pointing from the camera's spatial position to the corner point is used as the initial optical axis direction and normalized to become the z-axis vector of the camera coordinate system. The angle bisector direction vector is cross-multiplied with the optical axis direction vector to obtain a unit vector perpendicular to both vectors, which becomes the x-axis vector of the camera coordinate system. The optical axis direction vector is then cross-multiplied with the x-axis vector to obtain the y-axis vector of the camera coordinate system. These three pairwise orthogonal unit vectors (x, y, and z) constitute the camera's rotational posture, which, combined with the previously obtained camera spatial position, forms the complete 3D camera pose.

[0074] In some implementations, when calculating the camera 3D shooting pose corresponding to each shooting point, for each corner point to be photographed, the effective depth values ​​in the preset area around the corner point are extracted, and the minimum depth value among them is selected as the depth reference value of the corner point, which is used to calculate the 3D spatial coordinates of the corner point.

[0075] The effective depth value refers to the depth value within the preset area that conforms to the depth range and can truly reflect the distance information of the workpiece surface, excluding invalid zero values ​​or abnormal values ​​caused by imaging failure, occlusion, or noise. 3D spatial coordinates refer to the actual position coordinates of the corner point to be photographed in three-dimensional space.

[0076] Depth images commonly exhibit imaging noise and abrupt changes in depth values ​​at the edges of workpiece contours. Directly using single-pixel depth values ​​at corner locations can lead to significant errors or even invalid values, resulting in inaccurate subsequent 3D coordinate calculations. Employing a local region statistical approach can utilize the depth information of surrounding pixels to offset the interference of single-pixel noise, improving the stability of depth values. Furthermore, the depth value corresponds to the distance between the acquisition device and the workpiece surface; a smaller value indicates a closer distance. The weld seam of the mesh steel component is located on the upper surface of the profile, the closest point to the acquisition device. Selecting the minimum depth value within this region accurately corresponds to the actual weld seam location on the upper surface of the profile, avoiding interference from depth values ​​from the profile sides, workpiece grooves, or background areas in the weld seam location calculation.

[0077] Specifically, when calculating the camera pose corresponding to each corner point to be photographed, a local image region is first defined with the pixel coordinates of the corner point as the center and according to a preset region size; the depth values ​​corresponding to all pixels in this region are extracted, and invalid depth data with values ​​of zero or exceeding the reasonable depth range are filtered out, retaining all valid depth values; the minimum depth value is selected from all valid depth values ​​and used as the depth reference value of the current corner point to be photographed; finally, the three-dimensional spatial coordinates of the corner point are calculated by combining the camera intrinsic parameters with the depth reference value and the pixel coordinates of the corner point.

[0078] This invention achieves fully automated identification of weld seam locations on workpieces using global depth imaging, eliminating the need for standard CAD models and manual teaching. It adapts to workpiece dimensional errors, structural deformations, and placement deviations. By constructing a local coordinate system based on the workpiece's geometric features and performing clustering and sorting, it generates continuous and efficient scanning paths, effectively reducing equipment idle travel and reversal frequency. Combining contour geometric relationships, it automatically calculates the optimal shooting pose, ensuring the quality of weld seam image acquisition and ultimately outputting ordered pose data that the robot can directly execute. This invention achieves fully automated, robust, and efficient planning of weld seam imaging points on large workpieces, effectively solving the problems of traditional solutions such as reliance on preset models, poor workpiece adaptability, redundant motion paths, and low overall operational efficiency.

[0079] For example, taking the welding scenario of a large mesh-like industrial pallet steel component as an example, firstly, a large-field-of-view 3D camera fixed on a gantry is used to acquire the overall depth image of the pallet; after processing the depth image, the outer frame contour and internal mesh contour of the pallet are extracted, and the weld corner points at the intersection of all mesh profiles are identified to form a set of corner points to be photographed; a local coordinate system is established based on the long side of the pallet's outer contour, and all corner points are projected and clustered by column, with adjacent columns sorted in reverse order to form an ordered sequence of points that are continuously scanned along the short side of the workpiece; according to this sequence, the optimal shooting position and optical axis angle are calculated for each corner point in combination with the direction of the profile sides on both sides, and then converted into pose data that can be executed by the robotic arm through pre-completed hand-eye calibration parameters, finally outputting a complete set of ordered photographing poses for the robotic arm to drive the end-effector small-field-of-view camera to perform high-definition weld image acquisition.

[0080] Figure 4 The flowchart of the automatic planning method for weld seam photography points of industrial pallet steel components based on depth map includes steps 1 to 5.

[0081] Step 1: Input the panoramic depth map data (depth) of the workpiece acquired by the eye-to-hand large field-of-view camera. Perform hand-eye calibration on the eye-to-hand large field-of-view camera, the eye-to-hand small field-of-view camera, and the robotic arm to obtain the hand-eye calibration matrix. and .

[0082] Step 2: Perform 2D corner point recognition of the workpiece contour using the panoramic depth map data. The specific process is as follows: Figure 5 As shown, the depth image is filtered to remove environmental background interference; the depth image is binarized and then connected component filtering is performed to remove connected components smaller than a given threshold, retaining clear depth information of the industrial pallet steel components, and outputting a binary image mask; for the binary workpiece image mask, the contour information of the workpiece is calculated and the inner and outer contours are distinguished, the contour point sequence is fitted into a sequence of straight polygon vertices, and the convex hull of each contour is further calculated to filter the corner points for photography; for each polygon vertex sequence fitted to the inner contour, the corner points located on its convex hull are retained; all retained corner points are traversed, and if there are no other corner points less than a given distance threshold near the corner point, a perpendicular is drawn to the nearest outer contour edge, and the perpendicular point that meets the given direction constraint is added to the corner point sequence; the final 2D corner point sequence set points is output.

[0083] Step 3, Sorting 2D Corner Points to be Photographed: To achieve a continuous and efficient robot motion path, all generated photo point sets are sorted. The photo point sequence is sorted by each individual contour (e.g., ...). Figure 6 (as shown in a) is changed to a zigzag continuous scanning mode along the short side of the workpiece (e.g.) Figure 6 (as shown in b). This applies to the set of corner points to be photographed. Each point contains a depth Figure 2 3D plane coordinates First, check the number of points n. If n < 2, it indicates that the workpiece has insufficient feature points or only a single point. No complex sorting is needed; directly return the original index sequence as the result. Then, a local coordinate system is constructed based on the geometric features of the workpiece itself:

[0084] Reference vector extraction: Select the long side corner points on the outer contour. Calculate vector Calculate the vector magnitude The basis vectors of the new coordinate system are defined as follows:

[0085] X-axis direction: This direction is defined as the main scanning direction of the workpiece, which is parallel to the long side of the pallet.

[0086] Y-axis direction: This direction is perpendicular to the main scanning direction.

[0087] All points in the original point set 'points' Projecting onto the newly established local coordinate system eliminates interference caused by workpiece rotation: for any point Calculate relative to the origin vector Calculate the new coordinates using the dot product formula. :

[0088] , .

[0089] Global clustering segmentation based on the X-axis is used to achieve "column-style" scanning. All clicked points' in the new coordinate system are divided into several vertical columns: points' are sorted in ascending order by x'; the current cluster (current_cluster) is initialized; the sorted points are traversed, and the difference Δx between the current point and the last point in the previous cluster is calculated; a gap threshold D is set. If Δx > D, a new column is entered, the current cluster is sorted in ascending order by y' and added to the result list (result_clusters), and the current_cluster is cleared to start a new cluster; if Δx <= D, the current point is added to the current_cluster and the traversal continues; after the traversal, the remaining points in the current_cluster are sorted in ascending order by y' and added to the result list; the result list (result_clusters) is concatenated, even-numbered columns maintain the ascending order of the y-axis, and odd-numbered columns have their point sequences reversed. The final sorted result is shown below. Figure 6As shown in b, alternating scanning along the short side of the workpiece reduces the movement path of the robotic arm and lowers the risk of collision.

[0090] Step 4: For each 2D corner point to be photographed, perform 3D photographing pose calculation based on its surrounding contour information. First, obtain the depth value of the corner point to be photographed. For each 2D contour corner point p(x,y), in order to eliminate edge noise and improve depth estimation accuracy, the system does not directly use single-pixel depth, but adopts a local region statistical method.

[0091] Boundary calculation: Set a predefined search radius R. Calculate the boundary of the clipping region:

[0092] ,

[0093] ,

[0094] Where W and H are the width and height of the depth map, respectively.

[0095] Extracting depth values: Extract rectangular regions of interest (ROIs) from the global depth map. Retain all non-zero depth values ​​within the ROI and extract the minimum depth value d (the depth value represents the distance between the point and the large camera; the minimum value represents the highest point on the workpiece surface) as the depth value of the contour corner point. If there is no valid depth value within the ROI, skip that point.

[0096] According to 3D camera internal parameters Scale of depth sensor Convert the 2D corner pixel coordinates p(x,y) to 3D spatial coordinates P(X,Y,Z):

[0097] , ,

[0098] Calculate the 3D spatial pose of the eye in a small field of view camera on the hand, and calculate the 3D coordinates of the current point. and its adjacent points on the contour , The solution process is as follows: Figure 7 As shown:

[0099] Calculation of translation position T: Determine the direction of horizontal plane translation as the side. With edge The angle bisector, the unit vector in that direction The calculation formula is as follows:

[0100] ,

[0101] Based on the given horizontal translation length Horizontal offset point Then, according to the given vertical translation length... Calculate the final camera position ,in This is represented by the reverse direction of the Z-axis in the large camera coordinate system, which is the direction from the workpiece closer to the origin of the large camera.

[0102] Rotational pose R calculation: The rotational pose in 3D space can be represented by three unit vectors, representing the orientation of the x-axis, y-axis, and z-axis, respectively. The three unit vectors are paired and perpendicular. This refers to the orientation of the camera's optical axis. For example... Figure 7 As shown, the rotation pose of the 3D camera is calculated as follows: the z-axis orientation is the unit vector pointing from the camera position to the 3D corner point, i.e. The x-axis is perpendicular to ' Plane, that is, perpendicular to and ,pass and The cross product can be calculated. The last pass and Cross product is obtained Finally, the pose in coordinate system 1 of the large camera is obtained: =(( ), ).

[0103] Based on the hand-eye calibration information in step 1, the pose is... Calculate the pose of the corresponding robot base coordinate system. : .

[0104] Step 5: Output the robot pose corresponding to the image capture point of the eye-on-hand small field-of-view camera. The eye-on-hand small field-of-view camera moves to each image capture point one by one according to the pose, triggering the small camera to capture high-definition weld seam images, thereby guiding the welding robot to complete precise welding.

[0105] The technical solution of this embodiment acquires a global depth image of the workpiece to be welded. Through workpiece contour extraction and feature analysis, it identifies the set of all corner points corresponding to the weld seam to be photographed. A local reference coordinate system is established based on the geometric features of the workpiece's outer contour. All corner points to be photographed are projected and clustered to generate a continuously scanned, ordered sequence of photographing points. Then, the camera's 3D shooting pose is calculated according to the sequence order and the geometric relationship between adjacent contour edges of each corner point. This pose is then converted into an executable pose in the robot's base coordinate system and output. It can adapt to workpiece size deformation and placement deviations, fully automatically covering the entire process of photographing point recognition, path optimization, and pose calculation. This effectively improves the automation level and operational efficiency of weld seam photographing point planning, ensures stable imaging quality, and fully adapts to the automated welding needs of scenarios where workpiece deviations exist.

[0106] According to embodiments of the present invention, an automatic weld seam photographic point planning device corresponding to the automatic weld seam photographic point planning method is also provided. See also Figure 2 The schematic diagram of an embodiment of the device of the present invention shown herein may include: an acquisition unit 101, a point planning unit 102, and a pose calculation unit 103.

[0107] The acquisition unit 101 is configured to acquire a global depth image of the workpiece to be welded.

[0108] A global depth image is a two-dimensional depth mapping image that can fully reflect the overall shape and spatial distance information of the workpiece to be welded. The value of each pixel in the image corresponds to the straight-line spatial distance between the corresponding position on the workpiece surface and the acquisition device, which can directly reflect the three-dimensional shape characteristics of the workpiece.

[0109] In some implementations, the global depth image is acquired by a fixed first 3D camera; the 3D shooting pose of the camera corresponds to a second 3D camera mounted on the end effector of the robot; the first 3D camera and the second 3D camera are respectively pre-calibrated for coordinate system transformation.

[0110] The first 3D camera is fixedly mounted above the workpiece to be welded using a gantry bracket and a fixed frame, maintaining a constant spatial position and orientation during operation. The first 3D camera performs a single 3D imaging acquisition of the workpiece within the workstation, outputting a global depth image covering the entire shape of the workpiece. The calculated 3D shooting pose of the camera is entirely targeted at the second 3D camera mounted at the robot's end effector, with the pose data corresponding to the optical center position and optical axis orientation of the second 3D camera. When the robot moves the end effector to the designated pose, the second 3D camera is positioned in the preset shooting position and orientation, completing the acquisition of local weld seam images. This dual-camera collaborative architecture balances the dual requirements of global workpiece coverage and high-precision acquisition of local weld seams, resolving the technical contradiction of the inability to simultaneously achieve both the field of view and accuracy of a single camera.

[0111] Before the formal operation, a hand-eye calibration process was performed on both cameras: for the fixed first 3D camera, the transformation matrix between the first 3D camera coordinate system and the robot base coordinate system was obtained; for the second 3D camera installed at the robot's end effector, the transformation matrix between the second 3D camera coordinate system and the robot end effector coordinate system was obtained. The two sets of calibration matrices together constitute a complete coordinate transformation link, used to convert the point and pose data of global visual recognition into a base coordinate system pose that the robot can recognize and execute. Through the mechanism of hand-eye calibration of the two cameras, a complete coordinate transformation link was established, breaking down the coordinate barrier between visual perception and robot execution, and ensuring the accuracy of the planned pose execution.

[0112] The point planning unit 102 is configured to extract the workpiece contour and analyze its features from the global depth image to identify the set of corner points to be photographed corresponding to the weld.

[0113] The set of corner points to be photographed refers to the collection of coordinates of all contour feature corner points on the workpiece that need to be acquired for high-definition weld images. These corner points correspond to the weld positions at the junctions of steel component profiles and are the core target points for weld image acquisition.

[0114] Welds in mesh-like steel components are typically located at the corner points of the profile intersections, and there is a strong correspondence between the workpiece contour corner points and the weld acquisition points. By extracting contours and analyzing features, these points can be located directly from the actual acquired depth images, eliminating the need for pre-importing a standard CAD model of the workpiece or manually marking the points. This approach can adapt to actual dimensional deviations and structural deformations of the workpiece. Specifically, the global depth image is used to distinguish between the workpiece area and the background area, extracting the complete edge contour information of the workpiece. Combining the correspondence between the welds and contour corner points of the mesh-like steel components, feature points of all contours are analyzed and filtered to extract all valid feature corner points corresponding to the weld positions. Finally, the coordinates of the corner points to be photographed are compiled.

[0115] In some implementations, the point planning unit 102 performs workpiece contour extraction and feature analysis on the global depth image to identify the set of corner points to be photographed corresponding to the weld, including:

[0116] The point planning unit 102 is further configured to preprocess the global depth image to extract and distinguish the outer contour of the workpiece to be welded from at least one inner contour.

[0117] The purpose of preprocessing is to remove irrelevant and interfering information, retain the effective workpiece area, and improve the accuracy of subsequent feature extraction. The outer contour refers to the outermost closed edge contour of the workpiece to be welded; this contour constitutes the overall outer boundary of the workpiece and is not surrounded by other contours of the workpiece itself. The inner contour refers to the closed edge contour inside the workpiece, surrounded by the outer contour. This type of contour corresponds to the internal boundaries formed by internal holes and profile partitions in a mesh-like workpiece, and its corner points correspond to the weld positions inside the workpiece. For example... Figure 6 As shown in Figure a, the outer contour refers to the outermost contour, which includes the contour containing points 25, 26, 27, and 28; the rest of the contours are inner contours.

[0118] The original depth image contains a large amount of background information about the workstation environment and supporting structures, as well as tiny noise generated during the imaging process. Directly extracting the contours will result in a large number of invalid contours, interfering with corner point recognition. Mesh-shaped workpieces have multiple layers of closed contours, and different contours at different locations correspond to different weld seam distribution patterns. Distinguishing between inner and outer contours allows for the application of different corner point extraction strategies for the two types of contours, improving the accuracy and completeness of the extraction. Specifically, firstly, background areas with depth values ​​exceeding the workpiece's range are removed by depth threshold filtering. Then, the remaining workpiece area is binarized and connected component calculations are performed, deleting tiny interfering connected components with areas smaller than a set threshold. Subsequently, edge detection is performed on the processed workpiece area to extract all closed contour lines. Finally, contour inclusion relationships are determined to distinguish the outermost outer contour from all inner contours enclosed by it.

[0119] The point planning unit 102 is specifically configured to extract polygon features from each inner contour and filter out candidate corner points corresponding to the inner contour.

[0120] Polygon feature extraction specifically involves fitting continuous contour pixels into a polygonal structure composed of several line segments, and extracting the vertices of the polygon as contour feature points. Candidate corner points refer to contour vertices with weld seam corner features obtained from the initial screening of the inner contour, and are the basis for subsequent completion and screening.

[0121] The original contour consists of a large number of continuous pixels. Directly using all contour points would result in numerous redundant points and would fail to locate the true weld corner positions. The internal welds of the grid-like steel components are distributed at the corner positions where the profiles intersect, corresponding one-to-one with the polygon vertices of the inner contour. Through polygon fitting and feature filtering, redundant contour points can be eliminated, and the corner points corresponding to the internal welds can be accurately located. Specifically, for each inner contour, the continuous sequence of contour points is fitted into a polygon structure composed of straight line segments, and all vertices of the polygon are extracted. Then, feature filtering is performed on the polygon vertices, retaining vertices with obvious corner features as candidate corner points for the inner contour.

[0122] The point planning unit 102 is specifically configured to fill in missing corner points by making perpendiculars to the outer contour edge for candidate corner points that meet the preset distance constraints, and add the perpendicular fill points that meet the preset direction constraints to the set of corner points to be photographed.

[0123] Preset distance constraints refer to pre-defined distance thresholds used to determine whether other identified corner points exist around the candidate corner point, thus identifying any missing points. Perpendicular foot completion refers to drawing a perpendicular line from the candidate corner point to the contour edge, using the intersection of the perpendicular and the contour edge as the point to fill in any missing weld points on the contour edge. Preset direction constraints refer to pre-defined angle rules used to filter perpendicular foot completion points, retaining only those whose directions conform to the weld distribution pattern and filtering out invalid points.

[0124] Extracting only inner contour corner points will miss weld points on the outer contour edges, resulting in incomplete weld acquisition coverage. Triggering completion logic through distance constraints can accurately locate areas with missing points, avoiding redundant points generated by indiscriminate completion. Filtering perpendicular points through direction constraints can filter out invalid perpendicular points whose directions do not conform to the weld distribution, avoiding the mistaken addition of points in non-weld locations. Specifically, all candidate corner points are traversed to determine if there are other identified corner points within a preset distance range around the candidate corner point. If no other corner points are found within the range, it is determined that there is a missing weld point on the corresponding outer contour edge. A perpendicular line is drawn from the candidate corner point to the nearest outer contour edge to obtain the perpendicular point. Further, it is determined whether the direction of the line connecting the perpendicular point and the candidate corner point conforms to preset direction rules. Perpendicular points that conform to the rules are added to the set of corner points to be photographed as completion corner points.

[0125] The point planning unit 102 is further configured to construct a local reference coordinate system based on the geometric features of the outer contour of the workpiece to be welded, and then perform clustering and sorting on the set of corner points to be photographed to the local reference coordinate system to generate an ordered sequence of continuously scanned photographing points.

[0126] The local reference coordinate system is a two-dimensional coordinate system established based on the geometric features of the workpiece's outer contour. This coordinate system closely matches the workpiece's shape and is used to eliminate the interference of workpiece placement angle deviations on path planning, ensuring that the sorting logic adapts to the actual shape of the workpiece. Cluster sorting refers to a processing method that first groups points according to their distribution interval characteristics on the coordinate axes, and then sorts the points within each group. This is used to organize scattered points into a continuous scanning path.

[0127] The placement angle of the workpiece at the workstation exhibits random deviations. Directly using the world coordinate system for point sorting leads to a mismatch between the planned path and the actual shape of the workpiece, resulting in a large amount of redundant idle travel. Establishing a local coordinate system based on the workpiece's own outer contour can completely eliminate the interference of placement angles. Furthermore, clustering and sorting can organize scattered and disordered points into a continuous scanning path, reducing the robot's idle travel and the number of reversals. Specifically, the geometric features of the reference edge of the workpiece's outer contour are extracted, and a local reference coordinate system conforming to the workpiece's shape is established using this reference edge as the coordinate axis reference. Through coordinate transformation operations, the original coordinates of all corner points to be photographed are transformed and projected into this local reference coordinate system. Clustering and grouping are performed according to the distribution interval characteristics of the points on the coordinate axes, and finally, an ordered sequence of photographing points corresponding to a continuous scanning path is formed.

[0128] In some implementations, the point planning unit 102 projects the set of corner points to be photographed onto the local reference coordinate system and then performs clustering and sorting to generate an ordered sequence of photographed points for continuous scanning, including:

[0129] The point planning unit 102 is specifically configured to project each corner point to be photographed onto the local reference coordinate system, and then perform gap clustering along the first coordinate axis to obtain multiple sets of corner point columns.

[0130] Coordinate projection specifically involves transforming the coordinates of points in the original coordinate system into coordinates in a local reference coordinate system. After transformation, the relative positions of the points remain consistent with the shape of the workpiece. Gap clustering refers to an automatic grouping method that uses the coordinate intervals of points on a specified coordinate axis as the criterion, grouping consecutive points with coordinate differences less than a set threshold into the same group and points with coordinate differences exceeding the threshold into different groups, without the need to pre-set the number of groups. Corner point columns refer to each set of corner points obtained after clustering along the first coordinate axis. Corner points within the same group are close in position on the first coordinate axis and arranged along the second coordinate axis. The first coordinate axis corresponds to the main extension direction of the workpiece and is used to determine the arrangement direction of the scanning column; the second coordinate axis corresponds to the sorting direction of the corner points within the column and is used to determine the scanning order within a single column. The two coordinate axes are perpendicular to each other.

[0131] The weld corner points of a grid-like workpiece naturally exhibit a column-like distribution. However, when the workpiece's placement angle deviates, directly grouping them in the original coordinate system can lead to column misalignment and grouping chaos. Projecting to a local coordinate system that fits the workpiece ensures that the grouping logic matches the actual shape of the workpiece. Automatic grouping using gap clustering eliminates the need to pre-set the number of columns and column spacing, adapting to workpieces of different specifications and sizes and avoiding the limitations of manually preset parameters. Specifically, through coordinate transformation, all corner points to be photographed are converted from the original image coordinate system to the local reference coordinate system, obtaining the coordinates of each corner point in the new coordinate system. All corner points are then sorted in ascending order along the first coordinate axis. The sorted corner points are traversed, and the coordinate difference between the current corner point and the last corner point in the current group on the first coordinate axis is calculated sequentially. If the coordinate difference is greater than a preset gap threshold, the current corner point is assigned to a new group; if the coordinate difference is less than or equal to the threshold, the current corner point is assigned to the current group. After traversal, all groups consist of multiple columns of corner points arranged along the first coordinate axis.

[0132] The point planning unit 102 is further configured to sort each group of corner points along the second coordinate axis, with adjacent corner point groups using opposite sorting directions along the second coordinate axis, and each group of corner point groups connected end to end along the first coordinate axis to obtain a continuously scanned ordered sequence of photographing points.

[0133] The opposite sorting direction refers to the alternating ascending and descending sorting rules used for two adjacent sets of corner point columns, so that the last point of the previous column is spatially adjacent to the starting point of the next column. End-to-end connection refers to sequentially piecing together the sorting results of all corner point columns along the first coordinate axis to form a complete and continuous sequence of points.

[0134] If all corner point columns use the same sorting direction, after scanning each column, the robot needs to return from the end of the column to the beginning of the column before moving to the beginning of the next column, resulting in a large amount of invalid empty travel and increasing the number of robot reversals. Using reverse sorting of adjacent columns ensures that the end point of the previous column is spatially adjacent to the beginning point of the next column. After scanning the previous column, the robot can directly move to the next column, eliminating invalid return travel between columns, reducing robot reversals, improving motion continuity, and lowering the risk of equipment collisions. Specifically, for all corner points within each group of corner point columns, coordinate sorting is performed along the second coordinate axis. Along the arrangement order of the first coordinate axis, adjacent groups of corner point columns use opposite sorting directions; that is, when the previous group is arranged in ascending order, the next group is arranged in descending order, alternating in this manner. Finally, according to the arrangement order from left to right on the first coordinate axis, the sorting results of all corner point columns are sequentially spliced ​​together to form a complete and ordered sequence of image points. Figure 6As shown in b, the ascending order of the point numbers is the order in which the robotic arm drives the camera to perform the acquisition operation. All corner points to be photographed are clustered into multiple vertical corner point columns along the long side of the workpiece. Adjacent corner point columns are connected end to end in opposite sorting directions to form a continuous scanning path that goes back and forth along the short side of the workpiece. Among them, 0, 5, 23, and 28 are corner points of the outer contour of the pallet, and each corner point of the internal grid corresponds to the acquisition position of the weld seam at the intersection of the profiles.

[0135] The pose calculation unit 103 is configured to calculate the camera 3D shooting pose corresponding to each shooting point according to the order of the ordered shooting point sequence and the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed, and convert the camera 3D shooting pose into an executable pose in the robot base coordinate system and output it.

[0136] The camera 3D shooting pose refers to the complete pose data containing the camera's three-dimensional spatial position coordinates and three-axis rotation attitude. It is used to determine the shooting position and optical axis orientation of the second 3D camera 2, directly determining the acquisition angle and quality of the weld seam image. The robot base coordinate system refers to the three-dimensional coordinate system established with the fixed origin of the robot base as the reference. It is the reference coordinate system for the industrial robot to execute motion commands, and all robot-executable pose data must be transformed to this coordinate system.

[0137] The quality of weld seam image acquisition is directly related to the camera's shooting angle. Calculating the pose based on the geometric relationship of the contour edges ensures that the camera's optical axis matches the weld seam area, resulting in clear and complete weld seam features. Calculating and outputting the pose sequentially in an ordered order ensures that the robot executes the acquisition task along the optimal path, avoiding efficiency losses caused by disordered movement. Specifically, according to the ordered sequence of shooting points, for each corner point to be photographed, the 3D spatial position and rotational attitude of the camera at the corresponding shooting location are calculated by combining the geometric directions of the adjacent contour edges on both sides of the corner point. Through a pre-defined coordinate system transformation relationship, the pose data in the camera coordinate system is converted into an executable motion pose in the robot's base coordinate system, ultimately outputting the ordered pose data for all points.

[0138] In some implementations, the pose calculation unit 103 calculates the camera 3D shooting pose corresponding to each shooting point based on the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed, including:

[0139] The pose calculation unit 103 is specifically configured to calculate the angle bisector direction vector based on the direction vectors of the two adjacent contour edges of the corner point to be photographed.

[0140] Adjacent contour edges refer to the two contour line segments on either side of the corner to be photographed, which together form the corner, corresponding to the two profile edges on the steel component that intersect at the weld point. Angle bisector direction vector refers to the unit direction vector corresponding to the angle bisector of the angle between the two adjacent contour edges. This direction is located at the center of symmetry of the two contour edges and is the reference direction for the horizontal offset of the camera.

[0141] Specifically, two adjacent contour edges on both sides of the corner point to be photographed are extracted. The direction vectors corresponding to the two edges are calculated by using the coordinates of the two endpoints of each edge. The two direction vectors are then normalized. The two unit direction vectors are vector-added and then normalized again to finally obtain the angle bisector direction vector of the angle between the two contour edges.

[0142] The pose calculation unit 103 is further configured to offset a preset distance in the workpiece plane along the direction of the angle bisector, and then offset a preset height in the direction perpendicular to the workpiece plane to determine the spatial position of the camera.

[0143] The workpiece plane refers to the reference plane on which the main surface of the workpiece to be welded lies. The spatial position of the camera refers to the coordinate point of the camera's optical center in three-dimensional space, used to determine the camera's shooting position, and is the translation component of the camera's 3D pose.

[0144] Horizontal offset along the angle bisector allows the camera to be positioned away from directly above the weld, preventing interference between the camera tip and the workpiece structure while ensuring complete coverage of the weld area on both sides of the corner point. Vertical elevation by a preset height places the camera within its depth-of-field range, ensuring clear weld imaging and further mitigating the risk of collision between the camera and the workpiece. Specifically, starting from the corner point to be photographed, a preset horizontal distance is translated within the workpiece plane along the angle bisector to obtain the midpoint after horizontal offset. Then, starting from this midpoint, a preset vertical height is translated along a direction perpendicular to the workpiece plane and away from the workpiece to finally obtain the three-dimensional spatial coordinates of the camera's optical center.

[0145] The pose calculation unit 103 is further configured to calculate the three-axis rotational attitude of the camera by taking the direction of the camera pointing to the corner point to be photographed as the optical axis direction and combining the angle bisector direction, so as to obtain the 3D shooting pose of the camera.

[0146] The optical axis direction refers to the direction of the normal to the center of the camera lens, which is the direction of the central line of sight for the camera image. It corresponds to the z-axis of the camera coordinate system and directly determines the central area of ​​the shot and the orientation of the viewpoint. The three-axis rotational pose is a rotation matrix composed of three pairwise perpendicular unit vectors (corresponding to the camera's x, y, and z-axis directions, respectively). It is used to uniquely determine the camera's shooting orientation in three-dimensional space and is the rotational component of the camera's 3D pose.

[0147] Pointing the optical axis towards the corner point to be photographed ensures that the weld corner point is always centered in the camera's field of view, guaranteeing a clear and complete core acquisition area. Constructing two additional coordinate axes through vector cross products ensures both the orthogonality of the camera coordinate system and alignment of the camera's imaging direction with the weld contour direction. Specifically, the vector pointing from the camera's spatial position to the corner point is used as the initial optical axis direction and normalized to become the z-axis vector of the camera coordinate system. The angle bisector direction vector is cross-multiplied with the optical axis direction vector to obtain a unit vector perpendicular to both vectors, which becomes the x-axis vector of the camera coordinate system. The optical axis direction vector is then cross-multiplied with the x-axis vector to obtain the y-axis vector of the camera coordinate system. These three pairwise orthogonal unit vectors (x, y, and z) constitute the camera's rotational posture, which, combined with the previously obtained camera spatial position, forms the complete 3D camera pose.

[0148] In some implementations, when calculating the camera 3D shooting pose corresponding to each shooting point, for each corner point to be photographed, the effective depth values ​​in the preset area around the corner point are extracted, and the minimum depth value among them is selected as the depth reference value of the corner point, which is used to calculate the 3D spatial coordinates of the corner point.

[0149] The effective depth value refers to the depth value within the preset area that conforms to the depth range and can truly reflect the distance information of the workpiece surface, excluding invalid zero values ​​or abnormal values ​​caused by imaging failure, occlusion, or noise. 3D spatial coordinates refer to the actual position coordinates of the corner point to be photographed in three-dimensional space.

[0150] Depth images commonly exhibit imaging noise and abrupt changes in depth values ​​at the edges of workpiece contours. Directly using single-pixel depth values ​​at corner locations can lead to significant errors or even invalid values, resulting in inaccurate subsequent 3D coordinate calculations. Employing a local region statistical approach can utilize the depth information of surrounding pixels to offset the interference of single-pixel noise, improving the stability of depth values. Furthermore, the depth value corresponds to the distance between the acquisition device and the workpiece surface; a smaller value indicates a closer distance. The weld seam of the mesh steel component is located on the upper surface of the profile, the closest point to the acquisition device. Selecting the minimum depth value within this region accurately corresponds to the actual weld seam location on the upper surface of the profile, avoiding interference from depth values ​​from the profile sides, workpiece grooves, or background areas in the weld seam location calculation.

[0151] Specifically, when calculating the camera pose corresponding to each corner point to be photographed, a local image region is first defined with the pixel coordinates of the corner point as the center and according to a preset region size; the depth values ​​corresponding to all pixels in this region are extracted, and invalid depth data with values ​​of zero or exceeding the reasonable depth range are filtered out, retaining all valid depth values; the minimum depth value is selected from all valid depth values ​​and used as the depth reference value of the current corner point to be photographed; finally, the three-dimensional spatial coordinates of the corner point are calculated by combining the camera intrinsic parameters with the depth reference value and the pixel coordinates of the corner point.

[0152] This invention achieves fully automated identification of weld seam locations on workpieces using global depth imaging, eliminating the need for standard CAD models and manual teaching. It adapts to workpiece dimensional errors, structural deformations, and placement deviations. By constructing a local coordinate system based on the workpiece's geometric features and performing clustering and sorting, it generates continuous and efficient scanning paths, effectively reducing equipment idle travel and reversal frequency. Combining contour geometric relationships, it automatically calculates the optimal shooting pose, ensuring the quality of weld seam image acquisition and ultimately outputting ordered pose data that the robot can directly execute. This invention achieves fully automated, robust, and efficient planning of weld seam imaging points on large workpieces, effectively solving the problems of traditional solutions such as reliance on preset models, poor workpiece adaptability, redundant motion paths, and low overall operational efficiency.

[0153] Since the processing and functions implemented by the device in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0154] The technical solution of this invention uses a global depth image of the workpiece to be welded as input. Through contour extraction and feature filtering, it locates all corner points corresponding to the weld seams to be photographed. Combining the geometric features of the workpiece's outer contour, it constructs a local reference coordinate system to complete point projection and clustering, generating a continuous and ordered photographing path. Then, based on the contour geometric features of each corner point, it calculates the appropriate 3D shooting pose for the camera according to the path sequence. After coordinate transformation, it outputs pose data that the robot can directly execute. This approach can adapt to complex scenarios with workpiece size deformation and placement deviations, simultaneously achieving automatic point recognition, intelligent path optimization, and accurate pose calculation, significantly improving planning efficiency and imaging stability, and better meeting the on-site operation needs of automated welding.

[0155] According to an embodiment of the present invention, an automatic weld seam photographic point planning system corresponding to the automatic weld seam photographic point planning device is also provided. This automatic weld seam photographic point planning system may include the aforementioned automatic weld seam photographic point planning device.

[0156] Since the processing and functions implemented by the automatic planning system for weld seam photography points in this embodiment are basically the same as those of the aforementioned device embodiments, principles and examples, any details not covered in this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0157] The technical solution of this invention first acquires the global depth topography data of the workpiece to be welded, extracts the workpiece contour and completes feature analysis, identifies all corner points to be photographed at weld locations, establishes a local reference coordinate system based on the geometric features of the workpiece's outer contour, and performs projection, clustering, and sorting processing on all corner points to generate a continuously scanned, ordered sequence of photographing points. Then, according to the sequence order and combined with the geometric parameters of the adjacent contour edges of each corner point, the corresponding 3D camera pose is calculated, converted into an executable pose in the robot base coordinate system, and output. This eliminates the need for pre-importing a standard workpiece model and manually setting points, automatically adapting to workpiece size deformation and placement deviations, and simultaneously completing global point recognition, motion path optimization, and photographing pose calculation. This effectively improves the automation level and operational efficiency of the planning operation, enhances the stability of weld acquisition imaging, and improves the adaptability to on-site scenes.

[0158] According to an embodiment of the present invention, a storage medium corresponding to the automatic planning method for weld seam photographing points is also provided. The storage medium includes a stored program, wherein the program controls the device where the storage medium is located to execute the automatic planning method for weld seam photographing points described above when it is running.

[0159] Since the processing and functions implemented by the storage medium in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0160] The technical solution of this invention involves acquiring a global depth image of the workpiece to be welded, identifying all weld seam corners to be photographed through contour extraction and feature analysis, establishing a local reference coordinate system based on the geometric features of the workpiece's outer contour, and generating a continuous and efficient ordered sequence of photographing points through clustering and sorting after projecting the corner points. Following the sequence arrangement, the camera's 3D shooting pose is calculated based on the geometric relationships between adjacent contour edges of each corner point. After coordinate transformation, the robot-executable pose result is output. This method can adapt to the dimensional deformation and positional deviations of the workpiece, effectively improving the operational efficiency and robustness of weld seam photographing planning.

[0161] According to an embodiment of the present invention, a computer program product corresponding to the automatic planning method for weld seam photographing points is also provided. The computer program product includes a computer program that, when processed and executed, implements the steps of the above-described automatic planning method for weld seam photographing points.

[0162] Since the processing and functions implemented by the computer program product in this embodiment are basically corresponding to the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0163] The technical solution of this invention acquires a global depth image of the workpiece to be welded, performs workpiece contour extraction and feature analysis, identifies the set of corner points to be photographed corresponding to all weld seams, constructs a local reference coordinate system based on the geometric features of the workpiece's outer contour, projects the corner points onto this coordinate system, performs clustering and sorting, generates an ordered sequence of continuously scannable photographing points, and then calculates the camera 3D shooting pose corresponding to each point according to the sequence order and the geometric relationship of the adjacent contour edges of each corner point, converts it into an executable pose in the robot base coordinate system, and outputs it. This method can adapt to operating scenarios with workpiece size deformation and placement deviations, automatically completes global point identification, path optimization, and pose calculation, effectively improving planning efficiency and imaging quality stability, and better meeting the high-efficiency and stable operation requirements of automated welding.

[0164] In summary, it is readily understood by those skilled in the art that, without conflict, the aforementioned advantageous methods can be freely combined and superimposed.

[0165] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for automatically planning weld seam photographic points, characterized in that, include: Obtain the global depth image of the workpiece to be welded; The global depth image is used to extract the workpiece contour and perform feature analysis to identify the set of corner points to be photographed corresponding to the weld. A local reference coordinate system is constructed based on the geometric features of the outer contour of the workpiece to be welded. The set of corner points to be photographed is then projected onto the local reference coordinate system and clustered and sorted to generate an ordered sequence of photographed points for continuous scanning. According to the ordered sequence of shooting points, based on the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed, the camera 3D shooting pose corresponding to each shooting point is calculated, and the camera 3D shooting pose is converted into an executable pose in the robot base coordinate system and output.

2. The automatic planning method for weld seam photography points according to claim 1, characterized in that, The global depth image is used for workpiece contour extraction and feature analysis to identify the set of corner points to be photographed corresponding to the weld, including: The global depth image is preprocessed to extract and distinguish the outer contour and at least one inner contour of the workpiece to be welded. Polygon features are extracted from each inner contour, and candidate corner points corresponding to the inner contour are obtained by filtering. For candidate corner points that meet the preset distance constraints, make perpendicular feet to the outer contour edge to fill in the missing corner points, and add the perpendicular feet fill points that meet the preset direction constraints to the set of corner points to be photographed.

3. The automatic planning method for weld seam photography points according to claim 1 or 2, characterized in that, After projecting the set of corner points to be photographed onto the local reference coordinate system, clustering and sorting are performed to generate an ordered sequence of continuously scanned photograph points, including: After projecting each corner point to be photographed onto the local reference coordinate system, gap clustering is performed along the first coordinate axis to obtain multiple sets of corner point columns. Each set of corner points is sorted along the second coordinate axis, and adjacent sets of corner points are sorted in opposite directions along the second coordinate axis. Each set of corner points is then connected end to end along the first coordinate axis to obtain a continuously scanned, ordered sequence of photographic points.

4. The automatic planning method for weld seam photography points according to claim 1, characterized in that, Based on the geometric relationships of the adjacent contour edges corresponding to each corner point to be photographed, the 3D shooting pose of the camera corresponding to each shooting point is calculated, including: Calculate the angle bisector direction vector based on the direction vectors of the two adjacent contour sides of the corner point to be photographed; The camera's spatial position is determined by offsetting a preset distance within the workpiece plane along the direction of the angle bisector, and then offsetting a preset height along a direction perpendicular to the workpiece plane. Using the direction in which the camera points to the corner point to be photographed as the optical axis direction, and combining the direction of the angle bisector, the three-axis rotation attitude of the camera is calculated to obtain the 3D shooting pose of the camera.

5. The automatic planning method for weld seam photography points according to claim 4, characterized in that, When calculating the 3D shooting pose of the camera corresponding to each shooting point, for each corner point to be photographed, the effective depth value in the preset area around the corner point is extracted, and the minimum depth value is selected as the depth reference value of the corner point to calculate the 3D spatial coordinates of the corner point.

6. The automatic planning method for weld seam photographing points according to any one of claims 1 to 5, characterized in that, The global depth image is acquired by a fixed first 3D camera; the 3D shooting pose of the camera corresponds to a second 3D camera installed at the end of the robot; the first 3D camera and the second 3D camera have respectively completed hand-eye calibration in advance for coordinate system transformation.

7. An automatic planning device for weld seam photographing points, characterized in that, include: The acquisition unit is configured to acquire a global depth image of the workpiece to be welded; The point planning unit is configured to extract the workpiece contour and analyze its features from the global depth image, and identify the set of corner points to be photographed corresponding to the weld. The point planning unit is also configured to construct a local reference coordinate system based on the geometric features of the outer contour of the workpiece to be welded, and to perform clustering and sorting on the set of corner points to be photographed after projecting them onto the local reference coordinate system to generate an ordered sequence of photographing points for continuous scanning. The pose calculation unit is configured to calculate the camera 3D shooting pose corresponding to each shooting point according to the order of the ordered shooting point sequence and the geometric relationship of the adjacent contour edges corresponding to each corner point to be photographed, and convert the camera 3D shooting pose into an executable pose in the robot base coordinate system and output it.

8. An automatic planning system for weld seam photographing points, characterized in that, include: The automatic planning device for weld seam photography points as described in claim 7.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the automatic planning method for weld seam photographing points as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the automatic planning method for weld seam photographing points as described in any one of claims 1 to 6.

Images