Method, device, equipment, storage medium and program product for guiding hanging on sheet metal

Abstract

The application provides a sheet metal hanging guiding method, device, equipment, storage medium and program product, belongs to the intelligent manufacturing field, and through the introduction of the rigid structure identification piece template and the hook template combined template matching process, realizes the double template matching positioning mechanism, reduces the calculation amount of directly searching the small size hook with large deformation in the whole graph point cloud, effectively improves the operation beat; and, since the search space is accurately constrained in the target interested region, the accuracy of the hook point cloud matching can still be greatly improved under the condition of low matching threshold. In addition, the data recognition mode depending on the point cloud geometric structure instead of the two-dimensional image gray scale is adopted, the black and white color difference interference of the hook caused by the powder spraying is reduced, and finally the success rate and precision of the mechanical arm hanging position determination are significantly improved, realizing the high efficiency, low dropping rate and automatic hanging operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing, and in particular to a method, apparatus, equipment, storage medium, and program product for guiding sheet metal hanging. Background Technology

[0002] With the continuous development of intelligent manufacturing, industrial robots combined with vision guidance technology have been widely used in automated grasping and placing scenarios in factory production lines (such as sheet metal painting and assembly).

[0003] As the hanging fixtures circulate, the hooks are prone to deformation, and they become thicker and their color changes from black to white as the number of powder spraying cycles increases.

[0004] Existing visual guidance solutions cannot adapt to changes in the shape and color of the hangers, making visual guidance prone to failure, affecting the success rate of hanging sheet metal, and ultimately impacting work efficiency. Summary of the Invention

[0005] This invention provides a method, apparatus, equipment, storage medium, and program product for guiding sheet metal hanging, in order to solve the problem that visual guidance solutions in related technical solutions cannot adapt to changes in the shape and color of the hanging fixture, which makes visual guidance prone to failure, affects the success rate of sheet metal hanging, and ultimately affects work efficiency.

[0006] This invention provides a method for guiding sheet metal mounting, comprising: Acquire the point cloud data of the hanger, wherein the point cloud data of the hanger is the point cloud data of the hanger; A rigid structural identifier template is used to perform template matching on the hanger point cloud data. At least one first point cloud data corresponding to the rigid structural identifier is determined in the hanger point cloud data. The rigid structural identifier is a structural component in the hanger whose volume is larger than the hook volume. In the point cloud data of the hanging device, taking the first point cloud data as the starting point, a target region of interest is determined at a preset distance value in a preset direction. The preset direction and the preset distance value are determined based on the actual physical position of the rigid structural marker and the hook. The region of interest is matched with a hook template to obtain the hook pose corresponding to each hook. Based on the hook pose corresponding to the hook, the hanging pose of the robotic arm is determined; The robotic arm is controlled to perform sheet metal hanging operations based on the hanging posture.

[0007] The present invention provides a method for guiding sheet metal mounting, wherein the mounting fixture includes a first hook and a second hook, the first hook and the second hook being hooks located at both ends of the same bracket; the step of determining the mounting posture of the robotic arm based on the hook postures corresponding to the hooks includes: Obtain the first hook pose corresponding to the first hook and the second hook pose corresponding to the second hook; The upper hook position coordinates are determined based on the first position coordinates and the second position coordinates, where the first position coordinates are the position coordinates in the first hook pose and the second position coordinates are the position coordinates in the second hook pose. The X-axis of the hanging pose is defined by connecting the X-axis coordinates in the first position coordinates and the X-axis coordinates in the second position coordinates. The normal vector of the first fitting plane is used as the Z-axis of the hanging pose, and the Y-axis of the hanging pose is determined based on the right-hand rule. The first fitting plane is a plane obtained by fitting the point cloud data corresponding to the first hook and the point cloud data corresponding to the second hook. Based on the defined X-axis, Y-axis and Z-axis of the hanging pose, the hanging spatial posture of the robotic arm is determined. Based on the above-mounted position coordinates and the above-mounted spatial posture, the above-mounted posture of the robotic arm is determined.

[0008] The method for guiding sheet metal hanging provided by the present invention includes a first position coordinate that is the position coordinate of the apex of the hook in the first hook, and a second position coordinate that is the position coordinate of the apex of the hook in the second hook; The above hanging position coordinates are the coordinates of the center point between the first position coordinates and the second position coordinates.

[0009] The method for guiding sheet metal mounting provided by the present invention, wherein determining the mounting posture of the robotic arm based on the mounting position coordinates and the mounting spatial posture includes: Based on the above-mounted position coordinates and the above-mounted spatial attitude, determine the above-mounted pose in the camera coordinate system; The extrinsic parameters obtained from hand-eye calibration are used to transform the hanging pose in the camera coordinate system to obtain the hanging pose in the robotic arm coordinate system.

[0010] The method for guiding sheet metal mounting provided by the present invention further includes: The robotic arm is controlled to drive the camera to perform hand-eye calibration by taking pictures of the calibration plate from multiple positions and angles. External parameters are determined based on hand-eye calibration results; The extrinsic parameters are used to describe the transformation relationship between the camera coordinate system and the robotic arm coordinate system.

[0011] The method for guiding sheet metal mounting provided by the present invention further includes: Obtain the structural design parameters of the hanger, the structural design parameters including the first spatial position coordinates corresponding to the rigid structural marker and the second spatial position coordinates corresponding to the hook; The direction from the first spatial coordinate to the second spatial coordinate is taken as the preset direction, and the spatial distance between the first spatial coordinate and the second spatial coordinate is taken as the preset distance value.

[0012] The method for guiding sheet metal mounting provided by the present invention further includes: Obtain the structural design parameters of the hanger, the structural design parameters including the first spatial position coordinates corresponding to the rigid structural marker and the second spatial position coordinates corresponding to the hook; The direction from the first spatial coordinate to the second spatial coordinate is taken as the initial preset direction, and the spatial distance between the first spatial coordinate and the second spatial coordinate is taken as the initial preset distance value; Based on a preset number of historical matching results, the initial preset direction and the initial preset distance value are adaptively corrected to obtain the preset direction and the preset distance value.

[0013] The method for guiding sheet metal mounting provided by this invention includes, in the step of acquiring the point cloud data of the mounting fixture: The robotic arm is controlled to drive the camera to take pictures of the entire hanging device, thereby obtaining a two-dimensional image of the hanging device and a depth image of the hanging device; Point cloud data of the hanger is constructed based on the two-dimensional image and the depth image of the hanger.

[0014] The present invention provides a method for guiding sheet metal hanging, wherein the bracket is a cross bracket, and each cross bracket and the hook on the cross bracket constitute a hanging unit; The control of the robotic arm to perform sheet metal hanging operations based on the hanging posture includes: Perform a preset anomaly detection on each of the above-mentioned attached units; Determine the target attached unit that passes the preset anomaly detection, and each target attached unit corresponds to an attached pose; The robotic arm is controlled to perform the hanging operation sequentially according to the hanging posture corresponding to each target hanging unit; The preset anomaly includes one or more of the following: Abnormal hanging angle, missing or broken hooks, deformed hooks or brackets, abnormal hook spacing.

[0015] The method for guiding sheet metal mounting provided by the present invention further includes: Based on the pitch angle in the above-mounted space attitude exceeding the above-mounted angle range, an abnormal above-mounted angle is determined, and an abnormal above-mounted angle alert is output.

[0016] The method for guiding sheet metal mounting provided by the present invention further includes: The target region of interest is matched using a hook template to obtain the identified hook point cloud contour. If the hook point cloud outline is inconsistent with the hook outline corresponding to the hook template, it is determined that there is a missing hook or a broken hook, and an abnormal hook status reminder is output.

[0017] The method for guiding sheet metal mounting provided by the present invention further includes: Based on the target region of interest, determine the hook vertex region; Obtain the number of point clouds located within the hook vertex region; If the number of point clouds exceeds the preset point cloud number range, determine the hook deformation or support deformation, and output an abnormal reminder message for the hook deformation or support deformation.

[0018] The method for guiding sheet metal mounting provided by the present invention further includes: If the distance between the first hook and the second hook is determined to be greater than a preset tolerance threshold based on the first position coordinates and the second position coordinates, the hook distance is determined to be abnormal, and an abnormal hook distance reminder is output.

[0019] This invention provides a device for guiding sheet metal hanging, comprising: The acquisition unit is used to acquire the point cloud data of the hanger, wherein the point cloud data of the hanger is the point cloud data of the hanger; The first matching unit is used to perform template matching on the hanger point cloud data using a rigid structure identifier template, and to determine at least one first point cloud data corresponding to the rigid structure identifier in the hanger point cloud data. The rigid structure identifier is a structural component in the hanger whose volume is larger than the hook volume. The first determining unit is used to determine the target region of interest in the point cloud data of the hanger, with the first point cloud data as the starting point and at a preset distance value in a preset direction. The preset direction and the preset distance value are determined based on the actual physical position of the rigid structure marker and the hook. The second matching unit is used to perform template matching on the target region of interest using a hook template to obtain the hook pose corresponding to each hook. The second determining unit is used to determine the hanging posture of the robotic arm based on the hook posture corresponding to the hook. The control unit is used to control the robotic arm to perform sheet metal hanging operations based on the hanging posture.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of guiding sheet metal mounting as described above.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for guiding sheet metal mounting as described above.

[0022] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for guiding sheet metal mounting as described above.

[0023] The present invention provides a method, apparatus, device, storage medium, and program product for guiding sheet metal hanging. By introducing a template matching process combining rigid structural marker templates and hook templates, a dual template matching positioning mechanism is achieved. Specifically, this mechanism first matches large, non-deformable rigid structural marker features within a massive point cloud dataset to obtain stable and reliable first point cloud data. Subsequently, spatial translation is performed using inherent physical connection size relationships to directly generate a small-scale target region of interest. Finally, fine template matching is performed on hooks that are prone to deformation and discoloration within this local area. This process reduces the computational load of directly searching for small-sized hooks with large deformation in the entire point cloud, effectively improving the computational cycle time. Furthermore, because the search space is precisely constrained within the target region of interest, the accuracy of hook point cloud matching can still be significantly improved even with a low matching threshold. In addition, the use of a data recognition method relying on point cloud geometry rather than two-dimensional image grayscale reduces interference from black-and-white differences caused by powder coating on the hooks, ultimately significantly improving the success rate and accuracy of determining the hanging pose of the robotic arm, achieving highly efficient and low-drop-rate automated hanging operations. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is one of the flowcharts illustrating the method for guiding sheet metal mounting provided by the present invention; Figure 2 This is a flowchart illustrating the process of determining the hanging posture of a robotic arm based on the hook posture corresponding to the hook, as provided by the present invention. Figure 3 This is a flowchart illustrating the process of determining the hanging posture of a robotic arm based on the hanging position coordinates and the hanging spatial posture provided by the present invention. Figure 4 This is the second flowchart illustrating the method for guiding sheet metal hanging provided by the present invention; Figure 5 This is a schematic diagram of the process for obtaining point cloud data of the hanger provided by the present invention; Figure 6 This is the third flowchart illustrating the method for guiding sheet metal hanging provided by the present invention; Figure 7 This is the fourth flowchart illustrating the method for guiding sheet metal hanging provided by the present invention; Figure 8 This is a schematic diagram of a 2D image of the hanger provided by the present invention; Figure 9 This is a schematic diagram of the depth of the mounting bracket provided by the present invention; Figure 10 This is a schematic diagram of the point cloud data of the hanger provided by the present invention; Figure 11 This is a schematic diagram illustrating the segmentation and template matching of hanger point cloud data under a large field of view environment provided by the present invention; Figure 12 This is a schematic diagram of template matching of the hanger point cloud data using a rigid structure marker template provided by the present invention; Figure 13 This is a schematic diagram of the ROI including the hook, obtained by translating the first point cloud data according to the present invention. Figure 14 This is a schematic diagram of template matching of the target region of interest using a hook template, provided by the present invention; Figure 15 This is a schematic diagram of the two hook vertices provided by the present invention; Figure 16 This is a schematic diagram illustrating the determination of the X-axis, Y-axis, and Z-axis provided by the present invention; Figure 17 This is a schematic diagram of the abnormal hook spacing provided by the present invention; Figure 18 This is a schematic diagram of point cloud data with a normal hook status provided by the present invention; Figure 19 This is a schematic diagram of point cloud data showing an abnormal hook status provided by the present invention; Figure 20This is a schematic diagram of the point cloud data of the bending fixture provided by the present invention; Figure 21 This is a schematic diagram of point cloud data with an upward-sloping hanging angle provided by the present invention; Figure 22 This is a schematic block diagram of the device for guiding sheet metal hanging provided by the present invention; Figure 23 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0026] Figure label: 2201. Acquisition unit; 2202. First matching unit; 2203. First determining unit; 2204. Second matching unit; 2205. Second determining unit; 2206. Control unit; 2310. Processor; 2320. Communication interface; 2330. Memory; 2340. Communication bus. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0029] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0030] This invention provides a method for guiding sheet metal hanging. In practical applications, this method can be executed by an electronic device, such as an industrial control computer, server, or controller in a robotic arm system, which possesses data processing and computing capabilities. This embodiment does not specifically limit the executing entity. The following describes in detail the specific implementation steps of the method for guiding sheet metal hanging provided by this invention, using a control device as the executing entity.

[0031] The following is combined Figures 1-23 This invention describes the method, apparatus, equipment, storage medium, and program products for guiding sheet metal mounting.

[0032] Figure 1 This is one of the flowcharts illustrating the method for guiding sheet metal mounting provided by the present invention, such as... Figure 1 As shown, the method for guiding the hanging of sheet metal includes: Step 101: Obtain the point cloud data of the hanging device. The point cloud data of the hanging device is the point cloud data of the hanging device.

[0033] In industrial production lines for sheet metal painting or assembly, hangers typically refer to load-bearing fixtures mounted on an accumulation chain to suspend sheet metal parts. Control equipment can acquire the three-dimensional spatial information of the hangers using 3D vision sensors associated with robotic arms or the production line, thereby obtaining hanger point cloud data containing the three-dimensional coordinates of each point on the hanger's surface. This hanger point cloud data accurately reflects the geometric structural features of the hanger in three-dimensional space and is unaffected by changes in ambient light or the color of the hanger's surface.

[0034] Step 102: Use the rigid structure identifier template to perform template matching on the hanger point cloud data. In the hanger point cloud data, determine at least one first point cloud data corresponding to the rigid structure identifier. The rigid structure identifier is a structural component in the hanger whose volume is greater than that of the hook.

[0035] Rigid structural markers refer to large, simple, and rigid components on the hanger that are not easily deformed during transport. For example, in some specific applications, this rigid structural marker can be a fixed bracket on the hanger in the shape of a cross. Because the hooks on the hanger are prone to deformation and thickening during repeated powder spraying and handling, directly matching the entire hook image is often unsuccessful and time-consuming. Therefore, the control equipment pre-constructs a rigid structural marker template based on its three-dimensional geometric features. Using a 3D point cloud registration algorithm, the control equipment performs global or local feature comparison and search on the rigid structural marker template within the entire hanger point cloud data, identifying the 3D data region with the highest consistency with the template's geometric features. This successfully matched region is then extracted as the first point cloud data, thus achieving large-scale coarse positioning of the stable rigid structure on the hanger.

[0036] Step 103: In the point cloud data of the hanger, starting from the first point cloud data, determine the target region of interest at a preset distance value in a preset direction. The preset direction and preset distance value are determined based on the actual physical position of the rigid structure marker and the hook.

[0037] The preset direction and preset distance values ​​are determined based on the actual physical positions of the rigid structural marker and the hook. Specifically, since the rigid structural marker and the hook belong to the same hanger, there is a fixed physical structural connection between them. The preset direction and preset distance values ​​are three-dimensional spatial translation vectors that characterize this fixed rigid connection. After identifying the first point cloud data and acquiring its spatial pose, the control device translates the spatial position of the first point cloud data according to the three-dimensional spatial translation vector (i.e., the specified preset direction and preset distance values). This allows the control device to directly define a local point cloud region containing the hook to be located within the massive hanger point cloud data, i.e., the Region of Interest (ROI). Through this offset mapping based on physical topology, even if the hook itself experiences deformation errors of more than ±10 millimeters due to long-term circulation, the generated ROI can still accurately cover the actual location of the hook.

[0038] Step 104: Use the hook template to perform template matching on the target region of interest to obtain the hook pose corresponding to each hook.

[0039] The hook template is a pre-extracted 3D geometric model of a standard hook. Since the control device has already narrowed the search range from a vast full-map point cloud to a local region of interest in the preceding steps, it only needs to match the hook's 3D features within this small local point cloud using point cloud registration or feature comparison algorithms. In this process, because it relies on the distribution patterns of the point cloud rather than 2D grayscale or color features, it can effectively overcome the significant color difference interference caused by the hook being black before powder coating and white after powder coating. After successful matching, the control device can extract the spatial position and orientation of the hook within the target region of interest, thereby obtaining the precise hook pose.

[0040] Step 105: Determine the hanging position of the robotic arm based on the hook pose corresponding to the hook.

[0041] The hanging pose refers to the spatial position and angle that the end effector of the robotic arm needs to reach in order to accurately place the sheet metal part on the hook. The control equipment uses the actual pose of the hook in three-dimensional space as a reference, and through the corresponding spatial geometric coordinate transformation, calculates the optimal working position coordinates and spatial pose required for the robotic arm to align the sheet metal part and hang it on the hook, thereby generating the hanging pose data to guide the movement of the robotic arm.

[0042] Step 106: Control the robotic arm to perform sheet metal hanging operations based on the hanging posture.

[0043] The control device generates motion control commands containing the hanging posture information and sends them to the underlying controller of the robotic arm. This drives the robotic arm to move the clamped sheet metal part to the target spatial position and completes the assembly action of the sheet metal part onto the hook according to the hanging posture and trajectory, thereby realizing an automated hanging process.

[0044] In this embodiment, a dual template matching positioning mechanism is achieved by introducing a template matching process that combines rigid structural marker templates with hook templates. Specifically, this mechanism first matches the features of large, non-deformable rigid structural markers in the massive point cloud data to obtain stable and reliable first point cloud data. Subsequently, spatial translation is performed using the inherent physical connection size relationship to directly generate a small-scale target region of interest. Finally, fine template matching is performed on hooks that are easily deformable and discolored within this local area. In this process, the computational workload of directly searching for small-sized hooks with large deformation in the entire point cloud is reduced, effectively improving the computation cycle time. Furthermore, since the search space is precisely constrained within the target region of interest, the accuracy of hook point cloud matching can still be significantly improved even with a low matching threshold. In addition, the use of a data recognition method that relies on point cloud geometry rather than two-dimensional image grayscale reduces the black-and-white difference interference caused by powder spraying on the hooks, ultimately significantly improving the success rate and accuracy of determining the hook hanging pose of the robotic arm, and realizing highly efficient and low-drop-rate automated hanging operations.

[0045] In some embodiments, the hanger includes a first hook and a second hook, wherein the first hook and the second hook are hooks located at both ends of the same bracket; such as Figure 2 As shown, based on the hook pose corresponding to the hook, the hanging pose of the robotic arm is determined, including: Step 201: Obtain the first hook pose corresponding to the first hook and the second hook pose corresponding to the second hook.

[0046] In the physical structure of an actual hanger, a complete upper hanging position typically includes a first hook and a second hook located at opposite ends of the same rigid support, with the first and second hooks situated on the left and right sides of the support, respectively. Based on the template matching results in the target region of interest, the control device extracts the three-dimensional spatial data of the first hook and the second hook, respectively, thereby determining their poses. This provides the underlying data foundation for subsequent comprehensive consideration of the spatial geometry of the two associated hooks.

[0047] Step 202: Determine the upper hook position coordinates based on the first position coordinates and the second position coordinates. The first position coordinates are the position coordinates in the first hook pose, and the second position coordinates are the position coordinates in the second hook pose.

[0048] In this step, the control device extracts the first position coordinates from the first hook pose and the second position coordinates from the second hook pose. Based on this data, the control device performs numerical fusion calculations on the first and second position coordinates and uses the center point coordinates as the overall hanging position coordinates.

[0049] In some embodiments, the first position coordinate is the position coordinate of the hook vertex in the first hook, and the second position coordinate is the position coordinate of the hook vertex in the second hook; the upper hanging position coordinate is the coordinate of the center point between the first position coordinate and the second position coordinate.

[0050] Step 203: Define the X-axis of the hanging pose based on the line connecting the X-axis coordinates in the first position coordinates and the X-axis coordinates in the second position coordinates.

[0051] After determining the coordinates of the center point of the upper hook, the control device needs to construct a reference coordinate system for the upper hook movement in three-dimensional space. The control device calculates the vector connecting the first and second position coordinates in three-dimensional space and defines the direction of this vector as the X-axis direction of the upper hook pose. The X-axis generated in this way can adapt in real time to the relative tilt or height difference between the two hooks in space.

[0052] Step 204: Use the normal vector of the first fitting plane as the Z-axis of the hanging pose, and determine the Y-axis of the hanging pose based on the right-hand rule. The first fitting plane is a plane obtained by fitting the point cloud data corresponding to the first hook and the point cloud data corresponding to the second hook.

[0053] Local point cloud data belonging to the first hook and the second hook are extracted. A spatial plane fitting algorithm is then used to fit the overall point cloud data of these two hooks, resulting in a first fitting plane that characterizes the overall spatial orientation of the two hooks. Subsequently, the control device calculates the normal vector of this first fitting plane and uses its direction as the Z-axis of the hook's pose. After determining the X-axis and Z-axis vectors, the control device automatically derives and determines the Y-axis direction of the hook's pose based on the right-hand rule of the Cartesian coordinate system.

[0054] Step 205: Determine the hanging space posture of the robotic arm based on the defined X-axis, Y-axis, and Z-axis of the hanging pose.

[0055] Based on the above, the spatial rotation matrix required for the end effector of the robotic arm to simultaneously adapt to the two hooks is calculated using the three-axis (X-axis, Y-axis, Z-axis) direction vectors of the new coordinate system reconstructed from the geometric features of the two hooks. This determines the hanging spatial posture of the robotic arm.

[0056] Step 206: Determine the hanging pose of the robotic arm based on the hanging position coordinates and hanging spatial posture.

[0057] The control device mathematically combines the calculated upper-mounted position coordinates, representing spatial translation, with the upper-mounted spatial attitude, representing spatial rotation, to generate a complete upper-mounted pose transformation matrix with six degrees of freedom. This upper-mounted pose can then be directly used to guide the robotic arm in performing subsequent motion control.

[0058] In this embodiment, by introducing a data processing mechanism based on the geometric relationship of the two hooks, after obtaining the position coordinates and point cloud data of the first and second hooks, the coordinates of a single hook are not used as the absolute hanging reference. Instead, the center point is obtained by calculating the first and second position coordinates to generate the hanging position coordinates. The X-axis, Z-axis and Y-axis of the spatial coordinate system are reconstructed using the vector connecting the two positions and the normal vector of the hook point cloud fitting plane as the hanging spatial posture. This avoids the local coordinate error caused by the independent deformation of a single hook, and avoids the interference problem caused by the robotic arm rigidly moving according to a fixed teaching trajectory, which results in the workpiece being hooked on one hook and the other hooked. This ensures that the sheet metal workpiece can be accurately and simultaneously aligned with the two hooks, significantly improving the adaptability and success rate of the hanging operation.

[0059] In some embodiments, such as Figure 3As shown, based on the upper hanging position coordinates and the upper hanging spatial attitude, the upper hanging pose of the robotic arm is determined, including: Step 301: Determine the upper mount pose in the camera coordinate system based on the upper mount position coordinates and upper mount spatial attitude.

[0060] The camera coordinate system is a three-dimensional spatial coordinate reference system established with the camera's optical center as the origin. Since the point cloud data of the hanger acquired in the preliminary steps is generated directly by the camera, the hanger position coordinates and hanger spatial attitude extracted and calculated from this point cloud data are naturally based on the camera coordinate system in the initial state. At the data processing level, the control equipment fuses the translation data of the hanger position coordinates with the rotation features of the hanger spatial attitude through data stitching and matrix encapsulation to construct a transformation matrix or pose vector that can accurately describe the target's working state in the camera's field of view, thereby accurately determining the hanger pose in the camera coordinate system.

[0061] Step 302: Use the extrinsic parameters obtained from hand-eye calibration to transform the hanging pose in the camera coordinate system to obtain the hanging pose in the robotic arm coordinate system.

[0062] The robotic arm coordinate system refers to the physical space coordinate system used by the underlying control system of the robotic arm, specifically the robotic arm tool coordinate system or the base coordinate system. The extrinsic parameters obtained from hand-eye calibration refer to transformation matrix data pre-calculated and stored by the system, used to quantitatively describe the relative spatial position and angle transformation relationship between the camera coordinate system and the robotic arm tool coordinate system. In actual data processing, to enable the robotic arm to understand the coordinate commands given by the vision system, the control device calls this extrinsic parameter data and performs spatial geometric coordinate transformation operations, such as homogeneous matrix multiplication, between the mounted pose data in the camera coordinate system and this extrinsic parameter. This accurately maps the vision-perceived 3D coordinates and posture to the physical execution space of the robotic arm. The output result after the transformation calculation is the mounted pose in the robotic arm coordinate system, which is a low-level motion control parameter that can be directly recognized, parsed, and executed by the robotic arm controller.

[0063] In this embodiment, by introducing data conversion processing from the camera coordinate system to the robotic arm coordinate system, the spatial transformation matrix of the hanging pose in the camera coordinate system is calculated using the extrinsic parameters obtained from hand-eye calibration. This allows the virtual 3D target coordinates accurately identified and calculated by the camera to be seamlessly and accurately converted into physical execution parameters in the robotic arm tool coordinate system, eliminating coordinate system deviations in visual guidance and ensuring that the robotic arm can move precisely to the target hook position, thereby guaranteeing the execution accuracy of the entire automated hanging system in 3D physical space.

[0064] In some embodiments, such as Figure 4 As shown, the method for guiding sheet metal hanging also includes: Step 401: Control the robotic arm to drive the camera to perform hand-eye calibration by shooting the calibration plate from multiple positions and angles.

[0065] In actual industrial deployments, 3D vision sensors are typically mounted on the flange of a robotic arm using a bracket, forming an eye-in-hand vision guidance architecture. A calibration plate, a reference target with a known high-precision geometric pattern, is fixedly placed within the robotic arm's workspace. The control device sends calibration motion commands to the robotic arm, driving it to move the camera to multiple different spatial positions and angles above or around the calibration plate. During each stop, the camera acquires image and depth data from the fixed calibration plate. By collecting multiple sets of data pairs containing robotic arm joint position data and camera imaging data, the system can obtain a sufficient spatial sample size for subsequent data processing.

[0066] In some embodiments, the 3D vision sensor is a camera, such as a wide-field-of-view 3D camera.

[0067] Step 402: Determine the extrinsic parameters based on the hand-eye calibration results; whereby the extrinsic parameters are used to describe the transformation relationship between the camera coordinate system and the robotic arm coordinate system.

[0068] The control equipment runs a hand-eye calibration algorithm, extracting the position information of calibration feature points from multiple sets of acquired calibration board images. Combined with the pose matrix of the robotic arm's end-effector coordinate system fed back by the robotic arm control system at the corresponding moment, a system of homogeneous spatial transformation equations in the form "AX=XB" is constructed and solved simultaneously. The output of the equations is the hand-eye calibration result, which includes the camera's intrinsic and extrinsic parameters. The extrinsic parameter is essentially a three-dimensional rigid body transformation matrix containing rotation and translation vectors. By extracting this extrinsic parameter, the control equipment can accurately obtain the relative spatial mapping relationship between the origin and axis of the camera coordinate system and the robotic arm tool coordinate system.

[0069] In this embodiment, by introducing a data processing process that controls the robotic arm to drive the camera to take pictures of a fixed calibration board from multiple positions and angles and to calculate extrinsic parameters, data integration between the visual perception space and the physical execution space of the robotic arm is achieved. This extrinsic parameter data serves as a key spatial transformation benchmark, enabling the system to accurately and seamlessly convert the pose data of the hanging point calculated in the camera's point cloud data space into motion coordinate data in the robotic arm's tool coordinate system, thus achieving data conversion between different coordinate systems.

[0070] In some embodiments, the method for guiding sheet metal mounting further includes: Obtain the structural design parameters of the hanger, including the first spatial position coordinates corresponding to the rigid structural marker and the second spatial position coordinates corresponding to the hook; The direction from the first spatial coordinate to the second spatial coordinate is taken as the preset direction, and the spatial distance between the first spatial coordinate and the second spatial coordinate is taken as the preset distance value.

[0071] In this embodiment, the preset direction and preset distance value can be associated and stored with the rigid structure marker and hook so that they can be directly read during actual use without calculation before each use, thereby improving the efficiency of hanging.

[0072] In some embodiments, the method for guiding sheet metal mounting further includes: Obtain the structural design parameters of the hanger, including the first spatial position coordinates corresponding to the rigid structural marker and the second spatial position coordinates corresponding to the hook; The direction from the first spatial coordinate to the second spatial coordinate is taken as the initial preset direction, and the spatial distance between the first spatial coordinate and the second spatial coordinate is taken as the initial preset distance value; Based on a preset number of historical matching results, the initial preset direction and initial preset distance value are adaptively corrected to obtain the preset direction and preset distance value.

[0073] In this embodiment, the preset direction and preset distance value can be adaptively corrected based on historical matching results, thereby increasing the probability of successful hook-up.

[0074] In some embodiments, such as Figure 5 As shown, the point cloud data of the hanger is obtained, including: Step 501: Control the robotic arm to drive the camera to take pictures of the entire hanging device, and obtain a two-dimensional image and a depth image of the hanging device.

[0075] Specifically, the control device drives the robotic arm to move to the preset optimal global imaging position in one go, triggering a single-exposure image of the entire fixture at the current workstation using a wide-field-of-view 3D camera. Through this single-image operation, the control device can simultaneously extract two types of front-end visual data: one is a two-dimensional image of the fixture, which records the pixel distribution and surface features of the fixture as a whole from a two-dimensional perspective; the other is a depth image of the fixture, which records the absolute physical distance of each part of the fixture's surface from the camera lens within the field of view, i.e., depth information. This global wide-field-of-view imaging method ensures that all working areas on a fixture can be completely captured in a single image.

[0076] Step 502: Construct point cloud data of the hanger based on the two-dimensional image and depth image of the hanger.

[0077] Two-dimensional images only provide the two-dimensional pixel coordinates (X-axis and Y-axis features) of the hanger on a plane, while depth images supplement the crucial three-dimensional spatial Z-axis depth features. The control device extracts target pixels from the two-dimensional image and reads the depth values ​​corresponding to the same locations in the depth image. Then, it uses a pre-calibrated camera intrinsic parameter matrix to perform spatial geometric calculations on these data. Through this calculation, the control device maps and converts the pixel coordinates of the two-dimensional plane into true three-dimensional spatial coordinates in the camera coordinate system. The control device traverses and processes the effective pixels in the captured image, arraying and combining the calculated massive number of three-dimensional spatial points to ultimately generate hanger point cloud data containing the complete spatial morphology and three-dimensional structural information of the hanger.

[0078] In this embodiment, a large-field-of-view camera fixed to the robotic arm performs a single global image acquisition of the entire hanger. The two-dimensional matrix of the pixel plane is fused with the Z-axis features of the depth values, and point cloud data covering the entire working area is directly reconstructed via camera intrinsic parameter mapping. This replaces the traditional cyclical process of the robotic arm individually photographing, transmitting, calculating, and moving each hook. It reduces the number of image acquisitions and data transmissions by the vision system, saving tens of seconds of redundant time in actual tests. While ensuring a high-precision spatial data foundation for subsequent 3D matching, it overcomes the problem of difficulty in achieving the required hanging cycle time on the production line.

[0079] In some embodiments, the bracket is a cross bracket, and each cross bracket and the hook on the cross bracket constitute an upper hanging unit. The robotic arm is controlled to perform sheet metal hanging operations based on the hanging posture, including: Perform preset anomaly detection on each mounted unit; Identify the target attached units that pass the preset anomaly detection; each target attached unit corresponds to an attached pose. The robotic arm is controlled to perform the hanging operation sequentially according to the hanging posture corresponding to each target hanging unit; The preset exceptions include one or more of the following: Abnormal hanging angle, missing or broken hooks, deformed hooks or brackets, abnormal hook spacing.

[0080] In this embodiment, when multiple first point cloud data are identified in the hanging point cloud data, that is, when multiple cross brackets are identified, the cross brackets and the hooks set on them are used as hanging units. Pre-set anomaly detection is performed on each hanging unit to identify the target hanging unit that can perform the hanging operation. Finally, the robotic arm is driven to perform the hanging operation in sequence according to the hanging posture corresponding to each target hanging unit.

[0081] In this process, batch loading operations can be achieved even when there are different loading units on the same hanger, thereby improving loading efficiency.

[0082] In some embodiments, the method for guiding sheet metal mounting further includes: If the pitch angle in the upper-mounted space attitude exceeds the upper-mounted angle range, it is determined that the upper-mounted angle is abnormal, and an upper-mounted angle abnormality warning is output.

[0083] The hanging spatial attitude includes the spatial rotation angle information that the robotic arm's end effector must possess to complete the hanging operation. The hanging angle range is a set of safety threshold data pre-configured within the control equipment. This data defines the extreme safe physical boundaries of the tilting or rotation allowed for each joint or end flange when the robotic arm approaches the hanger and performs the hanging action. Because the hanger may experience various unforeseen severe bending or posture deviations during actual production, directly driving the robotic arm based on the extreme hanging spatial attitude calculated from the deformed hanger can easily lead to physical interference between the robotic arm and the hanger itself or other peripheral equipment on the production line.

[0084] To avoid the above situation, before generating control commands for the upper-mounted spatial attitude and sending them to the underlying servo system, the control device extracts angle values ​​from the upper-mounted spatial attitude, such as the pitch angle, and compares them with the preset upper and lower limit thresholds of the upper-mounted angle range. If the comparison calculation finds that the calculated pitch angle in the upper-mounted spatial attitude does not fall within the upper-mounted angle range, the control device determines that the current calculation result exceeds the safe execution boundary, that is, the upper-mounted spatial attitude exceeds the upper-mounted angle range.

[0085] If the error is determined to be outside the acceptable range, the control equipment will immediately intercept and stop issuing the current work execution command, and output an abnormal hanging angle alert to the external interface or production line control system. For example, in a practical hook status classification system, the control equipment can classify this type of abnormality into a specific abnormal status code (such as NG4, abnormal hanging angle status), highlight it on the human-machine interface or trigger an audible and visual alarm signal, and simultaneously control the robotic arm to abandon the current hanging action of the sheet metal part (i.e., not hang it), waiting for manual intervention or transfer to the next workstation.

[0086] In this embodiment, extreme hanging angles caused by excessive overall deviation of the hanger's position can be identified in advance, thereby effectively cutting off the transmission of dangerous control data, avoiding the risk of collision caused by the robotic arm forcibly hanging at an abnormal angle, and improving the equipment operation safety and fault tolerance of the automated assembly line.

[0087] In some embodiments, such as Figure 6 As shown, the method for guiding sheet metal hanging also includes: Step 601: Use the hook template to perform template matching on the target region of interest to obtain the identified hook point cloud outline.

[0088] In this embodiment, the hook template can be compared with the pattern formed by the point cloud in the target region of interest to identify the hook point cloud in the target region of interest. By connecting the point clouds located at the edge of the hook point cloud, the hook point cloud outline can be obtained.

[0089] Step 602: If the hook point cloud outline is inconsistent with the hook outline corresponding to the hook template, determine that there is a missing hook or a broken hook, and output a hook status abnormality reminder.

[0090] A missing or broken hook refers to a severe physical damage condition where the tip of the hook is broken or the entire hook has detached. If the control equipment finds that the hook point cloud outline does not match the hook outline corresponding to the hook template through feature comparison, it determines that the hook is missing or broken. At this time, an abnormal hook status alert is output. This alert data can carry information such as the location of the abnormal fixture and the abnormality type number to indicate that the current fixture is unavailable.

[0091] In this embodiment, the damage to the hook in the real physical world can be identified, effectively preventing the risk of workpiece falling, being scrapped, or even colliding with equipment caused by the robotic arm forcibly executing the hanging trajectory due to its inability to detect hook damage.

[0092] In some embodiments, such as Figure 7 As shown, the method for guiding sheet metal hanging also includes: Step 701: Determine the hook vertex region based on the target region of interest.

[0093] A bracket is a component on a hanger that connects multiple hooks together to provide spatial structural support, such as a fixed frame in the shape of a "cross".

[0094] In some embodiments, the hook vertex region can be determined in one of the following ways: Move the target region of interest a fixed distance from the bottom of the hook to the top of the hook to obtain the hook vertex region. Use a fixed position region in the target region of interest as the hook vertex region. With the target region of interest as the center, perform hook vertex template recognition within a preset radius to obtain the hook vertex region.

[0095] The values ​​for fixed distance, fixed location area, and preset radius can be selected according to actual usage needs, and will not be elaborated here.

[0096] Step 702: Obtain the number of point clouds located within the hook vertex region.

[0097] Step 703: If the number of point clouds exceeds the preset point cloud number range, determine the hook deformation or support deformation, and output the abnormal reminder information of hook deformation or support deformation.

[0098] In some embodiments, the preset point cloud quantity range can be set according to actual usage needs, and the specific values ​​will not be elaborated here.

[0099] In this embodiment, automatic detection and alerts can be provided for hook deformation or bracket deformation.

[0100] In some embodiments, the method for guiding the hanging of sheet metal further includes: If the distance between the first hook and the second hook is determined to be greater than a preset tolerance threshold based on the first position coordinates and the second position coordinates, the hook distance is determined to be abnormal, and an abnormal hook distance warning is output.

[0101] Specifically, the control device extracts the three-dimensional spatial coordinate values ​​from the first and second position coordinates, and uses a spatial distance calculation algorithm to calculate the actual physical distance between the first and second hooks. Subsequently, the control device compares the calculated actual physical distance with a preset tolerance threshold.

[0102] Among them, the preset tolerance threshold is a pre-set allowable range of distance fluctuation, which is used to characterize the maximum allowable deformation size of the sheet metal part that can be smoothly hooked into two hooks at the same time.

[0103] If the distance between the first hook and the second hook is determined to be within a preset tolerance threshold based on the first and second position coordinates, the control device determines that the geometric distance of the current hook is normal and belongs to the "OK" state where safe assembly is possible. At this time, the control device sends motion control commands including the hanging posture to the robotic arm controller, successfully controlling the robotic arm to perform sheet metal hanging operations based on the hanging posture, completing the assembly cycle of the normal production line.

[0104] Conversely, if the distance between the first and second hooks, determined based on the first and second position coordinates, exceeds a preset tolerance threshold, it indicates that the current fixture has undergone severe structural deformation during long-term operation, falling under the NG1 abnormal state. This means the hooks have been stretched outwards by external force, resulting in excessive distance deviation. If the fixture is forcibly hung in this condition, the sheet metal part cannot be aligned with both hooks simultaneously, easily leading to the workpiece being caught or falling off. Therefore, the robotic arm is controlled to stop the sheet metal hanging operation to avoid executing invalid or destructive motion trajectories. Simultaneously, the control equipment outputs a hook distance abnormality alert to the human-machine interface or the factory manufacturing execution system. This alert can be output as a pop-up window, alarm log, or a specific abnormal status code. By statistically analyzing this abnormality alert data, the factory can clearly understand the specific type of abnormal fixture (for example, if a certain distance abnormality alert is found to occur too frequently, it can be determined that the hooks in the corresponding batch have been severely stretched), thus providing accurate data for subsequent automatic fixture correction or targeted manual inspection and replacement.

[0105] The following is an example illustrating the method for guiding sheet metal mounting proposed in this invention.

[0106] like Figure 8 , Figure 9 and Figure 10 As shown, controlling the robotic arm to move the camera to the imaging position to obtain the image. Figure 8 and Figure 9 ,in, Figure 8 A 2D image of the hanger. Figure 9 For the depth map of the mount, based on Figure 8 and Figure 9 It can obtain point cloud data Figure 10 .

[0107] like Figure 11 As shown, the introduction of a large field of view environment for... Figure 10 By segmenting and template matching the point cloud data of the hangers, we can obtain, for example... Figure 12 The matching result shown (the white area in the image) is also the first point cloud data, where the blue area in the image is the rigid structure identifier template.

[0108] like Figure 13 As shown, the ROI (white rectangle) including the hook is obtained by translating the first point cloud data, and the hook is matched within the ROI. Through filtering and segmentation operations, the coordinates of the two hook vertices and the hook point cloud of each hook position are obtained.

[0109] like Figure 14 As shown, the white area in the image represents the point cloud data corresponding to the hook, while the blue area represents the hook template.

[0110] like Figure 15 and Figure 16As shown, the coordinates of the center point are calculated based on the two hook vertices, and the X-axis is determined by the line connecting the two vertices (1 and 2). Figure 16 (The brown line in the middle), based on the plane fitted by the two hook point clouds ( Figure 16 The yellow plane in the image), use the plane's normal vector as the Z-axis ( Figure 16 (The blue line in the image) The Y-axis is determined according to the right-hand rule. Figure 16 (The green line in the figure) and use extrinsic parameters to achieve coordinate transformation, so as to obtain the hanging pose in the coordinate system of the robotic arm.

[0111] If the above process runs normally, an OK status will be output; if the process fails to output the upper coordinates, an upper abnormal status will be output. Upper abnormal statuses include: NG1: Abnormal hook spacing (e.g.) Figure 17 (As shown): If the hook spacing deviation exceeds the threshold, the hook will not be hung. NG2: Hook status abnormal (e.g.) Figure 18 and Figure 19 As shown, where, Figure 18 For point cloud data in a normal hooking state, Figure 19 (Point cloud data indicating abnormal hook status): The hook tip is missing or broken, and it cannot be hooked; NG3: Hanger bending (e.g.) Figure 20 As shown in the image: the hanger is bent; the hanger position is too offset and it cannot be hung. NG4: Abnormal hanging angle (e.g.) Figure 21 As shown, if the upward mounting angle is raised: if the upward mounting angle exceeds the threshold range, avoid collision and do not mount the vehicle; NG5: Other abnormal status: to be manually identified, do not apply.

[0112] The device for guiding sheet metal hanging provided by the present invention will be described below. The device for guiding sheet metal hanging described below and the method for guiding sheet metal hanging described above can be referred to in correspondence.

[0113] In some embodiments, such as Figure 22 As shown, a device for guiding sheet metal hanging is provided, comprising the following modules: The acquisition unit 2201 is used to acquire the point cloud data of the hanger, which is the point cloud data of the hanger; The first matching unit 2202 is used to perform template matching on the hanger point cloud data using a rigid structure identifier template, and to determine at least one first point cloud data corresponding to the rigid structure identifier in the hanger point cloud data. The rigid structure identifier is a structural component in the hanger whose volume is greater than that of the hook. The first determining unit 2203 is used to determine the target region of interest in the hanger point cloud data, starting from the first point cloud data and at a preset distance value in a preset direction. The preset direction and preset distance value are determined based on the actual physical position of the rigid structure marker and the hook. The second matching unit 2204 is used to perform template matching on the target region of interest using a hook template to obtain the hook pose corresponding to each hook. The second determining unit 2205 is used to determine the hanging posture of the robotic arm based on the hook posture corresponding to the hook. Control unit 2206 is used to control the robotic arm to perform sheet metal hanging operations based on the hanging posture.

[0114] Figure 23 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 23 As shown, the electronic device may include: a processor 2310, a communication interface 2320, a memory 2330, and a communication bus 2340, wherein the processor 2310, the communication interface 2320, and the memory 2330 communicate with each other through the communication bus 2340. The processor 2310 can call logic instructions in the memory 2330 to execute a method for guiding sheet metal mounting. This method includes: acquiring point cloud data of the mounting fixture; performing template matching on the point cloud data of the mounting fixture using a rigid structural identifier template to determine at least one first point cloud data corresponding to a rigid structural identifier in the point cloud data, where the rigid structural identifier is a structural component in the mounting fixture with a volume larger than the hook volume; determining a target region of interest in the point cloud data of the mounting fixture, starting from the first point cloud data and at a preset distance value in a preset direction, where the preset direction and preset distance value are determined based on the actual physical positions of the rigid structural identifier and the hook; performing template matching on the target region of interest using a hook template to obtain a hook pose corresponding to each hook; determining the mounting pose of the robotic arm based on the hook pose; and controlling the robotic arm to perform sheet metal mounting operations based on the mounting pose.

[0115] Furthermore, the logical instructions in the aforementioned memory 2330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for guiding sheet metal hanging provided by the above methods. The method includes: acquiring hanger point cloud data, wherein the hanger point cloud data is the point cloud data of the hanger; performing template matching on the hanger point cloud data using a rigid structural identifier template, determining at least one first point cloud data corresponding to the rigid structural identifier in the hanger point cloud data, wherein the rigid structural identifier is a structural component in the hanger whose volume is larger than the hook volume; determining a target region of interest in the hanger point cloud data, starting from the first point cloud data and at a preset distance value in a preset direction, wherein the preset direction and preset distance value are determined based on the actual physical position of the rigid structural identifier and the hook; performing template matching on the target region of interest using a hook template to obtain a hook pose corresponding to each hook; determining the hanging pose of the robotic arm based on the hook pose; and controlling the robotic arm to perform sheet metal hanging operations based on the hanging pose.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for guiding sheet metal hanging provided by the methods described above. This method includes: acquiring point cloud data of a hanger, the point cloud data being the point cloud data of the hanger; performing template matching on the hanger point cloud data using a rigid structural identifier template, determining at least one first point cloud data corresponding to a rigid structural identifier in the hanger point cloud data, the rigid structural identifier being a structural component in the hanger with a volume larger than the hook volume; determining a target region of interest in the hanger point cloud data, starting from the first point cloud data and at a preset distance value in a preset direction, the preset direction and preset distance value being determined based on the actual physical positions of the rigid structural identifier and the hook; performing template matching on the target region of interest using a hook template to obtain a hook pose corresponding to each hook; determining the hanging pose of a robotic arm based on the hook pose; and controlling the robotic arm to perform sheet metal hanging operations based on the hanging pose.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for guiding sheet metal hanging, characterized in that, include: Acquire the point cloud data of the hanger, wherein the point cloud data of the hanger is the point cloud data of the hanger; A rigid structural identifier template is used to perform template matching on the hanger point cloud data. At least one first point cloud data corresponding to the rigid structural identifier is determined in the hanger point cloud data. The rigid structural identifier is a structural component in the hanger whose volume is larger than the hook volume. In the point cloud data of the hanging device, taking the first point cloud data as the starting point, a target region of interest is determined at a preset distance value in a preset direction. The preset direction and the preset distance value are determined based on the actual physical position of the rigid structural marker and the hook. The region of interest is matched with a hook template to obtain the hook pose corresponding to each hook. Based on the hook pose corresponding to the hook, the hanging pose of the robotic arm is determined; The robotic arm is controlled to perform sheet metal hanging operations based on the hanging posture.

2. The method for guiding sheet metal hanging according to claim 1, characterized in that, The hanger includes a first hook and a second hook, which are hooks located at both ends of the same bracket; Determining the hanging posture of the robotic arm based on the hook posture corresponding to the hook includes: Obtain the first hook pose corresponding to the first hook and the second hook pose corresponding to the second hook; The upper hook position coordinates are determined based on the first position coordinates and the second position coordinates, where the first position coordinates are the position coordinates in the first hook pose and the second position coordinates are the position coordinates in the second hook pose. The X-axis of the hanging pose is defined by connecting the X-axis coordinates in the first position coordinates and the X-axis coordinates in the second position coordinates. The normal vector of the first fitting plane is used as the Z-axis of the hanging pose, and the Y-axis of the hanging pose is determined based on the right-hand rule. The first fitting plane is a plane obtained by fitting the point cloud data corresponding to the first hook and the point cloud data corresponding to the second hook. Based on the defined X-axis, Y-axis and Z-axis of the hanging pose, the hanging spatial posture of the robotic arm is determined. Based on the above-mounted position coordinates and the above-mounted spatial posture, the above-mounted posture of the robotic arm is determined.

3. The method for guiding sheet metal hanging according to claim 2, characterized in that, The first position coordinates are the position coordinates of the hook vertex in the first hook, and the second position coordinates are the position coordinates of the hook vertex in the second hook; The above hanging position coordinates are the coordinates of the center point between the first position coordinates and the second position coordinates.

4. The method for guiding sheet metal hanging according to claim 2, characterized in that, Determining the hanging pose of the robotic arm based on the hanging position coordinates and the hanging spatial posture includes: Based on the above-mounted position coordinates and the above-mounted spatial attitude, determine the above-mounted pose in the camera coordinate system; The extrinsic parameters obtained from hand-eye calibration are used to transform the hanging pose in the camera coordinate system to obtain the hanging pose in the robotic arm coordinate system.

5. The method for guiding sheet metal hanging according to claim 4, characterized in that, The method for guiding the hanging of sheet metal also includes: The robotic arm is controlled to drive the camera to perform hand-eye calibration by taking pictures of the calibration plate from multiple positions and angles. External parameters are determined based on hand-eye calibration results; The extrinsic parameters are used to describe the transformation relationship between the camera coordinate system and the robotic arm coordinate system.

6. The method for guiding sheet metal hanging according to claim 1, characterized in that, The method for guiding the hanging of sheet metal also includes: Obtain the structural design parameters of the hanger, the structural design parameters including the first spatial position coordinates corresponding to the rigid structural marker and the second spatial position coordinates corresponding to the hook; The direction from the first spatial coordinate to the second spatial coordinate is taken as the preset direction, and the spatial distance between the first spatial coordinate and the second spatial coordinate is taken as the preset distance value.

7. The method for guiding sheet metal hanging according to claim 1, characterized in that, The method for guiding the hanging of sheet metal also includes: Obtain the structural design parameters of the hanger, the structural design parameters including the first spatial position coordinates corresponding to the rigid structural marker and the second spatial position coordinates corresponding to the hook; The direction from the first spatial coordinate to the second spatial coordinate is taken as the initial preset direction, and the spatial distance between the first spatial coordinate and the second spatial coordinate is taken as the initial preset distance value; Based on a preset number of historical matching results, the initial preset direction and the initial preset distance value are adaptively corrected to obtain the preset direction and the preset distance value.

8. The method for guiding sheet metal hanging according to claim 1, characterized in that, The acquisition of the hanger point cloud data includes: The robotic arm is controlled to drive the camera to take pictures of the entire hanging device, thereby obtaining a two-dimensional image of the hanging device and a depth image of the hanging device; Point cloud data of the hanger is constructed based on the two-dimensional image and the depth image of the hanger.

9. The method for guiding sheet metal hanging according to claim 2, characterized in that, The bracket is a cross bracket, and each cross bracket and the hook on the cross bracket form an upper hanging unit; The control of the robotic arm to perform sheet metal hanging operations based on the hanging posture includes: Perform a preset anomaly detection on each of the above-mentioned attached units; Determine the target attached unit that passes the preset anomaly detection, and each target attached unit corresponds to an attached pose; The robotic arm is controlled to perform the hanging operation sequentially according to the hanging posture corresponding to each target hanging unit; The preset anomaly includes one or more of the following: Abnormal hanging angle, missing or broken hooks, deformed hooks or brackets, abnormal hook spacing.

10. The method for guiding sheet metal hanging according to claim 9, characterized in that, The method for guiding the hanging of sheet metal also includes: Based on the pitch angle in the above-mounted space attitude exceeding the above-mounted angle range, an abnormal above-mounted angle is determined, and an abnormal above-mounted angle alert is output.

11. The method for guiding sheet metal hanging according to claim 9, characterized in that, The method for guiding the hanging of sheet metal also includes: The target region of interest is matched using a hook template to obtain the identified hook point cloud contour. If the hook point cloud outline is inconsistent with the hook outline corresponding to the hook template, it is determined that there is a missing hook or a broken hook, and an abnormal hook status reminder is output.

12. The method for guiding sheet metal hanging according to claim 9, characterized in that, The method for guiding the hanging of sheet metal also includes: Based on the target region of interest, determine the hook vertex region; Obtain the number of point clouds located within the hook vertex region; If the number of point clouds exceeds the preset point cloud number range, determine the hook deformation or support deformation, and output an abnormal reminder message for the hook deformation or support deformation.

13. The method for guiding sheet metal hanging according to claim 9, characterized in that, The method for guiding the hanging of sheet metal also includes: If the distance between the first hook and the second hook is determined to be greater than a preset tolerance threshold based on the first position coordinates and the second position coordinates, the hook distance is determined to be abnormal, and an abnormal hook distance reminder is output.

14. A device for guiding sheet metal hanging on a plate, characterized in that, include: The acquisition unit is used to acquire the point cloud data of the hanger, wherein the point cloud data of the hanger is the point cloud data of the hanger; The first matching unit is used to perform template matching on the hanger point cloud data using a rigid structure identifier template, and to determine at least one first point cloud data corresponding to the rigid structure identifier in the hanger point cloud data. The rigid structure identifier is a structural component in the hanger whose volume is larger than the hook volume. The first determining unit is used to determine the target region of interest in the point cloud data of the hanger, with the first point cloud data as the starting point and at a preset distance value in a preset direction. The preset direction and the preset distance value are determined based on the actual physical position of the rigid structure marker and the hook. The second matching unit is used to perform template matching on the target region of interest using a hook template to obtain the hook pose corresponding to each hook. The second determining unit is used to determine the hanging posture of the robotic arm based on the hook posture corresponding to the hook. The control unit is used to control the robotic arm to perform sheet metal hanging operations based on the hanging posture.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for guiding sheet metal mounting as described in any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for guiding sheet metal mounting as described in any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for guiding sheet metal mounting as described in any one of claims 1 to 13.

Images