Welding program generation method for humanoid robot and related device

Abstract

The present application relates to the technical field of welding, and provides a welding program generation method for a humanoid robot and related equipment, the welding program generation method for the humanoid robot comprising: acquiring a first workpiece image of a workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and a weld based on the first workpiece image; determining three-dimensional contour points of the weld based on a three-dimensional point cloud of the workpiece to be welded and the segmentation mask; acquiring welding task information, and inferring a welding step based on the welding task information and the three-dimensional contour points; determining welding process parameters based on the welding step, the segmentation mask, the three-dimensional contour points and an initial operation skill of the humanoid robot, and writing the welding process parameters into a welding initial program; and testing the welding initial program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot. Through the technical solution provided by the present application, safe automatic generation of a welding program for a humanoid robot is achieved.

Description

Technical Field

[0001] The present invention relates to the field of welding technology, and in particular to a welding program generation method for a humanoid robot and related equipment. Background Art

[0002] With the development of industry, flexible welding technology has gradually made breakthroughs, and industrial robots have been applied on a large scale in standard welding scenarios. Compared with traditional industrial or mobile robots, humanoid robots have unique advantages in welding tasks.

[0003] Welding programs are key to enabling flexible welding with humanoid robots. Manually programmed welding programs are primarily suitable for large-scale, standardized production scenarios and struggle to meet the demands of small-batch, high-variety, and non-standard flexible welding tasks. Welding path generation methods based on offline programming require manual or sensor calibration of the welding path, which is limited by modeling errors, resulting in low welding efficiency and difficulty ensuring welding quality. Furthermore, welding programs generated based on large models are not stable and secure enough for complex three-dimensional workpieces to support high-precision welding tasks performed by humanoid robots. Summary of the Invention

[0004] The present invention provides a welding program generation method for a humanoid robot and related equipment, which are used to solve the defects in the prior art.

[0005] The present invention provides a method for generating a welding program for a humanoid robot, comprising:

[0006] Acquire a first workpiece image of a workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image;

[0007] Determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask;

[0008] Acquire welding task information, and infer welding steps based on the welding task information and the three-dimensional contour points;

[0009] Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into an initial welding program;

[0010] The initial welding program is tested in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot.

[0011] According to a method for generating a welding program for a humanoid robot provided by the present invention, determining a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image includes:

[0012] Detecting a directional bounding box of the workpiece to be welded by using a multimodal large model;

[0013] Inputting the oriented bounding box and the first workpiece image into a visual basis model to obtain the segmentation mask; wherein the visual basis model includes a bounding box hint encoder, an image encoder, and a mask decoder;

[0014] The bounding box hint encoder is configured to encode the oriented bounding box;

[0015] The image encoder is configured to extract image embedding features from the first workpiece image;

[0016] The mask decoder is configured to process the encoded directional bounding box and the image embedding feature to obtain the segmentation mask.

[0017] According to a method for generating a welding program for a humanoid robot provided by the present invention, determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask comprises:

[0018] Acquire second workpiece images of the workpiece to be welded from multiple viewing angles, and three-dimensional point clouds corresponding to each of the second workpiece images;

[0019] Performing point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain a target three-dimensional point cloud and a three-dimensional bounding box of the workpiece to be welded;

[0020] The three-dimensional contour points of the weld are obtained according to the target three-dimensional point cloud, the three-dimensional bounding box and the segmentation mask.

[0021] According to a method for generating a welding program for a humanoid robot provided by the present invention, the method of inferring welding steps based on the welding task information and the three-dimensional contour points includes:

[0022] The welding task information and the three-dimensional contour points are input into a multimodal large model to obtain the welding steps.

[0023] According to a method for generating a welding program for a humanoid robot provided by the present invention, the method of testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot includes:

[0024] Loading the initial welding program into the simulation environment to simulate the welding process, and detecting the collision state and trajectory running quality of the humanoid robot and the workpiece to be welded during the simulated welding process;

[0025] The initial welding program is optimized according to the collision state and the trajectory running quality to obtain the target welding program.

[0026] According to a method for generating a welding program for a humanoid robot provided by the present invention, before testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot, the method further comprises:

[0027] A simulation environment is constructed according to the segmentation mask, the three-dimensional contour points and the humanoid robot, and the simulation environment is integrated to obtain a simulation engine; wherein the simulation environment includes a welding workbench, a humanoid robot model and a workpiece model.

[0028] The present invention also provides a welding system based on a humanoid robot, comprising:

[0029] A multimodal visual sensing module for acquiring workpiece images and 3D point clouds of the workpiece to be welded;

[0030] A welding program generation module, configured to implement the welding program generation method for a humanoid robot as described in any one of the above items, and obtain a target welding program for the humanoid robot;

[0031] The welding operation module includes a humanoid robot, which is used to drive the humanoid robot to weld the workpiece to be welded based on the target welding program.

[0032] The present invention also provides a welding program generating device for a humanoid robot, comprising:

[0033] a first determining module configured to acquire a first workpiece image of the workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image;

[0034] a second determining module configured to determine the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask;

[0035] an inference module configured to obtain welding task information and infer welding steps based on the welding task information and the three-dimensional contour points;

[0036] a third determining module configured to determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and write the welding process parameters into an initial welding program;

[0037] The testing module is configured to test the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for generating a welding program for a humanoid robot as described above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating a welding program for a humanoid robot as described in any one of the above is implemented.

[0040] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating a welding program for a humanoid robot as described above is implemented.

[0041] The present invention provides a welding program generation method and related equipment for a humanoid robot. First, a first workpiece image of a workpiece to be welded is acquired. Based on the first workpiece image, a segmentation mask corresponding to the workpiece to be welded and a weld is obtained. Three-dimensional contour points of the weld are obtained through the three-dimensional point cloud of the workpiece to be welded and the segmentation mask, thereby accurately identifying the workpiece to be welded and accurately understanding the welding scene. The welding steps are inferred based on the welding task information and the three-dimensional contour points. Then, the welding process parameters are determined based on the welding steps, the segmentation mask, the three-dimensional contour points and the initial operating skills of the humanoid robot. The welding process parameters are written into an initial welding program. To ensure the safety of the welding program, the initial welding program is tested in a simulation environment corresponding to the workpiece to be welded, and a target welding program for the humanoid robot is obtained, thereby realizing safe and automatic generation of the welding program for the humanoid robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a flow chart of the method for generating a welding program for a humanoid robot provided by the present invention.

[0044] Figure 2 FIG. 4 is a schematic diagram of determining a segmentation mask in an embodiment provided by the present invention.

[0045] Figure 3 It is a schematic diagram of generating and testing an initial welding program in an embodiment provided by the present invention.

[0046] Figure 4 Schematic diagram of a welding path generated by the humanoid robot provided by the present invention.

[0047] Figure 5 It is a structural schematic diagram of the humanoid robot-based welding system provided by the present invention.

[0048] Figure 6 It is a structural schematic diagram of the welding program generating device for a humanoid robot provided by the present invention.

[0049] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] In flexible production, the differences in the morphology and properties of welding workpieces lead to many limitations in traditional welding program generation methods: manual teaching programming mostly relies on empirical modeling, and its cognitive rigidity is in fundamental conflict with the dynamic requirements of small batches and multiple batches; although offline programming achieves digital mapping through CAD models, the model-to-entity deviation caused by processing and assembly errors makes it difficult to ensure welding quality; perception methods based on images and point clouds are limited by the ability to represent single-modal data, and have the limitations of low accuracy and poor adaptability; welding program generation methods based on large model interaction have high generalization, but the lack of protection mechanisms makes it difficult to ensure the stability and safety of humanoid robot welding systems. Therefore, how to establish a cross-modal semantic understanding mechanism for welding operation scenarios through the design of a multimodal large model architecture, form a multimodal data-driven humanoid robot welding program automatic generation method, and meet the complex and changing needs of flexible manufacturing, is the key problem that the present invention aims to solve.

[0052] Figure 1 FIG. 1 is a flow chart showing a method for generating a welding program for a humanoid robot according to an exemplary embodiment. Figure 1 As shown, in an exemplary embodiment, the method for generating a welding program for a humanoid robot includes steps 110 to 150, which are described in detail as follows.

[0053] Step 110 : Acquire a first workpiece image of a workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image.

[0054] In an embodiment of the present invention, a first workpiece image of a workpiece to be welded is acquired, and a segmentation mask corresponding to the workpiece to be welded and the weld is determined based on the first workpiece image.

[0055] Step 120 : determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask.

[0056] In this embodiment of the present invention, a 3D point cloud is a data structure representing an object in 3D space and is a collection of multiple 3D coordinate points. Based on the 3D point cloud of the workpiece to be welded and the segmentation mask, the 3D contour points of the weld are determined, providing a precise understanding of the welding scene.

[0057] Step 130: Obtain welding task information, and infer welding steps based on the welding task information and the three-dimensional contour points.

[0058] In the embodiment of the present invention, the welding task information is description information of the welding task, including the requirements to be met for welding. The detailed steps required for welding are accurately inferred based on the welding task information and the three-dimensional contour points.

[0059] Step 140 : determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into an initial welding program.

[0060] In an embodiment of the present invention, the initial operating skills of the humanoid robot are the operating skills of the humanoid robot controller, such as MOV, SET, ARC and other instructions. The above instructions can effectively avoid problems such as humanoid robot movement jitter and loss of control, and ensure the stability of the humanoid robot's welding task planning.

[0061] Based on the determined welding steps, segmentation mask, 3D contour points, and the initial operational skills of the humanoid robot, the welding process parameters are determined and written into the initial welding program, thereby achieving the initial generation of the welding program. The initial welding program is a pre-set basic program with pre-defined basic code.

[0062] Step 150 : testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot.

[0063] In an embodiment of the present invention, in order to ensure the safety of the welding program, a simulation environment is automatically built based on the understanding of the welding scene, and the code in the initial welding program is tested in the simulation environment corresponding to the workpiece to be welded, and then the target welding program is obtained after the test passes.

[0064] In an exemplary embodiment of the present invention, determining a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image includes:

[0065] Detecting a directional bounding box of the workpiece to be welded by using a multimodal large model;

[0066] inputting the directed bounding box and the first workpiece image into a visual base model to obtain the segmentation mask; wherein the visual base model comprises a bounding box prompt encoder, an image encoder and a mask decoder;

[0067] The bounding box prompt encoder is configured to encode the directed bounding box.

[0068] The image encoder is configured to extract image embedding features in the first workpiece image.

[0069] The mask decoder is configured to process the encoded directed bounding box and the image embedding features to obtain the segmentation mask.

[0070] In the embodiment of the application, a multi-modal large model such as GPT-4V, DeepSeek Janus, etc. is used to detect the directed bounding box of the workpiece to be welded. The multi-modal large model is a model with generalization understanding ability trained on a large amount of data, which can accurately identify the workpiece to be welded under the prompt of language instructions.

[0071] The directed bounding box of the workpiece to be welded comprises (x, y, w, h, θ), wherein (x, y) is the center point coordinate of the workpiece to be welded, , (x, y) is the center point coordinate of the workpiece to be welded, , respectively represent the length and width of the rotated target box of the workpiece to be welded, represents the angle between the long axis direction of the rotated target box and the horizontal direction.

[0072] The visual base model, such as SAM (Segment Anything Model), CLIP (Contrastive Language-Image Pre-training), etc., has been pre-trained on a large amount of different modal data. Based on this, appropriate fine-tuning parameters are set according to the characteristics and model features of the workpiece to be welded and the weld segmentation mask generation task, and then the corresponding workpiece image is generated through the workpiece to be welded and the weld segmentation mask. The pre-trained visual base model is further trained by adjusting the model weight and parameter to optimize the performance of the visual base model in the workpiece to be welded and the weld segmentation mask generation task, so that the fine-tuned visual base model can accurately generate the segmentation mask of the workpiece to be welded and the weld.

[0073] The visual foundation model includes a bounding box hint encoder, an image encoder, and a mask decoder. The bounding box hint encoder encodes the directional bounding box obtained by the multimodal large model. The image encoder processes the first workpiece image to obtain the image embedding features in the first workpiece image. The encoded directional bounding box and image embedding features are then fed into the mask decoder to generate a segmentation mask associated with the directional bounding box hint, ensuring the generalization and accuracy of the identification and positioning of the workpiece to be welded.

[0074] In one embodiment, if Figure 2 As shown in , the image encoder includes L converter modules composed of a multi-head attention mechanism. Each layer includes multi-head attention, a feedforward neural network, and a residual connection. Finally, the first workpiece image is input to the image encoder, and the embedded feature sequence of the image is output. The extraction task is issued to the multimodal large model, such as Figure 2 In the language input "Please output the center coordinates of the image of the workpiece to be welded and give the smallest rotation target box", the multimodal large model infers the rotation target box and center point in the first workpiece image, and obtains the parameters of the directed bounding box, including the center point (x, y), length and width ( , h), and the rotation angle θ. The bounding box hint encoder first encodes the position and rotation using a multilayer perceptron (MLP) and high-frequency Fourier feature mapping. It then introduces adaptive Gaussian position encoding and direction-corrected embedding to output a bounding box hint embedding vector. The directed bounding box is then input into the bounding box hint encoder for encoding. The output embedding vector, along with the image embedding features, is then input into the mask decoder to generate a segmentation mask for the workpiece to be welded and the weld, completing the understanding of the welding scene. Figure 2 The "Image & Hint Attention" in the mask decoder in

[15] includes an embedding fusion and transformer attention module and a mask prediction module. The embedding fusion and transformer attention module includes: adding the image embedding features and the hint features and inputting them into the transformer attention module to form a global context representation. The mask prediction module uses MLP to extract the intermediate mask representation from the context representation, and generates the final mask through matrix multiplication with the upsampled features of the transformer.

[0075] In an exemplary embodiment of the present invention, determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask includes:

[0076] Acquire second workpiece images of the workpiece to be welded from multiple viewing angles, and three-dimensional point clouds corresponding to each of the second workpiece images;

[0077] Performing point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain a target three-dimensional point cloud and a three-dimensional bounding box of the workpiece to be welded;

[0078] The three-dimensional contour points of the weld are obtained according to the target three-dimensional point cloud, the three-dimensional bounding box and the segmentation mask.

[0079] In the embodiment of the present invention, a multimodal vision sensor is used to capture a second workpiece image of the workpiece to be welded at multiple viewing angles, and a 3D point cloud corresponding to the second workpiece image is obtained at the same time. There is a one-to-one correspondence between the 3D point cloud and the pixels of the second workpiece image. The 3D point cloud is fused and registered to obtain a relatively complete target 3D point cloud and a 3D bounding box of the workpiece to be welded. , combined with the segmentation mask, to obtain the three-dimensional contour points of the weld , providing accurate data support for welding program generation and simulation environment establishment.

[0080] Point cloud fusion combines 3D point clouds from different perspectives to create a more complete and accurate 3D model. Point cloud registration uses spatial transformation to map the 3D point cloud corresponding to one workpiece image to the 3D point cloud of another workpiece image, ensuring a one-to-one correspondence between points in the two images corresponding to the same spatial location, thus achieving information fusion.

[0081] In an exemplary embodiment of the present invention, the inferring of welding steps based on the welding task information and the three-dimensional contour points includes:

[0082] The welding task information and the three-dimensional contour points are input into a multimodal large model to obtain the welding steps.

[0083] In an embodiment of the present invention, by means of the reasoning ability of the multimodal large model, three-dimensional contour points and welding task information are input into the multimodal large model, and the detailed process required for the humanoid robot's robotic arm to complete welding is output.

[0084] In an exemplary embodiment of the present invention, the testing of the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot includes:

[0085] Loading the initial welding program into the simulation environment to simulate the welding process, and detecting the collision state and trajectory running quality of the humanoid robot and the workpiece to be welded during the simulated welding process;

[0086] The initial welding program is optimized according to the collision state and the trajectory running quality to obtain the target welding program.

[0087] In this embodiment of the present invention, an initial welding program containing welding process parameters is loaded into a humanoid robot controller in a simulation environment. During the simulated welding process, a large multimodal model is used to assist in detecting the collision status and trajectory quality between the humanoid robot and the workpiece being welded. Errors and anomalies in these collision status and trajectory quality are recorded. The collision status indicates the collision between the humanoid robot's robotic arm and the workpiece being welded, while the trajectory quality indicates the trajectory quality of the humanoid robot's welding torch tip within the weld seam.

[0088] By analyzing the collision status and trajectory operation quality during the simulation process, errors such as humanoid robot collision and excessive trajectory deviation are detected. According to the analysis results, the instructions in the initial welding program are updated, and the above evaluation and correction process is repeated to achieve the safe execution of the humanoid robot welding trajectory in the simulation environment.

[0089] like Figure 3 As shown, in an embodiment of the present invention, based on the initial operating skills, welding task information and three-dimensional contour points of the humanoid robot, the welding steps are inferred through a multimodal large model, and then an initial welding program is generated. The generated initial welding program is simulated and understood through multimodal big data, that is, error analysis is performed, and prompt update information is generated according to the error analysis structure, and then the initial welding program is optimized to obtain the target welding program.

[0090] In an exemplary embodiment of the present invention, before testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot, the method further includes:

[0091] A simulation environment is constructed according to the segmentation mask, the three-dimensional contour points and the humanoid robot, and the simulation environment is integrated to obtain a simulation engine; wherein the simulation environment includes a welding workbench, a humanoid robot model and a workpiece model.

[0092] In an embodiment of the present invention, a simulation environment is established based on the three-dimensional reconstruction of the welding operation and the scene understanding results of the workpiece to be welded. For example, a simulation environment including a welding workbench, a humanoid robot model, and a workpiece model is constructed based on the Isaac Sim simulation platform, and a simulation engine such as PhysX is integrated to realize physical interaction and collision detection between the humanoid robot and the workpiece to be welded.

[0093] In the embodiment of the present invention, a first workpiece image of the workpiece to be welded is first obtained, a directed bounding box of the workpiece to be welded is obtained based on the multimodal large model, and a visual basic model is embedded to obtain a segmentation mask corresponding to the workpiece to be welded and the weld. A second workpiece image and the corresponding 3D point cloud taken from multiple perspectives are then used to fuse and register the point clouds to obtain a relatively complete target 3D point cloud and a 3D bounding box of the workpiece to be welded, and the 3D contour points of the weld are obtained in combination with the segmentation mask, thereby completing an accurate understanding of the welding scene and preliminarily generating an initial welding program for a humanoid robot based on the reasoning capability of the multimodal large model; relying on the scene understanding results, a simulation environment is automatically built, the generated code is tested in the simulation environment, the collision state of the humanoid robot and the workpiece to be welded is evaluated, errors are analyzed and correction prompts are given, thereby realizing the safe and automatic generation of the humanoid robot welding program, and the generation effect is as follows: Figure 4 As shown, the path, posture and position of the workpiece are respectively used for docking, overlapping and corner jointing. The technical solution provided by the present invention realizes the flexible operation of the humanoid robot and ensures the safety of the humanoid robot welding program generation.

[0094] Figure 5 FIG is a schematic diagram of a welding system based on a humanoid robot according to an exemplary embodiment. Figure 5 As shown, in an exemplary embodiment, the welding system based on a humanoid robot includes:

[0095] A multimodal visual sensing module 510 for acquiring a workpiece image and a three-dimensional point cloud of the workpiece to be welded;

[0096] A welding program generation module 520 is configured to implement any of the aforementioned methods for generating a welding program for a humanoid robot to obtain a target welding program for the humanoid robot;

[0097] The welding operation module 530 includes a humanoid robot, which is used to drive the humanoid robot to weld the workpiece to be welded based on the target welding program.

[0098] In an embodiment of the present invention, the multimodal visual sensing module includes a passive light vision submodule, a line structured light vision submodule, a coded structured light vision submodule, and an RGB-D sensor. The passive light vision submodule is used to acquire images of the workpiece to be welded, the RGB-D sensor is used to acquire images and point cloud information in a wide-area welding scene, and the line structured light vision submodule and the coded structured light vision submodule are used to acquire accurate point cloud information of the welded workpiece and weld in a small scene.

[0099] The welding program generation module includes an image / contour / point cloud information acquisition control module, a multimodal information processing module, a welding program automatic generation submodule, etc. The image / contour / point cloud information acquisition control module is used to control the various submodules in the multimodal visual sensing module to collect corresponding information. The multimodal information processing module is used to process the various collected information. The welding program automatic generation submodule is used to implement the welding program generation method of the humanoid robot as described in any of the above items to obtain the target welding program of the humanoid robot.

[0100] The welding operation module includes a humanoid robot motion control module, a humanoid robot, a water-cooled welding gun, a digital welding machine, an automatic wire feeder, and welding shielding gas. The humanoid robot motion control module drives the robot to weld the workpiece based on the target welding program. To reduce high temperatures and spatter interference, the humanoid robot is equipped with high-temperature protective clothing to ensure stable operation in harsh environments.

[0101] The following describes the humanoid robot welding program generation device provided by the present invention. The humanoid robot welding program generation device described below and the humanoid robot welding program generation method described above can be used in conjunction with each other. It should be noted that the device provided in the following embodiments and the method provided in the above embodiments share the same concept. The specific manner in which each module and unit performs operations has been described in detail in the method embodiments and will not be repeated here.

[0102] In an exemplary embodiment of the present invention, see Figure 6 , Figure 6 A welding program generating device for a humanoid robot is shown according to an exemplary embodiment, and includes the following modules.

[0103] A first determining module 610 is configured to obtain a first workpiece image of a workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image;

[0104] A second determining module 620 is configured to determine the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask;

[0105] an inference module 630 configured to obtain welding task information and infer welding steps based on the welding task information and the three-dimensional contour points;

[0106] A third determining module 640 is configured to determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and write the welding process parameters into an initial welding program;

[0107] The testing module 650 is configured to test the initial welding program in a simulation environment corresponding to the workpiece to be welded, and obtain a target welding program for the humanoid robot.

[0108] In an exemplary embodiment of the present invention, the first determining module 610 includes:

[0109] a detection submodule, configured to detect a directional bounding box of the workpiece to be welded using a multimodal large model;

[0110] An input submodule configured to input the oriented bounding box and the first workpiece image into a visual basis model to obtain the segmentation mask; wherein the visual basis model includes a bounding box hint encoder, an image encoder, and a mask decoder;

[0111] The bounding box hint encoder is configured to encode the oriented bounding box;

[0112] The image encoder is configured to extract image embedding features from the first workpiece image;

[0113] The mask decoder is configured to process the encoded directional bounding box and the image embedding feature to obtain the segmentation mask.

[0114] In an exemplary embodiment of the present invention, the second determining module 620 includes:

[0115] an acquisition submodule configured to acquire second workpiece images of the workpiece to be welded from multiple viewing angles, and a three-dimensional point cloud corresponding to each of the second workpiece images;

[0116] a point cloud processing submodule configured to perform point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain a target three-dimensional point cloud and a three-dimensional bounding box of the workpiece to be welded;

[0117] The contour point submodule is configured to obtain the three-dimensional contour points of the weld according to the target three-dimensional point cloud, the three-dimensional bounding box and the segmentation mask.

[0118] In an exemplary embodiment of the present invention, the reasoning module 630 includes:

[0119] The input submodule is configured to input the welding task information and the three-dimensional contour points into the multimodal large model to obtain the welding steps.

[0120] In an exemplary embodiment of the present invention, the testing module 650 includes:

[0121] a loading submodule configured to load the initial welding program into the simulation environment to simulate the welding process, and to detect the collision state and trajectory running quality of the humanoid robot and the workpiece to be welded during the simulated welding process;

[0122] The optimization submodule is configured to optimize the initial welding program according to the collision state and the trajectory running quality to obtain the target welding program.

[0123] In an exemplary embodiment of the present invention, the welding program generating device for a humanoid robot further includes:

[0124] A construction module is configured to construct a simulation environment based on the segmentation mask, the three-dimensional contour points and the humanoid robot, and integrate the simulation environment to obtain a simulation engine; wherein the simulation environment includes a welding workbench, a humanoid robot model and a workpiece model.

[0125] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call logic instructions in the memory 730 to execute a welding program generation method for a humanoid robot, the method comprising: acquiring a first workpiece image of a workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and a weld based on the first workpiece image;

[0126] Determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask;

[0127] Acquire welding task information, and infer welding steps based on the welding task information and the three-dimensional contour points;

[0128] Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into an initial welding program;

[0129] The initial welding program is tested in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot.

[0130] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0131] On the other hand, the present invention further provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer-readable storage medium, and the computer program being capable of executing the method for generating a welding program for a humanoid robot provided by the above methods when the computer program is executed by a processor, the method including: acquiring a first workpiece image of a workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and a weld seam based on the first workpiece image;

[0132] Determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask;

[0133] Acquire welding task information, and infer welding steps based on the welding task information and the three-dimensional contour points;

[0134] Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into an initial welding program;

[0135] The initial welding program is tested in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot.

[0136] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for generating a welding program for a humanoid robot provided by the above methods is implemented, the method comprising: acquiring a first workpiece image of a workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and a weld based on the first workpiece image;

[0137] Determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask;

[0138] acquire welding task information, and infer a welding step according to the welding task information and the three-dimensional contour points;

[0139] determine welding process parameters according to the welding step, the segmentation mask, the three-dimensional contour points, and an initial operation skill of the humanoid robot, and write the welding process parameters into a welding initial program;

[0140] test the welding initial program in a simulation environment corresponding to the workpiece to be welded, to obtain a target welding program of the humanoid robot.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0142] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary universal hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0143] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a welding program for a humanoid robot, characterized in that: include: Acquire a first workpiece image of a workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image; Determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; Acquire welding task information, and infer welding steps based on the welding task information and the three-dimensional contour points; Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into an initial welding program; Testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot; The determining of the segmentation mask of the workpiece to be welded and the weld based on the first workpiece image includes: Detecting a directional bounding box of the workpiece to be welded by using a multimodal large model; Inputting the oriented bounding box and the first workpiece image into a visual basis model to obtain the segmentation mask; wherein the visual basis model includes a bounding box hint encoder, an image encoder, and a mask decoder; The bounding box hint encoder is configured to encode the oriented bounding box; The image encoder is configured to extract image embedding features from the first workpiece image; The mask decoder is configured to process the encoded directional bounding box and the image embedding feature to obtain the segmentation mask; Among them, the bounding box hint encoder encodes position and rotation through a multi-layer perceptron and high-frequency Fourier feature mapping, introduces adaptive Gaussian position encoding and direction correction embedding, and outputs a bounding box hint embedding vector; The mask decoder includes an embedding fusion and Transformer attention module and a mask prediction module. The embedding fusion and Transformer attention module adds the image embedding features and the bounding box hint embedding vector and inputs the Transformer attention module to form a global context representation. The mask prediction module uses a multi-layer perceptron to extract the intermediate mask representation from the context representation, and generates a segmentation mask through matrix multiplication with the upsampled features of the Transformer attention module.

2. The method for generating a welding program for a humanoid robot according to claim 1, wherein: The step of determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask comprises: Acquire second workpiece images of the workpiece to be welded from multiple viewing angles, and three-dimensional point clouds corresponding to each of the second workpiece images; Performing point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain a target three-dimensional point cloud and a three-dimensional bounding box of the workpiece to be welded; The three-dimensional contour points of the weld are obtained according to the target three-dimensional point cloud, the three-dimensional bounding box and the segmentation mask.

3. The method for generating a welding program for a humanoid robot according to claim 1, wherein: The inferring of welding steps based on the welding task information and the three-dimensional contour points includes: The welding task information and the three-dimensional contour points are input into a multimodal large model to obtain the welding steps.

4. The method for generating a welding program for a humanoid robot according to claim 1, wherein: The testing of the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot includes: Loading the initial welding program into the simulation environment to simulate the welding process, and detecting the collision state and trajectory running quality of the humanoid robot and the workpiece to be welded during the simulated welding process; The initial welding program is optimized according to the collision state and the trajectory running quality to obtain the target welding program.

5. The method for generating a welding program for a humanoid robot according to any one of claims 1 to 4, wherein: Before testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot, the method further includes: A simulation environment is constructed according to the segmentation mask, the three-dimensional contour points and the humanoid robot, and the simulation environment is integrated to obtain a simulation engine; wherein the simulation environment includes a welding workbench, a humanoid robot model and a workpiece model.

6. A welding system based on a humanoid robot, characterized in that: include: A multimodal visual sensing module for acquiring workpiece images and 3D point clouds of the workpiece to be welded; a welding program generation module, configured to implement the welding program generation method for a humanoid robot as claimed in any one of claims 1 to 5, and obtain a target welding program for the humanoid robot; The welding operation module includes a humanoid robot, which is used to drive the humanoid robot to weld the workpiece to be welded based on the target welding program.

7. A welding program generating device for a humanoid robot, characterized in that: include: a first determining module configured to acquire a first workpiece image of the workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image; a second determining module configured to determine the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; an inference module configured to obtain welding task information and infer welding steps based on the welding task information and the three-dimensional contour points; a third determining module configured to determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and write the welding process parameters into an initial welding program; A testing module configured to test the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program for the humanoid robot; The first determination module includes: a detection submodule, configured to detect a directional bounding box of the workpiece to be welded using a multimodal large model; An input submodule configured to input the oriented bounding box and the first workpiece image into a visual basis model to obtain the segmentation mask; wherein the visual basis model includes a bounding box hint encoder, an image encoder, and a mask decoder; The bounding box hint encoder is configured to encode the oriented bounding box; The image encoder is configured to extract image embedding features from the first workpiece image; The mask decoder is configured to process the encoded directional bounding box and the image embedding feature to obtain the segmentation mask; Among them, the bounding box hint encoder encodes position and rotation through a multi-layer perceptron and high-frequency Fourier feature mapping, introduces adaptive Gaussian position encoding and direction correction embedding, and outputs a bounding box hint embedding vector; The mask decoder includes an embedding fusion and Transformer attention module and a mask prediction module. The embedding fusion and Transformer attention module adds the image embedding features and the bounding box hint embedding vector and inputs the Transformer attention module to form a global context representation. The mask prediction module uses a multi-layer perceptron to extract the intermediate mask representation from the context representation, and generates a segmentation mask through matrix multiplication with the upsampled features of the Transformer attention module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for generating a welding program for a humanoid robot according to any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a welding program for a humanoid robot according to any one of claims 1 to 5 is implemented.

Images