Welding program generation method of humanoid robot and related equipment

Abstract

The invention relates to the technical field of welding, and provides a welding program generation method of a humanoid robot and related equipment.The welding program generation method of the humanoid robot comprises the steps that a first workpiece image of a to-be-welded workpiece is obtained, and segmentation masks of the to-be-welded workpiece and a welding seam are determined based on the first workpiece image; determining three-dimensional contour points of the welding seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; welding task information is obtained, and welding steps are inferred according to the welding task information and the three-dimensional contour points; according to the welding steps, the segmentation masks, the three-dimensional contour points and the initial operation skill of the humanoid robot, welding process parameters are determined and written into an initial welding program; and the initial welding program is tested in the simulation environment corresponding to the to-be-welded workpiece, and the target welding program of the humanoid robot is obtained. Through the technical scheme provided by the invention, safe and automatic generation of the humanoid robot welding program is realized.

Description

Technical Field

[0001] The present invention relates to the field of welding technology, and particularly relates to a method for generating a welding program for a humanoid robot and related equipment. Background Art

[0002] With the development of industry, flexible welding technology has been gradually broken through, and industrial robots have been applied on a large scale in standard welding scenarios. Compared with traditional industrial or mobile robots, humanoid robots have unique advantages in welding tasks.

[0003] The welding program is the key to realizing the flexible welding of humanoid robots. The welding program obtained through manual programming is mainly applicable to large-scale and standardized production scenarios, and it is difficult to meet the flexible welding tasks of small batches, multiple varieties, and non-standard. The welding path generation method based on offline programming requires manual or sensor calibration of the welding path, which is limited by modeling errors, resulting in low welding efficiency and difficult to guarantee welding quality. And the welding program generated based on large models is not stable and safe enough on complex three-dimensional shaped workpieces to support humanoid robots to complete high-precision welding tasks. Summary of the Invention

[0004] The present invention provides a method for generating a welding program for a humanoid robot and related equipment to solve the defects in the prior art.

[0005] The present invention provides a method for generating a welding program for a humanoid robot, including: Obtaining a first workpiece image of the workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; Determining the three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; Obtaining welding task information, and inferring welding steps according to the welding task information and the three-dimensional contour points; Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points and the initial operation skills of the humanoid robot, and writing the welding process parameters into the initial welding program; Testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot.

[0006] According to the method for generating a welding program for a humanoid robot provided by the present invention, the determining the segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image includes: Detecting the oriented bounding box of the workpiece to be welded through a multimodal large model; Input the directed bounding box and the first workpiece image into a vision-based model to obtain the segmentation mask; wherein, the vision-based model includes a bounding box prompt encoder, an image encoder, and a mask decoder; The bounding box prompt encoder is configured to encode the directed bounding box; The image encoder is configured to extract image embedding features from the first workpiece image; The mask decoder is configured to process the encoded directed bounding box and the image embedding features to obtain the segmentation mask.

[0007] According to a method for generating a welding program of a humanoid robot provided by the present invention, determining the three-dimensional contour points of the weld seam according to the three-dimensional point cloud and the segmentation mask of the workpiece to be welded includes: Obtain second workpiece images of the workpiece to be welded from multiple perspectives, and the three-dimensional point cloud corresponding to each second workpiece image; Based on the second workpiece image, perform point cloud fusion and point cloud registration on the three-dimensional point cloud to obtain the target three-dimensional point cloud and three-dimensional bounding box of the workpiece to be welded; According to the target three-dimensional point cloud, the three-dimensional bounding box, and the segmentation mask, obtain the three-dimensional contour points of the weld seam.

[0008] According to a method for generating a welding program of a humanoid robot provided by the present invention, reasoning about the welding steps according to the welding task information and the three-dimensional contour points includes: Input the welding task information and the three-dimensional contour points into a multimodal large model to obtain the welding steps.

[0009] According to a method for generating a welding program of a humanoid robot provided by the present invention, testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot includes: Load the initial welding program into the simulation environment to simulate the welding process, and during the simulation welding process, detect the collision state and the trajectory running quality between the humanoid robot and the workpiece to be welded; Optimize the initial welding program according to the collision state and the trajectory running quality to obtain the target welding program.

[0010] According to a method for generating a welding program of a humanoid robot provided by the present invention, before testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot, the method further includes: Construct a simulation environment based on the segmentation mask, the three-dimensional contour points, and the humanoid robot, and integrate the simulation environment to obtain a simulation engine; wherein, the simulation environment includes a welding workbench, a humanoid robot model, and a workpiece model.

[0011] The present invention also provides a welding system based on a humanoid robot, including: A multi-modal visual sensing module for collecting a workpiece image and three-dimensional point cloud of a workpiece to be welded; A welding program generation module for implementing the welding program generation method of the humanoid robot as described in any one of the foregoing, to obtain a target welding program of the humanoid robot; A welding operation module, including a humanoid robot, for driving the humanoid robot to weld the workpiece to be welded based on the target welding program.

[0012] The present invention also provides a welding program generation device for a humanoid robot, including: A first determination module configured to obtain a first workpiece image of a workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; A second determination module configured to determine three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; An inference module configured to obtain welding task information, and infer welding steps according to the welding task information and the three-dimensional contour points; A third determination module configured to determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and write the welding process parameters into an initial welding program; A test module configured to test the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program of the humanoid robot.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the welding program generation method of the humanoid robot as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the welding program generation method of the humanoid robot as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the welding program generation method of the humanoid robot as described in any one of the above is implemented.

[0016] The welding program generation method and related equipment for a humanoid robot provided by the present invention first obtain a first workpiece image of the workpiece to be welded, obtain a segmentation mask corresponding to the workpiece to be welded and the weld seam based on the first workpiece image, and obtain the three-dimensional contour points of the weld seam through the three-dimensional point cloud and the segmentation mask of the workpiece to be welded, so as to accurately identify the workpiece to be welded and achieve a precise understanding of the welding scene. According to the welding task information and the three-dimensional contour points, the welding steps are inferred, and then according to the welding steps, the segmentation mask, the three-dimensional contour points and the initial operation skills of the humanoid robot, the welding process parameters are determined, and the welding process parameters are written into the initial welding program. To ensure the safety of the welding program, the initial welding program is tested in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot, realizing the safe and automatic generation of the welding program for the humanoid robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of the welding program generation method for a humanoid robot provided by the present invention.

[0019] Figure 2 It is a schematic diagram for determining the segmentation mask in the embodiment provided by the present invention.

[0020] Figure 3 It is a schematic diagram for generating and testing the initial welding program in the embodiment provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the welding path generated by the humanoid robot provided by the present invention.

[0022] Figure 5 It is a schematic structural diagram of the welding system based on the humanoid robot provided by the present invention.

[0023] Figure 6 It is a schematic structural diagram of the welding program generation device for a humanoid robot provided by the present invention.

[0024] Figure 7 It is a schematic structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In flexible production, the differences in the morphology and properties of welding workpieces lead to many limitations in traditional welding program generation methods: manual teaching programming mostly relies on experience modeling, and its cognitive solidification characteristics are in essential conflict with the dynamic needs of small batches and multiple batches; although offline programming realizes digital mapping through CAD models, the model-entity deviation caused by processing and assembly errors is difficult to ensure welding quality; the perception method based on images and point clouds is limited by the single-modal data representation capability, and has the limitations of low accuracy and poor adaptability; the welding program generation method based on large model interaction has high generalization, but the lack of protection mechanism makes it difficult to ensure the stability and safety of the humanoid robot welding system. Therefore, how to establish a cross-modal semantic understanding mechanism for welding operation scenarios through the design of a multi-modal large model architecture, form a multi-modal data-driven humanoid robot welding program automatic generation method, and meet the complex and changeable flexible manufacturing needs, is the key problem that the present invention intends to solve.

[0027] Figure 1 FIG. 1 is a flow chart showing a method for generating a welding program for a humanoid robot according to an exemplary embodiment. Figure 1 As shown, in an exemplary embodiment, the method for generating a welding program for a humanoid robot includes steps 110 to 150, which are described in detail as follows.

[0028] Step 110 : acquiring a first workpiece image of the workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and the weld based on the first workpiece image.

[0029] In the embodiment of the present invention, a first workpiece image of a workpiece to be welded is acquired, and a segmentation mask corresponding to the workpiece to be welded and the weld is determined based on the first workpiece image.

[0030] Step 120: determining the three-dimensional contour points of the weld according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask.

[0031] In the embodiment of the present invention, a 3D point cloud is a data structure representing an object in a 3D space, and is a set of multiple 3D coordinate points. The 3D contour points of the weld are determined based on the 3D point cloud of the workpiece to be welded and the segmentation mask, so as to achieve an accurate understanding of the welding scene.

[0032] Step 130, obtaining welding task information, and inferring welding steps based on the welding task information and the three-dimensional contour points.

[0033] In the embodiment of the present invention, the welding task information is the description information of the welding task, including the requirements to be met for welding. The detailed steps required for welding are accurately inferred based on the welding task information and the three-dimensional contour points.

[0034] Step 140, determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points and the initial operation skills of the humanoid robot, and writing the welding process parameters into the welding initial program.

[0035] In the embodiment of the present invention, the initial operating skills of the humanoid robot are the operating skills of the humanoid robot controller, such as MOV, SET, ARC and other instructions. The above instructions can effectively avoid problems such as motion jitter and loss of control of the humanoid robot, and ensure the stability of the welding task planning of the humanoid robot.

[0036] According to the determined welding steps, segmentation masks, three-dimensional contour points and the initial operation skills of the humanoid robot, the welding process parameters are determined, and the welding process parameters are written into the welding initial program to realize the preliminary generation of the welding program. The welding initial program is a pre-set basic program with pre-defined basic codes.

[0037] Step 150 , testing the initial welding program in a simulation environment corresponding to the workpiece to be welded, to obtain a target welding program for the humanoid robot.

[0038] In the embodiment of the present invention, in order to ensure the safety of the welding program, a simulation environment is automatically built based on the understanding of the welding scene, and the code in the initial welding program is tested in the simulation environment corresponding to the workpiece to be welded, and then the target welding program is obtained after the test passes.

[0039] In an exemplary embodiment of the present invention, determining the segmentation mask of the workpiece to be welded and the weld based on the first workpiece image includes: Detecting a directional bounding box of the workpiece to be welded by using a multimodal large model; Inputting the oriented bounding box and the first workpiece image into a visual basis model to obtain the segmentation mask; wherein the visual basis model includes a bounding box cue encoder, an image encoder and a mask decoder; The bounding box hint encoder is used to encode the oriented bounding box; The image encoder is used to extract image embedding features in the first workpiece image; The mask decoder is used to process the encoded directional bounding box and the image embedding feature to obtain the segmentation mask.

[0040] In the embodiments of the present invention, a multimodal large model, such as GPT-4V, DeepSeek Janus, etc., is used to detect the oriented bounding box of the workpiece to be welded. The multimodal large model is a model with generalization and understanding capabilities trained on a large amount of data, and can accurately identify the workpiece to be welded under the prompt of language instructions.

[0041] The oriented bounding box of the workpiece to be welded includes ( ), where ( ) is the coordinate of the center point of the workpiece to be welded, , respectively represent the length and width of the rotation target box of the workpiece to be welded, represents the angle between the long axis direction of the rotation target box and the horizontal direction.

[0042] Vision foundation models, such as SAM (Segment Anything Model), CLIP (Contrastive Language-Image Pre-training), etc., have been pre-trained on a large amount of different-modal data. Based on this, appropriate fine-tuning parameters are set according to the characteristics of the task of generating the segmentation masks of the workpiece to be welded and the weld seam and the characteristics of the model, and then corresponding workpiece images are generated through the segmentation masks of the workpiece to be welded and the weld seam, and the pre-trained vision foundation model is further trained. By adjusting the model weights and parameters, the performance of the vision foundation model in the task of generating the segmentation masks of the workpiece to be welded and the weld seam is optimized, so that the fine-tuned vision foundation model can accurately generate the segmentation masks of the workpiece to be welded and the weld seam.

[0043] The vision foundation model includes a bounding box prompt encoder, an image encoder, and a mask decoder. The oriented bounding box obtained by the multimodal large model is encoded by the bounding box prompt encoder, and the image encoder processes the first workpiece image to obtain the image embedding features in the first workpiece image. The encoded oriented bounding box and the image embedding features are sent into the mask decoder together to generate a segmentation mask related to the oriented bounding box prompt, ensuring the generalization and accuracy of the recognition and positioning of the workpiece to be welded.

[0044] In one embodiment, as Figure 2 shown, the image encoder includes L transformer modules composed of multi-head attention mechanisms. Each layer includes multi-head attention, a feed-forward neural network, and a residual connection. Finally, the first workpiece image is input into the image encoder, and a sequence of image embedding features is output. A extraction task is issued to the multimodal large model, such as Figure 2In response to the language input "Please output the image center coordinates of the workpiece to be welded and give the smallest rotated target box", the multimodal large model infers the rotated target box and the center point in the first workpiece image, and obtains the parameters of the oriented bounding box, including the center point (x, y), length and width ( , h), and rotation angle θ. The bounding box prompt encoder first encodes the position and rotation through a Multilayer Perceptron (MLP) and high-frequency Fourier feature maps, introduces adaptive Gaussian position encoding and direction correction embedding, and outputs a bounding box prompt embedding vector. The oriented bounding box is input into the bounding box prompt encoder for encoding, and the output embedding vector and the image embedding features are input into the mask decoder together to obtain the segmentation masks of the workpiece to be welded and the weld seam, completing the understanding of the welding scenario. Figure 2 The "image & prompt attention" in the mask decoder in

[0045] In an exemplary embodiment of the present invention, determining the three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask includes: Obtaining second workpiece images of the workpiece to be welded from multiple perspectives, and the corresponding three-dimensional point clouds of each of the second workpiece images; Performing point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain the target three-dimensional point cloud and three-dimensional bounding box of the workpiece to be welded; According to the target three-dimensional point cloud, the three-dimensional bounding box and the segmentation mask, obtaining the three-dimensional contour points of the weld seam.

[0046] In the embodiment of the present invention, a multimodal vision sensor is used to capture second workpiece images of the workpiece to be welded from multiple perspectives, and at the same time obtain the corresponding three-dimensional point clouds of the second workpiece images. There is a one-to-one correspondence between the three-dimensional point clouds and the pixels of the second workpiece images. Point cloud fusion and point cloud registration are performed on the three-dimensional point clouds to obtain a relatively complete target three-dimensional point cloud and three-dimensional bounding box of the workpiece to be welded , and then combined with the segmentation mask, the three-dimensional contour points of the weld seam are obtained , providing accurate data support for the generation of the welding program and the establishment of the simulation environment.

[0047] Point cloud fusion is to merge 3D point clouds from different perspectives to obtain a more complete and accurate 3D model. Point cloud registration is to map the 3D point cloud corresponding to a second workpiece image to the 3D point cloud of another second workpiece image through a spatial transformation, so that the points corresponding to the same spatial position in the two images are in one-to-one correspondence, thereby achieving the purpose of information fusion.

[0048] In an exemplary embodiment of the present invention, the inferring the welding steps according to the welding task information and the 3D contour points includes: Inputting the welding task information and the 3D contour points into a multimodal large model to obtain the welding steps.

[0049] In an embodiment of the present invention, by virtue of the inference ability of the multimodal large model, the 3D contour points and the welding task information are input into the multimodal large model, and the detailed process required for the robotic arm of the humanoid robot to complete welding is output.

[0050] In an exemplary embodiment of the present invention, the testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot includes: Loading the initial welding program into the simulation environment to simulate the welding process, and during the simulation welding process, detecting the collision state between the humanoid robot and the workpiece to be welded and the trajectory running quality; Optimizing the initial welding program according to the collision state and the trajectory running quality to obtain the target welding program.

[0051] In an embodiment of the present invention, the initial welding program written with welding process parameters is loaded into the humanoid robot controller in the simulation environment. During the simulation welding process, a multimodal large model is used to assist in detecting the collision state between the humanoid robot and the welding workpiece and the trajectory running quality, and information such as errors and abnormal conditions in the collision state and the trajectory running quality is recorded. The collision state characterizes the collision state between the robotic arm of the humanoid robot and the workpiece to be welded, and the trajectory running quality characterizes the trajectory running quality of the end of the welding torch of the humanoid robot on the weld seam.

[0052] By analyzing the collision state and the trajectory running quality, errors such as humanoid robot collision and excessive trajectory deviation in the simulation process are detected, and according to the analysis results, instructions in the initial welding program are prompted to be updated, and the above evaluation and correction process is repeated to realize the safe execution of the humanoid robot welding trajectory in the simulation environment.

[0053] Such as Figure 3As shown, in the embodiment of the present invention, based on the initial operation skills of the humanoid robot, the welding task information, and the three-dimensional contour points, the welding steps are inferred through a multimodal large model, and then an initial welding program is generated. The simulation effect of the generated initial welding program is understood through multimodal big data, that is, error analysis is carried out, and prompt update information is generated according to the error analysis result, and then the initial welding program is optimized to obtain the target welding program.

[0054] In an exemplary embodiment of the present invention, before testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot, the method further includes: According to the segmentation mask, the three-dimensional contour points, and the humanoid robot, a simulation environment is constructed and the simulation environment is integrated to obtain a simulation engine; wherein, the simulation environment includes a welding workbench, a humanoid robot model, and a workpiece model.

[0055] In the embodiment of the present invention, according to the three-dimensional reconstruction of the welding operation and the scene understanding result of the workpiece to be welded, a simulation environment is established. For example, a simulation environment including a welding workbench, a humanoid robot model, and a workpiece model is constructed based on a simulation platform such as Isaac Sim, and a simulation engine such as PhysX is integrated to realize the physical interaction and collision detection between the humanoid robot and the workpiece to be welded.

[0056] In the embodiment of the present invention, first, a first workpiece image of the workpiece to be welded is obtained. Based on a multimodal large model, an oriented bounding box of the workpiece to be welded is obtained, and the segmentation mask corresponding to the workpiece to be welded and the weld seam is obtained by embedding a visual foundation model. Through the second workpiece images taken from multiple perspectives and the corresponding three-dimensional point clouds, point cloud fusion and point cloud registration are carried out to obtain a relatively complete target three-dimensional point cloud and a three-dimensional bounding box of the workpiece to be welded, and the three-dimensional contour points of the weld seam are obtained in combination with the segmentation mask to complete the accurate understanding of the welding scene. Based on the inference ability of the multimodal large model, an initial welding program of the humanoid robot is initially generated; relying on the scene understanding result, a simulation environment is automatically built, the generated code is tested in the simulation environment, the collision state between the humanoid robot and the workpiece to be welded is evaluated, errors are analyzed and prompts are corrected to realize the safe and automatic generation of the welding program of the humanoid robot, and the generation effect is as Figure 4 shown, butt joint, lap joint, and fillet joint are respectively carried out on the path, posture, and position and posture of the workpiece. Through the technical solution provided by the present invention, the flexible operation of the humanoid robot is realized, and the safety of the generation of the welding program of the humanoid robot is guaranteed.

[0057] Figure 5 is a schematic diagram of a welding system based on a humanoid robot shown according to an exemplary embodiment. As Figure 5 shown, in an exemplary embodiment, the welding system based on a humanoid robot includes: A multi-modal visual sensing module 510, configured to collect workpiece images and three-dimensional point clouds of workpieces to be welded; A welding program generation module 520, configured to implement the welding program generation method of the humanoid robot as described in any one of the foregoing, to obtain a target welding program of the humanoid robot; A welding operation module 530, including a humanoid robot, configured to drive the humanoid robot to weld the workpiece to be welded based on the target welding program.

[0058] In an embodiment of the present invention, the multi-modal visual sensing module includes a passive light vision sub-module, a line structured light vision sub-module, a coded structured light vision sub-module, and an RGB-D sensor. Among them, the passive light vision sub-module is configured to collect workpiece images of workpieces to be welded, the RGB-D sensor is configured to collect image and point cloud information in a wide-area welding scene, and the line structured light vision sub-module and the coded structured light vision sub-module are configured to collect accurate point cloud information of welding workpieces and weld seams in a small scene; The welding program generation module includes an image / contour / point cloud information acquisition control module, a multi-modal information processing module, a welding program automatic generation sub-module, etc. The image / contour / point cloud information acquisition control module is configured to control each sub-module in the multi-modal visual sensing module to collect corresponding information, the multi-modal information processing module is configured to process various collected information, and the welding program automatic generation sub-module is configured to implement the welding program generation method of the humanoid robot as described in any one of the foregoing, to obtain a target welding program of the humanoid robot.

[0059] The welding operation module includes a humanoid robot motion control module, a humanoid robot, a water-cooled welding torch, a digital welding machine, an automatic wire feeder, welding shielding gas, etc. The humanoid robot motion control module is configured to drive the humanoid robot to weld the workpiece to be welded based on the target welding program. To reduce the interference of high temperature and spatter, the humanoid robot is equipped with a high-temperature resistant protective suit to ensure its stable operation in a harsh environment.

[0060] The welding program generation device of the humanoid robot provided by the present invention will be described below. The welding program generation device of the humanoid robot described below can be mutually referred to the welding program generation method of the humanoid robot described above. It should be noted that the device provided in the following embodiments and the method provided in the above embodiments belong to the same concept. The specific manners in which each module and unit perform operations have been described in detail in the method embodiments, and will not be repeated here.

[0061] In an exemplary embodiment of the present invention, please refer to Figure 6 , Figure 6 is a welding program generation device of a humanoid robot shown according to an exemplary embodiment, including the following modules.

[0062] The first determination module 610 is configured to obtain a first workpiece image of the workpiece to be welded and determine a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; The second determination module 620 is configured to determine three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; The inference module 630 is configured to obtain welding task information and infer welding steps according to the welding task information and the three-dimensional contour points; The third determination module 640 is configured to determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and write the welding process parameters into the initial welding program; The test module 650 is configured to test the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program of the humanoid robot.

[0063] In an exemplary embodiment of the present invention, the first determination module 610 includes: The detection sub-module is configured to detect an oriented bounding box of the workpiece to be welded through a multi-modal large model; The input sub-module is configured to input the oriented bounding box and the first workpiece image into a vision base model to obtain the segmentation mask; wherein, the vision base model includes a bounding box prompt encoder, an image encoder, and a mask decoder; The bounding box prompt encoder is used to encode the oriented bounding box; The image encoder is used to extract image embedding features in the first workpiece image; The mask decoder is used to process the encoded oriented bounding box and the image embedding features to obtain the segmentation mask.

[0064] In an exemplary embodiment of the present invention, the second determination module 620 includes: The acquisition sub-module is configured to acquire second workpiece images of the workpiece to be welded from multiple perspectives and the three-dimensional point cloud corresponding to each second workpiece image; The point cloud processing sub-module is configured to perform point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain a target three-dimensional point cloud and a three-dimensional bounding box of the workpiece to be welded; The contour point sub-module is configured to obtain three-dimensional contour points of the weld seam according to the target three-dimensional point cloud, the three-dimensional bounding box, and the segmentation mask.

[0065] In an exemplary embodiment of the present invention, the inference module 630 includes: An input sub-module, configured to input the welding task information and the three-dimensional contour points into a multi-modal large model to obtain the welding steps.

[0066] In an exemplary embodiment of the present invention, the test module 650 includes: A loading sub-module, configured to load the initial welding program into the simulation environment to simulate the welding process, and during the simulated welding process, detect the collision state and the trajectory running quality between the humanoid robot and the workpiece to be welded; An optimization sub-module, configured to optimize the initial welding program according to the collision state and the trajectory running quality to obtain the target welding program.

[0067] In an exemplary embodiment of the present invention, the welding program generation device of the humanoid robot further includes: A construction module, configured to construct a simulation environment according to the segmentation mask, the three-dimensional contour points, and the humanoid robot, and integrate the simulation environment to obtain a simulation engine; wherein, the simulation environment includes a welding workbench, a humanoid robot model, and a workpiece model.

[0068] Figure 7 The schematic physical structure diagram of an electronic device is exemplified, as Figure 7 shown. The electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logical instructions in the memory 730 to execute the welding program generation method of the humanoid robot. The method includes: obtaining a first workpiece image of the workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; Determining the three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; Obtaining welding task information, and reasoning welding steps according to the welding task information and the three-dimensional contour points; Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into the initial welding program; Testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot.

[0069] In addition, when the logical instructions in the above-mentioned memory 730 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0070] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for generating a welding program of a humanoid robot provided by the above-mentioned various methods. The method includes: acquiring a first workpiece image of a workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; Determining three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; Acquiring welding task information, and inferring welding steps according to the welding task information and the three-dimensional contour points; Determining welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and writing the welding process parameters into an initial welding program; Testing the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain a target welding program of the humanoid robot.

[0071] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the method for generating a welding program of a humanoid robot provided by the above-mentioned various methods. The method includes: acquiring a first workpiece image of a workpiece to be welded, and determining a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; Determining three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; Acquiring welding task information, and inferring welding steps according to the welding task information and the three-dimensional contour points; Determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points, and the initial operation skills of the humanoid robot, and write the welding process parameters into the initial welding program; Test the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0073] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a welding program for a humanoid robot, characterized in that, Including: Obtain a first workpiece image of the workpiece to be welded, and determine a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; Determine the three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; Obtain welding task information, and infer welding steps according to the welding task information and the three-dimensional contour points; Determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points and the initial operation skills of the humanoid robot, and write the welding process parameters into the initial welding program; Test the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot.

2. The method for generating a welding program of a humanoid robot according to claim 1, wherein The determining the segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image includes: Detect the oriented bounding box of the workpiece to be welded through a multimodal large model; Input the oriented bounding box and the first workpiece image into a vision-based model to obtain the segmentation mask; wherein, the vision-based model includes a bounding box prompt encoder, an image encoder and a mask decoder; The bounding box prompt encoder is used to encode the oriented bounding box; The image encoder is used to extract the image embedding features in the first workpiece image; The mask decoder is used to process the encoded oriented bounding box and the image embedding features to obtain the segmentation mask.

3. The method for generating a welding program of a humanoid robot according to claim 1, characterized in that The determining the three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask includes: Obtain second workpiece images of the workpiece to be welded from multiple perspectives, and the three-dimensional point cloud corresponding to each second workpiece image; Perform point cloud fusion and point cloud registration on the three-dimensional point cloud based on the second workpiece image to obtain the target three-dimensional point cloud and three-dimensional bounding box of the workpiece to be welded; Obtain the three-dimensional contour points of the weld seam according to the target three-dimensional point cloud, the three-dimensional bounding box and the segmentation mask.

4. The method for generating a welding program of a humanoid robot according to claim 1, characterized in that, The inferring welding steps according to the welding task information and the three-dimensional contour points includes: Input the welding task information and the three-dimensional contour points into a multimodal large model to obtain the welding steps.

5. The method for generating a welding program of a humanoid robot according to claim 1, wherein The testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot includes: Load the initial welding program into the simulation environment to simulate the welding process, and during the simulated welding process, detect the collision state and the trajectory running quality of the humanoid robot and the workpiece to be welded; Optimize the initial welding program according to the collision state and the trajectory running quality to obtain the target welding program.

6. The method for generating a welding program of a humanoid robot according to any one of claims 1 to 5, characterized in that, Before the testing the initial welding program in the simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot, the method further includes: Construct a simulation environment according to the segmentation mask, the three-dimensional contour points and the humanoid robot, and integrate the simulation environment to obtain a simulation engine; wherein, the simulation environment includes a welding workbench, a humanoid robot model and a workpiece model.

7. A welding system based on a humanoid robot, characterized in that, Including: A multi-modal visual sensing module for collecting workpiece images and three-dimensional point clouds of workpieces to be welded; A welding program generation module for implementing the welding program generation method of the humanoid robot according to any one of claims 1 to 6 to obtain the target welding program of the humanoid robot; A welding operation module, including a humanoid robot, for driving the humanoid robot to weld the workpiece to be welded based on the target welding program.

8. A welding program generation device for a humanoid robot, characterized in that, Comprising: A first determination module configured to obtain a first workpiece image of the workpiece to be welded and determine a segmentation mask of the workpiece to be welded and the weld seam based on the first workpiece image; A second determination module configured to determine the three-dimensional contour points of the weld seam according to the three-dimensional point cloud of the workpiece to be welded and the segmentation mask; An inference module configured to obtain welding task information and infer welding steps according to the welding task information and the three-dimensional contour points; A third determination module configured to determine welding process parameters according to the welding steps, the segmentation mask, the three-dimensional contour points and the initial operation skills of the humanoid robot, and write the welding process parameters into the initial welding program; A test module configured to test the initial welding program in a simulation environment corresponding to the workpiece to be welded to obtain the target welding program of the humanoid robot.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the welding program generation method of the humanoid robot according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the welding program generation method of the humanoid robot according to any one of claims 1 to 6.

Images