Industrial part assembling method and equipment and medium

Abstract

The invention discloses an industrial part assembling method and device and a medium, and relates to the technical field of robot control in industrial automation, and the method comprises the steps: obtaining the part category and the detection frame of each industrial part based on an RGB image and a depth image; obtaining a segmentation mask of each industrial part based on the detection frame and the RGB image, and obtaining a 3D model corresponding to each part category; the segmentation mask, the RGB image, the depth image and the 3D model are input into a pose estimation model, the 6D pose, output by the pose estimation model, of each industrial part is obtained, and the grabbing pose of the robot is determined based on the 6D poses; and based on the assembly sequence and the 6D poses of the industrial parts, the grabbing posture and the grabbing path of the robot are determined, and based on the grabbing path and the grabbing posture, the robot is controlled to conduct grabbing and assembly. The method is used for solving the problem that in the prior art, the assembly precision is poor due to the fact that the part recognition precision is poor, and precise recognition and precise assembly of industrial parts are achieved.

Description

Technical Field

[0001] This application relates to the technical field of robot control in industrial automation, and in particular to an assembly method, device, and medium for industrial parts. Background Art

[0002] In the manufacturing industry, the assembly of complex industrial parts is a key link in the production process. However, in the prior art, for the recognition of industrial components composed of multiple industrial parts, traditional visual recognition methods are still used, and their accuracy is poor. As a result, the accuracy of the robot's grasping and assembly is poor, affecting the operation efficiency and reliability of the automated production line. Summary of the Invention

[0003] In view of the above problems and technical requirements, the applicant has proposed an assembly method, device, and medium for industrial parts to solve the problem of poor assembly accuracy caused by poor accuracy of part recognition in the prior art, and to achieve accurate recognition and accurate assembly of industrial parts.

[0004] An embodiment of the application provides an assembly method for industrial parts, and the method includes:

[0005] Detecting an RGB image and a depth image including an industrial component, and obtaining the part category of each industrial part and the corresponding detection box in the RGB image, where the industrial component includes at least three industrial parts;

[0006] Obtaining a segmentation mask of each industrial part based on the detection box and the RGB image, and obtaining a 3D model corresponding to each part category;

[0007] Inputting the segmentation mask, the RGB image, the depth image, and the 3D model into a pre-trained pose estimation model, obtaining the 6D pose of each industrial part output by the pose estimation model, and determining the grasping pose of the robot based on the 6D pose, where the pose estimation model is trained based on segmentation mask samples, RGB image samples, depth image samples, 3D model samples, and 6D pose samples;

[0008] Determining the grasping pose and the grasping path of the robot based on the assembly sequence and the 6D pose of the industrial parts, and controlling the robot to complete the grasping and assembly of the industrial parts based on the grasping path and the grasping pose.

[0009] According to the assembly method for industrial parts provided by the embodiment of the application, the detecting an RGB image and a depth image including an industrial component, and obtaining the part category of each industrial part and the corresponding detection box in the RGB image includes:

[0010] Input the RGB image and the depth image into a pre-trained object detection model. Extract visual features from the RGB image and depth features from the depth image through the object detection model; fuse the visual features and the depth features to obtain fused features; obtain and output the part category and detection box output by the object detection model based on the fused features.

[0011] Among them, the object detection model is trained based on RGB image samples, depth image samples, part category samples, and detection box samples.

[0012] Among them, the visual features include: color features, edge features, and texture features.

[0013] According to the industrial part assembly method provided by the embodiments of the present application, after extracting visual features from the RGB image and depth features from the depth image through the object detection model, it further includes:

[0014] Judge whether the visual features belong to known features;

[0015] In the case where it is determined that the visual features belong to known features, execute the step of fusing the visual features and the depth features to obtain fused features;

[0016] In the case where it is determined that the visual features do not belong to known features, optimize the object detection model based on the incremental learning algorithm, and use the optimized object detection model to detect and output the part category and detection box.

[0017] According to the industrial part assembly method provided by the embodiments of the present application, before determining the grasping path of the robot based on the assembly sequence and 6D pose of the industrial parts, it further includes:

[0018] Obtain the assembly requirements corresponding to the industrial components;

[0019] Based on the assembly requirements and part categories, determine the assembly sequence and assembly positions of each industrial part, where the assembly position is the position where a certain industrial part is located;

[0020] Determine the grasping path of the robot based on the assembly sequence and 6D pose of the industrial parts, including:

[0021] Based on the assembly sequence and 6D pose, determine the grasping path for other industrial parts to be grasped and placed at the assembly position, where the other industrial parts are the remaining industrial parts after removing the industrial parts at the assembly position from the industrial components.

[0022] According to the industrial part assembly method provided by the embodiments of the present application, different industrial parts correspond to different grasping postures, and different industrial parts correspond to different grasping paths.

[0023] Based on the grasping path and grasping posture, control the robot to complete the grasping of industrial parts, including:

[0024] Perform the following grasping operations on each other industrial part:

[0025] Use the current grasping posture corresponding to the current other industrial part to control the robot to grasp the current other industrial part; use the current grasping path corresponding to the current other industrial part to control the robot to place the current other industrial part at the assembly position;

[0026] Judge whether all other industrial parts in the industrial component have been grasped; if so, determine that the industrial component assembly is completed; otherwise, determine the next other industrial part based on the assembly sequence, and perform the grasping operation on the next other industrial part;

[0027] Wherein, the other industrial parts are the remaining industrial parts in the industrial component after removing the industrial parts at the assembly position.

[0028] According to the industrial part assembly method provided by the embodiments of the present application, the method further includes:

[0029] During the process of controlling the robot to grasp industrial parts based on the grasping path and grasping posture, the grasping force and moving displacement during the robot's grasping operation are monitored in real time;

[0030] Based on the grasping force, the moving displacement and the 6D pose of the industrial parts at the assembly position, the grasping pose of the robot is adjusted in real time.

[0031] According to the industrial part assembly method provided by the embodiments of the present application, controlling the robot to complete the assembly of industrial parts includes:

[0032] Determine the assembly connection position between the current other industrial part and the target industrial part;

[0033] Based on the assembly connection position, the current pose of the current other industrial part and the 6D pose of the target industrial part, determine the assembly position;

[0034] Control the robot to place the current other industrial part at the assembly position to complete the assembly of the current other industrial part;

[0035] Wherein, the other industrial parts are the remaining industrial parts in the industrial component after removing the industrial parts at the assembly position, and the industrial parts at the assembly position are the target industrial parts.

[0036] According to the industrial part assembly method provided by the embodiments of the present application, after obtaining the detection frame based on the fusion feature, it further includes:

[0037] Remove redundant detection boxes based on the non-maximum suppression algorithm.

[0038] An embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the assembly method of industrial parts as described in any one of the above are implemented.

[0039] An embodiment of the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the assembly method of industrial parts as described in any one of the above are implemented.

[0040] The assembly method, device, and medium of industrial parts provided by the embodiments of the present application detect RGB images and depth images including industrial components, obtain the part categories of each industrial part, the corresponding detection boxes in the RGB images, and the segmentation masks of each industrial part. Among them, the industrial components include at least three industrial parts; obtain the 3D models corresponding to each part category; input the segmentation masks, RGB images, depth images, and 3D models into a pre-trained pose estimation model to obtain the 6D poses of each industrial part output by the pose estimation model, and determine the grasping poses of the robot based on the 6D poses. It can be seen that the present application can accurately obtain the positions and poses (6D poses) of each industrial part; furthermore, based on the assembly sequence and 6D poses of industrial parts, determine the grasping poses and grasping paths of the robot, and control the robot to complete the grasping and assembly of industrial parts based on the grasping paths and grasping poses. It can be seen that the present application accurately determines the corresponding grasping poses and grasping paths for each industrial part, ensures the accurate recognition, grasping, and assembly of each industrial part, and further improves the operation efficiency and operation reliability of the automated production line. Description of the Drawings

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 is one of the flow diagrams of the assembly method of industrial parts provided by the embodiments of the present application;

[0043] Figure 2 is the second flow diagram of the assembly method of industrial parts provided by the embodiments of the present application;

[0044] Figure 3 is the third flow diagram of the assembly method of industrial parts provided by the embodiments of the present application;

[0045] Figure 4 It is the fourth flow diagram of the assembly method of industrial parts provided by the embodiments of the present application;

[0046] Figure 5 It is the fifth flow diagram of the assembly method of industrial parts provided by the embodiments of the present application;

[0047] Figure 6 It is the sixth flow diagram of the assembly method of industrial parts provided by the embodiments of the present application;

[0048] Figure 7 It is the structural diagram of the electronic device provided by the embodiments of the present application. Specific Embodiments

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present invention.

[0050] The embodiments of the present application provide an assembly method for industrial parts. This method can be applied to intelligent terminals and can also be applied to servers. This application takes the example of this method being applied in a server for illustration. Here, it is for illustrative purposes only and is not used to limit the protection scope of the present application. Some other descriptions in the embodiments are also for illustrative purposes and will not be repeated one by one hereafter. The specific implementation of this method is as Figure 1 shown:

[0051] Step 101, detect the RGB image and depth image including industrial components, and obtain the part category of each industrial part and the corresponding detection box in the RGB image.

[0052] Among them, the industrial components include at least three industrial parts.

[0053] Step 102, obtain the segmentation mask of each industrial part based on the detection box and the RGB image, and obtain the 3D model corresponding to each part category.

[0054] Step 103, input the segmentation mask, RGB image, depth image, and 3D model into a pre-trained pose estimation model, obtain the 6D pose of each industrial part output by the pose estimation model, and determine the grasping pose of the robot based on the 6D pose.

[0055] Among them, the pose estimation model is trained based on segmentation mask samples, RGB image samples, depth image samples, 3D model samples, and 6D pose samples.

[0056] Among them, the pose estimation model includes the FoundationPose model, and the FoundationPose model is used for the estimation and tracking of 6D poses.

[0057] Step 104: Determine the grasping pose and grasping path of the robot based on the assembly sequence and 6D pose of the industrial parts, and control the robot to complete the grasping and assembly of the industrial parts based on the grasping path and grasping pose.

[0058] The industrial part assembly method provided by the embodiments of the present application detects RGB images and depth images including industrial components to obtain the part category of each industrial part, the corresponding detection box in the RGB image, and the segmentation mask of each industrial part. Among them, the industrial components include at least three industrial parts; obtain the 3D model corresponding to each part category; input the segmentation mask, RGB image, depth image, and 3D model into a pre-trained pose estimation model to obtain the 6D pose of each industrial part output by the pose estimation model, and determine the grasping pose of the robot based on the 6D pose. It can be seen that the present application can accurately obtain the position and pose (6D pose) of each industrial part; furthermore, based on the assembly sequence and 6D pose of the industrial parts, determine the grasping pose and grasping path of the robot, and control the robot to complete the grasping and assembly of the industrial parts based on the grasping path and grasping pose. It can be seen that the present application accurately determines the corresponding grasping pose and grasping path for each industrial part, ensuring the accurate recognition, grasping, and assembly of each industrial part, and further improving the operation efficiency and operation reliability of the automated production line.

[0059] In a specific embodiment, the specific implementation of detecting RGB images and depth images including industrial components to obtain the part category of each industrial part and the corresponding detection box in the RGB image is as Figure 2 shown:

[0060] Step 201: Input the RGB image and the depth image into a pre-trained object detection model.

[0061] Step 202: Extract visual features from the RGB image and depth features from the depth image through the object detection model; fuse the visual features and depth features to obtain fused features; obtain and output the part category and detection box output by the object detection model based on the fused features.

[0062] Among them, the object detection model is trained based on RGB image samples, depth image samples, part category samples, and detection box samples.

[0063] Among them, the visual features include: color features, edge features, and texture features.

[0064] Specifically, the object detection model is trained based on a mixed-precision training method and a distributed training method. The mixed-precision training method is used to balance the calculation speed and accuracy, and the distributed training method is used to improve the training efficiency and scalability. The combination of the two training methods ensures the accuracy of the object detection model.

[0065] Specifically, the object detection model includes the YOLOv8 model.

[0066] Specifically, this application makes partial improvements to the model structure of the existing YOLOv8 model. A convolutional block attention module is introduced in the neck structure part to enhance feature representation through spatial and channel attention and improve the feature fusion effect. The backbone network adopts the CAFormer module to enhance the feature extraction ability, adapt to complex industrial scenarios, and improve the accuracy of model prediction.

[0067] In a specific embodiment, after obtaining the detection boxes based on the fusion features, redundant detection boxes are removed based on the non-maximum suppression algorithm.

[0068] Specifically, during the prediction process of the object detection model, there may be multiple detection boxes for the same industrial part. Based on the fusion features, multiple corresponding detection boxes can be obtained. Furthermore, the non-maximum suppression algorithm is used to remove redundant detection boxes, specifically by suppressing the detection boxes with a high degree of overlap, thereby improving the accuracy and efficiency of the detection results.

[0069] Specifically, determine the probability that the detection box contains an industrial part, and sort the probabilities from high to low; take the one with the highest ranking as the reference box, and calculate the intersection over union (IoU) between the reference box and each detection box; delete the detection boxes with an IoU greater than the preset IoU.

[0070] In a specific embodiment, after extracting visual features from the RGB image and depth features from the depth image through the object detection model, predictions of the part category and detection boxes are made, specifically as Figure 3 shown:

[0071] Step 301, determine whether the visual features belong to known features. If so, execute Step 302; otherwise, execute Step 303.

[0072] Step 302, fuse the visual features and the depth features to obtain fusion features.

[0073] Furthermore, based on the fusion features, the part category and detection boxes output by the object detection model are obtained and output.

[0074] Step 303: Optimize the object detection model based on the incremental learning algorithm, and use the optimized object detection model to detect and output the part category and detection frame.

[0075] Specifically, in the case where it is determined that the visual feature does not belong to the known feature, optimize the object detection model based on the incremental learning algorithm, visual feature, and depth feature, and fuse the visual feature and the depth feature to obtain a fused feature; and use the optimized object detection model and the fused feature to detect and output the part category and detection frame.

[0076] Among them, the known feature is the feature that the object detection model has seen during the training phase.

[0077] Specifically, the incremental learning algorithm includes: a neural network for self-organizing incremental learning.

[0078] In this application, in the case where it is determined that the visual feature is not a known feature, new data samples are collected for online learning, and the model parameters of the object detection model are updated in real time. Through the incremental learning algorithm, the weights of the object detection model are dynamically adjusted, enabling the model to better adapt to new environments and workpiece changes. By continuously adapting to new environments and workpiece changes in the above manner, the detection accuracy and robustness are effectively improved.

[0079] In a specific embodiment, before determining the grasping path of the robot based on the assembly sequence and 6D pose of industrial parts, obtain the assembly requirements corresponding to the industrial components; based on the assembly requirements and part categories, determine the assembly sequence and assembly positions of each industrial part. Furthermore, based on the assembly sequence and 6D pose, determine the grasping path for other industrial parts to be grasped and placed at the assembly positions.

[0080] Among them, the assembly position is the position where a certain industrial part is located.

[0081] Among them, the other industrial parts are the remaining industrial parts after removing the industrial parts at the assembly position from the industrial components.

[0082] In a specific embodiment, different industrial parts correspond to different grasping postures, and different industrial parts correspond to different grasping paths.

[0083] For the specific implementation of controlling the robot to complete the grasping of industrial parts based on the grasping path and grasping posture, see Figure 4 :

[0084] Step 401: Perform the following grasping operations on each other industrial part: use the current grasping posture corresponding to the current other industrial part to control the robot to grasp the current other industrial part; use the current grasping path corresponding to the current other industrial part to control the robot to place the current other industrial part at the assembly position.

[0085] Step 402: Determine whether all other industrial parts in the industrial component have been grasped. If so, execute Step 403; otherwise, execute Step 404.

[0086] Step 403: Determine that the assembly of the industrial component is completed.

[0087] Here, the completion of the assembly of the industrial component means the completion of the assembly of the current industrial component.

[0088] Step 404: Determine the next other industrial part based on the assembly sequence, and perform a grasping operation on the next other industrial part.

[0089] Here, the other industrial parts are the remaining industrial parts in the industrial component after removing the industrial parts at the assembly positions.

[0090] This application can simultaneously identify the positions and postures of multiple industrial parts, and plan the grasping paths of each industrial part and control the robot based on the identified positions and postures during the grasping and assembly processes, so as to quickly and accurately complete the identification, grasping, and assembly of multiple industrial parts.

[0091] In a specific embodiment, the method also needs to adjust the grasping pose of the robot in real time. The specific implementation is as Figure 5 shown:

[0092] Step 501: During the process of controlling the robot to grasp the industrial part based on the grasping path and grasping pose, monitor the grasping force and moving displacement of the robot during the grasping operation in real time.

[0093] Step 502: Based on the grasping force, moving displacement, and 6D pose of the industrial part at the assembly position, adjust the grasping pose of the robot in real time.

[0094] Here, the grasping process at this time is the process of the robot grasping the industrial part and placing the grasped industrial part at the assembly position.

[0095] Specifically, during the grasping process, the movement of the robot and the friction between the robot and the industrial part may both cause changes in the grasping position (grasping pose). Therefore, during this process, the grasping force and moving displacement are monitored in real time, and based on the grasping force, moving displacement, and 6D pose of the industrial part at the assembly position, the grasping pose of the robot is adjusted in real time to ensure the stability and safety of the grasping process and ensure the precise assembly between industrial parts.

[0096] In a specific embodiment, the specific implementation of controlling the robot to complete the assembly of the industrial part is as Figure 6 shown:

[0097] Step 601: Determine the assembly connection position between the current other industrial part and the target industrial part.

[0098] Step 602: Determine the assembly position based on the assembly connection position, the current poses of current other industrial parts, and the 6D pose of the target industrial part.

[0099] Step 603: Control the robot to place the current other industrial part at the assembly position to complete the assembly of the current other industrial part.

[0100] Wherein, the other industrial parts are the remaining industrial parts after removing the industrial part at the assembly position from the industrial component, and the industrial part at the assembly position is the target industrial part.

[0101] Specifically, determine the assembly connection position between the other industrial part and the target industrial part based on a preset visual recognition model. The assembly connection positions of different other industrial parts and the target industrial part are different.

[0102] In addition, in this application, the other industrial parts are variable, and the target industrial part is also variable.

[0103] For example, the industrial component is composed of three industrial parts A, B, and C, and its assembly sequence is to assemble A onto C, and then assemble B onto A (or the combination of A and C). Among them, in the process of assembling A onto C, the other industrial part is A, and the target industrial part is C; in the process of assembling B onto A (or the combination of A and C), the other industrial part is B, and the target industrial part is A (or the combination of A and C).

[0104] Another example, the industrial component is composed of four industrial parts A, B, C, and D, and its assembly sequence is to assemble A onto C, then assemble B onto A (or the combination of A and C), and finally assemble D onto B (or the combination of B, A, and C). Among them, in the process of assembling A onto C, the other industrial part is A, and the target industrial part is C; in the process of assembling B onto A (or the combination of A and C), the other industrial part is B, and the target industrial part is A (or the combination of A and C); in the process of assembling D onto B (or the combination of B, A, and C), the other industrial part is D, and the target industrial part is B (or the combination of B, A, and C).

[0105] Specifically, as the assembly work progresses, the assembly position will change. Therefore, in the assembly process of each other industrial part, it is necessary to re-determine the assembly position, and perform the assembly operation of the industrial part based on this assembly position and the corresponding grasping path.

[0106] This application solves the problems of accurate recognition, grasping, and assembly of complex industrial parts through multi-modal feature fusion, deep learning algorithms, and pose estimation models, improving the assembly efficiency and reliability of the robot in complex industrial scenarios.

[0107] Specifically, by fusing multi-modal features and based on deep learning algorithms, precise positioning and recognition of complex components are achieved. With the SAM pre-trained model, without additional training, the segmentation of complex industrial parts can be quickly realized, and the pixel-level position information of the components can be obtained. The 6D pose data of the workpiece is obtained through the FoundationPose model. And according to the characteristics of the workpiece, the grasping pose is adjusted accordingly, so as to control the robot to achieve precise recognition and assembly of complex industrial parts.

[0108] Figure 7 An example of the physical structure diagram of an electronic device is shown in Figure 7 As shown, the electronic device may include: a processor 701, a communications interface 702, a memory 703, and a communication bus 704. Among them, the processor 701, the communications interface 702, and the memory 703 communicate with each other through the communication bus 704. The processor 701 can call the logic instructions in the memory 703 to execute the assembly method of industrial parts.

[0109] In addition, when the logic instructions in the above-mentioned memory 703 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks, etc., which can store program codes.

[0110] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the assembly method of industrial parts provided by the above-mentioned various methods.

[0111] On yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the assembly method of industrial parts provided by the above-mentioned various embodiments.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above is only the preferred implementation of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the scope of protection of the present application.

Claims

1. An assembly method for industrial parts, characterized in that, The method includes: Detecting an RGB image and a depth image including industrial components to obtain the part category of each industrial part and the corresponding detection box in the RGB image, where the industrial components include at least three industrial parts; Obtaining a segmentation mask for each industrial part based on the detection box and the RGB image, and acquiring a 3D model corresponding to each part category; Inputting the segmentation mask, the RGB image, the depth image, and the 3D model into a pre-trained pose estimation model to obtain the 6D pose of each industrial part output by the pose estimation model, and determining the grasping pose of the robot based on the 6D pose, where the pose estimation model is trained based on segmentation mask samples, RGB image samples, depth image samples, 3D model samples, and 6D pose samples; Determining the grasping pose and the grasping path of the robot based on the assembly sequence and the 6D pose of the industrial parts, and controlling the robot to complete the grasping and assembly of the industrial parts based on the grasping path and the grasping pose.

2. The assembly method of the industrial part according to claim 1, characterized in that, The detecting the RGB image and the depth image including industrial components to obtain the part category of each industrial part and the corresponding detection box in the RGB image includes: Inputting the RGB image and the depth image into a pre-trained object detection model, extracting visual features from the RGB image through the object detection model, and extracting depth features from the depth image; fusing the visual features and the depth features to obtain fused features; obtaining and outputting the part category and the detection box output by the object detection model based on the fused features; where the object detection model is trained based on RGB image samples, depth image samples, part category samples, and detection box samples; where the visual features include: color features, edge features, and texture features.

3. The assembly method of the industrial parts according to claim 2, characterized in that, After extracting visual features from the RGB image and depth features from the depth image through the object detection model, it further includes: Judging whether the visual features belong to known features; In the case of determining that the visual features belong to known features, performing the step of fusing the visual features and the depth features to obtain fused features; In the case of determining that the visual features do not belong to known features, optimizing the object detection model based on an incremental learning algorithm, and detecting and outputting the part category and the detection box using the optimized object detection model.

4. The assembly method of the industrial parts according to any one of claims 1-3, characterized in that, Before determining the grasping path of the robot based on the assembly sequence and the 6D pose of the industrial parts, it further includes: Obtaining the assembly requirements corresponding to the industrial components; Determining the assembly sequence and the assembly position of each industrial part based on the assembly requirements and the part category, where the assembly position is the position where a certain industrial part is located; Determining the grasping path of the robot based on the assembly sequence and the 6D pose of the industrial parts includes: Determining the grasping path for other industrial parts to be grasped and placed at the assembly position based on the assembly sequence and the 6D pose, where the other industrial parts are the remaining industrial parts after removing the industrial parts at the assembly position from the industrial components.

5. The assembly method of the industrial parts according to any one of claims 1-3, characterized in that, Different industrial parts correspond to different grasping poses and different grasping paths; Controlling a robot to complete the grasping of industrial parts based on a grasping path and a grasping posture, including: Performing the following grasping operations on each other industrial part: Controlling the robot to grasp the current other industrial part by using the current grasping posture corresponding to the current other industrial part; controlling the robot to place the current other industrial part at the assembly position by using the current grasping path corresponding to the current other industrial part; Judging whether all other industrial parts in the industrial component have been grasped; if so, determining that the industrial component assembly is completed; otherwise, determining the next other industrial part based on the assembly sequence and performing a grasping operation on the next other industrial part; Wherein, the other industrial parts are the remaining industrial parts in the industrial component after removing the industrial parts at the assembly position.

6. The assembly method of the industrial parts according to any one of claims 1-3, characterized in that, The method further includes: During the process of controlling the robot to grasp the industrial parts based on the grasping path and the grasping posture, real-time monitoring of the grasping force and the moving displacement during the robot's grasping operation; Based on the grasping force, the moving displacement and the 6D pose of the industrial parts at the assembly position, real-time adjusting the grasping pose of the robot.

7. The assembly method of the industrial part according to any one of claims 1-3, characterized in that, Controlling the robot to complete the assembly of industrial parts, including: Determining the assembly connection position between the current other industrial part and the target industrial part; Based on the assembly connection position, the current pose of the current other industrial part and the 6D pose of the target industrial part, determining the assembly position; Controlling the robot to place the current other industrial part at the assembly position to complete the assembly of the current other industrial part; Wherein, the other industrial parts are the remaining industrial parts in the industrial component after removing the industrial parts at the assembly position, and the industrial parts at the assembly position are the target industrial parts.

8. The assembling method of the industrial parts according to claim 2, characterized in that, After obtaining the detection frame based on the fusion feature, it further includes: Removing redundant detection frames based on the non-maximum suppression algorithm.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the industrial part assembly method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the industrial part assembly method according to any one of claims 1 to 8.

Images