Industrial part sorting system and method based on deep learning
By combining deep learning with intelligent adsorption-type end effectors, the problems of recognition accuracy and grasping stability of traditional methods in highly mixed and random scenarios are solved, enabling rapid and accurate sorting of industrial parts, adapting to complex scenarios and improving flexible adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-07
- Publication Date
- 2026-05-15
AI Technical Summary
Traditional rule-based 2D vision detection methods and mechanical positioning schemes that rely on fixed fixtures are difficult to adapt to the highly mixed and random industrial parts sorting scenarios, resulting in low recognition accuracy, poor gripping stability, and low flexibility, which cannot meet the production needs of flexible production lines.
An industrial parts sorting system based on deep learning is adopted, including a conveying mechanism, a vision acquisition module, an actuator, and a control and computing module. It uses depth cameras and color cameras to acquire images in real time, identifies parts information through deep neural networks and pose estimation algorithms, and combines intelligent adsorption-type end effectors to adaptively adjust the suction force to achieve precise grasping and sorting.
It enables rapid and accurate sorting of mixed industrial parts of various categories, improves recognition accuracy and gripping stability, adapts to complex scenarios, and enhances sorting efficiency and flexible adaptability.
Smart Images
Figure CN122032898A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of industrial intelligent manufacturing and machine vision, and in particular to an industrial parts sorting system and method based on deep learning. Background Technology
[0002] Mixed loading and stacking of multiple product categories are common upstream processes in flexible production lines. Their complexity is mainly reflected in the large variety of parts, random spatial orientation, and disordered placement. The numerous types of parts, with significant differences in surface morphology and material between different categories, along with their random spatial orientation, disordered placement, and lack of fixed arrangement rules, necessitate adaptation to frequent production line switching during production, thus placing high demands on the flexibility of the sorting system.
[0003] Traditional rule-based two-dimensional visual inspection methods or mechanical positioning schemes that rely on fixed fixtures are difficult to adapt to this highly mixed and random work scenario.
[0004] Traditional solutions often employ rule-based two-dimensional visual inspection methods or mechanical positioning methods that rely on fixed fixtures, which cannot cope with highly mixed and random operating environments. Parts are prone to mutual occlusion and stacking, and metal and other materials are prone to reflective interference, resulting in a significant decrease in the accuracy of target detection and recognition. The movement of the conveyor belt introduces spatiotemporal registration errors, causing a deviation between the target detection results and the motion control of the robotic arm, affecting the accuracy of grasping.
[0005] Different parts have different surface morphologies, and some parts have large curvature, sharp edges or through-hole structures. Traditional suction cup end effectors have insufficient sealing and adsorption stability. When dealing with parts with rough surfaces and porous structures, problems such as leakage and falling are likely to occur, which cannot meet the requirements of stable sorting.
[0006] Fixed fixtures or special jigs can only be used for a single type of part. When switching production lines, they need to be readjusted, resulting in low flexibility. Problems such as insufficient visual inspection accuracy and poor adsorption stability further lead to slow gripping cycle and unreasonable path planning, making it difficult to improve overall sorting efficiency and match the production rhythm of flexible production lines.
[0007] In summary, existing technologies suffer from prominent problems such as low recognition accuracy, poor gripping stability, insufficient flexibility, and low sorting efficiency in flexible production line scenarios with complex stacking and mixed product handling. There is an urgent need for an intelligent sorting system that can balance accurate recognition, stable gripping, flexible adaptation, and efficient sorting. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an industrial parts sorting system and method based on deep learning, which realizes the rapid and accurate sorting of industrial parts with multiple categories mixed together.
[0009] The objective of this invention can be achieved through the following technical solutions: A deep learning-based industrial parts sorting system, the system comprising: a conveying mechanism, a vision acquisition module, an actuator, a control and computing module, and a sorting unit; The conveying mechanism includes an arc-shaped conveyor belt for conveying various industrial parts and a straight section of the conveyor belt that smoothly connects with it. The vision acquisition module includes a camera for real-time acquisition of color images of industrial parts and a depth camera for real-time acquisition of depth images of industrial parts, and is deployed above the straight section of the conveyor belt. The actuator includes a robotic arm with six degrees of freedom and an intelligent adsorption-type end effector, wherein the intelligent adsorption-type end effector has a built-in negative pressure sensor and a pressure sensor; The control and computing module includes a host computer controller, a visual reasoning unit, and an intelligent grasping planner. The sorting unit includes several sorting bins classified by category or process rules; The visual reasoning unit receives images acquired in real time by the visual acquisition module and identifies and analyzes the images using a deep neural network and pose estimation algorithm to obtain part category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity, and part mass. The grasping planner obtains the optimal adsorption point and smooth grasping path based on the output data of the visual reasoning unit and transmits it to the host computer controller to control the actuator to grasp the target part into the target sorting bin. The actuator adaptively adjusts the suction force of the intelligent adsorption end effector based on the part surface curvature value, surface roughness, porosity, and part mass.
[0010] Furthermore, the depth camera is a binocular depth camera; the depth camera acquires the depth information and color image information of the object being detected through active infrared structured light or binocular parallax principle.
[0011] Furthermore, the intelligent adsorption-type end effector includes a multi-specification suction cup assembly that can be quickly replaced, a high-response micro air valve, a negative pressure resistant silicone air tube, and a negative pressure stabilizing device. The multi-specification suction cup assembly, the high-response micro air valve, and the negative pressure resistant silicone air tube are connected by a flexible structure. The negative pressure stabilizing device controls the multi-specification suction cup assembly, the high-response micro air valve, and the negative pressure resistant silicone air tube to adaptively adjust the suction force according to the curvature, surface roughness, and pore characteristics of the adsorbed part, so as to adsorb the part in a coordinated manner.
[0012] Furthermore, the process by which the negative pressure stabilizing device controls the multi-specification suction cup assembly, the high-response micro air valve, and the negative pressure-resistant silicone air tube to collaboratively adsorb the components includes: The curvature, surface roughness, and porosity characteristics of the adsorbed parts are obtained. Based on the preset grading standard, the curvature, surface roughness, and porosity of the adsorbed parts are quantified and graded according to the characteristic parameters to obtain the grading characteristics of the current adsorbed parts. Based on a preset suction cup specification matching matrix, and according to the grading characteristics of the adsorbed parts, an appropriate suction cup assembly is selected. The suction cup assembly includes a planar hard suction cup, a flexible corrugated suction cup, and a multi-chamber vacuum suction cup. Based on the surface roughness, pore characteristics and mass of the adsorbed parts, the target negative pressure corresponding to the minimum stable adsorption force is calculated. Based on the pressure sensor built into the intelligent adsorption end effector, the initial pressure is obtained. Based on the initial pressure and the target negative pressure, the valve opening duty cycle of the high-response micro air valve is calculated and obtained. The response frequency of the high-response micro air valve is obtained according to the pore characteristics of the adsorbed parts. Based on the valve opening duty cycle and response frequency control, the opening of the air inlet / exhaust channel of the high-response micro air valve is adjusted, and it works in conjunction with the suction cup assembly to adsorb the parts being adsorbed; wherein... During the adsorption process, the built-in negative pressure sensor of the intelligent adsorption end effector collects pressure data in real time, calculates the average pressure and standard deviation of fluctuation, and continuously monitors the contact pressure between the suction cup and the part. Based on the average pressure, standard deviation of fluctuation, and contact pressure, the target negative pressure, valve opening duty cycle, and response frequency are adjusted in real time.
[0013] Furthermore, the specific process of adjusting the target negative pressure, valve opening duty cycle, and response frequency in real time based on the pressure mean, fluctuation standard deviation, and fitting pressure includes: Compare the average pressure with the preset suction threshold. If the average pressure is less than the preset minimum suction threshold, it is determined that the suction is insufficient, and the valve opening duty cycle is increased. If the average pressure is greater than the preset maximum suction threshold, it is determined that the suction is too large, and the valve opening duty cycle is reduced or the pressure relief valve of the high-response micro air valve is briefly opened. The bonding pressure is compared with the preset bonding threshold. If the bonding pressure is less than the preset bonding threshold, it is determined that the bonding is insufficient. The robot arm posture is then finely adjusted and the target negative pressure is reduced. Once the bonding pressure is greater than the preset minimum bonding threshold, the target negative pressure is restored. Compare the standard deviation of the fluctuation with the preset fluctuation threshold. If the standard deviation of the fluctuation is greater than the preset fluctuation threshold, increase the target negative pressure and the response frequency of the high-response micro valve.
[0014] Furthermore, when the vision acquisition module acquires color and depth images of industrial parts in real time, it also uses an adaptive lighting compensation algorithm to suppress reflection interference and noise errors based on the ambient lighting environment, and removes noise points in the depth image through a spatial filtering algorithm.
[0015] Furthermore, the visual reasoning unit, based on a deep neural network constructed with a multi-scale feature pyramid and an attention enhancement mechanism, performs joint feature extraction on the color image and depth image, and combines a pose estimation algorithm to obtain the category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality of the stacked industrial parts.
[0016] Furthermore, the process of obtaining category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity, and part quality of stacked industrial parts by combining joint feature extraction with pose estimation algorithm includes: Cross-modal registration: Based on the intrinsic and extrinsic parameters of each camera in the vision acquisition module, the 3D coordinates of the acquired depth image are aligned with the pixel coordinates of the color image; Multi-scale feature extraction: Low-level detail features of industrial parts in the color image and depth image are extracted through shallow convolutional layers of a deep neural network. The low-level detail features include part edges, textures, local protrusions and local depressions. The low-level detail features extracted by the shallow convolutional layers are fused through deep convolutional layers of a deep neural network to generate color texture features and depth geometric features. Cross-modal fusion: The color texture features and depth geometric features are fused channel by channel by channel weighted summation to obtain joint features; In the channel weighted summation, for the case of stacking and occlusion of industrial parts in the color image and depth image, the effective area that is not occluded is focused by the spatial attention mechanism of the deep neural network, and the feature channel weights related to part recognition and pose calculation are strengthened by the channel attention mechanism. Target recognition and segmentation: Based on the joint features, the classification branch of the deep neural network is used to determine the category of industrial parts and output the category information of industrial parts. The segmentation branch of the deep neural network is used to perform pixel-level segmentation of the target parts and obtain the pixel-level mask of the target parts. Calculate the initial spatial pose: Based on the pixel-level mask of the target part, the three-dimensional coordinate information provided by the depth image, and the joint features, the pose parameters are solved by the PnP algorithm with feature alignment, and the initial spatial pose is output. Surface normal information calculation: Based on the pixel-level mask of the target part, according to the preset neighborhood window size, fit the local plane and output the surface normal vector of each pixel in the depth image of the part region; Surface curvature value calculation: Based on the three-dimensional coordinate information provided by the depth image and the calculated surface normal vector, a local differential geometric model of the part surface is constructed, and the surface curvature value is calculated based on the local differential geometric model; Surface roughness calculation: Divide the pixel-level mask regions of the depth image and color image into micro-blocks of a preset size, and calculate the gray-level variance, texture entropy, depth value variance and height range of each block to obtain roughness features; Based on the pre-established feature-roughness mapping network, roughness is mapped through the roughness features, and the roughness of all blocks is averaged to obtain the surface roughness. Porosity calculation: Mark the color regions with gray values below the gray threshold in the pixel-level mask region of the color image and the depth regions with abrupt changes in depth values that exceed the depth range of the main body of the part in the pixel-level mask region of the depth image; take the intersection of the color region and the depth region to obtain the candidate porosity region; calculate the area of the candidate porosity region and the total area of the pixel-level mask region; and output the porosity as the ratio of the area of the candidate porosity region to the total area of the pixel-level mask region. Part mass calculation: Based on the depth image and initial spatial pose, the two-dimensional depth map of the part is reconstructed into a three-dimensional point cloud. The Poisson surface reconstruction algorithm is used to transform the three-dimensional point cloud into a closed three-dimensional mesh model. The mesh volume is calculated, and the effective volume is obtained by subtracting the pore volume obtained based on porosity from the mesh volume. Based on the category information of industrial parts, the material and density of industrial parts are obtained, and the part mass is calculated.
[0017] Furthermore, the process by which the grasping planner obtains the optimal adsorption point and smooth grasping path based on the output data of the visual reasoning unit includes: The image data acquired by the vision acquisition module is converted into base coordinate system data of the robotic arm, and all adsorption points on the surface of the part are obtained; Based on a preset multi-index adsorption point scoring standard, the adsorption points on the surface of the part are scored according to the initial spatial pose of the part, surface normal information, surface curvature value, surface roughness, porosity and part mass, so as to obtain the optimal adsorption point. Using the initial position of the robotic arm, the optimal adsorption point, and the target sorting bin position as path nodes, and adding transition points between the initial position of the robotic arm and the optimal adsorption point, as well as between the optimal adsorption point and the target sorting bin position, a smooth grasping path that satisfies basic motion constraints is generated; wherein, the basic motion constraints include joint motion constraints, safety constraints, and smoothness constraints.
[0018] An industrial parts sorting method based on the deep learning-based industrial parts sorting system described above, the method comprising: Step S1: Start the conveyor mechanism and put the mixed industrial parts to be sorted into the arc-shaped conveyor belt. The random posture of the parts is automatically straightened by the spatial geometric guidance structure of the arc segment. In step S2, after the industrial parts are conveyed by the arc-shaped conveyor belt, they enter the straight section of the conveyor belt, and the color and depth images of the industrial parts are acquired in real time by the vision acquisition module. Step S3: The acquired image data is transmitted to the control and calculation module. The visual reasoning unit performs joint feature extraction on the color image and depth image to obtain the category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality of the industrial parts. Step S4: Based on the category information, initial spatial pose, surface normal information, surface curvature value, surface roughness, porosity, and part mass of the industrial parts, the intelligent grasping planner generates the optimal adsorption point and planned path for the robotic arm and sends them to the execution mechanism. The execution mechanism adaptively adjusts the suction force of the intelligent adsorption end effector based on the optimal adsorption point and the surface curvature value, surface roughness, porosity, and part mass to adsorb the corresponding industrial parts and classify them into designated sorting bins according to the planned path. Step S5: After the current industrial parts are sorted, a reset command is sent to the actuator through the host computer controller, so that the actuator returns to the initial position along the planned path; Step S6: Continuously acquire images through the vision acquisition module. If no image of industrial parts is acquired within a preset time, it is determined that all sorting tasks have been completed and the current sorting ends; otherwise, return to step S2.
[0019] Compared with the prior art, the beneficial effects of the present invention include: 1. This invention addresses the core pain points of existing industrial parts sorting systems in terms of adaptability to complex scenarios, grasping stability, flexibility, and sorting efficiency through deep visual perception, intelligent adsorption execution, and multi-dimensional grasping planning. It achieves rapid and accurate sorting of mixed industrial parts of various categories. In this invention, a visual acquisition module acquires color and depth images of industrial parts in real time, and a visual inference unit processes the images to obtain part category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity, and part mass. It obtains the optimal adsorption point and smooth grasping path, and adaptively adjusts the suction force of the intelligent adsorption end effector. Automatic extraction requires no manual pre-setting of rules, resulting in a high degree of automation. Furthermore, it can adaptively grasp parts based on various parameter settings, achieving wide-range part compatibility and covering mainstream industrial scenarios.
[0020] 2. This invention uses a binocular depth camera to simultaneously acquire color and depth images. Through a multi-scale feature pyramid and attention enhancement mechanism, it fuses the global semantics and local details of the two types of image features and performs feature enhancement on stacked occluded areas. This solves the problem of traditional two-dimensional vision being unable to see or distinguish objects. It maintains a high recognition rate and high recognition accuracy even in scenarios with multiple stacked objects and severe occlusion.
[0021] 3. This invention suppresses the reflection of highlights and uneven lighting on metal parts through an adaptive lighting compensation algorithm and removes noise from depth images through a spatial filtering algorithm. Compared with traditional fixed-parameter visual filtering, it can dynamically adapt to the changing lighting and dust interference in industrial environments, and significantly improves the accuracy of 3D reconstruction and the robustness of recognition.
[0022] 4. This invention uses modular multi-specification suction cups, which can automatically switch according to the curvature and roughness output by visual reasoning. Combined with a flexible sealing structure and a negative pressure stabilizing device, it can dynamically adjust the adsorption shape and suction force, compensate for surface unevenness errors, and solve the problems of traditional fixed suction cups not being able to adhere to curved surfaces and poor sealing on rough surfaces, thereby improving the success rate of grasping high curvature parts.
[0023] 5. Based on a preset multi-index adsorption point scoring standard, this invention scores all adsorption points on the surface of a part according to the initial spatial pose of the part, surface normal information, surface curvature value, surface roughness, porosity and part mass, to obtain the optimal adsorption point, which can avoid areas with risk of leakage and improve adsorption stability.
[0024] 6. This invention extracts the porosity and surface roughness of parts through visual reasoning, and adjusts the pressure replenishment frequency and target negative pressure accordingly. It achieves dynamic air leakage compensation for high-porosity and porous parts, solving the pain points of traditional systems that cannot firmly grasp porous parts and have rapid pressure loss. Attached Figure Description
[0025] Figure 1 This is a structural diagram of the system device of the present invention; Figure 2 This is a flowchart of the method of the present invention; In the diagram, 110 - arc-shaped conveyor belt, 112 - straight section conveyor belt, 120 - industrial parts, 130 - robotic arm, 131 - intelligent adsorption end effector, 140 - vision acquisition module, 150 - control and computing module, 160a - sorting bins classified by category, 160b - sorting bins classified by process rules. Detailed Implementation
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0027] Example 1 This embodiment discloses an industrial parts sorting system based on deep learning. The system is as follows: Figure 1As shown, it includes: a conveying mechanism, a vision acquisition module 140, an actuator, a control and calculation module 150, and a sorting unit.
[0028] The conveying mechanism includes an arc-shaped conveyor belt 110 for conveying various industrial parts and a straight section of conveyor belt 112 that smoothly connects with it.
[0029] The vision acquisition module 140 includes a camera for real-time acquisition of color images of industrial parts and a depth camera for real-time acquisition of depth images of industrial parts, and is positioned above the straight section of the conveyor belt 112.
[0030] The depth camera is a binocular depth camera; the depth camera acquires the depth information and color image information of the object being detected through active infrared structured light or binocular parallax principle.
[0031] When the vision acquisition module 140 acquires color and depth images of industrial parts in real time, it also uses an adaptive lighting compensation algorithm to suppress reflection interference and noise errors based on the ambient lighting environment, preserving the texture details of the parts. It also uses a spatial filtering algorithm to remove noise points in the depth image and preserve the geometric contours of the part surface.
[0032] The actuators include a robotic arm 130 with six degrees of freedom and an intelligent adsorption-type end effector 131.
[0033] The intelligent adsorption end effector 131 has a built-in negative pressure sensor and a pressure sensor.
[0034] The control and computing module 150 includes a host computer controller, a visual reasoning unit, and an intelligent grasping planner.
[0035] The sorting unit includes several sorting bins 160a classified by category or sorting bins 160b classified by process rules.
[0036] The visual reasoning unit receives images acquired in real time by the visual acquisition module 140, and identifies and analyzes the images through deep neural networks and pose estimation algorithms to obtain part category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality.
[0037] The grasping planner obtains the optimal adsorption point and smooth grasping path based on the output data of the visual reasoning unit, and transmits it to the host computer controller to control the actuator to grasp the target part into the target sorting bin.
[0038] During the grasping process, the actuator also adaptively adjusts the suction force of the intelligent adsorption end effector 131 based on the surface curvature value, surface roughness, porosity and part mass of the part.
[0039] The visual reasoning unit is based on a deep neural network constructed with a multi-scale feature pyramid and an attention enhancement mechanism. It performs joint feature extraction on color and depth images and combines pose estimation algorithms to obtain part category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality.
[0040] The process of obtaining part category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity, and part quality by combining joint feature extraction with pose estimation algorithms specifically includes: Cross-modal registration: Based on the intrinsic and extrinsic parameters of each camera in the vision acquisition module 140, the three-dimensional coordinates of the acquired depth image are aligned with the pixel coordinates of the color image to ensure that the texture features and geometric features of the same part area correspond one-to-one.
[0041] Multi-scale feature extraction: Low-level detail features of industrial parts in color and depth images are extracted through shallow convolutional layers of deep neural networks. These low-level detail features include part edges, textures, local protrusions, and local depressions. The low-level detail features extracted by the shallow convolutional layers are then fused through deep convolutional layers of deep neural networks to generate color texture features and depth geometric features.
[0042] Cross-modal fusion: By channel-wise weighted summation, color texture features and depth geometric features are fused channel by channel to obtain joint features; In channel-wise weighted summation, for the case of stacking and occlusion of industrial parts in color images and depth images, the effective areas that are not occluded are focused through the spatial attention mechanism of deep neural networks, and the feature channel weights related to part recognition and pose calculation are strengthened through the channel attention mechanism.
[0043] Target recognition and segmentation: Based on joint features, the classification branch of the deep neural network is used to classify industrial parts and output the category information of the industrial parts. The segmentation branch of the deep neural network is used to perform pixel-level segmentation of the target parts and obtain the pixel-level mask of the target parts.
[0044] Specifically, the joint features are input into the network classification branch (fully connected layer + Softmax activation), and the network outputs a class probability distribution through a pre-trained class label library. The label with the highest probability is taken as the final class information.
[0045] Initial spatial pose calculation: Based on the pixel-level mask of the target part, the 3D coordinate information provided by the depth image, and joint features, the pose parameters are solved using the PnP algorithm with feature alignment, and the initial spatial pose is output. The initial spatial pose specifically includes the rotation matrix R, the translation vector t, i.e., the position x, y, z of the part in the camera coordinate system, as well as the roll angle, pitch angle, and yaw angle.
[0046] Surface normal information calculation: Based on the pixel-level mask of the target part, and according to the preset neighborhood window size, the local plane is fitted, and the surface normal vector of each pixel in the depth image of the part region is output, which also provides a basis for curvature calculation.
[0047] Surface curvature value calculation: Based on the three-dimensional coordinate information provided by the depth image and the calculated surface normal vector, a local differential geometric model of the part surface is constructed, and the surface curvature value is calculated based on the local differential geometric model.
[0048] Specifically, after constructing a local differential geometric model of the part surface, the gradients of the surface normal vector in the x and y directions are calculated to reflect the rate of change of the normal direction; based on the differential geometric formula, the principal curvature is calculated through the normal gradient, and the average curvature is calculated as a representative curvature value of the part surface; Gaussian filtering is applied to the average curvature to suppress curvature abrupt changes caused by noise, and the final surface curvature value is output.
[0049] Surface roughness calculation: The pixel-level mask regions of the depth image and color image are divided into micro-blocks of a preset size, and the gray-level variance, texture entropy, depth value variance and height range of each block are calculated to obtain roughness features; Based on the pre-established feature-roughness mapping network, roughness is mapped through roughness features, and the surface roughness is obtained by averaging the roughness of all blocks.
[0050] Specifically, the feature-roughness mapping network is a lightweight CNN. This network is trained on industrial standard samples with known roughness to establish a mapping relationship between image features and actual surface roughness.
[0051] Porosity calculation: Mark the color regions with gray values below the gray threshold in the pixel-level mask region of the color image and the depth regions with abrupt changes in depth values in the pixel-level mask region of the depth image that exceed the depth range of the main body of the part by ±0.5mm; take the intersection of the color region and the depth region to obtain the candidate porosity region, calculate the area of the candidate porosity region and the total area of the pixel-level mask region, and output the ratio of the area of the candidate porosity region to the total area of the pixel-level mask region as the porosity.
[0052] Specifically, after obtaining the candidate pore regions, morphological opening operations are used to remove noise pores with too small an area.
[0053] Part mass calculation: Based on the depth image and initial spatial pose, the two-dimensional depth map of the part is reconstructed into a three-dimensional point cloud. The Poisson surface reconstruction algorithm is used to transform the three-dimensional point cloud into a closed three-dimensional mesh model. The mesh volume is calculated, and the effective volume is obtained by subtracting the pore volume obtained based on porosity from the mesh volume. Based on the category information of industrial parts, the material and density of industrial parts are obtained, and the part mass is calculated.
[0054] Specifically, after calculating the mass of the part, a safety redundancy coefficient is added and multiplied by the part mass to compensate for the volume estimation error, so that a slightly heavier part mass is obtained as the final output data.
[0055] In this embodiment, after calculating and obtaining the above data, mean filtering is performed on curvature, roughness and porosity to remove outliers; the seven types of information are integrated into structured data and transmitted to the crawling planner in real time through the data interface.
[0056] The process by which the grasping planner obtains the optimal snap-in point and smooth grasping path based on the output data of the visual reasoning unit includes: The image data acquired by the vision acquisition module 140 is converted into base coordinate system data of the robotic arm 130, and all adsorption points on the surface of the part are obtained. Based on a preset multi-index adsorption point scoring standard, the adsorption points on the surface of the part are scored according to the initial spatial pose of the part, surface normal information, surface curvature value, surface roughness, porosity and part mass, so as to obtain the optimal adsorption point. Using the initial position of the robotic arm 130, the optimal adsorption point, and the target sorting bin position as path nodes, and adding transition points between the initial position of the robotic arm 130 and the optimal adsorption point, as well as between the optimal adsorption point and the target sorting bin position, a smooth grasping path that satisfies basic motion constraints is generated; among which, basic motion constraints include joint motion constraints, safety constraints, and smoothness constraints.
[0057] Specifically, the safety constraint requires that the minimum distance between all points on the path and obstacles must be less than a preset safety distance, and the smoothness constraint requires that joint velocity, acceleration, and jerk be continuous.
[0058] The intelligent adsorption end effector 131 includes a multi-size suction cup assembly that can be quickly replaced, a high-response miniature air valve, a negative pressure resistant silicone air tube, and a negative pressure flow stabilizing device.
[0059] Multi-specification suction cup components, high-response micro air valves, and negative pressure resistant silicone air tubes are connected by a flexible structure. The negative pressure stabilizing device controls the multi-specification suction cup components, high-response micro air valves, and negative pressure resistant silicone air tubes to adaptively adjust the suction force according to the curvature, surface roughness, and pore characteristics of the adsorbed parts, so as to adsorb the parts in a coordinated manner.
[0060] The process by which a negative pressure flow stabilizing device controls the adsorption components of multi-specification suction cup assemblies, high-response miniature air valves, and negative pressure-resistant silicone air tubes includes: The curvature, surface roughness, and porosity characteristics of the adsorbed parts are obtained. Based on the preset grading standard, the curvature, surface roughness, and porosity of the adsorbed parts are quantified and graded according to characteristic parameters to obtain the grading characteristics of the current adsorbed parts.
[0061] In this embodiment, the grading features include: low curvature, medium curvature, and high curvature; low roughness, medium roughness, and high roughness; low porosity, medium porosity, and high porosity.
[0062] Based on a preset suction cup specification matching matrix, and according to the classification characteristics of the parts to be adsorbed, an appropriate suction cup assembly is selected. The suction cup assembly includes a flat hard suction cup, a flexible corrugated suction cup, and a multi-chamber vacuum suction cup.
[0063] In this embodiment, the preset matching criteria are: Low curvature + low roughness: The flat hard suction cup is selected, which has a large adsorption area and good sealing performance; Medium curvature + medium roughness: Flexible corrugated tube suction cups are used to conform to the deformation of curved surfaces and compensate for surface unevenness; High curvature + high roughness: Multi-chamber vacuum suction cups are used for zoned adsorption to avoid local air leakage affecting the overall seal.
[0064] Based on the surface roughness, pore characteristics, and mass of the adsorbed parts, the target negative pressure corresponding to the minimum stable adsorption force is calculated.
[0065] The formula for calculating the target negative pressure corresponding to the minimum stable adsorption force is: in, For the quality of the parts, It is the acceleration due to gravity. For safety, redundant friction force, For porosity The determined leakage compensation pressure, The effective adsorption area of the suction cup. The surface roughness of the adsorbed parts The sealing efficiency is determined by this.
[0066] The higher, The larger the value, the range is −5kPa to −20kPa, and the negative sign indicates negative pressure; The lower, The higher the value, the greater the range is 0.7 to 0.95.
[0067] In addition, there is a target negative pressure constraint during the calculation: , To preset the minimum negative pressure value, The maximum negative pressure is preset to avoid damage to parts due to excessively high negative pressure or detachment due to excessively low negative pressure.
[0068] Based on the pressure sensor built into the intelligent adsorption end effector 131, the initial pressure is obtained. Based on the initial pressure and the target negative pressure, the valve opening duty cycle of the high-response micro air valve is calculated, and the response frequency of the high-response micro air valve is obtained according to the pore characteristics of the adsorbed parts.
[0069] Based on valve opening duty cycle and response frequency control, the opening of the air inlet / exhaust channel of the high-response micro air valve is adjusted, and the suction cup assembly is used to adsorb the parts to be adsorbed.
[0070] During the adsorption process, the built-in negative pressure sensor of the intelligent adsorption end effector 131 collects pressure data in real time, calculates the average pressure and standard deviation of fluctuation, and continuously monitors the contact pressure between the suction cup and the part. Based on the average pressure, standard deviation of fluctuation, and contact pressure, the target negative pressure, valve opening duty cycle, and response frequency are adjusted in real time.
[0071] The specific process of adjusting the target negative pressure, valve opening duty cycle, and response frequency in real time based on the pressure mean, standard deviation of fluctuation, and fitting pressure includes: Compare the average pressure with the preset suction threshold. If the average pressure is less than the preset minimum suction threshold, it is determined that the suction is insufficient, and the valve opening duty cycle is increased. If the average pressure is greater than the preset maximum suction threshold, it is determined that the suction is too large, and the valve opening duty cycle is reduced or the pressure relief valve of the high-response micro air valve is briefly opened. The bonding pressure is compared with the preset bonding threshold. If the bonding pressure is less than the preset bonding threshold, it is determined that the bonding is insufficient. The robotic arm 130 posture is then finely adjusted and the target negative pressure is reduced. The target negative pressure is restored after the bonding pressure is greater than the preset minimum bonding threshold. Compare the standard deviation of the fluctuation with the preset fluctuation threshold. If the standard deviation of the fluctuation is greater than the preset fluctuation threshold, increase the target negative pressure and the response frequency of the high-response micro valve.
[0072] Furthermore, different suction cups adapt to different shapes during the adsorption process: Flexible bellows suction cup: The bellows is extended and retracted by internal pressure difference to fit the medium curvature surface, forming a tight contact between the curved surface and the suction cup, compensating for ±3mm surface unevenness error. Multi-chamber vacuum chuck: The device independently monitors the pressure of each chamber. If a chamber leaks air due to porosity / roughness (pressure higher than normal), the device will detect the leak. If the pressure drops, the corresponding miniature sub-valve of that chamber will be closed, focusing on the effective adsorption area and preventing a drop in overall pressure.
[0073] Example 2 This embodiment, based on Embodiment 1 above, discloses an industrial parts sorting method, as follows: Figure 2 As shown, it includes: S1, start the conveyor mechanism and put the mixed industrial parts to be sorted into the arc-shaped conveyor belt 110. Through the spatial geometric guidance structure of the arc segment, the random posture of the parts is automatically straightened.
[0074] S2, after being transported by the arc-shaped conveyor belt 110, the industrial parts enter the straight section of the conveyor belt 112, and the color and depth images of the industrial parts are collected in real time by the vision acquisition module 140.
[0075] S3 transmits the acquired image data to the control and computing module 150. The visual reasoning unit performs joint feature extraction on the color image and depth image to obtain the category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality of the industrial parts.
[0076] S4. Based on the category information, initial spatial pose, surface normal information, surface curvature value, surface roughness, porosity, and part mass of the industrial parts, the intelligent grasping planner generates the optimal adsorption point and planned path for the robotic arm 130 and sends it to the actuator. The actuator adaptively adjusts the suction force of the intelligent adsorption end effector 131 according to the optimal adsorption point and based on the surface curvature value, surface roughness, porosity, and part mass to adsorb the corresponding industrial parts and classify them into the designated sorting bins according to the planned path.
[0077] S5: After the current industrial parts are sorted, a reset command is sent to the actuator through the host computer controller, so that the actuator returns to the initial position along the planned path.
[0078] S6: The vision acquisition module 140 continuously acquires images. If no image of the industrial parts is acquired within a preset time, the sorting task is considered complete and the current sorting ends. Otherwise, return to S2.
[0079] Example 3 Based on Embodiments 1 and 2, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned industrial parts sorting method.
[0080] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned industrial parts sorting method. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0081] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0082] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A deep learning-based industrial parts sorting system, characterized in that, The system includes: a conveying mechanism, a vision acquisition module, an execution mechanism, a control and calculation module, and a sorting unit; The conveying mechanism includes an arc-shaped conveyor belt for conveying various industrial parts and a straight section of the conveyor belt that smoothly connects with it. The vision acquisition module includes a camera for real-time acquisition of color images of industrial parts and a depth camera for real-time acquisition of depth images of industrial parts, and is deployed above the straight section of the conveyor belt. The actuator includes a robotic arm with six degrees of freedom and an intelligent adsorption-type end effector, wherein the intelligent adsorption-type end effector has a built-in negative pressure sensor and a pressure sensor; The control and computing module includes a host computer controller, a visual reasoning unit, and an intelligent grasping planner. The sorting unit includes several sorting bins classified by category or process rules; The visual reasoning unit receives images acquired in real time by the visual acquisition module and identifies and analyzes the images using a deep neural network and pose estimation algorithm to obtain part category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity, and part mass. The grasping planner obtains the optimal adsorption point and smooth grasping path based on the output data of the visual reasoning unit and transmits it to the host computer controller to control the actuator to grasp the target part into the target sorting bin. The actuator adaptively adjusts the suction force of the intelligent adsorption end effector based on the part surface curvature value, surface roughness, porosity, and part mass.
2. The deep learning-based industrial parts sorting system according to claim 1, characterized in that, The depth camera is a binocular depth camera; the depth camera acquires the depth information and color image information of the object being detected through active infrared structured light or binocular parallax principle.
3. The deep learning-based industrial parts sorting system according to claim 1, characterized in that, The intelligent adsorption-type end effector includes a multi-specification suction cup assembly that can be quickly replaced, a high-response micro air valve, a negative pressure resistant silicone air tube, and a negative pressure stabilizing device. The multi-specification suction cup assembly, the high-response micro air valve, and the negative pressure resistant silicone air tube are connected by a flexible structure. The negative pressure stabilizing device controls the multi-specification suction cup assembly, the high-response micro air valve, and the negative pressure resistant silicone air tube to adaptively adjust the suction force according to the curvature, surface roughness, and porosity characteristics of the adsorbed parts, so as to adsorb the parts in a coordinated manner.
4. The deep learning-based industrial parts sorting system according to claim 3, characterized in that, The process by which the negative pressure stabilizing device controls the multi-specification suction cup assembly, the high-response micro air valve, and the negative pressure resistant silicone air tube to collaboratively adsorb the components includes: The curvature, surface roughness, and porosity characteristics of the adsorbed parts are obtained. Based on the preset grading standard, the curvature, surface roughness, and porosity of the adsorbed parts are quantified and graded according to the characteristic parameters to obtain the grading characteristics of the current adsorbed parts. Based on a preset suction cup specification matching matrix, and according to the grading characteristics of the adsorbed parts, an appropriate suction cup assembly is selected. The suction cup assembly includes a planar hard suction cup, a flexible corrugated suction cup, and a multi-chamber vacuum suction cup. Based on the surface roughness, pore characteristics and mass of the adsorbed parts, the target negative pressure corresponding to the minimum stable adsorption force is calculated. Based on the pressure sensor built into the intelligent adsorption end effector, the initial pressure is obtained. Based on the initial pressure and the target negative pressure, the valve opening duty cycle of the high-response micro air valve is calculated and obtained. The response frequency of the high-response micro air valve is obtained according to the pore characteristics of the adsorbed parts. Based on the valve opening duty cycle and response frequency control, the opening of the air inlet / exhaust channel of the high-response micro air valve is adjusted, and it works in conjunction with the suction cup assembly to adsorb the parts being adsorbed; wherein... During the adsorption process, the built-in negative pressure sensor of the intelligent adsorption end effector collects pressure data in real time, calculates the average pressure and standard deviation of fluctuation, and continuously monitors the contact pressure between the suction cup and the part. Based on the average pressure, standard deviation of fluctuation, and contact pressure, the target negative pressure, valve opening duty cycle, and response frequency are adjusted in real time.
5. The deep learning-based industrial parts sorting system according to claim 4, characterized in that, The specific process of adjusting the target negative pressure, valve opening duty cycle, and response frequency in real time based on the average pressure, standard deviation of fluctuation, and fitting pressure includes: Compare the average pressure with the preset suction threshold. If the average pressure is less than the preset minimum suction threshold, it is determined that the suction is insufficient, and the valve opening duty cycle is increased. If the average pressure is greater than the preset maximum suction threshold, it is determined that the suction is too large, and the valve opening duty cycle is reduced or the pressure relief valve of the high-response micro air valve is briefly opened. The bonding pressure is compared with the preset bonding threshold. If the bonding pressure is less than the preset bonding threshold, it is determined that the bonding is insufficient. The robot arm posture is then finely adjusted and the target negative pressure is reduced. Once the bonding pressure is greater than the preset minimum bonding threshold, the target negative pressure is restored. Compare the standard deviation of the fluctuation with the preset fluctuation threshold. If the standard deviation of the fluctuation is greater than the preset fluctuation threshold, increase the target negative pressure and the response frequency of the high-response micro valve.
6. The industrial parts sorting system based on deep learning according to claim 1, characterized in that, When the vision acquisition module acquires color and depth images of industrial parts in real time, it also uses an adaptive lighting compensation algorithm to suppress reflection interference and noise errors based on the ambient lighting environment, and removes noise points in the depth image through a spatial filtering algorithm.
7. The deep learning-based industrial parts sorting system according to claim 1, characterized in that, The visual reasoning unit is based on a deep neural network constructed with a multi-scale feature pyramid and an attention enhancement mechanism. It performs joint feature extraction on the color image and the depth image, and combines the pose estimation algorithm to obtain the category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality of the stacked industrial parts.
8. The deep learning-based industrial parts sorting system according to claim 7, characterized in that, The process of obtaining category information, initial spatial pose, surface normal information, surface curvature value, surface roughness, porosity, and part quality of stacked industrial parts by combining joint feature extraction with pose estimation algorithm includes: Cross-modal registration: Based on the intrinsic and extrinsic parameters of each camera in the vision acquisition module, the 3D coordinates of the acquired depth image are aligned with the pixel coordinates of the color image; Multi-scale feature extraction: Low-level detail features of industrial parts in the color image and depth image are extracted through shallow convolutional layers of a deep neural network. The low-level detail features include part edges, textures, local protrusions and local depressions. The low-level detail features extracted by the shallow convolutional layers are fused through deep convolutional layers of a deep neural network to generate color texture features and depth geometric features. Cross-modal fusion: The color texture features and depth geometric features are fused channel by channel by channel weighted summation to obtain joint features; In the channel weighted summation, for the case of stacking and occlusion of industrial parts in the color image and depth image, the effective area that is not occluded is focused by the spatial attention mechanism of the deep neural network, and the feature channel weights related to part recognition and pose calculation are strengthened by the channel attention mechanism. Target recognition and segmentation: Based on the joint features, the classification branch of the deep neural network is used to determine the category of industrial parts and output the category information of industrial parts. The segmentation branch of the deep neural network is used to perform pixel-level segmentation of the target parts and obtain the pixel-level mask of the target parts. Calculate the initial spatial pose: Based on the pixel-level mask of the target part, the three-dimensional coordinate information provided by the depth image, and the joint features, the pose parameters are solved by the PnP algorithm with feature alignment, and the initial spatial pose is output. Surface normal information calculation: Based on the pixel-level mask of the target part, according to the preset neighborhood window size, fit the local plane and output the surface normal vector of each pixel in the depth image of the part region; Surface curvature value calculation: Based on the three-dimensional coordinate information provided by the depth image and the calculated surface normal vector, a local differential geometric model of the part surface is constructed, and the surface curvature value is calculated based on the local differential geometric model; Surface roughness calculation: Divide the pixel-level mask regions of the depth image and color image into micro-blocks of a preset size, and calculate the gray-level variance, texture entropy, depth value variance and height range of each block to obtain roughness features; Based on the pre-established feature-roughness mapping network, roughness is mapped through the roughness features, and the roughness of all blocks is averaged to obtain the surface roughness. Porosity calculation: Mark the color regions with gray values below the gray threshold in the pixel-level mask region of the color image and the depth regions with abrupt changes in depth values that exceed the depth range of the main body of the part in the pixel-level mask region of the depth image; take the intersection of the color region and the depth region to obtain the candidate porosity region; calculate the area of the candidate porosity region and the total area of the pixel-level mask region; and output the porosity as the ratio of the area of the candidate porosity region to the total area of the pixel-level mask region. Part mass calculation: Based on the depth image and initial spatial pose, the two-dimensional depth map of the part is reconstructed into a three-dimensional point cloud. The Poisson surface reconstruction algorithm is used to transform the three-dimensional point cloud into a closed three-dimensional mesh model. The mesh volume is calculated, and the effective volume is obtained by subtracting the pore volume obtained based on porosity from the mesh volume. Based on the category information of industrial parts, the material and density of industrial parts are obtained, and the part mass is calculated.
9. The industrial parts sorting system based on deep learning according to claim 1, characterized in that, The process by which the grasping planner obtains the optimal grabbing point and smooth grasping path based on the output data of the visual reasoning unit includes: The image data acquired by the vision acquisition module is converted into base coordinate system data of the robotic arm, and all adsorption points on the surface of the part are obtained; Based on a preset multi-index adsorption point scoring standard, the adsorption points on the surface of the part are scored according to the initial spatial pose of the part, surface normal information, surface curvature value, surface roughness, porosity and part mass, so as to obtain the optimal adsorption point. Using the initial position of the robotic arm, the optimal adsorption point, and the target sorting bin position as path nodes, and adding transition points between the initial position of the robotic arm and the optimal adsorption point, as well as between the optimal adsorption point and the target sorting bin position, a smooth grasping path that satisfies basic motion constraints is generated; wherein, the basic motion constraints include joint motion constraints, safety constraints, and smoothness constraints.
10. An industrial parts sorting method based on the deep learning-based industrial parts sorting system as described in any one of claims 1-9, characterized in that, The method includes: Step S1: Start the conveyor mechanism and put the mixed industrial parts to be sorted into the arc-shaped conveyor belt. The random posture of the parts is automatically straightened by the spatial geometric guidance structure of the arc segment. In step S2, after the industrial parts are conveyed by the arc-shaped conveyor belt, they enter the straight section of the conveyor belt, and the color and depth images of the industrial parts are acquired in real time by the vision acquisition module. Step S3: The acquired image data is transmitted to the control and calculation module. The visual reasoning unit performs joint feature extraction on the color image and depth image to obtain the category information, initial spatial pose value, surface normal information, surface curvature value, surface roughness, porosity and part quality of the industrial parts. Step S4: Based on the category information, initial spatial pose, surface normal information, surface curvature value, surface roughness, porosity, and part mass of the industrial parts, the intelligent grasping planner generates the optimal adsorption point and planned path for the robotic arm and sends them to the execution mechanism. The execution mechanism adaptively adjusts the suction force of the intelligent adsorption end effector based on the optimal adsorption point and the surface curvature value, surface roughness, porosity, and part mass to adsorb the corresponding industrial parts and classify them into designated sorting bins according to the planned path. Step S5: After the current industrial parts are sorted, a reset command is sent to the actuator through the host computer controller, so that the actuator returns to the initial position along the planned path; Step S6: Continuously acquire images through the vision acquisition module. If no image of industrial parts is acquired within a preset time, it is determined that all sorting tasks have been completed and the current sorting ends; otherwise, return to step S2.