Gearbox sorting method and system based on vision

Abstract

The invention discloses a gearbox sorting method and system based on vision, and relates to the related field of machine vision control, and the method comprises the steps: collecting a three-dimensional point cloud image and bar code information of a gearbox; the collected three-dimensional point cloud image of the gearbox is preprocessed; segmenting the three-dimensional point cloud image of the gearbox by adopting a point cloud image segmentation model based on deep learning; based on the segmented three-dimensional point cloud image of the gearbox, obtaining position information of the target gearbox; the position information and the bar code information of the target gearbox are transmitted to a control unit, and path planning data of the sorting robot are generated; and an instruction sent by the control unit is received, the sorting robot is designated to execute sorting operation, and the target gearbox is conveyed to a designated area. According to the invention, precise segmentation of the three-dimensional point cloud image of the gearbox is realized, and the accurate coordinate of the target gearbox is obtained. The problems that an existing sorting method is low in efficiency and high in sorting error rate are solved, and the technical effect of improving the sorting accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the field related to machine vision control, and in particular to a vision-based gearbox sorting method and system. Background Art

[0002] In scenarios such as logistics, warehouse management, and industrial production, product sorting is a key link in the production process. According to order requirements or delivery plans, products need to be quickly and accurately picked out from storage areas, and classified and centrally processed according to established rules to ensure that they arrive at designated locations efficiently. On automobile production lines, real-time sorting of various parts is crucial and directly related to whether assembly tasks can be completed on time. Among them, the sorting of gearboxes is particularly critical, and its high efficiency and high accuracy are important prerequisites for ensuring the smooth progress of automobile production.

[0003] At present, common gearbox sorting methods mainly include manual sorting, mechanical sorting and automatic sorting. Although manual sorting has high flexibility and can adapt to small-scale production scenarios, it is inefficient and prone to sorting errors due to human factors, which in turn affects the production progress of products. At the same time, it requires a lot of manpower costs. Mechanical sorting uses tools such as conveyors and slides to assist in sorting tasks, which can improve sorting efficiency to a certain extent, but its equipment maintenance costs are high, and its flexibility is insufficient, making it difficult to cope with complex and changing sorting needs. Automatic sorting is a way to achieve efficient and accurate sorting with the help of automated equipment, such as barcode scanning, electronic tags and manipulators, which is suitable for large-scale sorting scenarios. However, barcode scanning and electronic tags can only play an auxiliary role and cannot completely ensure the accuracy of sorting; the range of activities of the manipulator is limited and its adaptability in complex working environments is poor. These factors limit the scope of application of automatic sorting systems to a certain extent.

[0004] With the continuous advancement of technology, many new sorting technologies have emerged. Among them, sorting technology based on machine vision has gradually emerged. This technology processes the image of the gearbox and uses visual inspection methods to obtain its production information and location information, thereby providing powerful assistance for sorting. In recent years, this method has been widely used, however, its accuracy and robustness still need to be further improved to better meet the increasingly complex and changing production needs. Summary of the invention

[0005] In response to the technical problems existing in the above-mentioned prior art, the present application provides a vision-based gearbox sorting method, processes the collected gearbox three-dimensional point cloud image, obtains the spatial position of the gearbox, and uploads the acquired data to the system in real time, accurately plans the sorting route, and controls the sorting robot to quickly and accurately transport the gearbox to the designated location, thereby reducing the sorting error rate and greatly improving the sorting efficiency.

[0006] The present application provides a vision-based gearbox sorting method, the specific steps of which include:

[0007] (1) Collect the 3D point cloud image and barcode information of the gearbox;

[0008] (2) Preprocessing the collected gearbox three-dimensional point cloud image;

[0009] (3) Use a deep learning-based point cloud image segmentation model to segment the gearbox three-dimensional point cloud image;

[0010] (4) Based on the segmented gearbox three-dimensional point cloud image, obtain the position information of the target gearbox;

[0011] (5) transmitting the location information and barcode information of the target gearbox to the control unit to generate path planning data for the sorting robot;

[0012] (6) Receive instructions sent by the control unit, designate the sorting robot to perform sorting operations, and transport the target gearbox to the designated area.

[0013] Furthermore, an RGB-D camera (3D camera) is used to obtain the color image and depth information of the gearbox. This camera combines the color capture capability of the traditional RGB camera and the three-dimensional information acquisition capability of the depth camera. At the same time, a fixed barcode scanner is used to scan the barcode of each gearbox to obtain the model, production date and customer order information. According to the color image and depth information of the gearbox, the two-dimensional coordinates (x, y) of each image pixel and the z coordinate in the camera coordinate system (i.e., the distance between the gearbox and the camera) are obtained respectively. Combined with the camera parameters, this information is converted into a three-dimensional point cloud image in the world coordinate system.

[0014] Furthermore, the specific steps of converting the collected gearbox three-dimensional image into a three-dimensional point cloud image include:

[0015] Determine the parameters of the RGB-D camera used, including the focal length and base point when acquiring the 3D image of the gearbox;

[0016] Using a computer to read the color image and depth information of the gearbox, specifically involving the OpenCV image processing library in a Python program;

[0017] According to the camera parameters and the coordinate information of each pixel point obtained from the three-dimensional image of the gearbox, the three-dimensional coordinates of each pixel are calculated to generate point cloud data.

[0018] Call the Python program and use the Open3D library to save the point cloud data of the gearbox as a PCD file;

[0019] The visualization function in the Open3D library is called to process the point cloud data and generate a three-dimensional point cloud image of the gearbox.

[0020] Furthermore, the gearbox three-dimensional point cloud image is preprocessed, and the preprocessing process includes the following steps:

[0021] Firstly, the statistical filtering method is used to remove noise points in the point cloud image, and the average distance and standard deviation in the neighborhood of each point cloud are calculated. If the neighborhood distance of a point cloud is higher than the multiple of the average and standard deviation, the point cloud is determined to be noise and is removed.

[0022] Secondly, the normal vector filtering method is used to smooth the point cloud image and reduce data redundancy. The principal component analysis is performed on the neighborhood point set of each point cloud, the normal vector of each point cloud is calculated, and the Z vectors of all point cloud normal vectors are adjusted to positive values; plane fitting is performed on each point cloud and its K nearest neighbor points, and the original normal vector is corrected using the normal vector of the fitting plane; the position of the point cloud is adjusted to the intersection of the normal vector and the fitting plane to achieve smooth processing of the point cloud.

[0023] Finally, the point cloud image is downsampled to reduce the size of the point cloud data, thereby reducing the computational complexity. The downsampling method adopted is voxel downsampling, which divides the point cloud image into a fixed-size three-dimensional grid, and uses the centroid of each grid point to represent all the points in the network.

[0024] Furthermore, PointNet is used as a point cloud image segmentation model based on deep learning to achieve the segmentation of the gearbox three-dimensional point cloud image. PointNet directly processes each point in the point cloud image without converting the point cloud data into a regular grid or voxel, and aggregates global features through symmetric functions. This method is insensitive to the order of the point cloud and can process any number of input points. The model includes an input layer, a feature extraction layer, a global feature extraction layer, and a segmentation layer. It can not only maintain the disorder of the point cloud features, but also extract global and local information from it to meet the needs of the segmentation task.

[0025] Furthermore, the position information of the gearbox is obtained based on the 3D point cloud segmentation image of the gearbox processed by the deep learning-based point cloud image segmentation model. The segmentation model assigns a label to each point in the 3D point cloud image of the gearbox, indicating the category to which the point belongs. Based on the 3D point cloud segmentation image of the gearbox, the number of gearboxes in the image and the area of ​​the target gearbox are determined, the geometric center of the target gearbox area is calculated, and the coordinates of the geometric center are used as the position information of the target gearbox.

[0026] Furthermore, the location information and barcode information of the target gearbox are transmitted to the control unit, and the acquired information is compared with the information in the data storage system to determine the transportation information of the target gearbox, namely the user order information, the transportation destination coordinates and the transportation quantity. Then, the transportation path of the sorting robot is planned according to the location of the gearbox and the destination location.

[0027] The present application also provides a vision-based gearbox sorting system, comprising:

[0028] An image acquisition unit, used to acquire a three-dimensional point cloud image and barcode information of the gearbox;

[0029] An image preprocessing unit, used for preprocessing the collected gearbox three-dimensional point cloud image;

[0030] An image segmentation unit, which uses a deep learning-based point cloud image segmentation model to segment the gearbox three-dimensional point cloud image;

[0031] A position information processing unit, used for acquiring position information of a target gearbox based on the segmented three-dimensional point cloud image of the gearbox;

[0032] A control unit, used to transmit the position information and barcode information of the target gearbox to the control unit to generate path planning data for the sorting robot;

[0033] The execution unit is used to receive the instructions sent by the control unit, designate the sorting robot to perform the sorting operation, and transport the target gearbox to the designated area.

[0034] The present invention discloses the following technical effects:

[0035] The visual inspection method of the present invention obtains the necessary information required in the sorting process by collecting images of the gearbox. Unlike traditional two-dimensional images that can only provide planar projection information, the present invention uses an RGB-D camera to capture a three-dimensional image of the gearbox, so that depth information can be obtained and the accurate position and distance relationship of the gearbox in space can be determined. Further, the three-dimensional image of the gearbox is converted into a three-dimensional point cloud image, and its three-dimensional coordinates are directly recorded to provide accurate position information for the sorting process. In addition, the present invention also uses a point cloud image segmentation model based on deep learning to segment the three-dimensional point cloud image of the gearbox, which can accurately identify the specific number and type of gearboxes. This method effectively solves the problem of information loss caused by gearbox stacking and occlusion, and significantly improves the accuracy and reliability of detection. By extracting the image area of ​​the target gearbox from the three-dimensional point cloud image of the gearbox, and further determining its geometric center as the position coordinate of the target gearbox, the accuracy of positioning is further improved. The deep learning model performs well in processing speed and robustness to point cloud data. It can not only accurately locate the target gearbox, but also has stronger stability and adaptability than traditional visual inspection methods. This provides strong data support for the path planning of the sorting robot, significantly improves the sorting efficiency and accuracy, and brings significant improvements to industrial automation production. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0037] Figure 1 A schematic flow chart of a vision-based gearbox sorting method provided in an embodiment of the present application.

[0038] Figure 2 A detailed structural diagram of the deep learning-based point cloud image segmentation model provided in an embodiment of the present application.

[0039] Figure 3 A schematic structural diagram of a vision-based gearbox sorting system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0042] In the following description, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application.

[0043] Embodiment 1: This embodiment of the present application provides a vision-based gearbox sorting method, such as Figure 1 As shown, the method includes:

[0044] Step S10, collect the three-dimensional point cloud image and barcode information of the gearbox. Use an RGB-D camera (3D camera) to obtain the color image and depth information of the gearbox. The camera combines the color capture capability of the traditional RGB camera and the three-dimensional information acquisition capability of the depth camera; the RGB-D camera uses structured light, time of flight (ToF), binocular stereo vision and phase detection technology to calculate the distance by emitting modulated light signals and measuring the time it takes for the light signals to return to the camera; the camera uses two cameras to simulate human binocular vision and calculates the depth by comparing the parallax of the left and right images. At the same time, a fixed barcode scanner is used to scan the barcode of each gearbox to obtain the model, production date and customer order information.

[0045] According to the color image and depth information of the gearbox, the two-dimensional coordinates (x, y) of each image pixel and the z coordinate in the camera coordinate system (i.e., the distance between the gearbox and the camera) are obtained respectively, and combined with the camera parameters, these information are converted into a three-dimensional point cloud image in the world coordinate system. The specific steps for converting the collected gearbox three-dimensional image into a three-dimensional point cloud image include:

[0046] Determine the parameters of the RGB-D camera used, including the focal length and base point when acquiring the 3D image of the gearbox;

[0047] Using a computer to read the color image and depth image of the gearbox, specifically involving the OpenCV image processing library in a Python program;

[0048] According to the camera parameters and the coordinate information of each pixel point obtained from the gearbox three-dimensional image, the three-dimensional coordinates of each pixel are calculated to generate point cloud data: For each pixel point (u, v) in the image, its depth value is D(u, v), and its three-dimensional coordinates (X, Y, Z) in the world coordinate system are calculated using the following formula:

[0049]

[0050] Z=D(u,v)

[0051] Among them, c x and c y are the coordinates of the center of the image, f x and f y are the focal lengths along the x-axis and y-axis, respectively. It is worth noting that when calculating the depth value, the unit is converted to meters.

[0052] Call the Python program and use the Open3D library to save the point cloud data of the gearbox as a PCD file;

[0053] The visualization function in the Open3D library is called to process the point cloud data and generate a three-dimensional point cloud image of the gearbox.

[0054] Step S20, preprocessing the acquired three-dimensional point cloud image of the gearbox, the preprocessing process includes the following steps:

[0055] Firstly, the statistical filtering method is used to remove noise points in the point cloud image, and the average distance and standard deviation in the neighborhood of each point cloud are calculated. If the neighborhood distance of a point cloud is higher than the multiple of the average and standard deviation, the point cloud is determined to be noise and is removed.

[0056] Secondly, the normal vector filtering method is used to smooth the point cloud image and reduce data redundancy. The principal component analysis is performed on the neighborhood point set of each point cloud to calculate the normal vector of each point cloud. The specific steps are as follows:

[0057] Select the K nearest neighbor point set for each point;

[0058] Calculate the covariance matrix of the neighborhood point set and solve its eigenvalues ​​and eigenvectors;

[0059] Select the eigenvector corresponding to the minimum eigenvalue as the normal vector of the point.

[0060] To ensure that the normal vectors of all points are in the same direction, the Z vectors of all point cloud normal vectors are adjusted to positive values. A plane is fitted for each point cloud and its K nearest neighbor points, and the original normal vector is corrected using the normal vector of the fitted plane. The position of the point cloud is adjusted to the intersection of the normal vector and the fitted plane to achieve smooth processing of the point cloud.

[0061] Finally, the point cloud image is downsampled to reduce the size of the point cloud data, thereby reducing the computational complexity. The downsampling method adopted is voxel downsampling, which divides the point cloud image into a fixed-size three-dimensional grid, and uses the centroid of each grid point to represent all the points in the network.

[0062] Step S30, using a deep learning-based point cloud image segmentation model to segment the gearbox three-dimensional point cloud image. The deep learning-based point cloud image segmentation model is PointNet. PointNet directly processes each point in the point cloud image without converting the point cloud data into a regular grid or voxel, and aggregates global features through symmetric functions. This method is insensitive to the order of the point cloud and can process any number of input points. The model includes an input layer, a feature extraction layer, a global feature extraction layer, and a segmentation layer, which can not only maintain the disorder of the point cloud features, but also extract global and local information from it to meet the needs of the segmentation task.

[0063] The detailed structure of PointNet is as follows Figure 2 As shown in the figure, the input feature of the model is the 3D point cloud data of the gearbox. Assume that it contains N points, each with 3D coordinates (X, Y, Z). The input feature is an n×3 matrix. PointNet contains multiple multi-layer perceptron layers to extract features of each point. Figure 2 The numbers in brackets below each multilayer perceptron in the figure represent its output feature dimension. The multilayer perceptron layer maps each point from 3 dimensions to a higher dimension. This point-by-point operation maintains the disorder of the point cloud. Each multilayer perceptron uses a parameter sharing mechanism to reduce computational complexity.

[0064] In the input transformation and feature transformation stages of the model, the T-Net structure is used to realize the rotation and translation transformation of the point cloud. The purpose of T-Net is to learn an affine transformation matrix to align the input points. T-Net is a pre-trained model used to estimate the parameters of the affine transformation matrix, and perform matrix multiplication operations on the estimated affine transformation matrix and the point cloud to realize the point cloud transformation. The T-Net structure in the two stages is used to align the input points and local features respectively. This structure reduces the impact of rotation and translation changes on feature extraction.

[0065] The local features obtained after feature transformation are further extracted through multiple multi-layer perceptrons to extract high-dimensional local features, mapping the dimension of each point from 64 to 1024. The maximum pooling function (symmetric function) is applied to the high-dimensional local features to compress the features of all points into a 1024-dimensional global feature vector as the global feature of the point cloud. The maximum pooling function ensures that the network does not depend on the order of the points and ensures the disorder of the point cloud.

[0066] The global features are copied to obtain global features of size n×1024, and then fused with local features of size n×64. Features of size n×1088 are concatenated along the last dimension, and the features are refined through a multi-layer perceptron to reduce the feature dimension to 128 dimensions. The combination of global features and local features can help the model understand the overall structure and details of the point cloud. Secondly, the features are mapped into a score matrix of size n×m through a multi-layer perceptron as the output of the entire network. The matrix contains the probability distribution of the predicted category label for each point, where m represents the number of gearbox categories. Based on the probability distribution results, the category label is predicted for each point, and a three-dimensional point cloud segmentation image of the gearbox is generated, with point clouds of different colors representing different categories of gearboxes.

[0067] Step S40, according to the 3D point cloud segmentation image of the gearbox processed by the deep learning-based point cloud image segmentation model, the position information of the target gearbox is obtained. According to the 3D point cloud segmentation image of the gearbox, the number of gearboxes in the image and the area of ​​the target gearbox are determined, and the geometric center of the target gearbox area is calculated. The calculation process of the geometric center is as follows:

[0068] In this embodiment, it is assumed that there are k points in the image area of ​​the target gearbox, and the coordinates of each point are (x i ,y i ,z i ), geometric center (x c ,y c ,z c ) is calculated as follows:

[0069]

[0070]

[0071] Where i is the index of the point, ranging from 1 to k. The coordinates of the geometric center are used as the position coordinates of the target gearbox to accurately locate the position of the target gearbox.

[0072] Step S50, the location information and barcode information of the target gearbox are transmitted to the control unit, and the acquired information is compared with the information in the data storage system to determine the transportation information of the target gearbox, i.e., the user order information, the coordinates of the transportation destination and the transportation quantity. Then, the transportation path of the sorting robot is planned according to the location of the gearbox and the destination location.

[0073] Step S60, receiving the instruction sent by the control unit, designating the sorting robot to perform the sorting operation, and transporting the target gearbox to the designated area.

[0074] Embodiment 2: A vision-based gearbox sorting system, such as Figure 3 As shown, including:

[0075] An image acquisition unit, used to acquire a three-dimensional point cloud image and barcode information of the gearbox;

[0076] An image preprocessing unit, used for preprocessing the collected gearbox three-dimensional point cloud image;

[0077] An image segmentation unit, which uses a deep learning-based point cloud image segmentation model to segment the gearbox three-dimensional point cloud image;

[0078] A position information processing unit, used for acquiring position information of a target gearbox based on the segmented three-dimensional point cloud image of the gearbox;

[0079] A control unit, used to transmit the position information and barcode information of the target gearbox to the control unit to generate path planning data for the sorting robot;

[0080] The execution unit is used to receive the instructions sent by the control unit, designate the sorting robot to perform the sorting operation, and transport the target gearbox to the designated area.

[0081] A vision-based gearbox sorting system provided in an embodiment of the present invention can execute a vision-based gearbox sorting method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0082] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0083] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A vision-based gearbox sorting method, characterized in that: The method comprises: (1) Collect the 3D point cloud image and barcode information of the gearbox; (2) Preprocessing the collected gearbox three-dimensional point cloud image; (3) Use a deep learning-based point cloud image segmentation model to segment the gearbox three-dimensional point cloud image; (4) Based on the segmented gearbox three-dimensional point cloud image, obtain the position information of the target gearbox; (5) transmitting the location information and barcode information of the target gearbox to the control unit to generate path planning data for the sorting robot; (6) Receive instructions sent by the control unit, designate the sorting robot to perform sorting operations, and transport the target gearbox to the designated area.

2. A vision-based gearbox sorting method as claimed in claim 1, characterized in that: An RGB-D camera is used to obtain the color image and depth information of the gearbox. A fixed barcode scanner is used to scan the barcode of each gearbox to obtain the model, production date and customer order information. The two-dimensional coordinates (x, y) of each image pixel and the z coordinate in the camera coordinate system are obtained based on the color image and depth information of the gearbox. Combined with the camera parameters, this information is converted into a three-dimensional point cloud image in the world coordinate system.

3. A vision-based gearbox sorting method as claimed in claim 2, characterized in that: The specific steps of converting the gearbox 3D image into a 3D point cloud image include: Determine the parameters of the RGB-D camera used, including the focal length and base point when acquiring the 3D image of the gearbox; Using a computer to read the color image and depth information of the gearbox, specifically involving the OpenCV image processing library in a Python program; According to the camera parameters and the coordinate information of each pixel point obtained from the three-dimensional image of the gearbox, the three-dimensional coordinates of each pixel are calculated to generate point cloud data; Call the Python program and use the Open3D library to save the point cloud data of the gearbox as a PCD file; The visualization function in the Open3D library is called to process the point cloud data and generate a three-dimensional point cloud image of the gearbox.

4. A vision-based gearbox sorting method as claimed in claim 3, characterized in that: The process of calculating the three-dimensional coordinates of each pixel according to the camera parameters and the coordinate information of each pixel point obtained from the three-dimensional image of the gearbox is as follows: For each pixel point (u, v) in the image, its depth value is D(u, v), and its three-dimensional coordinates (X, Y, Z) in the world coordinate system are calculated using the following formula: Z=D(u,v) Among them, c x and c y are the coordinates of the center of the image, f x and f y are the focal lengths along the x-axis and y-axis respectively.

5. A vision-based gearbox sorting method as claimed in claim 1, characterized in that: The method for preprocessing the three-dimensional point cloud image of the gearbox includes: using a statistical filtering method to remove noise points in the point cloud image, calculating the average distance and standard deviation in the neighborhood of each point cloud, and setting a neighborhood threshold to remove noise points; using a normal vector filtering method to smooth the point cloud image, adjusting the position of the point cloud to the intersection of the normal vector and the fitting plane to achieve smoothing of the point cloud; using a voxel downsampling method to divide the point cloud image into a three-dimensional grid of a fixed size, and using the centroid of each grid point to represent all points in the network.

6. A vision-based gearbox sorting method as claimed in claim 1, characterized in that: PointNet is used as a point cloud image segmentation model based on deep learning to realize the segmentation of the gearbox three-dimensional point cloud image; PointNet directly processes each point in the point cloud image without converting the point cloud data into a regular grid or voxel, and aggregates global features through symmetric functions; the model includes an input layer, a feature extraction layer, a global feature extraction layer and a segmentation layer, fusing global and local information.

7. A vision-based gearbox sorting method as claimed in claim 1, characterized in that: The process of obtaining the target gearbox location information from the segmented image is as follows: the deep learning-based point cloud image segmentation model assigns a label to each point in the gearbox three-dimensional point cloud image to indicate the category to which the point belongs, and determines the number of gearboxes in the image and the area of ​​the target gearbox based on the three-dimensional point cloud segmentation image of the gearbox, calculates the geometric center of the target gearbox area, and uses the coordinates of the geometric center as the location information of the target gearbox.

8. A vision-based gearbox sorting method as claimed in claim 1, characterized in that: The process of the control unit generating the path planning data for the sorting robot is as follows: the location information and barcode information of the target gearbox are transmitted to the control unit, the acquired information is compared with the information in the data storage system, the transportation information of the target gearbox, i.e., the user order information, the transportation destination coordinates and the transportation quantity are determined, and the transportation path of the sorting robot is planned according to the location of the gearbox and the destination location.

9. A vision-based gearbox sorting system, characterized in that: The system is used to implement a vision-based gearbox sorting method according to any one of claims 1 to 8, and the system comprises: An image acquisition unit, used to acquire a three-dimensional point cloud image and barcode information of the gearbox; An image preprocessing unit, used for preprocessing the collected gearbox three-dimensional point cloud image; An image segmentation unit, which uses a deep learning-based point cloud image segmentation model to segment the gearbox three-dimensional point cloud image; A position information processing unit, used for acquiring position information of a target gearbox based on the segmented three-dimensional point cloud image of the gearbox; A control unit, used to transmit the position information and barcode information of the target gearbox to the control unit to generate path planning data for the sorting robot; The execution unit is used to receive the instructions sent by the control unit, designate the sorting robot to perform the sorting operation, and transport the target gearbox to the designated area.

Images