A three-dimensional vision-based production method and system for disc-shaped shoe manufacturing

Abstract

The present invention discloses a disc shoe manufacturing method and system based on 3D vision, which relates to the technical field of image processing. The method includes: obtaining image data of a first shoe mold based on a corresponding shoe mold camera, where the first shoe mold is a left-foot shoe mold or a right-foot shoe mold; extracting shoe mold feature points based on the image data, and calculating the deviation between the shoe mold feature points and the reference shoe mold feature points to generate the running trajectories of a plurality of robots for shoe manufacturing; during the shoe manufacturing process, denoising the acquired point cloud data based on a deep learning method to obtain the denoised point cloud data; performing point cloud segmentation on the denoised point cloud data to obtain the point cloud data after point cloud segmentation; extracting edges from the point cloud data after point cloud segmentation to obtain edge information; and performing speed adaptive adjustment processing on the robot based on the edge information. The present invention improves production efficiency and the quality of sole products, and reduces production costs at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a disc shoe-making production method and system based on three-dimensional vision. Background Art

[0002] There are several problems in the current disc shoe-making production process. The first problem is that the relative positions of each robotic arm and work station are adjusted to be the same by means of hardware adjustment, but this method has a large workload and high difficulty, and it is difficult to ensure that the relative positions of each disc device are the same, and at the same time, the hardware adjustment is difficult. The second problem is that there are many processes and it is necessary to manually teach multiple robots to spray glue points on the trajectories of all shoe molds, resulting in a large workload and low efficiency. The third problem is that the noise commonly present in the point cloud data cannot be well removed. The fourth problem is that due to the different depths of the molds, the speed of the robot cannot be adaptively adjusted.

[0003] Three-dimensional vision technology uses depth sensors or stereo vision systems. By capturing multiple perspective images or video data of the object surface and combining advanced algorithm processing, it can recover the three-dimensional structure and information of the object or scene in real time.

[0004] Introducing three-dimensional vision technology, which can intelligently identify the position and posture of the sole mold and guide the robotic arm to complete the glue spraying action. Its highly automated, high-efficiency, high-precision and ability to adapt to a poor environment open up a new vision for disc shoe-making automation, thereby improving production efficiency, enhancing the quality of sole products and reducing production costs. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a disc shoe-making production method and system based on three-dimensional vision, which can improve the reliability of trajectory planning and robot speed adjustment in the shoe-making production process, improve production efficiency and the quality of sole products, and at the same time reduce production costs.

[0006] To solve the above technical problems, the present invention provides a disc shoe-making production method based on three-dimensional vision, and the method includes:

[0007] Based on the disc machine, move the first shoe mold to the lower part of the corresponding shoe mold camera, and obtain the image data of the first shoe mold based on the corresponding shoe mold camera, where the first shoe mold is a left-foot shoe mold or a right-foot shoe mold;

[0008] Extract the shoe mold feature points based on the image data, obtain the shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate the running trajectories of several robots based on the deviation, and each robot performs shoe-making production based on the corresponding running trajectory;

[0009] During the shoe manufacturing process where each robot operates based on its corresponding running trajectory, point cloud data is acquired using a 3D depth camera, and the point cloud data is denoised using a deep learning method to obtain denoised point cloud data.

[0010] The denoised point cloud data is subjected to point cloud segmentation to obtain point cloud data after point cloud segmentation.

[0011] Edge extraction is performed on the point cloud data after point cloud segmentation to obtain edge information.

[0012] Based on the edge information, speed adaptive adjustment processing is performed on the robot.

[0013] Optionally, calculating the deviation between the shoe mold feature points and the reference shoe mold feature points, and generating the running trajectories of several robots based on the deviation includes:

[0014] Obtaining reference shoe mold images from an offline workstation, extracting shoe mold feature points from the reference shoe mold images to obtain reference shoe mold feature points, and calculating the deviation between the shoe mold feature points and the reference shoe mold feature points.

[0015] Based on the deviation, a spraying trajectory for the first robot, a scraping edge trajectory for the second robot, and a straight injection trajectory for the third robot are generated using the first reference trajectory.

[0016] Optionally, the denoising the point cloud data using a deep learning method to obtain denoised point cloud data includes:

[0017] Obtaining point cloud sample data using the DataLoader component in the pytorch deep learning framework, and converting the point cloud sample data into the format of (N, D, 3) to obtain the point cloud sample data after format conversion, where N is the number of point cloud sample data input to the network for one training, and D is the number of points in each point cloud sample data.

[0018] Generating a transformation matrix for the point cloud sample data after format conversion using the principal component analysis method, and performing alignment processing on the point cloud sample data after format conversion using matrix multiplication based on the transformation matrix to obtain the point cloud sample data after alignment processing.

[0019] Training a point cloud denoising model based on the point cloud sample data after alignment processing to obtain a trained point cloud denoising model.

[0020] Inputting the point cloud data into the trained point cloud denoising model for denoising processing to obtain denoised point cloud data.

[0021] Optionally, the point cloud denoising model uses a PointNet convolutional neural network as the backbone network. The PointNet convolutional neural network includes several layers of multi-layer perceptrons and a decoder. Each layer of the multi-layer perceptron includes five hidden layers, and each hidden layer includes a Batch Norm1d normalization layer, a ReLU activation layer, and a max pooling layer. The decoder includes several layers of fully connected layers, a normalization layer, and an activation layer, and there is a Dropout layer between the fully connected layers.

[0022] Optionally, the expression of the loss function of the point cloud denoising model is:

[0023]

[0024] where L is the loss function, p i ′ is the nearest point in the clean point cloud, p i is the sample point cloud, and I is the number of point clouds.

[0025] Optionally, the point cloud data after denoising processing is subjected to point cloud segmentation processing to obtain the point cloud data after point cloud segmentation processing, including:

[0026] Generating a local tangent plane based on the least squares fitting method using the point cloud data after denoising processing. The expression of the local tangent plane is:

[0027]

[0028] where is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the coordinate origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point;

[0029] Calculating the centroid of all neighborhoods in the local tangent plane and calculating the covariance matrix based on the centroid of the neighborhood. The expression of the covariance matrix is:

[0030]

[0031] where M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, and T is the transpose;

[0032] Obtaining the normal vector of the neighborhood points based on the covariance matrix and calculating the normal angle using the normal vector of the seed points based on the normal vector of the neighborhood points;

[0033] Performing point cloud segmentation processing on the point cloud data after denoising processing based on the normal angle to obtain the point cloud data after point cloud segmentation processing.

[0034] Optionally, the expression for the normal angle is:

[0035]

[0036] where ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood.

[0037] Optionally, edge extraction is performed on the point cloud data after point cloud segmentation processing to obtain edge information, including:

[0038] Select the current point in the point cloud data after point cloud segmentation processing, project the tangent plane based on the neighborhood of the current point, and obtain the projected neighboring points;

[0039] Take the current point as a corner point, connect the projected neighboring points with the corner point based on a preset order to obtain a number of included angles;

[0040] Select the largest included angle among a number of included angles, compare the largest included angle with a preset threshold to obtain a comparison result, and judge whether the current point is an edge point based on the comparison result.

[0041] Optionally, the speed adaptive adjustment processing of the robot based on the edge information includes:

[0042] Obtain the depth information of the current trajectory point of the robot based on the edge information, and calculate the adjustment speed of the robot based on the depth information. Among them, the expression for the adjustment speed is:

[0043] v′ = v(1 + k*(z - z0))

[0044] where v′ is the adjustment speed, v is the original speed, k is the proportionality coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height;

[0045] Perform adaptive speed adjustment on the robot based on the adjustment speed.

[0046] In addition, the present invention also provides a three-dimensional vision-based disc shoe production system, and the system includes:

[0047] Image data acquisition module: used to move the first shoe mold to the lower part of the corresponding shoe mold camera based on the disc machine, and acquire the image data of the first shoe mold based on the corresponding shoe mold camera. The first shoe mold is a left shoe mold or a right shoe mold;

[0048] Running trajectory generation module: used to extract the feature points of the shoe mold based on the image data, obtain the feature points of the shoe mold, calculate the deviation between the feature points of the shoe mold and the reference shoe mold feature points, generate the running trajectories of several robots based on the deviation, and each robot performs shoe manufacturing production based on the corresponding running trajectory;

[0049] Point cloud denoising module: used to obtain point cloud data based on a 3D depth camera during the process of each robot performing shoe manufacturing production based on the corresponding running trajectory, and perform denoising processing on the point cloud data based on a deep learning method to obtain the denoised point cloud data;

[0050] Point cloud segmentation module: used to perform point cloud segmentation processing on the denoised point cloud data to obtain the point cloud data after point cloud segmentation processing;

[0051] Edge extraction module: used to extract the edges of the point cloud data after point cloud segmentation processing to obtain edge information;

[0052] Speed adaptive adjustment module: used to perform speed adaptive adjustment processing on the robot based on the edge information.

[0053] In the embodiment of the present invention, calculating the deviation between the left and right shoe mold feature points and the reference shoe mold feature points, and generating the corresponding running trajectory of the robot according to the deviation solves the problem of the movement trajectory planning of the robotic arm during the disk shoe manufacturing process, has better robustness and trajectory planning accuracy, does not require adjusting the relative positions of each robotic arm and workstation, and improves the efficiency of trajectory planning. Denoising the point cloud data based on a deep learning method improves the accuracy of subsequent edge contour detection. Adaptive adjustment of the movement speed of the robotic arm according to the depth information obtained from the edge information of the point cloud data during the movement of the robotic arm can improve production efficiency, improve the quality of the sole product, and reduce production costs at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0055] Figure 1 is a schematic flowchart of the disk shoe manufacturing production method based on 3D vision in the first embodiment of the present invention;

[0056] Figure 2 is a schematic flowchart of the disk shoe manufacturing production method based on 3D vision in the second embodiment of the present invention;

[0057] Figure 3 It is a schematic flow chart of the disc shoe manufacturing production method based on 3D vision in the third embodiment of the present invention

[0058] Figure 4 It is a schematic diagram of the structural composition of the disc shoe manufacturing production system based on 3D vision in the embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of the equipment of the disc shoe manufacturing production method based on 3D vision in the embodiment of the present invention. Detailed implementation manners

[0060] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0061] Embodiment 1

[0062] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of the disc shoe manufacturing production method based on 3D vision in the first embodiment of the present invention. The method includes:

[0063] S01: Move the first shoe mold to the lower part of the corresponding shoe mold camera based on the turntable machine, and obtain the image data of the first shoe mold based on the corresponding shoe mold camera. The first shoe mold is a left-foot shoe mold or a right-foot shoe mold;

[0064] S02: Extract the shoe mold feature points based on the image data, obtain the shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate the running trajectories of several robots based on the deviation, and each robot performs shoe manufacturing production based on the corresponding running trajectory;

[0065] S03: During the process of each robot performing shoe manufacturing production based on the corresponding running trajectory, obtain the point cloud data based on the 3D depth camera, and perform denoising processing on the point cloud data based on the deep learning method to obtain the denoised point cloud data;

[0066] S04: Perform point cloud segmentation processing on the denoised point cloud data to obtain the point cloud data after point cloud segmentation processing;

[0067] S05: Extract the edges of the point cloud data after point cloud segmentation processing to obtain edge information;

[0068] S06: Perform speed adaptive adjustment processing on the robot based on the edge information.

[0069] In the embodiment of the present invention, the deviation between the left and right shoe mold feature points and the reference shoe mold feature points is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of the movement trajectory planning of the robotic arm in the disc shoe-making process, has better robustness and trajectory planning accuracy, and does not require adjusting the relative positions of each robotic arm and work station, thus improving the efficiency of trajectory planning. The point cloud data is denoised based on the deep learning method, improving the accuracy of subsequent edge contour detection. During the movement of the robotic arm, the movement speed of the robotic hand is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, enhance the quality of the sole products, and reduce production costs at the same time.

[0070] Embodiment Two

[0071] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of the disc shoe-making production method based on 3D vision in the second embodiment of the present invention. The method includes

[0072] S100: Move the left shoe mold under the first camera and the right shoe mold under the second camera based on the disc machine;

[0073] In the specific implementation process of the present invention, the disc rotating machine supplies materials, moves the left shoe mold under the first camera, and moves the right shoe mold under the second camera.

[0074] S110: The first camera acquires the first image of the left shoe mold;

[0075] S120: The second camera acquires the second image of the right shoe mold;

[0076] In the specific implementation process of the present invention, the first camera takes a photo of the left shoe mold for detection to obtain the corresponding first image, and the second camera takes a photo of the right shoe mold for detection to obtain the corresponding second image.

[0077] S111: Extract the shoe mold feature points based on the first image to obtain the first shoe mold feature points, calculate the first deviation between the first shoe mold feature points and the reference shoe mold feature points, generate the first running trajectories of several robots based on the first deviation, and each robot performs shoe-making production based on the corresponding first running trajectory;

[0078] In the specific implementation process of the present invention, calculating the first deviation between the first shoe mold feature points and the reference shoe mold feature points, and generating the first running trajectories of a plurality of robots based on the first deviation includes: obtaining a reference shoe mold image based on an offline station, extracting shoe mold feature points from the reference shoe mold image to obtain reference shoe mold feature points, and calculating the first deviation between the first shoe mold feature points and the reference shoe mold feature points; generating the spraying trajectory of the first robot, the edge scraping trajectory of the second robot, and the linear feeding trajectory of the third robot based on the first deviation using a first reference trajectory.

[0079] Specifically, the reference shoe mold feature points have been obtained in advance at the offline station. The offline station uses the same shoe mold fixture to fix the shoe mold, obtains the reference left and right shoe mold images through the camera at the offline station, extracts the shoe mold feature points from the reference left shoe mold image to obtain the reference left shoe mold feature points, and the reference left shoe mold feature points are the boundary contour points of the reference left shoe mold. Extracting the shoe mold feature points based on the first image to obtain the first shoe mold feature points. The first shoe mold feature points are the boundary contour points of the left shoe mold. Obtaining the local plane of the first image, randomly selecting a pixel point as the current pixel point in the local plane, constructing a target plane with three adjacent points around the current pixel point, calculating the average distance from the current pixel point to the target plane, projecting the neighboring points of the current pixel point onto the reference plane based on k-neighborhood search, calculating the normal vector of the projected neighboring points, calculating the vector angle according to the average distance, the normal vector of the neighboring points, and the normal vector of the current pixel point, and extracting the boundary contour points according to the vector angle. Calculating the first deviation between the first shoe mold feature points and the reference shoe mold feature points, that is, calculating the deviation distance between each shoe mold feature point and the corresponding reference shoe mold feature point, generating the trajectory point deviation of the robot according to the deviation distance, adjusting the first reference trajectory according to the trajectory point deviation, and obtaining the spraying trajectory of the first robot, the edge scraping trajectory of the second robot, and the linear feeding trajectory of the third robot. Each robot performs shoe manufacturing production based on the corresponding first running trajectory. The specific equipment layout is as Figure 5 shown. The first robot is robot A in the figure, the second robot is robot B in the figure, the third robot is robot D in the figure, and the first camera is camera 1 in the figure.

[0080] S121: Extracting the second shoe mold feature points based on the second image, calculating the second deviation between the second shoe mold feature points and the reference shoe mold feature points, generating the second running trajectories of a plurality of robots based on the second deviation, and each robot performs shoe manufacturing production based on the corresponding second running trajectory;

[0081] In the specific implementation process of the present invention, generating the second running trajectories of a plurality of robots based on the second deviation includes: generating the spraying trajectory of the fourth robot and the linear feeding trajectory of the fifth robot based on the second deviation using a second reference trajectory.

[0082] Specifically, based on the second image, the feature points of the shoe mold are extracted to obtain the second shoe mold feature points, and the second deviation between the second shoe mold feature points and the reference shoe mold feature points is calculated, that is, the deviation distance between each shoe mold feature point and the corresponding reference shoe mold feature point is calculated. According to the deviation distance, the trajectory point deviation of the robot is generated, and the second reference trajectory is adjusted according to the trajectory point deviation to obtain the spraying trajectory of the fourth robot and the linear feeding trajectory of the fifth robot. Each robot performs shoe manufacturing production based on the corresponding second operating trajectory. The specific equipment layout is as Figure 5 shown. The fourth robot is robot C in the figure, the fifth robot is robot D in the figure, and the second camera is camera 2 in the figure.

[0083] S112: During the process of each robot performing shoe manufacturing production based on the corresponding first operating trajectory, the first point cloud data is obtained based on the first three-dimensional depth camera, and the first point cloud data is denoised based on the deep learning method to obtain the denoised first point cloud data. Then, the denoised first point cloud data is subjected to point cloud segmentation processing to obtain the first point cloud data after point cloud segmentation processing;

[0084] In the specific implementation process of the present invention, the denoising of the first point cloud data based on the deep learning method to obtain the denoised first point cloud data includes: obtaining point cloud sample data based on the DataLoader component in the pytorch deep learning framework, and converting the point cloud sample data into the format of (N, D, 3) to obtain the point cloud sample data after format conversion, where N is the number of point cloud sample data input into the network for one training, and D is the number of points in each point cloud sample data; generating a transformation matrix for the point cloud sample data after format conversion based on the principal component analysis method, and performing alignment processing on the point cloud sample data after format conversion using matrix multiplication based on the transformation matrix to obtain the aligned point cloud sample data; training the point cloud denoising model based on the aligned point cloud sample data to obtain a trained point cloud denoising model; inputting the first point cloud data into the trained point cloud denoising model for denoising processing to obtain the denoised first point cloud data.

[0085] Further, the point cloud denoising model uses a pointnet convolutional neural network as the backbone network. The pointnet convolutional neural network includes several layers of multi-layer perceptrons and a decoder. Each layer of the multi-layer perceptron includes five hidden layers, and each hidden layer includes a Batch Norm1d normalization layer, a ReLU activation layer, and a max pooling layer. The decoder includes several layers of fully connected layers, a normalization layer, and an activation layer, and there is a Dropout layer between the fully connected layers.

[0086] Further, the expression of the loss function of the point cloud denoising model is:

[0087]

[0088] where L is the loss function, p i ′ is the clean point cloud, p i is the sample point cloud, and I is the number of point clouds.

[0089] Further, performing point cloud segmentation processing on the first point cloud data after denoising processing to obtain the first point cloud data after point cloud segmentation processing includes:

[0090] Generating a local tangent plane based on the least squares fitting method using the first point cloud data after denoising processing. The expression of the local tangent plane is:

[0091]

[0092] where is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the coordinate origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point;

[0093] Calculating the centroid of all neighborhoods in the local tangent plane and calculating the covariance matrix based on the centroid of the neighborhood. The expression of the covariance matrix is:

[0094]

[0095] where M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, and T is the transpose;

[0096] Obtaining the normal vector of the neighborhood point based on the covariance matrix and calculating the normal angle using the normal vector of the seed point based on the normal vector of the neighborhood point. The expression of the normal angle is:

[0097]

[0098] where ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood;

[0099] Performing point cloud segmentation processing on the first point cloud data after denoising processing based on the normal angle to obtain the first point cloud data after point cloud segmentation processing.

[0100] Specifically, during the shoe manufacturing process where each robot operates based on the corresponding first operating trajectory, the first point cloud data is real-time acquired based on the first 3D depth camera on the production line. The point cloud sample data is obtained based on the DataLoader component in the pytorch deep learning framework, and the point cloud sample data is converted into the format of (N, D, 3) to obtain the point cloud sample data after format conversion. Here, N is the number of point cloud sample data input into the network for one training, and D is the number of points in each point cloud sample data. Before inputting into the neural network, the main axis needs to be aligned with the Cartesian coordinate system for each input data. A transformation matrix for the point cloud sample data after format conversion is generated based on the principal component analysis method, and the point cloud sample data after format conversion is aligned based on the transformation matrix using matrix multiplication. Through matrix multiplication of the transformation matrix and the input data, the point cloud sample data after alignment processing is obtained. The point cloud denoising model is trained based on the point cloud sample data after alignment processing to obtain a trained point cloud denoising model. The point cloud denoising model uses the pointnet convolutional neural network as the backbone network. The pointnet convolutional neural network includes several layers of multi-layer perceptrons and a decoder. The local features of the point cloud data are extracted through the multi-layer perceptrons. Each layer of the multi-layer perceptron includes five hidden layers. The number of neurons in the first hidden layer is 64, the number of neurons in the second hidden layer is 128, the number of neurons in the third hidden layer is 256, the number of neurons in the fourth hidden layer is 512, and the number of neurons in the fifth hidden layer is 1024. Each hidden layer includes a Batch Norm1d normalization layer, a ReLU activation layer, and a max pooling layer. The features of each data point are aggregated through the max pooling layer, so as to obtain a global feature vector representing the entire point cloud data. The decoder includes several layers of fully connected layers, normalization layers, and activation layers. There are three fully connected layers, and the number of neurons in them are 1024, 512, 256, and 3 respectively. There is a Dropout layer between the fully connected layers, and the probability of the Dropout layer is 0.3. The expression of the loss function of the point cloud denoising model is:

[0101]

[0102] where L is the loss function, p i ′ is the nearest point in the clean point cloud, p i is the sample point cloud, and I is the number of point clouds. This loss function is the distance from the sample point to the nearest point on the clean point cloud. The first point cloud data is input into the trained point cloud denoising model for denoising processing to obtain the first point cloud data after denoising processing. The denoising is performed using a deep learning-based method, moving the noise points as close as possible to the position of the clean point cloud, thereby learning the distance function of the space. The trained neural network can then regularize the input noisy point cloud into a clean point cloud.

[0103] Call the region growing algorithm in the point cloud library for point cloud segmentation. The region growing algorithm determines whether point cloud data belongs to the same cluster based on information such as normal vectors, curvature, and color. The goal of the algorithm is to merge points that are close enough according to the smoothness consistency constraint conditions. Therefore, the output of the algorithm is a set of clusters, and the points on each cluster are located on the same smooth surface. Generate a local tangent plane based on the first point cloud data after denoising using the least squares fitting method. The expression of the local tangent plane is:

[0104]

[0105] where, is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the coordinate origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point;

[0106] Calculate the centroid of all neighborhoods in the local tangent plane, and calculate the covariance matrix based on the centroid of the neighborhood. The expression of the covariance matrix is:

[0107]

[0108] where, M is the covariance matrix, k is the number of neighborhood points, is the centroid of the neighborhood, is the neighborhood point, T is the transpose;

[0109] Obtain the normal vector of the neighborhood point based on the covariance matrix, perform eigenvalue decomposition according to the covariance matrix, and the eigenvector corresponding to the smallest eigenvalue of the covariance matrix is the normal vector of the point. Calculate the normal angle based on the normal vector of the neighborhood point and the normal vector of the seed point. The expression of the normal angle is:

[0110]

[0111] where, ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood; Sort the input first point cloud data according to the curvature value of the point cloud, and select the point with the smallest average angle between the normals as the seed point. Since the area where the seed point is located is the smoothest area, starting from the smoothest area for growth can effectively reduce the total number of segmentation segments and improve the segmentation efficiency.

[0112] Perform point cloud segmentation on the denoised first point cloud data based on the normal angle. Compare the normal angle with a preset smoothing threshold. If it is less than the preset smoothing threshold, the neighborhood points corresponding to the normal angle are regarded as part of the same smooth surface until all the neighborhood points of the smooth surfaces are clustered. Then, point cloud region segmentation can be performed to obtain the first point cloud data after point cloud segmentation processing.

[0113] S113: Extract edges from the first point cloud data after point cloud segmentation processing to obtain first edge information, and perform speed adaptive adjustment processing on the robot based on the first edge information;

[0114] In the specific implementation process of the present invention, extracting edges from the first point cloud data after point cloud segmentation processing to obtain first edge information includes: selecting a current point in the first point cloud data after point cloud segmentation processing, performing projection of the tangent plane based on the neighborhood of the current point to obtain the projected neighboring points; taking the current point as a corner point, connecting the projected neighboring points with the corner point in a preset order to obtain several included angles; selecting the largest included angle among the several included angles, comparing the largest included angle with a preset threshold to obtain a comparison result, and judging whether the current point is an edge point based on the comparison result.

[0115] Further, performing speed adaptive adjustment processing on the robot based on the first edge information includes: obtaining the depth information of the current trajectory point of the robot based on the first edge information, and calculating the adjustment speed of the robot based on the depth information, where the expression of the adjustment speed is:

[0116] v′ = v(1 + k*(z - z0))

[0117] where v′ is the adjustment speed, v is the original speed, k is the proportionality coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height;

[0118] Perform adaptive speed adjustment on the robot based on the adjustment speed.

[0119] Specifically, select a current point in the first point cloud data after point cloud segmentation processing. The current point is randomly selected. Perform projection of the tangent plane based on the neighborhood of the current point to obtain the projected neighboring points. It should be noted that after projection, the position of the current point does not change; take the current point as a corner point, connect the projected neighboring points with the corner point in a preset order to obtain several included angles θ = {θ1, θ2, …, θ n}, where n = ||N p || - 1, N pis the number of projection points, and the preset order can be counterclockwise or clockwise; select the largest included angle among several included angles. The larger the included angle, the more likely the current point is an edge point. Compare the largest included angle with a preset threshold to obtain a comparison result, and determine whether the current point is an edge point based on the comparison result. That is, when the largest included angle is greater than the preset threshold, it is determined that the current point is an edge point. All edge points are extracted according to the above steps. After obtaining all edge points, connect the edge points to obtain the first edge information.

[0120] All points on the running trajectories of the robots include not only the x-axis and y-axis information but also the z-axis information. The speed can be adjusted through the z-axis information. Based on the first edge information, obtain the depth information of the current trajectory point of the robot, and calculate the adjusted speed of the robot based on the depth information. The expression of the adjusted speed is:

[0121] v′ = v(1 + k*(z - z0))

[0122] where v′ is the adjusted speed, v is the original speed, k is the proportionality coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height; the proportionality coefficient is used to control the sensitivity of the speed change with the z-axis. Set k > 0, and as the z-axis information increases, the speed will increase; z0 is set to the average value of z. Perform adaptive speed adjustment on the robot based on the adjusted speed. During the movement of the robotic arm, adaptively adjust the movement speed of the robotic arm according to the depth information, which can improve production efficiency.

[0123] S122: During the shoe-making production process of each robot based on the corresponding second running trajectory, obtain the second point cloud data based on the second three-dimensional depth camera, perform denoising processing on the second point cloud data based on the deep learning method to obtain the denoised second point cloud data, and perform point cloud segmentation processing on the denoised second point cloud data to obtain the second point cloud data after point cloud segmentation processing;

[0124] In the specific implementation process of the present invention, during the shoe-making production process of each robot based on the corresponding second running trajectory, obtain the second point cloud data based on the second three-dimensional depth camera. The method steps for denoising processing and point cloud segmentation processing of the second point cloud data are the same as those for denoising processing and point cloud segmentation processing of the first point cloud data, and will not be elaborated here.

[0125] S123: Perform edge extraction on the second point cloud data after point cloud segmentation processing to obtain the second edge information, and perform speed adaptive adjustment processing on the robot based on the second edge information.

[0126] In the specific implementation process of the present invention, edge extraction is performed on the second point cloud data after point cloud segmentation to obtain second edge information, and based on the second edge information, speed adaptive adjustment processing is performed on the robot, so as to adaptively adjust the moving speeds of the fourth robot and the fifth robot corresponding to the right shoe mold.

[0127] In the embodiment of the present invention, the deviation between the feature points of the left and right shoe molds and the feature points of the reference shoe mold is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of the movement trajectory planning of the robotic arm in the disk shoe-making process, has better robustness and trajectory planning accuracy, and does not require adjusting the relative positions of each robotic arm and workstation, improving the efficiency of trajectory planning. The point cloud data is denoised based on a deep learning method, improving the accuracy of subsequent edge contour detection. During the movement of the robotic arm, the moving speed of the robotic hand is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, enhance the quality of the sole products, and reduce production costs at the same time.

[0128] Embodiment III

[0129] Please refer to Figure 3 , Figure 3 which is a schematic flow chart of the disk shoe-making production method based on three-dimensional vision in the third embodiment of the present invention. The method includes:

[0130] S200: Move the left shoe mold to the lower part of the first camera and the right shoe mold to the lower part of the second camera based on the disk machine;

[0131] S210: The first camera acquires the first image of the left shoe mold;

[0132] S220: The second camera acquires the second image of the right shoe mold;

[0133] S211: Extract the feature points of the shoe mold based on the first image to obtain the first shoe mold feature points, and calculate the first deviation between the first shoe mold feature points and the reference shoe mold feature points;

[0134] S212: Generate the spraying trajectory of the first robot, the scraping trajectory of the second robot, and the linear injection trajectory of the third robot based on the first deviation using the first reference trajectory, and each robot performs shoe-making production based on the corresponding running trajectory;

[0135] S213: During the process of each robot performing shoe-making production based on the corresponding first running trajectory, acquire the first point cloud data based on the first three-dimensional depth camera;

[0136] S214: Obtain point cloud sample data using the DataLoader component in the PyTorch deep learning framework, and convert the format of the point cloud sample data to (N, D, 3) to obtain the point cloud sample data after format conversion;

[0137] S215: Generate a transformation matrix for the point cloud sample data after format conversion based on the principal component analysis method, and perform alignment processing on the point cloud sample data after format conversion using matrix multiplication based on the transformation matrix to obtain the point cloud sample data after alignment processing;

[0138] S216: Train a point cloud denoising model based on the point cloud sample data after alignment processing to obtain a trained point cloud denoising model;

[0139] S217: Input the first point cloud data into the trained point cloud denoising model for denoising processing to obtain the first point cloud data after denoising processing;

[0140] S218: Perform point cloud segmentation processing on the first point cloud data after denoising processing to obtain the first point cloud data after point cloud segmentation processing;

[0141] S219: Extract the edges of the first point cloud data after point cloud segmentation processing to obtain the first edge information;

[0142] S2110: Perform speed adaptive adjustment processing on the robot based on the first edge information;

[0143] S221: Extract the feature points of the shoe mold based on the second image to obtain the second shoe mold feature points, and calculate the second deviation between the second shoe mold feature points and the reference shoe mold feature points;

[0144] S222: Generate the spraying trajectory of the fourth robot and the linear feeding trajectory of the fifth robot based on the second deviation using the second reference trajectory, and each robot performs shoe manufacturing production based on the corresponding operation trajectory;

[0145] S223: During the shoe manufacturing production process of each robot based on the corresponding second operation trajectory, obtain the second point cloud data using the second 3D depth camera;

[0146] S224: Obtain point cloud sample data using the DataLoader component in the PyTorch deep learning framework, and convert the format of the point cloud sample data to (N, D, 3) to obtain the point cloud sample data after format conversion;

[0147] S225: Generate a transformation matrix for the point cloud sample data after format conversion based on the principal component analysis method, and perform alignment processing on the point cloud sample data after format conversion using matrix multiplication based on the transformation matrix to obtain the point cloud sample data after alignment processing;

[0148] S226: Train a point cloud denoising model based on the aligned point cloud sample data to obtain a trained point cloud denoising model;

[0149] S227: Input the second point cloud data into the trained point cloud denoising model for denoising processing to obtain the second point cloud data after denoising processing;

[0150] S228: Perform point cloud segmentation processing on the second point cloud data after denoising processing to obtain the second point cloud data after point cloud segmentation processing;

[0151] S229: Extract edges from the second point cloud data after point cloud segmentation processing to obtain second edge information;

[0152] S2210: Perform speed adaptive adjustment processing on the robot based on the second edge information.

[0153] In the embodiment of the present invention, the deviation between the left and right shoe mold feature points and the reference shoe mold feature points is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of the movement trajectory planning of the robotic arm in the disk shoe-making process, has better robustness and trajectory planning accuracy, does not require adjusting the relative positions of each robotic arm and work station, and improves the efficiency of trajectory planning. The point cloud data is denoised based on a deep learning method, which improves the accuracy of subsequent edge contour detection. During the movement of the robotic arm, the movement speed of the robotic arm is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, improve the quality of the sole products, and reduce production costs at the same time.

[0154] Embodiment Four

[0155] Please refer to Figure 4 , Figure 4 which is a schematic structural composition diagram of the disk shoe-making production system based on 3D vision in the embodiment of the present invention. The system includes:

[0156] Image data acquisition module 41: Used to move the first shoe mold to the lower part of the corresponding shoe mold camera based on the disk machine, and acquire the image data of the first shoe mold based on the corresponding shoe mold camera. The first shoe mold is a left-foot shoe mold or a right-foot shoe mold;

[0157] Running trajectory generation module 42: Used to extract shoe mold feature points based on the image data, obtain shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate the running trajectories of several robots based on the deviation, and each robot performs shoe-making production based on the corresponding running trajectory;

[0158] Point cloud denoising module 43: Used for obtaining point cloud data based on a three-dimensional depth camera during the shoe-making production process where each robot operates based on its corresponding operation trajectory, and performing denoising processing on the point cloud data based on a deep learning method to obtain the denoised point cloud data;

[0159] Point cloud segmentation module 44: Used for performing point cloud segmentation processing on the denoised point cloud data to obtain the point cloud data after point cloud segmentation processing;

[0160] Edge extraction module 45: Used for extracting edges from the point cloud data after point cloud segmentation processing to obtain edge information;

[0161] Speed adaptive adjustment module 46: Used for performing speed adaptive adjustment processing on the robot based on the edge information.

[0162] In the specific implementation process of the present invention, the specific implementation manners of the system items can refer to the implementation manners of the above method items, and will not be elaborated here.

[0163] In the embodiments of the present invention, the deviation between the feature points of the left and right shoe molds and the reference shoe mold feature points is calculated, and the corresponding operation trajectory of the robot is generated according to this deviation, which solves the problem of the movement trajectory planning of the robotic arm during the disc shoe-making process, has better robustness and trajectory planning accuracy, does not require adjusting the relative positions of each robotic arm and work station, and improves the efficiency of trajectory planning. The denoising processing of the point cloud data based on the deep learning method improves the accuracy of subsequent edge contour detection. During the movement of the robotic arm, the movement speed of the robotic hand is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, enhance the quality of the sole products, and reduce production costs at the same time.

[0164] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium can include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0165] In addition, the above has introduced in detail a disc shoe-making production method and system based on three-dimensional vision provided by the embodiments of the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A disc-shaped shoe production method based on 3D vision, characterized in that The method includes: Moving a first shoe mold to below the corresponding shoe mold camera based on a disk machine, and acquiring image data of the first shoe mold based on the corresponding shoe mold camera, where the first shoe mold is a left-foot shoe mold or a right-foot shoe mold; Extracting shoe mold feature points based on the image data, obtaining shoe mold feature points, calculating the deviation between the shoe mold feature points and the reference shoe mold feature points, generating the running trajectories of several robots based on the deviation, and each robot performs shoe manufacturing production based on the corresponding running trajectory; During the process of each robot performing shoe manufacturing production based on the corresponding running trajectory, acquiring point cloud data based on a three-dimensional depth camera, and performing denoising processing on the point cloud data based on a deep learning method to obtain denoised point cloud data; Performing point cloud segmentation processing on the denoised point cloud data to obtain point cloud data after point cloud segmentation processing; Performing edge extraction on the point cloud data after point cloud segmentation processing to obtain edge information; Performing speed adaptive adjustment processing on the robot based on the edge information; Among them, the performing speed adaptive adjustment processing on the robot based on the edge information includes: Acquiring the depth information of the current trajectory point of the robot based on the edge information, and calculating the adjustment speed of the robot based on the depth information, where the expression of the adjustment speed is: v ′ = v(1 + k*(z - z0)) Among them, v ′ is the adjusted speed, v is the original speed, k is the proportionality coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height; Performing adaptive speed adjustment on the robot based on the adjustment speed.

2. The method for producing shoes on a disc based on three-dimensional vision according to claim 1, wherein The calculating the deviation between the shoe mold feature points and the reference shoe mold feature points, and generating the running trajectories of several robots based on the deviation includes: Acquiring a reference shoe mold image based on an offline station, extracting shoe mold feature points from the reference shoe mold image, obtaining reference shoe mold feature points, and calculating the deviation between the shoe mold feature points and the reference shoe mold feature points; Generating a spraying trajectory of a first robot, a scraping trajectory of a second robot, and a linear injection trajectory of a third robot based on the deviation using a first reference trajectory.

3. The three-dimensional vision-based disc shoe production method according to claim 1, characterized in that The performing denoising processing on the point cloud data based on a deep learning method to obtain denoised point cloud data includes: Acquiring point cloud sample data based on the DataLoader component in the pytorch deep learning framework, and performing format conversion on the point cloud sample data into the format of (N, D, 3) to obtain format-converted point cloud sample data, where N is the number of point cloud sample data input into the network for one training, and D is the number of points in each point cloud sample data; Generating a transformation matrix of the format-converted point cloud sample data based on the principal component analysis method, and performing alignment processing on the format-converted point cloud sample data using matrix multiplication based on the transformation matrix to obtain aligned point cloud sample data; Training a point cloud denoising model based on the aligned point cloud sample data to obtain a trained point cloud denoising model; Inputting the point cloud data into the trained point cloud denoising model for denoising processing to obtain denoised point cloud data.

4. The three-dimensional vision-based disc shoe production method according to claim 3, characterized in that, The point cloud denoising model uses a PointNet convolutional neural network as the backbone network. The PointNet convolutional neural network includes several layers of multi-layer perceptrons and a decoder. Each layer of the multi-layer perceptron includes five hidden layers, and each hidden layer includes a Batch Norm1d normalization layer, a ReLU activation layer, and a max pooling layer. The decoder includes several layers of fully connected layers, a normalization layer, and an activation layer, and there is a Dropout layer between the fully connected layers.

5. The method for producing shoes on a disc based on three-dimensional vision according to claim 4, characterized in that, The expression of the loss function of the point cloud denoising model is: Among them, L is the loss function, p i ′ is the nearest point in the clean point cloud, p i is the sample point cloud, and I is the number of point clouds.

6. The method for producing shoes on a disc based on three-dimensional vision according to claim 1, characterized in that, Performing point cloud segmentation processing on the denoised point cloud data to obtain the point cloud data after point cloud segmentation processing, including: Generating a local tangent plane using the denoised point cloud data based on the least squares fitting method. The expression of the local tangent plane is: Among them, is a local tangent plane, is the normal vector of the local tangent plane, d is the distance from the coordinate origin to the local tangent plane, and k is the number of neighborhood points, is a neighborhood point; Calculating the centroid of all neighborhoods in the local tangent plane, and calculating the covariance matrix based on the centroid of the neighborhoods. The expression of the covariance matrix is: where M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, and T is the transpose; Obtaining the normal vector of the neighborhood points based on the covariance matrix, and calculating the normal angle using the normal vector of the neighborhood points and the normal vector of the seed points; Performing point cloud segmentation processing on the denoised point cloud data based on the normal angle to obtain the point cloud data after point cloud segmentation processing.

7. The three-dimensional vision-based disc shoe production method according to claim 6, characterized in that The expression of the normal angle is: Among them, ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood.

8. The three-dimensional vision-based disc shoe production method according to claim 1, wherein Performing edge extraction on the point cloud data after point cloud segmentation processing to obtain edge information, including: Selecting the current point in the point cloud data after point cloud segmentation processing, and performing projection of the tangent plane based on the neighborhood of the current point to obtain the projected neighboring points; Taking the current point as a corner point, and connecting the projected neighboring points with the corner point in a preset order to obtain several included angles; Selecting the maximum included angle among several included angles, comparing the maximum included angle with a preset threshold to obtain a comparison result, and judging whether the current point is an edge point based on the comparison result.

9. A disc-shaped shoe production system based on 3D vision, characterized in that, The system includes: An image data acquisition module: used to move the first shoe mold to the lower part of the corresponding shoe mold camera based on a turntable machine, and acquire the image data of the first shoe mold based on the corresponding shoe mold camera. The first shoe mold is a left shoe mold or a right shoe mold; A running trajectory generation module: used to extract shoe mold feature points based on the image data, obtain the shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate the running trajectories of several robots based on the deviation, and each robot performs shoe manufacturing production based on the corresponding running trajectory; A point cloud denoising module: used to acquire point cloud data based on a three-dimensional depth camera during the process of each robot performing shoe manufacturing production based on the corresponding running trajectory, and perform denoising processing on the point cloud data based on a deep learning method to obtain the denoised point cloud data; A point cloud segmentation module: used to perform point cloud segmentation processing on the denoised point cloud data to obtain the point cloud data after point cloud segmentation processing; An edge extraction module: used to perform edge extraction on the point cloud data after point cloud segmentation processing to obtain edge information; A speed adaptive adjustment module: used to perform speed adaptive adjustment processing on the robot based on the edge information; Among them, the speed adaptive adjustment processing of the robot based on the edge information includes: Obtain the depth information of the current trajectory point of the robot based on the edge information, and calculate the adjustment speed of the robot based on the depth information, where the expression of the adjustment speed is: v ′ = v(1 + k*(z - z0)) Among them, v ′ is the adjusted speed, v is the original speed, k is the proportionality coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height; Perform adaptive speed adjustment on the robot based on the adjustment speed.

Images