Disc shoemaking production method and system based on three-dimensional vision

Abstract

The invention discloses a disc shoemaking production method and system based on three-dimensional vision, and relates to the technical field of image processing.The method comprises the steps that image data of a first shoe mold is obtained based on a corresponding shoe mold camera, and the first shoe mold is a left shoe mold or a right shoe mold; shoe mold feature points are extracted based on the image data, and the deviation between the shoe mold feature points and the reference shoe mold feature points is calculated to generate a plurality of moving tracks of the robot for shoemaking production; in the shoemaking production process, denoising the obtained point cloud data based on a deep learning method to obtain denoised point cloud data; performing point cloud segmentation processing on the denoised point cloud data to obtain point cloud data after point cloud segmentation processing; performing edge extraction on the point cloud data after the point cloud segmentation processing to obtain edge information; and performing speed adaptive adjustment processing on the robot based on the edge information. The production efficiency and the sole product quality are improved, and meanwhile the production cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a disc shoe production method and system based on three-dimensional vision. Background Art

[0002] There are several problems in the current disc shoe production process. The first problem is that the relative position of each robot arm and workstation is adjusted to the same through hardware adjustment, but this method has a large workload and high difficulty. It is difficult to ensure that the relative position of each disc device is the same, and the hardware adjustment is difficult. The second problem is that there are many processes that require manual teaching of multiple robots to spray glue points on all shoe mold trajectories, which is a large workload and low efficiency. The third problem is that the noise that is prevalent in the point cloud data cannot be removed well. The fourth problem is that the speed of the robot cannot be adjusted adaptively due to the different depths of the mold.

[0003] Three-dimensional vision technology uses depth sensors or stereo vision systems to capture multiple perspective images or video data of the surface of an object and combines it with advanced algorithm processing to restore the three-dimensional structure and information of an object or scene in real time.

[0004] The introduction of 3D vision technology can intelligently identify the position and posture of the sole mold and guide the robot arm to complete the glue spraying action. Its high degree of automation, high efficiency, high precision and ability to adapt to poor environments have opened up a new horizon for disc shoemaking automation, thereby improving production efficiency, improving the quality of sole products and reducing production costs. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a disc shoe production method and system based on three-dimensional vision, which can improve the reliability of trajectory planning and robot speed adjustment in the shoe production process, improve production efficiency and sole product quality, and reduce production costs.

[0006] In order to solve the above technical problems, the present invention provides a disc shoe production method based on three-dimensional vision, the method comprising:

[0007] The first shoe model is moved to the bottom of the corresponding shoe model camera based on the disc machine, and image data of the first shoe model is acquired based on the corresponding shoe model camera, where the first shoe model is a left foot shoe model or a right foot shoe model;

[0008] Extracting shoe mold feature points based on the image data to obtain shoe mold feature points, and calculating deviations between the shoe mold feature points and reference shoe mold feature points, generating running trajectories of several robots based on the deviations, and each robot performs shoe production based on the corresponding running trajectory;

[0009] In the process of shoe production performed by each robot based on the corresponding running trajectory, point cloud data is acquired based on the three-dimensional depth camera, and the point cloud data is denoised based on the deep learning method to obtain the denoised point cloud data;

[0010] Performing point cloud segmentation processing on the denoised point cloud data to obtain point cloud data after point cloud segmentation processing;

[0011] Perform edge extraction on the point cloud data after point cloud segmentation to obtain edge information;

[0012] The speed of the robot is adaptively adjusted based on the edge information.

[0013] Optionally, calculating the deviation between the shoe mold feature point and the reference shoe mold feature point, and generating a plurality of robot running trajectories based on the deviation, includes:

[0014] Acquire a reference shoe mold image based on an offline workstation, extract shoe mold feature points from the reference shoe mold image, obtain reference shoe mold feature points, and calculate deviations between the shoe mold feature points and the reference shoe mold feature points;

[0015] Based on the deviation, a spraying trajectory of the first robot, a scraping trajectory of the second robot, and a linear injection trajectory of the third robot are generated using the first reference trajectory.

[0016] Optionally, performing denoising on the point cloud data based on a deep learning method to obtain denoised point cloud data includes:

[0017] Based on the DataLoader component in the pytorch deep learning framework, point cloud sample data is obtained, and the point cloud sample data is converted into a format of (N, D, 3) to obtain point cloud sample data after format conversion, where N is the number of point cloud sample data input to the network for one training, and D is the number of points in each point cloud sample data;

[0018] Generate a conversion matrix of the format-converted point cloud sample data based on the principal component analysis method, and perform alignment processing on the format-converted point cloud sample data using matrix multiplication based on the conversion matrix to obtain the aligned point cloud sample data;

[0019] The point cloud denoising model is trained based on the aligned point cloud sample data to obtain a trained point cloud denoising model;

[0020] The point cloud data is input into a trained point cloud denoising model for denoising to obtain denoised point cloud data.

[0021] Optionally, the point cloud denoising model adopts a pointnet convolutional neural network as a backbone network, and the pointnet convolutional neural network includes several layers of multilayer perceptrons and decoders, each layer of multilayer perceptron includes five hidden layers, each hidden layer includes a Batch Norm1d normalization layer, a ReLU activation layer and a maximum pooling layer, and the decoder includes several layers of fully connected layers, normalization layers and activation layers, and there are Dropout layers between the fully connected layers.

[0022] Optionally, the loss function of the point cloud denoising model is expressed as:

[0023]

[0024] Among them, L is the loss function, p i ′ is the closest point in the clean point cloud, p i is the sample point cloud, and I is the number of point clouds.

[0025] Optionally, performing point cloud segmentation processing on the denoised point cloud data to obtain point cloud data after point cloud segmentation processing includes:

[0026] The local tangent plane is generated by using the denoised point cloud data based on the least squares fitting method. The expression of the local tangent plane is:

[0027]

[0028] in, is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point;

[0029] The centroids of all neighborhoods in the local tangent plane are calculated, and the covariance matrix is ​​calculated based on the centroids of the neighborhoods. The expression of the covariance matrix is:

[0030]

[0031] Among them, M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, T is the transpose;

[0032] Obtaining the normal vector of the neighborhood point based on the covariance matrix, and calculating the normal angle using the normal vector of the seed point based on the normal vector of the neighborhood point;

[0033] Point cloud segmentation processing is performed on the point cloud data after denoising processing based on the normal angle to obtain point cloud data after point cloud segmentation processing.

[0034] Optionally, the expression of the normal angle is:

[0035]

[0036] Among them, ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood.

[0037] Optionally, performing edge extraction on the point cloud data after point cloud segmentation processing to obtain edge information includes:

[0038] Select the current point in the point cloud data after point cloud segmentation processing, project the tangent plane based on the neighborhood of the current point, and obtain the neighboring points after projection;

[0039] Taking the current point as a corner point, connecting the projected neighboring points with the corner point based on a preset order to obtain a plurality of angles;

[0040] A maximum angle among several angles is selected, and the maximum angle is compared with a preset threshold to obtain a comparison result, and based on the comparison result, it is determined whether the current point is an edge point.

[0041] Optionally, the adaptively adjusting the speed of the robot based on the edge information includes:

[0042] The depth information of the current trajectory point of the robot is obtained based on the edge information, and the adjustment speed of the robot is calculated based on the depth information, wherein the expression of the adjustment speed is:

[0043] v′=v(1+k*(z-z0))

[0044] Where v′ is the adjusted speed, v is the original speed, k is the proportional coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height;

[0045] The robot is adaptively adjusted in speed based on the adjustment speed.

[0046] In addition, the present invention also provides a disc shoe production system based on three-dimensional vision, the system comprising:

[0047] An image data acquisition module is used to move the first shoe model to the bottom of the corresponding shoe model camera based on the disc machine, and acquire image data of the first shoe model based on the corresponding shoe model camera, wherein the first shoe model is a left foot shoe model or a right foot shoe model;

[0048] An operation trajectory generation module is used to extract shoe mold feature points based on the image data, obtain shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate operation trajectories of several robots based on the deviation, and each robot performs shoe production based on the corresponding operation trajectory;

[0049] Point cloud denoising module: used to obtain point cloud data based on a three-dimensional depth camera during the shoemaking production process of each robot based on the corresponding running trajectory, and to denoise the point cloud data based on a deep learning method to obtain denoised point cloud data;

[0050] Point cloud segmentation module: used to perform point cloud segmentation processing on the point cloud data after denoising, and obtain the point cloud data after point cloud segmentation processing;

[0051] Edge extraction module: used to extract the edge of the point cloud data after point cloud segmentation to obtain edge information;

[0052] Speed ​​adaptive adjustment module: used for performing speed adaptive adjustment processing on the robot based on the edge information.

[0053] In an embodiment of the present invention, the deviation between the feature points of the left and right shoe molds and the feature points of the reference shoe mold is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of trajectory planning of the robot arm in the disc shoemaking process, has better robustness and trajectory planning accuracy, and does not need to adjust the relative position of each robot arm and workstation, thereby improving the efficiency of trajectory planning. The point cloud data is denoised based on the deep learning method, which improves the accuracy of subsequent edge contour detection. During the movement of the robot arm, the robot arm movement speed is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, improve the quality of the sole product, and reduce production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a schematic flow chart of a disc shoe production method based on three-dimensional vision in a first embodiment of the present invention;

[0056] Figure 2 is a schematic flow chart of a disc shoe production method based on three-dimensional vision in a second embodiment of the present invention;

[0057] Figure 3 Schematic diagram of the process of the disc shoe production method based on three-dimensional vision in the third embodiment of the present invention

[0058] Figure 4 Schematic diagram of the structure of a disc shoe production system based on three-dimensional vision in an embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of the equipment of the disc shoe production method based on three-dimensional vision in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Embodiment 1

[0062] See also Figure 1 , Figure 1 : is a schematic flow chart of a disc shoe production method based on three-dimensional vision in a first embodiment of the present invention, the method comprising:

[0063] S01: moving a first shoe model to below a corresponding shoe model camera based on a disc machine, and acquiring image data of the first shoe model based on the corresponding shoe model camera, wherein the first shoe model is a left foot shoe model or a right foot shoe model;

[0064] S02: extracting shoe mold feature points based on the image data to obtain shoe mold feature points, and calculating deviations between the shoe mold feature points and reference shoe mold feature points, generating operation trajectories of several robots based on the deviations, and each robot performs shoe production based on the corresponding operation trajectory;

[0065] S03: in the process of shoe production performed by each robot based on the corresponding running trajectory, point cloud data is acquired based on the three-dimensional depth camera, and denoising is performed on the point cloud data based on the deep learning method to obtain denoised point cloud data;

[0066] S04: performing point cloud segmentation processing on the denoised point cloud data to obtain point cloud data after point cloud segmentation processing;

[0067] S05: extracting edges of the point cloud data after point cloud segmentation to obtain edge information;

[0068] S06: Performing adaptive speed adjustment processing on the robot based on the edge information.

[0069] In an embodiment of the present invention, the deviation between the feature points of the left and right shoe molds and the feature points of the reference shoe mold is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of trajectory planning of the robot arm in the disc shoemaking process, has better robustness and trajectory planning accuracy, and does not need to adjust the relative position of each robot arm and workstation, thereby improving the efficiency of trajectory planning. The point cloud data is denoised based on the deep learning method, which improves the accuracy of subsequent edge contour detection. During the movement of the robot arm, the robot arm movement speed is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, improve the quality of the sole product, and reduce production costs.

[0070] Embodiment 2

[0071] See also Figure 2 , Figure 2 is a flow chart of a disc shoe production method based on three-dimensional vision in a second embodiment of the present invention, the method comprising

[0072] S100: moving the left shoe model to below the first camera and the right shoe model to below the second camera based on the disc machine;

[0073] In the specific implementation process of the present invention, the disc rotating machine receives materials, moves the left shoe mold to the bottom of the first camera, and moves the right shoe mold to the bottom of the second camera.

[0074] S110: The first camera acquires a first image of the left shoe model;

[0075] S120: The second camera acquires a second image of the right shoe model;

[0076] In the specific implementation process of the present invention, the first camera takes a photo of the left shoe model to obtain a corresponding first image, and the second camera takes a photo of the right shoe model to obtain a corresponding second image.

[0077] S111: extracting shoe mold feature points based on the first image to obtain first shoe mold feature points, and calculating first deviations between the first shoe mold feature points and reference shoe mold feature points, generating first running trajectories of a plurality of robots based on the first deviations, and each robot performs shoe production based on the corresponding first running trajectory;

[0078] In the specific implementation process of the present invention, the first deviation between the first shoe mold feature point and the reference shoe mold feature point is calculated, and the first operation trajectory of several robots is generated based on the first deviation, including: obtaining a reference shoe mold image based on an offline workstation, and extracting shoe mold feature points from the reference shoe mold image to obtain reference shoe mold feature points, and calculating the first deviation between the first shoe mold feature point and the reference shoe mold feature point; based on the first deviation, the first reference trajectory is used to generate the spraying trajectory of the first robot, the scraping trajectory of the second robot, and the straight injection trajectory of the third robot.

[0079] Specifically, the reference shoe mold feature points are obtained in advance at the offline workstation, and the offline workstation uses the same shoe mold fixture to fix the shoe mold. The reference left and right shoe mold images are obtained through the camera of the offline workstation, and the shoe mold feature points are extracted from the reference left shoe mold image to obtain the reference left shoe mold feature points, which are the boundary contour points of the reference left shoe mold. The shoe mold feature points are extracted based on the first image to obtain the first shoe mold feature points, which are the boundary contour points of the left shoe mold. The local plane of the first image is obtained, and a pixel point is randomly selected as the current pixel point in the local plane. A target plane with three adjacent points is constructed around the current pixel point, and the average distance from the current pixel point to the target plane is calculated. The neighboring points of the current pixel point are projected onto the reference plane based on the k-neighborhood search, and the normal vectors of the neighboring points after projection are calculated. The vector angle is calculated based on the average distance, the normal vectors of the neighboring points, and the normal vector of the current pixel point, and the boundary contour points are extracted based on the vector angle. Calculate the first deviation between the first shoe mold feature point and the reference shoe mold feature point, that is, calculate the deviation distance between each shoe mold feature point and the corresponding reference shoe mold feature point, generate the robot's trajectory point deviation according to the deviation distance, adjust the first reference trajectory according to the trajectory point deviation, and obtain the spraying trajectory of the first robot, the scraping trajectory of the second robot, and the linear injection trajectory of the third robot. Each robot performs shoemaking production based on the corresponding first running trajectory. The specific equipment layout is as follows: Figure 5 As shown, the first robot is robot A in the figure, the second robot is robot B in the figure, the third robot is robot D in the figure, and the first camera is camera 1 in the figure.

[0080] S121: extracting shoe mold feature points based on the second image to obtain second shoe mold feature points, and calculating second deviations between the second shoe mold feature points and reference shoe mold feature points, generating second running trajectories of a plurality of robots based on the second deviation, and each robot performs shoe production based on the corresponding second running trajectory;

[0081] In a specific implementation of the present invention, generating the second operation trajectories of the plurality of robots based on the second deviation includes: generating a spraying trajectory of the fourth robot and a linear injection trajectory of the fifth robot using a second reference trajectory based on the second deviation.

[0082] Specifically, the shoe mold feature points are extracted based on the second image to obtain the second shoe mold feature points, and the second deviation between the second shoe mold feature points and the reference shoe mold feature points is calculated, that is, the deviation distance between each shoe mold feature point and the corresponding reference shoe mold feature point is calculated, and the robot's trajectory point deviation is generated according to the deviation distance. The second reference trajectory is adjusted according to the trajectory point deviation to obtain the spraying trajectory of the fourth robot and the linear injection trajectory of the fifth robot. Each robot performs shoemaking production based on the corresponding second running trajectory. The specific equipment layout is as follows: Figure 5 As shown, the fourth robot is the C robot in the figure, the fifth robot is the D robot in the figure, and the second camera is the camera 2 in the figure.

[0083] S112: in a shoemaking production process performed by each robot based on the corresponding first running trajectory, first point cloud data is acquired based on the first three-dimensional depth camera, denoising is performed on the first point cloud data based on a deep learning method to obtain first point cloud data after denoising, and point cloud segmentation is performed on the first point cloud data after denoising to obtain first point cloud data after point cloud segmentation;

[0084] In the specific implementation process of the present invention, the first point cloud data is denoised based on the deep learning method to obtain the first point cloud data after denoising, including: obtaining point cloud sample data based on the DataLoader component in the pytorch deep learning framework, and converting the point cloud sample data into a format of (N, D, 3) to obtain point cloud sample data after format conversion, wherein N is the number of point cloud sample data input into the network for one training, and D is the number of points in each point cloud sample data; generating a conversion matrix of the point cloud sample data after format conversion based on the principal component analysis method, and aligning the point cloud sample data after format conversion using matrix multiplication based on the conversion matrix to obtain aligned point cloud sample data; training a point cloud denoising model based on the aligned point cloud sample data to obtain a trained point cloud denoising model; inputting the first point cloud data into the trained point cloud denoising model for denoising to obtain the first point cloud data after denoising.

[0085] Furthermore, the point cloud denoising model adopts a pointnet convolutional neural network as the backbone network, and the pointnet convolutional neural network includes several layers of multilayer perceptrons and decoders, each layer of multilayer perceptron includes five hidden layers, each hidden layer includes a Batch Norm1d normalization layer, a ReLU activation layer and a maximum pooling layer, and the decoder includes several layers of fully connected layers, normalization layers and activation layers, and there are Dropout layers between the fully connected layers.

[0086] Furthermore, the loss function of the point cloud denoising model is expressed as:

[0087]

[0088] Among them, L is the loss function, p i ′ is the clean point cloud, p i is the sample point cloud, and I is the number of point clouds.

[0089] Furthermore, performing point cloud segmentation processing on the first point cloud data after the denoising processing to obtain the first point cloud data after the point cloud segmentation processing includes:

[0090] The local tangent plane is generated by using the first point cloud data after denoising based on the least squares fitting method. The expression of the local tangent plane is:

[0091]

[0092] in, is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point;

[0093] The centroids of all neighborhoods in the local tangent plane are calculated, and the covariance matrix is ​​calculated based on the centroids of the neighborhoods. The expression of the covariance matrix is:

[0094]

[0095] Among them, M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, T is the transpose;

[0096] The normal vector of the neighborhood point is obtained based on the covariance matrix, and the normal angle is calculated based on the normal vector of the neighborhood point using the normal vector of the seed point. The expression of the normal angle is:

[0097]

[0098] Among them, ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood;

[0099] Point cloud segmentation processing is performed on the first point cloud data after denoising processing based on the normal angle to obtain the first point cloud data after point cloud segmentation processing.

[0100] Specifically, in the process of shoe production based on the corresponding first running track of each robot, the first point cloud data is acquired in real time based on the first three-dimensional depth camera on the production line, and the point cloud sample data is acquired based on the DataLoader component in the pytorch deep learning framework, and the point cloud sample data is converted into (N, D, 3) format to obtain the point cloud sample data after format conversion, wherein N is the number of point cloud sample data input to the network for one training, and D is the number of points in each point cloud sample data. Before inputting into the neural network, it is necessary to align each input data with the principal axis and the Cartesian coordinate system, generate a conversion matrix of the point cloud sample data after format conversion based on the principal component analysis method, and align the point cloud sample data after format conversion based on the conversion matrix using matrix multiplication, and obtain the point cloud sample data after alignment by performing matrix multiplication on the conversion matrix and the input data. The point cloud denoising model is trained based on the aligned point cloud sample data to obtain a trained point cloud denoising model. The point cloud denoising model uses a pointnet convolutional neural network as a backbone network. The pointnet convolutional neural network includes several layers of multilayer perceptrons and decoders. The local features of the point cloud data are extracted by the multilayer perceptrons. Each layer of the multilayer perceptron includes five hidden layers. The neuron size of the first hidden layer is 64, the neuron size of the second hidden layer is 128, the neuron size of the third hidden layer is 256, the neuron size of the fourth hidden layer is 512, and the neuron size of the fifth hidden layer is 1024. Each hidden layer includes Batch Norm1d normalization layer, ReLU activation layer and maximum pooling layer, the maximum pooling layer is used to aggregate the features of each data point, so as to obtain a global feature vector representing the entire point cloud data. The decoder includes several layers of fully connected layers, normalization layers and activation layers. There are three fully connected layers, and their neuron sizes are 1024, 512, 256 and 3 respectively. There is a Dropout layer between the fully connected layers, and the probability of the Dropout layer is 0.3. The loss function of the point cloud denoising model is expressed as:

[0101]

[0102] Among them, L is the loss function, p i ′ is the closest point in the clean point cloud, p i is the sample point cloud, and I is the number of point clouds. The loss function is the distance from the sample point to the nearest point on the clean point cloud. The first point cloud data is input into the trained point cloud denoising model for denoising to obtain the first point cloud data after denoising. The deep learning-based method is used for denoising, and the noise points are moved as close to the clean point cloud as possible, so as to learn the distance function of the space. The trained neural network can then regularize the input noise point cloud into a clean point cloud.

[0103] The regional growing algorithm in the point cloud library is called to perform point cloud segmentation. The regional growing algorithm uses information such as normals, curvatures, and colors to determine whether the point cloud data is clustered into one category. The goal of the algorithm is to merge sufficiently close points according to smooth consistency constraints. Therefore, the output of the algorithm is a set of clusters, and the points on each cluster are located on the same smooth surface. The local tangent plane is generated using the first point cloud data after denoising based on the least squares fitting method. The expression of the local tangent plane is:

[0104]

[0105] in, is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point;

[0106] The centroids of all neighborhoods in the local tangent plane are calculated, and the covariance matrix is ​​calculated based on the centroids of the neighborhoods. The expression of the covariance matrix is:

[0107]

[0108] Among them, M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, T is the transpose;

[0109] Based on the covariance matrix, the normal vector of the neighborhood point is obtained, and the eigenvalue decomposition is performed according to the covariance matrix. The eigenvector corresponding to the minimum eigenvalue of the covariance matrix is ​​the normal vector of the point, and the normal angle is calculated based on the normal vector of the neighborhood point using the normal vector of the seed point. The expression of the normal angle is:

[0110]

[0111] Among them, ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood; the first point cloud data input is sorted according to the curvature value of the point cloud, and the point with the smallest average angle between normals is selected as the seed point. Because the area where the seed point is located is the smoothest area, starting to grow from the smoothest area can effectively reduce the total number of segmented fragments and improve segmentation efficiency.

[0112] Based on the normal angle, point cloud segmentation is performed on the first point cloud data after denoising. The normal angle is compared with a preset smoothing threshold. If it is less than the preset smoothing threshold, the neighborhood points corresponding to the normal angle are regarded as part of the same smooth surface. This process is repeated until all neighborhood points of the smooth surface are clustered. Then, regional segmentation of the point cloud can be performed to obtain the first point cloud data after point cloud segmentation.

[0113] S113: extracting edges of the first point cloud data after the point cloud segmentation process to obtain first edge information, and performing speed adaptive adjustment processing on the robot based on the first edge information;

[0114] In the specific implementation process of the present invention, the edge extraction is performed on the first point cloud data after the point cloud segmentation processing to obtain the first edge information, including: selecting a current point in the first point cloud data after the point cloud segmentation processing, projecting a tangent plane based on the neighborhood of the current point, and obtaining a neighboring point after the projection; taking the current point as a corner point, connecting the projected neighboring points and the corner point based on a preset order to obtain a number of angles; selecting the maximum angle among the several angles, and comparing the maximum angle with a preset threshold to obtain a comparison result, and judging whether the current point is an edge point based on the comparison result.

[0115] Further, the adaptive speed adjustment processing of the robot based on the first edge information includes: acquiring the depth information of the current trajectory point of the robot based on the first edge information, and calculating the adjustment speed of the robot based on the depth information, wherein the expression of the adjustment speed is:

[0116] v′=v(1+k*(z-z0))

[0117] Where v′ is the adjusted speed, v is the original speed, k is the proportional coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height;

[0118] The robot is adaptively adjusted in speed based on the adjustment speed.

[0119] Specifically, a current point is selected from the first point cloud data after the point cloud segmentation process. The current point is randomly selected, and a tangent plane is projected based on the neighborhood of the current point to obtain a neighboring point after projection. It should be noted that after projection, the position of the current point has not changed. The current point is used as a corner point, and the neighboring point after projection is connected with the corner point based on a preset order to obtain a number of angles θ={θ1,θ2,…,θ n}, where n = || N p ||-1, N pis the number of projection points, and the preset order can be counterclockwise or clockwise; select the maximum angle among several angles, the larger the angle, the more likely the current point is an edge point, and compare the maximum angle with the preset threshold to obtain a comparison result, and determine whether the current point is an edge point based on the comparison result, that is, when the maximum angle is greater than the preset threshold, the current point is determined to be an edge point. All edge points are extracted according to the above steps, and after all edge points are obtained, the edge points are connected to obtain the first edge information.

[0120] All points on the running track of each robot include not only x-axis and y-axis information, but also z-axis information, and the speed can be adjusted by the z-axis information. The depth information of the current track point of the robot is obtained based on the first edge information, and the adjustment speed of the robot is calculated based on the depth information, wherein the expression of the adjustment speed is:

[0121] v′=v(1+k*(z-z0))

[0122] Where v' is the adjusted speed, v is the original speed, k is the proportional coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height; the proportional coefficient is used to control the sensitivity of the speed as the z-axis changes. If k>0 is set, the speed will increase as the z-axis information increases; z0 is set to the average value of z, and the robot is adaptively adjusted based on the adjusted speed. Adaptively adjusting the movement speed of the robot according to the depth information during the movement of the robot arm can improve production efficiency.

[0123] S122: in the process of shoemaking production performed by each robot based on the corresponding second running trajectory, second point cloud data is acquired based on the second three-dimensional depth camera, the second point cloud data is denoised based on the deep learning method to obtain second point cloud data after denoising, and point cloud segmentation is performed on the second point cloud data after denoising to obtain second point cloud data after point cloud segmentation;

[0124] In the specific implementation process of the present invention, when each robot is performing shoe production based on the corresponding second operating trajectory, second point cloud data is acquired based on the second three-dimensional depth camera, and the method steps of denoising and point cloud segmentation processing of the second point cloud data are the same as the method steps of denoising and point cloud segmentation processing of the first point cloud data, and will not be repeated here.

[0125] S123: Perform edge extraction on the second point cloud data after the point cloud segmentation processing to obtain second edge information, and perform speed adaptive adjustment processing on the robot based on the second edge information.

[0126] During the specific implementation of the present invention, edge extraction is performed on the second point cloud data after point cloud segmentation processing to obtain second edge information, and the speed of the robot is adaptively adjusted based on the second edge information, so as to adaptively adjust the movement speeds of the fourth robot and the fifth robot corresponding to the right shoe mold.

[0127] In an embodiment of the present invention, the deviation between the feature points of the left and right shoe molds and the feature points of the reference shoe mold is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of trajectory planning of the robot arm in the disc shoemaking process, has better robustness and trajectory planning accuracy, and does not need to adjust the relative position of each robot arm and workstation, thereby improving the efficiency of trajectory planning. The point cloud data is denoised based on the deep learning method, which improves the accuracy of subsequent edge contour detection. During the movement of the robot arm, the robot arm movement speed is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, improve the quality of the sole product, and reduce production costs.

[0128] Embodiment 3

[0129] See also Figure 3 , Figure 3 : is a schematic flow chart of a disc shoe production method based on three-dimensional vision in a third embodiment of the present invention, the method comprising:

[0130] S200: moving the left shoe model to below the first camera and the right shoe model to below the second camera based on the disc machine;

[0131] S210: The first camera acquires a first image of the left shoe model;

[0132] S220: The second camera acquires a second image of the right shoe model;

[0133] S211: extracting shoe mold feature points based on the first image to obtain first shoe mold feature points, and calculating first deviations between the first shoe mold feature points and reference shoe mold feature points;

[0134] S212: generating a spraying trajectory of the first robot, a scraping trajectory of the second robot, and a linear injection trajectory of the third robot based on the first deviation and using the first reference trajectory, and each robot performs shoe production based on the corresponding operation trajectory;

[0135] S213: Acquiring first point cloud data based on the first three-dimensional depth camera during the shoe production process of each robot based on the corresponding first running trajectory;

[0136] S214: acquiring point cloud sample data based on the DataLoader component in the pytorch deep learning framework, and performing format conversion on the point cloud sample data to (N, D, 3) to obtain point cloud sample data after format conversion;

[0137] S215: generating a conversion matrix of the format-converted point cloud sample data based on a principal component analysis method, and performing alignment processing on the format-converted point cloud sample data using matrix multiplication based on the conversion matrix to obtain aligned point cloud sample data;

[0138] S216: training a point cloud denoising model based on the aligned point cloud sample data to obtain a trained point cloud denoising model;

[0139] S217: Inputting the first point cloud data into a trained point cloud denoising model for denoising, to obtain first point cloud data after denoising;

[0140] S218: performing point cloud segmentation processing on the first point cloud data after the denoising processing to obtain the first point cloud data after the point cloud segmentation processing;

[0141] S219: performing edge extraction on the first point cloud data after the point cloud segmentation process to obtain first edge information;

[0142] S2110: performing speed adaptive adjustment processing on the robot based on the first edge information;

[0143] S221: extracting shoe mold feature points based on the second image to obtain second shoe mold feature points, and calculating second deviations between the second shoe mold feature points and reference shoe mold feature points;

[0144] S222: generating a spraying trajectory of a fourth robot and a linear injection trajectory of a fifth robot based on the second deviation and using the second reference trajectory, and each robot performs shoe production based on the corresponding operation trajectory;

[0145] S223: Acquiring second point cloud data based on the second three-dimensional depth camera during the shoe production process of each robot based on the corresponding second running trajectory;

[0146] S224: acquiring point cloud sample data based on the DataLoader component in the pytorch deep learning framework, and performing format conversion on the point cloud sample data to (N, D, 3) to obtain point cloud sample data after format conversion;

[0147] S225: generating a conversion matrix of the format-converted point cloud sample data based on a principal component analysis method, and performing alignment processing on the format-converted point cloud sample data using matrix multiplication based on the conversion matrix to obtain aligned point cloud sample data;

[0148] S226: training a point cloud denoising model based on the aligned point cloud sample data to obtain a trained point cloud denoising model;

[0149] S227: Inputting the second point cloud data into the trained point cloud denoising model for denoising, to obtain the denoised second point cloud data;

[0150] S228: performing point cloud segmentation processing on the second point cloud data after the denoising processing to obtain second point cloud data after the point cloud segmentation processing;

[0151] S229: extracting edges from the second point cloud data after the point cloud segmentation process to obtain second edge information;

[0152] S2210: Performing adaptive speed adjustment processing on the robot based on the second edge information.

[0153] In an embodiment of the present invention, the deviation between the feature points of the left and right shoe molds and the feature points of the reference shoe mold is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of trajectory planning of the robot arm in the disc shoemaking process, has better robustness and trajectory planning accuracy, and does not need to adjust the relative position of each robot arm and workstation, thereby improving the efficiency of trajectory planning. The point cloud data is denoised based on the deep learning method, which improves the accuracy of subsequent edge contour detection. During the movement of the robot arm, the robot arm movement speed is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, improve the quality of the sole product, and reduce production costs.

[0154] Embodiment 4

[0155] See also Figure 4 , Figure 4 : is a schematic diagram of the structure of a disc shoe production system based on three-dimensional vision in an embodiment of the present invention, the system includes:

[0156] The image data acquisition module 41 is used to move the first shoe model to the bottom of the corresponding shoe model camera based on the disc machine, and acquire the image data of the first shoe model based on the corresponding shoe model camera, wherein the first shoe model is a left foot shoe model or a right foot shoe model;

[0157] The running track generation module 42 is used to extract shoe mold feature points based on the image data, obtain shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate running tracks of several robots based on the deviation, and each robot performs shoe production based on the corresponding running track;

[0158] Point cloud denoising module 43: used for acquiring point cloud data based on a three-dimensional depth camera and denoising the point cloud data based on a deep learning method to obtain denoised point cloud data during the shoemaking process of each robot based on the corresponding running trajectory;

[0159] Point cloud segmentation module 44: used to perform point cloud segmentation processing on the point cloud data after the denoising process to obtain point cloud data after the point cloud segmentation process;

[0160] Edge extraction module 45: used to extract edges from point cloud data after point cloud segmentation processing to obtain edge information;

[0161] The speed adaptive adjustment module 46 is used to perform speed adaptive adjustment processing on the robot based on the edge information.

[0162] In the specific implementation process of the present invention, the specific implementation method of the system item can refer to the implementation method of the above-mentioned method item, which will not be repeated here.

[0163] In an embodiment of the present invention, the deviation between the feature points of the left and right shoe molds and the feature points of the reference shoe mold is calculated, and the corresponding running trajectory of the robot is generated according to the deviation, which solves the problem of trajectory planning of the robot arm in the disc shoemaking process, has better robustness and trajectory planning accuracy, and does not need to adjust the relative position of each robot arm and workstation, thereby improving the efficiency of trajectory planning. The point cloud data is denoised based on the deep learning method, which improves the accuracy of subsequent edge contour detection. During the movement of the robot arm, the robot arm movement speed is adaptively adjusted according to the depth information obtained from the edge information of the point cloud data, which can improve production efficiency, improve the quality of the sole product, and reduce production costs.

[0164] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0165] In addition, the above is a detailed introduction to a disc shoe production method and system based on three-dimensional vision provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A disc shoe production method based on three-dimensional vision, characterized in that: The method comprises: The first shoe model is moved to the bottom of the corresponding shoe model camera based on the disc machine, and image data of the first shoe model is acquired based on the corresponding shoe model camera, where the first shoe model is a left foot shoe model or a right foot shoe model; Extracting shoe mold feature points based on the image data to obtain shoe mold feature points, and calculating deviations between the shoe mold feature points and reference shoe mold feature points, generating running trajectories of several robots based on the deviations, and each robot performs shoe production based on the corresponding running trajectory; In the process of shoe production performed by each robot based on the corresponding running trajectory, point cloud data is acquired based on the three-dimensional depth camera, and the point cloud data is denoised based on the deep learning method to obtain the denoised point cloud data; Performing point cloud segmentation processing on the denoised point cloud data to obtain point cloud data after point cloud segmentation processing; Perform edge extraction on the point cloud data after point cloud segmentation to obtain edge information; The speed of the robot is adaptively adjusted based on the edge information.

2. The disc shoe production method based on three-dimensional vision according to claim 1 is characterized in that: The calculating the deviation between the shoe mold feature point and the reference shoe mold feature point, and generating a plurality of robot running trajectories based on the deviation, comprises: Acquire a reference shoe mold image based on an offline workstation, extract shoe mold feature points from the reference shoe mold image, obtain reference shoe mold feature points, and calculate deviations between the shoe mold feature points and the reference shoe mold feature points; Based on the deviation, a spraying trajectory of the first robot, a scraping trajectory of the second robot, and a linear injection trajectory of the third robot are generated using the first reference trajectory.

3. The disc shoe production method based on three-dimensional vision according to claim 1 is characterized in that: The step of performing denoising on the point cloud data based on a deep learning method to obtain denoised point cloud data includes: Based on the DataLoader component in the pytorch deep learning framework, point cloud sample data is obtained, and the point cloud sample data is converted into a format of (N, D, 3) to obtain point cloud sample data after format conversion, where N is the number of point cloud sample data input to the network for one training, and D is the number of points in each point cloud sample data; Generate a conversion matrix of the format-converted point cloud sample data based on the principal component analysis method, and perform alignment processing on the format-converted point cloud sample data using matrix multiplication based on the conversion matrix to obtain the aligned point cloud sample data; The point cloud denoising model is trained based on the aligned point cloud sample data to obtain a trained point cloud denoising model; The point cloud data is input into a trained point cloud denoising model for denoising to obtain denoised point cloud data.

4. The disc shoe production method based on three-dimensional vision according to claim 3 is characterized in that: The point cloud denoising model adopts the pointnet convolutional neural network as the backbone network, and the pointnet convolutional neural network includes several layers of multilayer perceptrons and decoders, each of which includes five hidden layers, each of which includes a Batch Norm1d normalization layer, a ReLU activation layer and a maximum pooling layer, and the decoder includes several layers of fully connected layers, normalization layers and activation layers, and there are Dropout layers between the fully connected layers.

5. The disc shoe production method based on three-dimensional vision according to claim 4 is characterized in that: The loss function of the point cloud denoising model is expressed as: Among them, L is the loss function, p i ′ is the closest point in the clean point cloud, p i is the sample point cloud, and I is the number of point clouds.

6. The disc shoe production method based on three-dimensional vision according to claim 1 is characterized in that: The step of performing point cloud segmentation processing on the denoised point cloud data to obtain the point cloud data after the point cloud segmentation processing includes: The local tangent plane is generated by using the denoised point cloud data based on the least squares fitting method. The expression of the local tangent plane is: in, is the local tangent plane, is the normal vector of the local tangent plane, d is the distance from the origin to the local tangent plane, k is the number of neighborhood points, is the neighborhood point; The centroids of all neighborhoods in the local tangent plane are calculated, and the covariance matrix is ​​calculated based on the centroids of the neighborhoods. The expression of the covariance matrix is: Among them, M is the covariance matrix, k is the number of neighborhood points, is the neighborhood centroid, is the neighborhood point, T is the transpose; Obtaining the normal vector of the neighborhood point based on the covariance matrix, and calculating the normal angle using the normal vector of the seed point based on the normal vector of the neighborhood point; Point cloud segmentation processing is performed on the point cloud data after denoising processing based on the normal angle to obtain point cloud data after point cloud segmentation processing.

7. The disc shoe production method based on three-dimensional vision according to claim 6 is characterized in that: The expression of the normal angle is: Among them, ∝ m is the normal angle, N is the number of neighborhood points of the normal angle, is the normal vector of the seed point; is the normal vector of any point in the neighborhood.

8. The disc shoe production method based on three-dimensional vision according to claim 1 is characterized in that: The step of extracting edges from the point cloud data after the point cloud segmentation process to obtain edge information includes: Select the current point in the point cloud data after point cloud segmentation processing, project the tangent plane based on the neighborhood of the current point, and obtain the neighboring points after projection; Taking the current point as a corner point, connecting the projected neighboring points with the corner point based on a preset order to obtain a plurality of angles; A maximum angle among several angles is selected, and the maximum angle is compared with a preset threshold to obtain a comparison result, and based on the comparison result, it is determined whether the current point is an edge point.

9. The disc shoe production method based on three-dimensional vision according to claim 1 is characterized in that: The method of adaptively adjusting the speed of the robot based on the edge information includes: The depth information of the current trajectory point of the robot is obtained based on the edge information, and the adjustment speed of the robot is calculated based on the depth information, wherein the expression of the adjustment speed is: v ′ =v(1+k*(z-z0)) Among them, v ′ To adjust the speed, v is the original speed, k is the proportional coefficient, z is the z coordinate of the current trajectory, and z0 is the reference height; The robot is adaptively adjusted in speed based on the adjustment speed.

10. A disc shoe production system based on three-dimensional vision, characterized in that: The system comprises: An image data acquisition module is used to move the first shoe model to the bottom of the corresponding shoe model camera based on the disc machine, and acquire image data of the first shoe model based on the corresponding shoe model camera, wherein the first shoe model is a left foot shoe model or a right foot shoe model; An operation trajectory generation module is used to extract shoe mold feature points based on the image data, obtain shoe mold feature points, calculate the deviation between the shoe mold feature points and the reference shoe mold feature points, generate operation trajectories of several robots based on the deviation, and each robot performs shoe production based on the corresponding operation trajectory; Point cloud denoising module: used to obtain point cloud data based on a three-dimensional depth camera during the shoemaking production process of each robot based on the corresponding running trajectory, and to denoise the point cloud data based on a deep learning method to obtain denoised point cloud data; Point cloud segmentation module: used to perform point cloud segmentation processing on the point cloud data after denoising, and obtain the point cloud data after point cloud segmentation processing; Edge extraction module: used to extract the edge of the point cloud data after point cloud segmentation to obtain edge information; Speed ​​adaptive adjustment module: used for performing speed adaptive adjustment processing on the robot based on the edge information.

Images