A sole edge extraction method based on three-dimensional vision

Abstract

The application provides a sole edge extraction method based on three-dimensional vision, comprising the following steps: S1, shooting a sole image and obtaining a corresponding sole edge point cloud; S2, detecting whether the integrity of the sole edge point cloud meets preset requirements, if yes, entering S4, otherwise, entering S3; S3, locating a point cloud missing position, shooting a supplementary image, obtaining a supplementary point cloud according to the supplementary image, supplementing the point cloud missing position, and entering S2; S4, judging whether the obtained sole edge point cloud reaches a preset number, if yes, entering S5, otherwise, controlling the sole to rotate by a set angle and entering S1; S5, fusing the multiple sole edge point clouds to obtain fused edge point clouds, and generating a sole edge curve. The application balances speed and accuracy, and meets the adaptability of sole diversification under the premise of controlling cost.

Description

Technical Field

[0001] This invention relates to a method for extracting the edge of a shoe sole based on three-dimensional vision. Background Technology

[0002] In mature footwear manufacturing, rubber soles are typically produced using injection molding technology. This results in newly formed soles containing injection residue or burrs. Traditionally, this involves manual visual inspection and removal of these residues or burrs from the sole edges. However, with rising labor costs and increasing efficiency requirements in this process, fully manual operation is no longer sufficient. Furthermore, due to the high elasticity of rubber soles, traditional automated equipment's visual imaging systems cannot adaptively detect and acquire accurate sole edge data.

[0003] To overcome the shortcomings of the aforementioned shoe sole edge detection technologies, 3D cameras are being used in industry to replace 2D cameras, acquiring and identifying shoe sole edges through 3D reconstruction. Current technologies either identify shoe sole edges using single-frame 3D imaging or continuous multi-frame 3D imaging. However, the diversity and elastic deformation of shoe soles make it impossible for 3D cameras to accurately capture the shoe sole edge in a single frame. While multi-frame solutions can adapt to diverse shoe shapes, the algorithms are complex and require expensive high-speed 3D cameras, resulting in slow speed and high cost. Summary of the Invention

[0004] The purpose of this invention is to propose a method for extracting the edges of shoe soles based on three-dimensional vision, which balances speed and accuracy, while also satisfying the adaptability of diverse shoe soles under the premise of cost control.

[0005] This invention is achieved through the following technical solution:

[0006] A method for extracting shoe sole edges based on 3D vision includes the following steps:

[0007] Step S1: Use a 3D camera to capture a frame of shoe sole image, and perform 3D reconstruction based on the frame of shoe sole image to obtain a segment of shoe sole edge point cloud corresponding to the frame of shoe sole image;

[0008] Step S2: Check whether the integrity of the point cloud at the edge of the sole meets the preset requirements. If yes, proceed to step S4; otherwise, proceed to step S3.

[0009] Step S3: Locate the missing point cloud position, control the sole rotation according to the missing point cloud position, so that the position of the sole corresponding to the missing point cloud position is within the shooting range of the 3D camera, and use the 3D camera to take a supplementary image, perform three-dimensional reconstruction based on the supplementary image to obtain the supplementary point cloud, fill the missing point cloud position according to the supplementary point cloud, and proceed to step S2.

[0010] Step S4: Determine whether the number of obtained shoe sole edge point cloud segments has reached the preset number. If yes, proceed to step S5; otherwise, control the shoe sole to rotate at a set angle and proceed to step S1.

[0011] Step S5: Fuse the multi-segment shoe sole edge point cloud to obtain the fused edge point cloud. Match the fused edge point cloud with the template curve in the database, select the template curve with the highest matching degree, optimize the fused edge point cloud according to the template curve, and generate the shoe sole edge curve according to the optimized point cloud.

[0012] Furthermore, in step S2, detecting whether the integrity of the point cloud at the edge of the shoe sole meets the preset requirements specifically involves: counting the number of effective structured light encoded pixels P1 in the depth map corresponding to the shoe sole image frame, and counting the number of point clouds at the edge of the shoe sole P2. P2 / P1 is the point cloud density of the point cloud at the edge of the shoe sole. If the point cloud density is less than the set first threshold, it is determined that the integrity of the point cloud at the edge of the shoe sole does not meet the requirements.

[0013] Furthermore, in step S2, locating the missing point cloud location specifically involves: dividing the depth map corresponding to the shoe sole image into several regions, counting the number of effective structured light encoded pixels P3 in each region, and counting the number of point clouds P4 in each region. If P4 / P3 is less than the set second threshold, it is determined that the point cloud of that region is incomplete; establishing a coordinate system with the shoe sole rotation center as the origin, determining the coordinate value corresponding to the geometric center of that region, and determining the shoe sole rotation angle based on the coordinate value.

[0014] Furthermore, in step S3, the point supplementation specifically involves: based on the relationship between the pose corresponding to the supplemented point cloud and the pose corresponding to the missing point cloud at the edge of the shoe sole, the supplemented point cloud is spliced ​​and merged into the point cloud at the edge of the shoe sole.

[0015] Furthermore, in step S5, the point clouds of each shoe sole edge are spliced ​​and fused together based on the pose relationship between the point clouds of each shoe sole edge.

[0016] Furthermore, after obtaining the fused edge point cloud, it is filtered by a Gaussian filtering algorithm, and then interpolated by a multiple interpolation algorithm.

[0017] Furthermore, in step S5, optimizing the fused edge point cloud based on the template curve specifically involves: performing elastic compensation and smoothing optimization on the fused edge point cloud using the template curve.

[0018] Furthermore, in step S5, the database is obtained through shoe sole design drawings, manual drawing, and / or deep learning training methods.

[0019] Furthermore, the 3D camera includes a monocular structured light 3D camera or a binocular structured light 3D camera.

[0020] Furthermore, the sole is placed on a rotating platform, which is located inside a support. A 3D camera is rotated and arranged on the support, positioned diagonally above the sole, to take inward-tilted pictures of the sole. The rotating platform and the 3D camera are respectively connected to a control mechanism.

[0021] The present invention has the following beneficial effects:

[0022] 1. This invention obtains multiple shoe sole edge point clouds through multiple shots, and then fuses these multiple shoe sole edge point clouds to obtain the final shoe sole edge curve. It combines the high efficiency of single-frame 3D imaging with the high adaptability of multi-frame continuous 3D imaging. The algorithm is simple and does not require a high-speed 3D camera, which improves both accuracy and speed while controlling costs. It is applicable to various shoe soles. For each shoe sole edge point cloud, if the point cloud is found to be incomplete, it is reshot to further improve accuracy. Template matching is used to optimize the fusion of edge point clouds, which can avoid errors caused by elastic shoe sole deformation and improve the accuracy of edge point sampling. Attached Figure Description

[0023] The present invention will now be described in further detail with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart of the present invention.

[0025] Figure 2 This is a schematic diagram of the imaging structure of the present invention.

[0026] Figure 3 This is a schematic diagram illustrating the effects of the present invention.

[0027] The components include: 1. 3D camera; 2. Rotary stage; 3. Support; 4. Shoe sole; 5. Pneumatic pressure claw. Detailed Implementation

[0028] like Figure 1 and Figure 2 As shown, the method for extracting shoe sole edges based on 3D vision includes the following steps:

[0029] Step S1: Use 3D camera 1 to capture a frame of shoe sole image, and perform 3D reconstruction based on the frame of shoe sole image to obtain a segment of shoe sole edge point cloud corresponding to the frame of shoe sole image;

[0030] The 3D camera 1 is mounted on a support 3, which is a hollow rectangular frame. The 3D camera 1 is rotatably mounted on a crossbeam on the upper section of the hollow rectangular frame. A rotating platform 2 is installed inside the hollow rectangular frame. The shoe sole 4 is placed on the upper end of the rotating platform 2 and fixed by a pneumatic gripper 5. The rotating platform 2 is a high-precision controllable automatic rotating platform 2. The rotating platform 2 and the 3D camera 1 are respectively connected to the control mechanism. During shooting, the 3D camera tilts inward to shoot the shoe sole. The inward tilt angle varies depending on the shoe type and can be adjusted during actual shooting. More specifically, the control mechanism includes a PLC and a vision processor (GPU). The 3D camera 1 is bidirectionally connected to the vision processor. The vision processor controls the 3D camera 1 to shoot. The image captured by the 3D camera 1 is fed back to the vision processor. The vision processor is bidirectionally connected to the PLC. The vision processor sends the rotation angle to the PLC. The output of the PLC is connected to the rotating platform 2 to control the rotation of the rotating platform 2 according to the rotation angle. The PLC feeds back the shooting posture to the vision processor. Among them, the fixing mechanism of 3D camera 1, the structure of rotary table 2, the structure of pneumatic pressure claw 5, and the structure of pneumatic pressure claw 5 fixing shoe sole 4 are all existing technologies.

[0031] In this embodiment, a monocular structured light 3D camera 1 based on the parallax principle is used for 3D reconstruction. The structured light method relies on a projector to project coded structured light onto the object being photographed, which is then captured by the camera. Because different parts of the object have different distances and orientations relative to the camera, the size and shape of the structured light-coded pattern will change. This change can be captured by the camera and then converted into depth information by a processing unit, thereby obtaining the 3D contour information of the sole 4, i.e., a depth map. The edge features of the sole are obtained by slicing the depth map at the pixel level, and edge points are located using these edge features, thus obtaining the sole edge point cloud. The slices of the depth map can be XY-axis slices in a Cartesian coordinate system or angle slices in a polar coordinate system. In this embodiment, for the Cartesian coordinate system, the depth information of each row or column of pixels is extracted as a dataset for analysis. Edge features vary depending on the shoe type, such as sole edge angle, sole edge curvature, sole edge drop, and sole edge roughness. Edge features can be defined manually or trained using deep learning. This process is existing technology.

[0032] Step S2: Check whether the integrity of the point cloud at the edge of the sole meets the preset requirements. If yes, proceed to step S4; otherwise, proceed to step S3.

[0033] Due to shooting angle or occlusion issues, blind spots may occur, resulting in regional missing points in the 3D reconstructed point cloud, causing incomplete point clouds. To detect whether the integrity of the point cloud at the edge of the sole meets the preset requirements, the following steps are taken: count the number of effective structured light encoded pixels P1 in the sole image frame, and count the number of point clouds at the edge of the sole P2. P2 / P1 is the point cloud density of the point cloud at the edge of the sole. If the point cloud density is less than the set first threshold, it is determined that the integrity of the point cloud at the edge of the sole does not meet the requirements.

[0034] Step S3: Locate the missing point cloud position, control the sole 4 to rotate according to the missing point cloud position, so that the position of the sole 4 corresponding to the missing point cloud position is within the shooting range of the 3D camera 1, and use the 3D camera 1 to take a supplementary image, perform three-dimensional reconstruction based on the supplementary image to obtain the supplementary point cloud, fill the missing point cloud position according to the supplementary point cloud, and proceed to step S2.

[0035] The specific method for locating missing point cloud locations is as follows: by dividing the X and Y axis coordinates of the shoe sole image into regions, the depth map is divided into several regions. Depending on the shoe type, the regions can be divided equally or unequally. The number of effective structured light encoded pixels P3 in each region is counted, and the number of point clouds P4 in each region is counted. If P4 / P3 is less than the set second threshold, the point cloud of that region is determined to be incomplete.

[0036] Among them, the first threshold and the second threshold are values ​​set manually after actual debugging. The first threshold is a global judgment, and the second threshold is a local judgment. The first threshold > the second threshold.

[0037] The initial values ​​of the first and second thresholds are set after sampling tests during algorithm debugging. In actual production, they are adjusted by floating based on the yield rate of different styles of soles 4 during the actual cutting process.

[0038] By calibrating the position of the rotary table 2 in advance, the rotation center of the rotary table 2 (i.e. the rotation center of the shoe sole 4) is obtained. A Cartesian coordinate system or a polar coordinate system is established with the rotation center of the shoe sole 4 as the origin. For this segment, the coordinate value corresponding to its geometric center is determined. The rotation angle of the shoe sole 4 is determined according to the coordinate value. After the PLC controls the rotary table 2 to rotate by the rotation angle, the 3D camera 1 takes another picture.

[0039] The specific process for supplementing the points involves: based on the relationship between the pose of the supplemented point cloud and the pose of the missing point cloud at the edge of the sole, stitching and merging the supplemented point cloud into the point cloud at the edge of the sole.

[0040] Step S4: Determine whether the number of point cloud segments obtained on the edge of the sole has reached the preset number. If yes, proceed to step S5; otherwise, control the sole 4 to rotate at a set angle and proceed to step S1. The preset number can be 2 segments, 3 segments, or 4 segments.

[0041] Step S5: Based on the pose relationship between the point clouds of each shoe sole edge segment, the point clouds of each shoe sole edge segment are stitched and fused to obtain a fused edge point cloud. This fused edge point cloud is then filtered using a Gaussian filtering algorithm, and interpolated using a multiple interpolation algorithm. The filtered and interpolated fused edge point cloud is then matched with template curves in the database. The template curve with the highest matching degree is selected, and elastic compensation and smoothing optimization are performed on the fused edge point cloud using the template curve. Finally, a shoe sole edge curve is generated based on the optimized point cloud. Figure 3 As shown. The database was obtained through shoe sole design drawings, manual drawing, and / or deep learning training methods.

[0042] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the present invention. All equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the specification of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A method for extracting shoe sole edges based on three-dimensional vision, characterized in that: Includes the following steps: Step S1: Use a 3D camera to capture a frame of shoe sole image, and perform 3D reconstruction based on the frame of shoe sole image to obtain a segment of shoe sole edge point cloud corresponding to the frame of shoe sole image; Step S2: Check whether the integrity of the point cloud at the edge of the sole meets the preset requirements. If yes, proceed to step S4; otherwise, proceed to step S3. Step S3: Locate the missing point cloud position, control the sole rotation according to the missing point cloud position, so that the position of the sole corresponding to the missing point cloud position is within the shooting range of the 3D camera, and use the 3D camera to take a supplementary image, perform three-dimensional reconstruction based on the supplementary image to obtain the supplementary point cloud, fill the missing point cloud position according to the supplementary point cloud, and proceed to step S2. Step S4: Determine whether the number of obtained shoe sole edge point cloud segments has reached the preset number. If yes, proceed to step S5; otherwise, control the shoe sole to rotate at a set angle and proceed to step S1. Step S5: Fuse the multi-segment shoe sole edge point cloud to obtain the fused edge point cloud. Match the fused edge point cloud with the template curve in the database, select the template curve with the highest matching degree, optimize the fused edge point cloud according to the template curve, and generate the shoe sole edge curve according to the optimized point cloud. In step S2, detecting whether the integrity of the point cloud at the edge of the sole meets the preset requirements specifically involves: counting the number of effective structured light encoded pixels P1 in the depth map corresponding to the sole image of the frame, and counting the number of point clouds at the edge of the sole P2. P2 / P1 is the point cloud density of the point cloud at the edge of the sole. If the point cloud density is less than the set first threshold, it is determined that the integrity of the point cloud at the edge of the sole does not meet the requirements.

2. The method for extracting shoe sole edges based on three-dimensional vision according to claim 1, characterized in that: In step S3, the location of missing point cloud positions and the control of sole rotation based on the missing point cloud positions are specifically as follows: the depth map corresponding to the sole image of the frame is divided into several regions, the number of effective structured light encoded pixels P3 in each region is counted, and the number of point clouds P4 in each region is counted. If P4 / P3 is less than the set second threshold, it is determined that the point cloud of that region is incomplete; a coordinate system is established with the sole rotation center as the origin, and for that region, the coordinate value corresponding to its geometric center is determined, and the sole rotation angle is determined based on the coordinate value.

3. The method for extracting shoe sole edges based on three-dimensional vision according to claim 2, characterized in that: In step S3, the point supplementation specifically involves: based on the relationship between the pose corresponding to the supplemented point cloud and the pose corresponding to the missing point cloud at the edge of the shoe sole, the supplemented point cloud is spliced ​​and merged into the point cloud at the edge of the shoe sole.

4. A method for extracting shoe sole edges based on three-dimensional vision according to claim 1, 2, or 3, characterized in that: In step S5, the point clouds of each shoe sole edge are spliced ​​and fused together based on the pose relationship between the point clouds of each shoe sole edge.

5. A method for extracting shoe sole edges based on three-dimensional vision according to claim 1, 2, or 3, characterized in that: After obtaining the fused edge point cloud, it is filtered by a Gaussian filtering algorithm, and then interpolated by a multiple interpolation algorithm.

6. A method for extracting shoe sole edges based on three-dimensional vision according to claim 1, 2, or 3, characterized in that: In step S5, optimizing the fused edge point cloud based on the template curve specifically involves: performing elastic compensation and smoothing optimization on the fused edge point cloud using the template curve.

7. A method for extracting shoe sole edges based on three-dimensional vision according to claim 1, 2, or 3, characterized in that: In step S5, the database is obtained through shoe sole design drawings, manual drawing, and / or deep learning training.

8. A method for extracting shoe sole edges based on three-dimensional vision according to claim 1, 2, or 3, characterized in that: The 3D camera includes a monocular structured light 3D camera or a binocular structured light 3D camera.

9. A method for extracting shoe sole edges based on three-dimensional vision according to claim 1, 2, or 3, characterized in that: The sole is placed on a rotating platform, which is located inside a support. A 3D camera is rotated and positioned on the support, diagonally above the sole, to take inward-tilted pictures of the sole. The rotating platform and the 3D camera are respectively connected to a control mechanism.

Images