A method for reconstructing a three-dimensional foot model and parameter measurement using a depth camera

Abstract

The application discloses a kind of three-dimensional foot model and parameter measurement method of reconstruction using depth camera, and point cloud is collected to foot by depth camera and is carried out point cloud preprocessing, and the point cloud after preprocessing is combined as a group two by two according to the source of collection, input PREDATOR network and is carried out rough matching, then NICP algorithm is used to carry out accurate matching, and the point cloud of accurate matching is fused into a point cloud;Surface point cloud of the point cloud after fusion is carried out smoothing processing, then alpha-shape surface reconstruction algorithm is used to obtain three-dimensional foot model;Data measurement is carried out to the point cloud after fusion, and the length, width, height and girth of foot are obtained.The application is based on four depth cameras as foot three-dimensional image acquisition device, so that hardware cost is reduced while accuracy is guaranteed within usable range;Rough matching is carried out using PREDATOR network and accurate matching is carried out using NICP algorithm, which is suitable for human foot point cloud, and the reconstruction accuracy is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional foot model measurement, and particularly relates to a three-dimensional foot model reconstruction method and parameter measurement method using a depth camera. BACKGROUND

[0002] When people buy shoes, they must go through the shoe size matching stage, and sometimes they have to try on repeatedly to determine the appropriate shoes. At present, the matching of shoe type codes is segmented according to sales, and in order to match the foot shapes of most people as much as possible, each standard shoe size is about 5-10 mm different. However, this often leads to people not being able to choose completely suitable shoes when choosing shoes. Moreover, there are some patients with deformed foot shapes in China, and the segmented shoe size design cannot meet their shoe wearing needs, and unsuitable shoes may even damage their knees and waists. Therefore, in order to more efficiently solve the personalized needs of shoes, it is necessary to introduce an automatic foot measurement device.

[0003] At present, three-dimensional reconstruction technology is continuously developing, and the reconstruction accuracy of objects is continuously improving, which makes the accuracy of measuring the three-dimensional foot model after three-dimensional reconstruction completely comparable to the accuracy of manual measurement. Since the skin of the human body is a flexible object, and the user will inevitably shake when standing, how to quickly and comprehensively obtain the three-dimensional information of the whole foot is a problem that needs to be solved in three-dimensional foot reconstruction. In three-dimensional reconstruction, a depth camera is usually used to obtain the distance and depth information between each part of the object and the lens. After obtaining the depth information at multiple angles using the depth camera, the depth information in each direction is integrated into the same coordinate, and the three-dimensional information of the object as a whole can be obtained.

[0004] In the prior art, there are usually the following ways to perform three-dimensional reconstruction and measurement on the foot: one is to perform three-dimensional reconstruction of the foot through a laser line scanning platform, which is commonly used in medical diagnosis and is expensive and large in size, and is not suitable for offline store promotion; two is to place a motor on the platform, rotate the depth camera, obtain the omnidirectional point cloud, and synthesize the same point cloud. However, since the human body is not static, in order to obtain stable images, the motor rotates slowly, and if the foot is moved during the waiting process of the camera rotation, the accuracy will be reduced; three is to place multiple fixed-position depth cameras on the platform, and use multiple cameras instead of the effect of motor rotation, but since high-precision depth cameras are relatively expensive, it is not possible to place too many devices, and when using point cloud matching algorithms on the point clouds collected by multiple depth cameras, the point clouds to be matched need to have a large amount of overlapping parts, and as the number of depth cameras increases, the cost also increases, in addition, the method of selecting fixed camera positions needs to be calibrated through pose calibration for reconstruction, which leads to the need to ensure that the assembly position of the camera is absolutely accurate during mass production, thereby increasing the production cost and difficulty. Therefore, the existing technology needs to be improved. SUMMARY

[0005] The technical problem solved by the present application is to provide a method for reconstructing a three-dimensional foot model and measuring parameters by using a depth camera, so as to quickly and accurately reconstruct a three-dimensional foot model and measure foot parameters.

[0006] To solve the above technical problems, the present application provides a method for reconstructing a three-dimensional foot model and measuring parameters by using a depth camera, and the specific process is as follows:

[0007] S1, a three-dimensional foot image acquisition device is built, which includes depth cameras in different acquisition directions, and then a calibration block is used to calibrate the depth cameras and synchronize them to the same coordinate system;

[0008] S2, the foot point cloud in different directions is acquired by the depth camera and sent to the upper computer for point cloud preprocessing, and the preprocessed point cloud in the same coordinate system is obtained;

[0009] S3, the preprocessed point cloud is combined into a group according to the acquisition source, input into the PREDATOR network for rough matching, the rough matching transformation matrix is obtained, and the rough matching point cloud in the same coordinate system is obtained by using the rough matching transformation matrix and the preprocessed point cloud;

[0010] S4, the rough matching point cloud is combined into a group according to the acquisition source, the NICP algorithm based on normal vector and curvature is used to accurately match each group of rough matching point cloud, the accurate matching transformation matrix is obtained, and the accurate matching point cloud in the same coordinate system is obtained by using the accurate matching transformation matrix to transform the rough matching point cloud; then the accurate matching point cloud is fused into one point cloud;

[0011] S5, the surface point cloud of the fused point cloud is smoothed, and then the alpha-shape surface reconstruction algorithm is used to obtain a three-dimensional foot model;

[0012] S6, the length, width, height and girth of the foot are obtained by measuring the data of the fused point cloud.

[0013] As an improvement of the method for reconstructing a three-dimensional foot model and measuring parameters by using a depth camera of the present application:

[0014] The foot three-dimensional image acquisition device in step S1 includes a square platform, two depth cameras are arranged on the square platform in front and back, and the front No. 1 camera, the front No. 2 camera, the rear No. 1 camera and the rear No. 2 camera are sequentially arranged in a clockwise direction from the upper left corner of the square platform. The front No. 1 camera and the front No. 2 camera are arranged at the upper left corner and the upper right corner of the square platform respectively, the rear No. 1 camera and the rear No. 2 camera are arranged at the bottom edge of the square platform and the included angle between the rear No. 1 camera and the rear No. 2 camera and the central axis of the square platform is 15°, and the vertex of the included angle is the center of the square platform; the lenses of the depth cameras all point to the center of the platform, and the lens height is 30 cm.

[0015] The two feet of the collected person are respectively stepped on the middle positions of the square platform and are arranged on the two sides of the central axis, the toes are towards the two depth cameras in front, and the heels are towards the two depth cameras in back. The foot point cloud collected by each depth camera includes left foot point cloud and right foot point cloud.

[0016] As a further improvement of the three-dimensional foot model reconstruction and parameter measurement method using depth cameras of the present application:

[0017] The specific method for calibrating the depth cameras to the same coordinate system by using the calibration block is:

[0018] S1.1, at least three calibration blocks are placed on the square platform, and the height of the calibration block is lower than the lens of the depth camera; three-dimensional point clouds of each calibration block are obtained by the front No. 1 camera, the front No. 2 camera, the rear No. 1 camera and the rear No. 2 camera respectively;

[0019] S1.2, Harris 3D corner point detection is performed on the three-dimensional point clouds of each calibration block at the same time to obtain the end points of the calibration block in the three-dimensional point cloud;

[0020] S1.3, the four depth cameras are calibrated two by two through the end points of the calibration block:

[0021] The coordinate transformation from the front No. 2 camera to the front No. 1 camera is:

[0022] C1=R1C2+T1 (6)

[0023] The coordinate transformation from the rear No. 1 camera to the front No. 2 camera is:

[0024] C2=R2C3+T2 (7)

[0025] The coordinate transformation from the rear No. 2 camera to the rear No. 1 camera is:

[0026] C3=R3C4+T3 (8)

[0027] The coordinate transformation from the rear No. 1 camera to the front No. 1 camera is:

[0028] C1=R1(R2C3+T2)+T1 (9)

[0029] The coordinate transformation from the second back camera to the first camera is:

[0030] C1=R1(R2(R3C4+T3)+T2)+T1 (10)

[0031] Wherein, C1, C2, C3 and C4 are the coordinates of the point clouds obtained by the first camera, the second camera, the third camera and the fourth camera respectively, the transformation matrix from the second camera to the first camera is R1, T1, the transformation matrix from the third camera to the second camera is R2, T2, and the transformation matrix from the fourth camera to the third camera is R3, T3.

[0032] As a further improvement of the application of the depth camera to reconstruct a three-dimensional foot model and a parameter measurement method:

[0033] The specific process of the point cloud preprocessing in step S2 is that the foot point clouds in different directions are all subjected to straight-through filtering and Euclidean clustering segmentation, and then synchronized to the same coordinate system according to formula (6)-formula (10), so as to obtain the preprocessed point clouds in the same coordinate system:

[0034] (1), straight-through filtering: first, specify a dimension and the value range in the dimension, second, traverse each point in the point cloud, judge whether the value of the point in the specified dimension is in the value range, delete the points whose values are not in the value range; the points left after the traversal are the point clouds after the straight-through filtering;

[0035] (2), Euclidean clustering segmentation: the straight-through filtered point cloud is segmented according to the Euclidean distance between points, the connected regions are sorted in descending order, and the connected regions containing more than 2000 points are retained, and the remaining connected regions are deleted as a large amount of fly noise in space.

[0036] As a further improvement of the application of the depth camera to reconstruct a three-dimensional foot model and a parameter measurement method:

[0037] The specific process of the accurate matching in step S4 is:

[0038] The features of each point in the two input rough matched point clouds are calculated, the matching point pairs in the two rough matched point clouds are found according to the distance and features of the points, and the conditions that the rotation angle is less than 5° and the Euclidean distance of displacement is less than 10mm are added; the least square method is used to minimize and minimize the objective function to solve the point cloud to obtain the two accurate matched transformation matrices.

[0039] As a further improvement of the application of the depth camera to reconstruct a three-dimensional foot model and a parameter measurement method:

[0040] The specific process of step S5 to obtain the three-dimensional foot model is as follows:

[0041] S5.1, using a moving least square method to fit the surface point cloud of the fused point cloud to achieve smoothing processing;

[0042] S5.2, the alpha-shape surface reconstruction algorithm is as follows: using two alpha spheres with a radius of alpha to roll on the inner and outer surfaces of the point cloud, when the trajectories of the two spheres rolling in and out of the two spheres have contact, any point in the two spheres satisfies that the distance from the center of the ball in one of the two ball trajectories is greater than alpha, then it is determined as a boundary, thereby obtaining the three-dimensional foot model.

[0043] As a further improvement of the three-dimensional foot model reconstruction and parameter measurement method using a depth camera of the present application:

[0044] The specific method of data measurement in step S6 is as follows:

[0045] S6.1, the square platform is used as the XOY plane of the three-dimensional coordinate system, the orientation of the toes is used as the positive direction of the y-axis, and the foot width is parallel to the x-axis, the fused point cloud includes the left foot point cloud and the right foot point cloud, and data measurement is performed on the left foot point cloud and the right foot point cloud respectively;

[0046] S6.2, traverse the left foot point cloud (or the right foot point cloud), the point with the smallest y value is determined as the heel point H(x h , y h , z h ), the point with the largest y value is determined as the second toe point T(x t , y t , z t ), and the heel point is used as the origin of the three-dimensional coordinate system, then the x, y and z values of all point clouds P(x i , y i , z i ) are subtracted by the x, y and z values of the point H, and the whole foot is translated to the origin:

[0047] P′=(x i -x h , y i -y h , z i -z h )(i=1,2,3...n) (12)

[0048] S6.3, according to the x and z values of the second toe point T, the foot point cloud is rotated around the axis to be exactly parallel to the y-axis, and the points T and H are on the y-axis:

[0049]

[0050] Rotate around the x-axis

[0051]

[0052] Rotate ω° around z axis;

[0053] S6.4, find the maximum point L(x l , y l , z l ) and the minimum point R(x r , y r , z r ) of x axis, rotate the foot point cloud around the angle to parallel to x axis:

[0054]

[0055] Rotate θ° around y axis;

[0056] S6.5, in the points between 0% and 30% of the distance between point T and point H, and the y axis is less than 5cm, find the maximum point HL(x hl , y hl , z hl ) and the minimum point HR(x hr , y hr , z hr ) of x axis, calculate the length, width, height and metatarsophalangeal girth of the foot:

[0057] (1), the foot length is the distance between point T and point H;

[0058] (2), the foot width is the distance between point L and point R;

[0059] (3), the heel width is the distance between point HL and point HR;

[0060] (4), according to the found points L and R, project the points with y value between the two points to XOY plane, extract the point cloud on the plane, find the outermost point on the plane by convex hull algorithm, connect them, calculate the distance between each point PP i (x i , y i , z i ) in clockwise direction, and the sum of the distance is the metatarsophalangeal girth:

[0061]

[0062] As a further improvement of the three-dimensional foot model reconstruction and parameter measurement method using depth camera:

[0063] The two-by-two combination according to the collection sources is specifically combining the first camera and the second camera, the second camera and the third camera, and the third camera and the fourth camera respectively.

[0064] The beneficial effects of the present application mainly embody in:

[0065] 1、The present application uses four depth cameras as a foot three-dimensional image acquisition device, introduces a new algorithm in the matching stage to improve the original algorithm, so that the hardware cost can be reduced while ensuring the accuracy within the usable range; the depth camera pose estimation is collected using a calibration block, which solves the problem of two-dimensional calibration of integrated depth cameras, and can be used on site without prior calibration of the depth camera on the production line, which is convenient and simple to use;

[0066] 2、The point cloud matching of the present application adopts a combination of coarse matching and accurate matching, coarse matching of point cloud is carried out through the PREDATOR network, which solves the problem of low point cloud overlap degree and difficult to complete coarse matching; the NICP algorithm is used instead of the traditional ICP algorithm for accurate matching, which is more suitable for human foot point cloud, and the reconstruction accuracy is higher;

[0067] 3、The reconstruction of the three-dimensional foot model of the present application uses the alpha-shape surface reconstruction method instead of Poisson reconstruction, which makes the algorithm more stable and the effect not worse than Poisson reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0068] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0069] Figure 1 It is a structure schematic view of a foot three-dimensional image acquisition device in a three-dimensional foot model reconstruction and parameter measurement method using a depth camera according to the present application;

[0070] Figure 2 It is a schematic view of Harris 3D corner point detection of a calibration block according to the present application;

[0071] Figure 3 It is an effect schematic view of end point obtained by Harris 3D corner point detection of a calibration block for iterative closest point algorithm matching according to the present application (the left figure is before matching, and the right figure is after matching);

[0072] Figure 4 It is an effect schematic view of point cloud before and after preprocessing according to the present application (the left figure is before point cloud preprocessing, and the right figure is after point cloud preprocessing);

[0073] Figure 5 It is a structure schematic view of PREDATOR network;

[0074] Figure 6 It is an effect schematic view of point cloud after accurate matching according to the present application;

[0075] Figure 7A three-dimensional foot model effect schematic diagram of the three-dimensional reconstruction of the present application

[0076] Figure 8 A photograph of a shoe last used in the experiment of the present application;

[0077] Figure 9 A three-dimensional reconstruction effect schematic diagram of a shoe last of the present application. DETAILED DESCRIPTION

[0078] The present application is further described below in conjunction with specific embodiments, but the scope of protection of the present application is not limited to this:

[0079] Example 1, a three-dimensional foot model reconstruction and parameter measurement method using a depth camera, the specific process is as follows:

[0080] 1, build a collection device

[0081] The present application uses a depth camera based on the Time-of-Flight principle to collect three-dimensional image information of the foot. The depth camera ranging principle is to send light pulses to the target continuously, and then use the sensor to receive the light returned from the object, and obtain the distance of the target object by detecting the flight (round trip) time of the light pulse. The depth camera can collect the distance information of each point of the foot, plus the X, Y coordinates on the 2D plane, to calculate the three-dimensional coordinates of each point, and all three-dimensional points are collected in the same three-dimensional space to obtain the foot point cloud.

[0082] The foot three-dimensional image collection device, as shown in Figure 1 , includes a square platform and four depth cameras, two depth cameras are arranged on the square platform in front and back, starting from the upper left corner of the square platform, in clockwise direction, they are front No. 1 camera, front No. 2 camera, rear No. 1 camera and rear No. 2 camera, front No. 1 camera and front No. 2 camera are respectively arranged on the left and right corners of the diagonal line of the square platform (i.e. the upper left corner and the upper right corner of the square platform), rear No. 1 camera and rear No. 2 camera are arranged near the bottom edge of the square platform and the included angle with the central axis of the square platform is 15° (the vertex of the included angle is the center of the square platform). The lenses of the four depth cameras all point to the center of the platform, and the lens height is 30 cm. When collecting, the feet of the collector are respectively stepped on the middle position of the square platform, and are arranged on both sides of the central axis, and the toes are towards the front No. 1 camera and the front No. 2 camera, and the heels are towards the rear No. 1 camera and the rear No. 2 camera, the point cloud collected by each depth camera includes left foot point cloud and right foot point cloud.

[0083] 2, pose estimation of the depth camera

[0084] Since the point cloud is obtained from four different direction depth cameras, the obtained point cloud is not in the same coordinate system. Meanwhile, since the depth camera used in the application uses the TOF principle, which measures the distance by calculating the time difference of the back and forth, it does not rely on the visible lens, so it is difficult to obtain the internal and external parameters of the camera using the traditional plane calibration board. In order to unify the coordinate system and solve the problem of difficult two-dimensional calibration, the application first uses the calibration block to estimate the coordinates of the camera. The calibration block is a rectangular block made of rigid material with accurate dimensions, commonly used in industrial production for the calibration of mechanical arms, that is, for correcting the true distance value of the object from the camera. It is because the position and coordinates of the calibration block are very accurate and will not move, and it has obvious corner points, that is, the vertices of the calibration block, so in the application, the two-dimensional calibration board can be replaced by the camera calibration, solving the problem of difficult two-dimensional calibration of integrated depth cameras. Since the calibration block is a standard part and is separated from the three-dimensional image acquisition device of the foot part of the application, the depth camera can be calibrated at any time, which is convenient and simple to use. The specific process is as follows:

[0085] 2.1, place multiple calibration blocks of different heights on the square platform, the height of the calibration block is lower than the lens of the four depth cameras, so that each depth camera can shoot the upper top surface of the calibration block, and the three-dimensional point cloud of each calibration block is obtained by using the first camera, the second camera, the third camera and the fourth camera;

[0086] 2.2, the three-dimensional point cloud of each calibration block obtained by each depth camera is detected by Harris 3D corner point detection at the same time, as shown in Figure 2 The idea of Harris 3D is to find the end point coordinates of the block by moving to detect the number change of the point cloud in the block, with the help of the normal of the discrete point cloud. The detection steps are:

[0087] (1), the normal of the point cloud is solved, and the normal covariance matrix is constructed, and the expression of the normal covariance matrix is

[0088]

[0089] Where M 3d is the covariance matrix, n is the number of points in the point cloud, n x , n y , n z are the normal vectors of the points in x, y and z directions respectively;

[0090] (2), the corner point response value of each point in the point cloud is calculated according to the corner point response function, and the calculation formula is

[0091] R 3d = detM 3d -0.04(traceM 3d )2 (2)

[0092] where detM 3d = n x 2 n y 2 n z 2 , traceM 3d = n x 2 + n y 2 + n z 2 - n x 2 (n y n z ) 2 - n y 2 (n z n x ) 2 - n z 2 (n x n y ) 2 , R 3d is the corner response value;

[0093] (3) Set a corner response threshold, when there is a point whose R 3d is greater than the corner response threshold, and it is also a local maximum point, it is considered that the normal vector has a large change in x, y, z directions, and it is the most obvious point relative to the surrounding points, then it is determined that the point is a corner point of the point cloud; otherwise, the current point is not a corner point.

[0094] Through Harris 3D corner detection, the end points of the calibration block can be found in the three-dimensional point cloud of the calibration block obtained by the four cameras.

[0095] 2.3, after extracting the end points of the calibration block in each three-dimensional point cloud, the four depth cameras are calibrated two by two

[0096] Since the position of the calibration block in the world coordinate system is fixed, the found end points are matched by the iterative closest point algorithm (ICP), and the effect is shown in Figure 3 , and the formula is as follows:

[0097]

[0098] where the end point set before matching is P s , and after matching is P t , where and This involves matching corresponding points in the set of front-end points and the set of back-end points. The camera's transformation matrix in the spatial coordinate system includes R and T, where R is the rotation matrix and T is the translation matrix. Assuming a rotation of α° around the x-axis results in a translation of t1 on the x-axis, a rotation of β° around the y-axis results in a translation of t2 on the y-axis, and a rotation of γ° around the z-axis results in a translation of t3 on the z-axis, the specific formula is:

[0099]

[0100] R = R x (θ)·R y (θ)·R z (θ)

[0101]

[0102] Assume the coordinates of the point cloud of the calibration block acquired by the first camera, the second camera, the first camera, and the second camera are C1, C2, C3, and C4, respectively. The transformation matrix from the second camera to the first camera is R1, T1, the transformation matrix from the first camera to the second camera is R2, T2, and the transformation matrix from the second camera to the first camera is R3, T3.

[0103] First, calibrate the first and second cameras:

[0104] The point clouds of the calibration blocks acquired by the first and second cameras are as follows: Figure 3 As shown on the left, solid circles mark point cloud 1, which is the point cloud acquired by the first camera, while dashed circles mark point cloud 2, which is the point cloud acquired by the second camera. The original coordinates of each discrete point in the point clouds acquired by the second camera are [x′, y′, z′]. T Multiply the rotation matrix R1 on the left by the original coordinates, and then add it to the translation matrix T1 to obtain the new coordinates [x, y, z]. T (The principle behind all subsequent applications of transformation matrices is the same).

[0105]

[0106] That is, the coordinate transformation from the second camera to the first camera is:

[0107] C1=R1C2+T1 (6)

[0108] At this point, the point clouds acquired by the first two cameras and the point cloud acquired by the first camera are roughly located on the same coordinate system.

[0109] Similarly, pairwise calibration is performed between the first two cameras and the last camera, and between the last camera and the second last camera:

[0110] The coordinate transformation from the second camera to the first two cameras is as follows:

[0111] C2 = R2C3 + T2 (7)

[0112] The coordinate transformation from the second back camera to the first back camera is:

[0113] C3 = R3C4 + T3 (8)

[0114] Thus, the coordinate transformation from the first back camera to the first front camera is:

[0115] C1 = R1(R2C3 + T2) + T1 (9)

[0116] The coordinate transformation from the second back camera to the first front camera is:

[0117] C1 = R1(R2(R3C4 + T3) + T2) + T1 (10)

[0118] At this point, the coordinate systems of the first front camera, the second front camera, the first back camera, and the second back camera are unified (the order of matching the point clouds obtained in each direction in the rough matching and accurate matching below is also the same), and the entire coordinate system is unified, which reduces the calculation amount for the subsequent point cloud processing, provides a better reference position for point cloud matching, and improves the operation speed.

[0119] 3. Point cloud preprocessing

[0120] After the coordinate system of the depth camera is unified, the basic construction of the device is completed. After obtaining the point cloud using the depth camera, it is found that there are not only foot point clouds in space, but also a large amount of invalid background and a large number of fly noise in the distance. In order to reduce the matching error in the following, the foot point clouds collected by the four depth cameras need to be preprocessed respectively.

[0121] 3.1. By using the method of straight-through filtering, the useless background and ground point clouds are removed, and the number of point clouds is reduced. First, a dimension and the value range under the dimension are specified, second, each point in the point cloud is traversed, it is judged whether the value of the point in the specified dimension is within the value range, the point whose value is not within the value range is deleted, finally, the traversal is ended, and the remaining points constitute the filtered point cloud;

[0122] 3.2. Use Euclidean clustering segmentation, segment the straight-through filtered point cloud according to the Euclidean distance between points, sort the connected regions from large to small, and retain the connected regions containing more than 2000 points, i.e. the left and right feet of the person, and delete the remaining connected regions, i.e. a large number of fly noise in the air; Figure 4 As shown in the left image, the processing result is as shown in the right image; Figure 4

[0123] ​3.3、Four foot point clouds collected by four depth cameras are respectively subjected to straight-through filtering and Euclidean clustering segmentation, and then are synchronized to the same coordinate system according to the formulae (6)-(10) in step 2.3, so as to obtain the preprocessed point clouds in the same coordinate system.

[0124] 4、Point cloud rough matching

[0125] Point cloud matching refers to that due to the geometric characteristics of the object itself, the whole point cloud information needs to be acquired by shooting point clouds from multiple directions, and the coordinate systems of these point clouds are different, so the point clouds acquired under each view angle need to be converted in coordinates and unified to the global coordinate system. The point cloud matching process refers to that a linear transformation is obtained, which, when acting on one of the two point clouds, can make the coordinate systems of the two point clouds sufficiently unified. The process of point cloud matching generally has two steps of rough matching and accurate matching, because the gradient descent iterative algorithm is generally used in the accurate matching algorithm to obtain the transformation matrix, and for point clouds with unclear features, it is easy to fall into a local optimal solution and cause matching failure. In order to solve this problem, the point clouds must be roughly matched before accurate matching to provide a good initial position for accurate matching and avoid falling into a local optimal solution, that is, the process of rough matching.

[0126] The point cloud matching in industrial production generally directly uses the ICP algorithm for matching, which will cause problems in the foot matching done by the present application. This is because since four point clouds in different directions are acquired, the overlapping part of each point cloud is only about 30%, so the use of the ICP algorithm will cause serious mismatching, resulting in instability of the whole algorithm, for example, the ICP algorithm identifies the left and right sides of the foot point cloud as the same side, thereby causing mismatching.

[0127] To solve this problem, the present application first uses the PREDATOR network based on deep learning for rough matching. The structure of the PREDATOR network is as follows: Figure 5The PREDATOR (PREDiction of 3D point clouds with low overlap) is a method for registration of 3D point clouds with low overlap, as shown in (Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, Konrad Schindler, ETH Zurich). The main principle of PREDATOR is to use early information exchange between two point cloud encodings, which can extract features point by point to predict which points are feature points in the overlapping part and then match them. Therefore, PREDATOR is a good solution for the application of the present application. First, the acquired point cloud is down-sampled once to make the point cloud more uniform, and then the voxel grid point cloud P and Q are sent to the encoder, and the encoder extracts special points P' and Q' and their latent features X P′ Q′ The overlap attention module uses a series of graph convolution modules (GNN) and cross-attention module (CA) blocks to update the features using the common context information and map them to the overlap score o P′ Q′ and the cross overlap score Finally, the decoder converts the conditional features and the overlap score into the point-wise feature descriptor F P Q , the overlap score o P Q and the matching score m P Q .

[0128] The steps for training the PREDATOR network are as follows:

[0129] 4.1, use a separate depth camera to acquire point clouds of 20 directions of the foot as a basic data set, since the direction of the point cloud in three-dimensional space is uncertain, the point cloud is rotated in the x, y, z three-axis to enhance the training data, and fly noise is added on the point cloud, and Gaussian filtering is used as an extended training set, and the basic training set and the extended training set are combined into the same training data set.

[0130] 4.2, build the training environment of PREDATOR: use Ubuntu system and Python language to create a deep learning environment Predator, import PyTorch, Open3D, Scikit-leam, MinkowskiEnginge and other dependent libraries on the terminal, and use RTX 1080ti graphics card for CUDA acceleration;

[0131] ​​​​​4.3, from the training data set obtained in step 4.1, select point clouds with an overlap degree of 30%-50% as a point cloud pair to input the PREDATOR network to obtain the transformation matrix R and T,

[0132] 4.4 Obtain the matching error:

[0133]

[0134] wherein Ω * represents the number of points in the point cloud pair that are considered to be the same feature point, x * , y * represents a pair of points belonging to Ω * R and T represent the transformation matrix obtained by training the network in step 4.3.

[0135] Apply R and T to x * to calculate the Euclidean distance with y * , and divide all the Euclidean distances by Ω * , then take the square root to obtain E RMSE . If E RMSE <0.2, it indicates a successful match, and if the number of successfully matched point cloud pairs is more than 90% of all point cloud pairs, the training is completed and the training weights are saved.

[0136] The preprocessed point cloud obtained in step 3 is input into the trained PREDATOR network for coarse matching according to the combination of the first camera and the second camera, the second camera and the third camera, and the third camera and the fourth camera, and three coarse matching transformation matrices are obtained. Then, the coarse matching transformation matrix is transformed with the corresponding preprocessed point cloud, and finally four coarse matching point clouds in the same coordinate system are obtained.

[0137] 5. Precise matching of point clouds

[0138] Due to the difference between individual feet, precise matching must be performed after coarse matching. Since the surface of the human foot can be approximated as a smooth curved surface, the improved ICP algorithm based on normal vectors and curvature, i.e., the Normal ICP (NICP algorithm), is used for precise matching. The NICP algorithm is characterized in that, when matching two point clouds, it does not consider the Euclidean distance between the matching point clouds, but uses the local features of the point cloud surface as the matching criterion for point pairs and the transformation calculation.

[0139] ​The rough matched point cloud obtained in step 4 is taken as a group of inputs for accurate matching according to the two-by-two combination of the first camera and the second camera, the second camera and the third camera, and the third camera and the fourth camera, and then three accurate matching transformation matrices are obtained through accurate matching, and then the accurate matching transformation matrices are transformed with the corresponding rough matched point clouds, and finally four accurate matched point clouds in the same coordinate system are obtained.

[0140] The specific process of accurate matching is as follows: the features of each point in the two input rough matched point clouds, i.e. the normal and curvature of the field of each point, are calculated to mark each point; the matching point pairs in the two input rough matched point clouds are found according to the distance and features of the points, and since rough matching has been completed, the transformation matrix for accurate matching will not produce large data, and in order to avoid iteration into local optimum, the conditions of limiting the rotation angle to less than 5° and the Euclidean distance of displacement to less than 10 mm are added in the program; the least square method is used to minimize the objective function to solve the accurate matching transformation matrix. The objective function here includes point-plane projection and normal rotation error.

[0141] The effect of the accurate matched point cloud is shown in Figure 6 , and then the four point clouds obtained through accurate matching are fused into one point cloud.

[0142] 6. Three-dimensional reconstruction

[0143] The point cloud is composed of a large number of unordered points, in order to achieve good reconstruction effect and facilitate the user to observe, the point cloud needs to be three-dimensionally reconstructed to be planarized:

[0144] 6.1. The mobile least square method (mls) is used to fit the surface point cloud of the fused point cloud in step 5 to achieve the purpose of smoothing the point cloud;

[0145] 6.2. Since the foot does not have a bottom feature, the traditional Poisson reconstruction effect is poor, and the shape of the section will appear, so the alpha-shape surface reconstruction algorithm is used in the present application, the principle of which is that two alpha spheres with a radius of alpha are used to roll on the inner and outer surfaces of the point cloud, and when the trajectories of the two spheres inside and outside the two spheres have contact, any point in the two spheres satisfies that the distance from the center of the ball in the trajectory to the center of the ball is greater than alpha, then it is determined as the boundary, so the foot can be reconstructed without matching the bottom surface, thereby obtaining a three-dimensional foot model for visual presentation to the user.

[0146] The final effect of the three-dimensional foot model after three-dimensional reconstruction is shown in Figure 7 .

[0147] 7. Data measurement

[0148] Since the three-dimensional model after three-dimensional reconstruction obtained in step 6 loses the information of points in the point cloud, the fused point cloud obtained in step 5 is used for data measurement, and the steps are as follows:

[0149] 7.1, determine the coordinate axis, take the direction of the toes as the positive direction of the y axis, and take the foot width as approximately parallel to the x axis, that is, take the stepped square platform as the XOY plane on the three-dimensional coordinate system; take a plane perpendicular to the XOY plane and passing through the centroid of the fused point cloud, and divide the fused point cloud into left foot point cloud and right foot point cloud for measurement respectively, and the measurement methods of the left foot point cloud and the right foot point cloud are the same;

[0150] 7.2, traverse the left foot (or right foot) point cloud, find the point with the minimum y value as the heel point H(x h , y h , z h ), find the point with the maximum y value as the second toe point T(x t , y t , z t ), and subtract the x, y and z values of point H from the x, y and z values of all point clouds P(x i , y i , z i ), so that the whole foot is moved to the origin, and the heel point is the origin:

[0151] P′=(x i -x h , y i -y h , z i -z h )(i=1,2,3...n) (12)

[0152] 7.3, rotate the foot point cloud around the axis according to the x and z values of point T to be exactly parallel to the y axis, and points T and H are on the y axis:

[0153]

[0154] Rotate around the x axis

[0155]

[0156] Rotate around the z axis by ω°

[0157] 7.4, find the maximum point L(x l , y l , z l ) and the minimum point R(x r , y r , z r ) of the x axis, and rotate the foot point cloud around the included angle to be parallel to the x axis:

[0158]

[0159] Rotate θ° around y axis

[0160] 7.5, in the distance of 0%~30% between point T and point H, and the point of y axis less than 5cm, find the maximum point HL(x hl , y hl , z hl ) and the minimum point HR(x hr , y hr , z hr ) of x axis; according to the special points, lines and surfaces extracted, calculate the length, width and height of the foot and the metatarsophalangeal girth:

[0161] (1), the foot length is the distance between point T and point H;

[0162] (2), the foot width is the distance between point L and point R;

[0163] (3), the heel width is the distance between point HL and point HR;

[0164] (4), according to the special points L and R found, project the points between the two points in y value to XOY plane, extract the point cloud on the plane, find the outermost points on the plane with convex hull algorithm, connect them, calculate the distance between each point PP i (x i , y i , z i ) in clockwise direction, and the sum of the distances is the metatarsophalangeal girth:

[0165]

[0166] 8, online use

[0167] The tested person stands in the center area of the square platform of the foot three-dimensional image acquisition device, the feet are located on the left and right sides of the central axis of the square platform, then the four depth cameras of the foot three-dimensional image acquisition device respectively collect point clouds in four directions of the feet and send them to the upper computer for processing; in the upper computer, the point clouds in four directions are preprocessed according to step 3 to obtain preprocessed point clouds in the same coordinate system, so as to reduce the influence of noise; then the PREDATOR network trained in step 4 is used to perform rough matching on the four preprocessed point clouds, obtain a transformation matrix after rough matching, and use the transformation matrix after rough matching to transform the preprocessed point clouds correspondingly to obtain point clouds after rough matching, so as to provide a better initial state for accurate matching. Then, the improved ICP algorithm based on normal vector and curvature (NICP) is used to accurately match the four point clouds after rough matching, obtain three transformation matrices after accurate matching, and use the transformation matrices after accurate matching to transform the corresponding point clouds after rough matching to obtain four point clouds after accurate matching in the same coordinate system; then the four point clouds after accurate matching are fused into one point cloud; the fused point cloud is smoothed and reconstructed by alpha-shape to obtain a three-dimensional foot model, and then the method of step 7 is used to calculate the length, width, height and circumferences of each part of the foot based on the fused point cloud, so as to complete the reconstruction of the three-dimensional foot model and the measurement of the foot parameters.

[0168] Experiment:

[0169] The foot parameters obtained by the three-dimensional foot model reconstruction and measurement method of the application using a depth camera are compared with the foot parameters obtained by manual measurement to verify the accuracy of the foot parameters obtained by the application. Since human skin is an elastic material, manual measurement of the human foot will have a large error, so 9 shoe lasts of different sizes are used instead of real human feet. The shoe lasts are made of hard plastic and imitate the shape of real human feet, so they will not deform significantly during measurement, minimizing measurement error. The photos of the shoe lasts are shown in Figure 8 The three-dimensional foot model after reconstruction is shown in Figure 9 To reduce the influence of measurement error, the measurement is performed 5 times and the average value is taken as the final manual and application measurement data, as shown in Table 1.

[0170] Table 1 Comparison of manual measurement and application measurement data

[0171]

[0172] Then, the manual measurement values in Table 1 are subtracted from the measurement values obtained by the three-dimensional foot model reconstruction and parameter measurement method of the application using a depth camera as the absolute error of manual measurement and the application, and the average and standard deviation of the absolute error are counted, as shown in Table 2.

[0173] Table 2 Error Absolute Value Table

[0174]

[0175] According to Table 2, it can be seen that the average error of the selected shoe size main parameters foot length, foot width and heel width is less than 2mm, and the standard deviation is less than 1.5mm. The metatarsophalangeal girth length is mainly measured by wrapping a tape around the foot, and the metatarsophalangeal features of the shoe last are not as obvious as the actual foot, which will cause a larger error, so the standard deviation of the absolute error will be higher, but the overall error is also less than 3mm, and the accuracy is sufficient for shoe size measurement.

[0176] Finally, it should be noted that the above only lists several specific embodiments of the application. Obviously, the application is not limited to the above embodiments, and there can be many variations. All variations that can be directly derived or inferred from the disclosed content by those of ordinary skill in the art should be considered within the scope of the application.

Claims

1. A method for reconstructing a 3D foot model and measuring its parameters using a depth camera, characterized in that... The process includes the following: S1. Build a foot 3D image acquisition device, which includes depth cameras with different acquisition directions. Then, use a calibration block to calibrate and synchronize the depth cameras to the same coordinate system. S2. Use a depth camera to collect foot point clouds from different directions and send them to the host computer for point cloud preprocessing to obtain preprocessed point clouds in the same coordinate system. S3. The preprocessed point cloud is grouped into pairs according to the acquisition source, and input into the PREDATOR network for coarse matching to obtain the coarse matching transformation matrix. The coarse matching transformation matrix is ​​then used to transform the preprocessed point cloud to obtain the coarse matching point cloud in the same coordinate system. S4. The coarsely matched point clouds are combined in pairs according to the acquisition source. The NICP algorithm based on normal vector and curvature is used to perform precise matching on each pair of coarsely matched point clouds to obtain the precise matching transformation matrix. The precise matching transformation matrix is ​​then used to transform the coarsely matched point clouds to obtain precisely matched point clouds located in the same coordinate system. Then, the precisely matched point clouds are fused into a single point cloud. The specific process of exact matching is as follows: Calculate the features of each point in the two coarsely matched point clouds, find matching point pairs in the two coarsely matched point clouds based on the distance and features of the points, and add the condition that the rotation angle is less than 5° and the Euclidean distance of the displacement is less than 10mm; use the least squares method to minimize the objective function to solve the point cloud and obtain the transformation matrix of the two precisely matched points. S5. Smooth the surface point cloud of the fused point cloud, and then use the alpha-shape surface reconstruction algorithm to obtain the three-dimensional foot model; The specific process of obtaining the three-dimensional foot model is as follows: S5.

1. Use the moving least squares method to fit the surface point cloud of the fused point cloud to achieve smoothing. S5.2 The alpha-shape surface reconstruction algorithm is as follows: Two spheres with radius alpha are rolled on the inner and outer surfaces of the point cloud. When the trajectories of the two spheres are in contact, any point on the two spheres is determined to be a boundary if the distance to the center of one of the spheres is greater than alpha, thereby obtaining the three-dimensional foot model. S6. Perform data measurement on the fused point cloud to obtain the length, width, height, and circumference of the foot.

2. The method for reconstructing a 3D foot model and measuring parameters using a depth camera according to claim 1, characterized in that: The foot 3D image acquisition device described in step S1 includes a square platform with two depth cameras positioned at the front and two at the back. Starting from the upper left corner of the square platform, the cameras are arranged clockwise as follows: front camera 1, front camera 2, rear camera 1, and rear camera 2. The front camera 1 and front camera 2 are located at the upper left and upper right corners of the square platform, respectively. The rear camera 1 and rear camera 2 are located on the bottom edge of the square platform and form an angle of 15° with the central axis of the square platform. The vertex of the angle is the center of the square platform. The lenses of the depth cameras all point towards the center of the platform, and the lens height is 30cm. The subject's feet were positioned in the middle of a square platform, on either side of the central axis, with the toes pointing towards the two depth cameras in front and the heels towards the two depth cameras behind. Each depth camera captured a point cloud of the feet, including the point cloud of the left foot and the point cloud of the right foot.

3. The method for reconstructing a 3D foot model and measuring parameters using a depth camera according to claim 2, characterized in that: The specific method for calibrating and synchronizing the depth camera to the same coordinate system using a calibration block is as follows: S1.1 At least three calibration blocks are placed on the square platform, and the height of each calibration block is lower than the lens of the depth camera; the three-dimensional point cloud of each calibration block is obtained by the first front camera, the second front camera, the first rear camera, and the second rear camera respectively; S1.2 Simultaneously perform Harris3D corner detection on the 3D point cloud of each calibration block to obtain the endpoints of the calibration block in the 3D point cloud; S1.

3. Perform pairwise calibration of the four depth cameras using the endpoints of the calibration block: The coordinate transformation from the second-to-first camera to the first-to-first camera is as follows: C1=R1C2+T1 (6) The coordinate transformation from the second camera to the first two cameras is as follows: C2=R2C3+T2 (7) The coordinate transformation from the second camera to the first camera is as follows: C3=R3C4+T3 (8) The coordinate transformation from the second camera to the first camera is as follows: C1=R1(R2C3+T2)+T1 (9) The coordinate transformation from the second camera to the first camera is as follows: C1=R1(R2(R3C4+T3)+T2)+T1 (10) Wherein, C1, C2, C3 and C4 are the coordinates of the point cloud acquired by the first camera, the second camera, the last camera and the second camera, respectively. The transformation matrix from the second camera to the first camera is R1,T1, the transformation matrix from the last camera to the second camera is R2,T2, and the transformation matrix from the second camera to the last camera is R3,T3.

4. The method for reconstructing a three-dimensional foot model and measuring parameters using a depth camera according to claim 3, characterized in that: The specific process of point cloud preprocessing in step S2 is as follows: the foot point clouds in different directions are all processed by pass-through filtering and Euclidean clustering, and then synchronized to the same coordinate system according to equations (6)-(10), so as to obtain the preprocessed point cloud on the same coordinate system: (1) Pass-through filtering: First, specify a dimension and the value range under that dimension. Second, traverse each point in the point cloud and determine whether the value of the point in the specified dimension is within the value range. Delete points whose values ​​are not within the value range. The points left after the traversal are the point cloud after the pass-through filtering. (2) Euclidean clustering segmentation: The point cloud after the through-filter is segmented according to the Euclidean distance between the points. The connected regions are sorted from largest to smallest, and the connected regions with more than 2000 points are retained. A large amount of fly noise in the air in the remaining connected regions is deleted.

5. The method for reconstructing a three-dimensional foot model and measuring parameters using a depth camera according to claim 4, characterized in that: The specific method for data measurement in step S6 is as follows: S6.1 The square platform is used as the XOY plane in the three-dimensional coordinate system. The direction of the toes is taken as the positive direction of the y-axis, and the foot width is parallel to the x-axis. The fused point cloud includes the left foot point cloud and the right foot point cloud. Data measurements are performed on the left foot point cloud and the right foot point cloud respectively. S6.

2. Traverse the left foot point cloud (or right foot point cloud), and define the point with the smallest y-value as the heel point H(x). h ,y h ,z h The point with the largest y-value is the second toe point T(x). t ,y t ,z t The heel point is taken as the origin of the three-dimensional coordinate system, and then all point clouds P(x) are calculated. i ,y i ,z i Subtracting the x, y, and z values ​​of point H from the x, y, and z values ​​of the foot will shift the entire foot to the origin. P′=(x i -x h ,y i -y h ,z i -z h )(i=1,2,3…n) (12) S6.

3. Based on the x and z values ​​of the second toe point T, rotate the foot point cloud around the axis until it is precisely parallel to the y-axis, with points T and H both lying on the y-axis: Rotate about the x-axis Rotate ω° around the z-axis; S6.4 Find the maximum point L(x) on the x-axis. l ,y l ,z l Points and minimum points R(x) r ,y r ,z r Rotate the foot point cloud around its included angle until it is parallel to the x-axis: Rotate θ° around the y-axis; S6.

5. Among the points within 0% to 30% of the distance between points T and H, and where the y-axis distance is less than 5cm, find the point HL(x) with the maximum x-axis distance. hl ,y hl ,z hl Minimum and HR point (x) hr ,y hr ,z hr Calculate the length, width, height, and metatarsophalangeal circumference of the foot: (1) The length of the foot is the distance between points T and H; (2) The foot width is the distance between points L and R; (3) The heel width is the distance between point HL and point HR; (4) Based on the found points L and R, project the points whose y-values ​​are between the two points onto the XOY plane, extract the point cloud on the plane, use the convex hull algorithm to find the outermost points of the plane, connect them, and calculate the PP of each point in a clockwise direction. i (x i ,y i ,z i The distance between them, the sum of the distances is the metatarsophalangeal circumference:

6. The method for reconstructing a three-dimensional foot model and measuring parameters using a depth camera according to claim 5, characterized in that: The specific combination based on the source of data acquisition is as follows: combining the first camera with the second camera, the second camera with the last camera, and the last camera with the second camera.

Images