A depth camera based three-dimensional foot shape measurement system

Abstract

This invention provides a three-dimensional foot measurement system based on a depth camera. The technical solution involves using a depth camera to obtain at least six images of each foot, including at least three images on each side of the midline of each foot. A data processing module performs the following processing on the images obtained for each foot: first, register the images on the same side; then, register the images on opposite sides to obtain a reconstructed foot image; finally, use the reconstructed foot image to obtain foot shape parameters. This invention has high accuracy. Simultaneously, it has low computation time and good real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional reconstruction technology, and relates to a measurement system that uses a camera to perform three-dimensional reconstruction of foot shape and then calculates relevant parameters of foot shape. Background Technology

[0002] In recent years, with the development of e-commerce platforms such as Taobao and JD.com, online shopping has become a lifestyle. However, receiving shoes purchased online often results in the shoes not fitting or differing from the seller's pictures, requiring returns and exchanges, affecting normal use, causing inconvenience for both consumers and merchants, and wasting logistical resources. Special groups such as military personnel, professional athletes, people with foot defects, and the elderly require customized footwear services to meet their specific needs. Furthermore, with the improvement of people's living standards and aesthetic tastes, online personalized footwear is expected to become a new development trend.

[0003] The existing three-dimensional foot measurement system mainly uses fully automatic three-dimensional foot scanners to measure foot shape, such as the Footscan 3D foot scanner from Jingyixun Company and the JD-Foot 3DScanner foot three-dimensional scanner from Emaisi Intelligent Technology Company, etc. See reference 1: [1] Ye Xiaolu, Shi Kai. Research on the application of three-dimensional foot scanning technology in footwear products [J]. China Leather, 2015. They use multiple lasers, multiple lenses and multiple sensors to work synchronously. Although the product development technology is perfect and there is no need to attach markers, the disadvantages are also significant. Scanning based on laser points or lines takes a long time, and long-term contact with lasers is also harmful to the body. In addition, the use of multiple sensors is expensive. For some small and micro enterprises or even individual retailers, the cost of purchasing a fully automatic three-dimensional foot scanner is high. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a three-dimensional foot measurement system based on a depth camera. The three-dimensional foot measurement system is characterized by moderate price, simple structure, high security, high accuracy, small size and easy operation. It can not only allow consumers to purchase or customize footwear products online, but also provide footwear customization services online and offline for shoe manufacturers / sellers.

[0005] The technical solution of the present invention is a three-dimensional foot measurement system based on a depth camera, characterized in that it includes a depth camera and a data processing module.

[0006] At least six images of each foot are obtained using a depth camera; the images are obtained by taking at least three images on each side of the midline, which is the line connecting the midpoint of the tip of the second toe and the midpoint of the heel; images on the same side of the midline are called ipsilateral images; images on different sides of the midline are called heterolateral images.

[0007] The data processing module implements the following process:

[0008] The images obtained for each foot are processed as follows:

[0009] First, the images on the same side are registered; then the images on the opposite side are registered to obtain a reconstructed image of the foot; the parameters of the foot type are obtained using the reconstructed image.

[0010] Furthermore, the original images obtained for each foot are downsampled and preprocessed, with the preprocessing including background segmentation, discrete point filtering, and image smoothing.

[0011] Furthermore, when registering images on the same side, a coarse registration process is performed first, followed by a fine registration process.

[0012] Furthermore, when registering images from opposite sides, a coarse registration process is performed first, followed by a fine registration process.

[0013] The present invention has the following advantages and effects compared with the prior art:

[0014] (1) High reliability. Experiments show that the parameter error of the foot shape obtained using this invention can be controlled within 5mm, and the system has a high accuracy. At the same time, the calculation time of this system is less than 5 minutes, and the real-time performance is good.

[0015] (2) Low price. Existing 3D foot measurement systems are expensive and vary in price, while our 3D foot measurement system costs only 1 / 10 of the price of a depth camera.

[0016] (3) Small size. The three-dimensional foot measurement system provided by this invention can be implemented using only a handheld depth camera and a data processing module. The data processing module is implemented using hardware such as a DSP and is fixed to the handheld depth camera. Therefore, the system provided by this invention is much smaller than a large fixed three-dimensional foot measurement system.

[0017] (4) Easy to operate. Simply follow the simple shooting rules to collect three-dimensional surface data of the human foot shape, start the data processing module, and you can achieve accurate reconstruction of the three-dimensional foot shape. Attached Figure Description

[0018] Figure 1A schematic diagram of the process for calculating foot type parameters using the measurement system provided by this invention;

[0019] Figure 2 This is a schematic diagram of the original image acquisition.

[0020] Figure 3 This is a schematic diagram of the same-side registration process for the data processing module.

[0021] Figure 4 This is a schematic diagram of the cross-side registration process in the data processing module;

[0022] Figure 5 This is a schematic diagram illustrating the reconstruction effect of the shoe shape by the data processing module of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention, and the specific implementation is not limited thereto.

[0024] Figure 1 This is a flowchart illustrating the process of calculating foot type parameters using the measurement system provided by this invention. As shown, it includes six steps. The following is a detailed explanation... Figure 1 The present invention will be described in detail below.

[0025] First, use a depth camera to obtain at least six raw images of each foot. The shooting environment must meet the following three requirements: general indoor lighting conditions; a flat, non-reflective surface (it is recommended to place a flat piece of white paper under the subject's feet); and no other objects in the field of view besides the subject's bare feet. During the acquisition of raw images for each foot, use the line connecting the midpoint of the second toe and the midpoint of the heel as the center line, and take at least three images on each side of this center line. For example, to capture six images... Figure 2 As shown, six images of the left foot were obtained. The three images taken to the left of the center line were taken at angles of approximately 45° to the left front, left center, and left back, respectively. The three images taken to the right of the center line were taken at angles of approximately 45° to the right front, right center, and right back, respectively.

[0026] Secondly, the acquired raw images are downsampled. The purpose of downsampling is to improve computational efficiency and ensure real-time performance when dealing with large amounts of data. A voxel grid filter method is recommended for downsampling. In this embodiment, the depth camera is a MEMS-3D sensor, and a 40x subtraction reduces the number of pixels in a single raw image from millions to over 20,000. Experiments show that as long as the downsampled image contains over 20,000 pixels, the subsequent foot shape parameter calculations can be completed.

[0027] Secondly, the downsampled image undergoes downsampled image preprocessing. This preprocessing includes background segmentation, discrete point filtering, and image smoothing. Background segmentation refers to removing all objects from the image except for the feet, especially the ground where the feet are located. In this invention, if the method recommended above is used to acquire the original image, the background is relatively clean, free of debris, with only a piece of white paper needing segmentation and removal. The recommended background segmentation method is as follows: first, use the RANSAC (Random Sample Consensus) method to find a planar model; then, rotate the image using a rigid body transformation matrix to filter out the ground background, thus completing the background segmentation. Discrete point filtering mainly removes singular points unrelated to the feet, using statistical filtering. Image smoothing is used to filter out noise interference caused by the properties of the measured object, the depth camera itself, and uneven lighting in the application environment, using Gaussian filtering. It is worth noting that the background segmentation process involved in this invention includes the following steps:

[0028] (1) Using the RANSAC method, calculate the coefficients A, B, C, and D of the planar model corresponding to a downsampled image. The planar model is AX+BY+CZ+D=0. The coordinate system of the planar model is as follows: the line connecting the midpoint of the tip of the second toe and the midpoint of the heel is the midline, the midpoint of the midline is the origin, the straight line containing the midline and pointing to the midpoint of the tip of the second toe is the X-axis, the straight line perpendicular to the X-axis and coinciding with the plane of the foot is the Y-axis, and the straight line perpendicular to the plane of the foot and pointing upward is the Z-axis. The X-axis, Y-axis and Z-axis constitute a right-handed coordinate system.

[0029] (2) From the above plane model, obtain the normal vector of the plane. The plane normal vector is obtained by using the Rodriguez formula, along with the Z-axis offset D. with vector The rotation matrix R between them is used to construct the translation vector T = (0, 0, -D) through the Z-axis offset;

[0030] (3) A rigid transformation matrix is ​​formed by the rotation matrix R and the translation vector T. A rigid transformation matrix is ​​used to perform a rigid body transformation on the downsampled image, meaning that all points in the downsampled image are rotated and translated to be above the XOY plane; all points with Z coordinate values ​​less than a threshold are considered ground background points and filtered out from the downsampled image. The threshold value is determined based on the actual situation; in this embodiment, it is set to 6.5 mm.

[0031] The background segmentation can be completed by following the four steps above.

[0032] Secondly, same-side image registration is performed on the preprocessed images. Same-side image registration includes two processes: coarse same-side image registration and fine same-side image registration. Coarse same-side image registration provides favorable initial conditions for the fine same-side image registration process. The coarse same-side image registration process includes PCA (Principal Component Analysis) and principal direction correction. See reference 2: Liu Zhe, Zhou Tian, ​​Peng Dongdong, Feng Chen. Research on an improved PCA-based ICP point cloud registration algorithm [J]. Journal of Natural Science of Heilongjiang University, 2019. PCA is a rigid body transformation that finds the principal directions of multiple same-side images to coincide by projecting the image onto the plane of the ground and utilizing the shape characteristics of the foot itself. In addition, since the same-side images are taken from different angles on the same side, the images are not uniform and may cause the principal direction to be reversed. Therefore, a set of four rotation matrices is used to correct the principal direction and complete the coarse same-side image registration. Fine registration of the same-side images further improves the registration accuracy based on the coarse registration of the same-side images. The method used is the ICP (Iterative Closest Point) algorithm. After completing the same-side image registration, two images are obtained for each foot, one on each side of the midline.

[0033] Next, heterolateral image registration is performed, which involves registering the two images for each foot. Heterolateral image registration includes coarse heterolateral image registration and fine heterolateral image registration. Coarse heterolateral image registration uses PCA and principal orientation correction methods, while fine heterolateral image registration uses the ICP algorithm, which is exactly the same as the ipsilateral image registration process, and will not be elaborated further here. After heterolateral image registration, a foot reconstruction image is obtained for each foot.

[0034] Finally, foot parameters are calculated. Measurements are taken from each reconstructed foot image to obtain key parameters for each foot—foot length, foot width, and foot height.

[0035] Figures 3 to 5 It is the result of the experiment.

[0036] Figure 3 This is a schematic diagram of ipsilateral image registration according to the present invention. Taking the registration of the left side of the left foot as an example, as shown in the figure, Figure 3 (a) is a diagram showing the effect of using Cloudcompare software to display three images of the same side of a certain side after preprocessing the downsampled image. Figure 3 (b) is the result image displayed by Cloudcompare software after coarse registration of the above three images from the same side. Figure 3 (c) The image displayed by Cloudcompare software after fine registration of the three images from the same side mentioned above. It should be noted that, because... Figure 3(a) is a grayscale image, so the outline of the foot in the lower right part of the image cannot be clearly displayed.

[0037] Figure 4 This is a schematic diagram of the heterolateral image registration involved in this invention. Taking the registration of the left and right sides of the left foot as an example, the images shown are all effect images displayed by Cloudcompare software. As shown in the figure, Figure 4 (a) is the image of the opposite side after being registered on the same side on both sides (including one image on each side obtained after same-side registration). Figure 4 (b) is the image after coarse registration of the images on opposite sides. Figure 4 (c) is the image after fine registration of the contralateral image, which is the complete foot reconstruction image obtained.

[0038] Figure 5 This is a schematic diagram illustrating the information processing results of the shoe type according to the present invention. (Through...) Figure 5 The experiments shown demonstrate that this invention can not only reconstruct foot shapes, but also reconstruct footwear models. Figure 5 The original image of the footwear is acquired, and then processed using the method described in this invention to obtain a reconstructed image of the footwear, which can also yield important parameters of the footwear.

[0039] To demonstrate the accuracy of this invention, we conducted an experiment using left and right foot data from 10 participants (5 men and 5 women). During the experiment, foot parameters were measured using a soft measuring tape (the smallest unit of measurement for a soft measuring tape is 1 mm), and the foot parameters were also calculated using the method provided in this invention. According to the People's Republic of China National Standard GB / T3294-1998—"New Shoe Size"—the difference between adjacent shoe sizes is fixed at 5 mm. Therefore, this system uses 5 mm as the error standard. If the error between the system's measurement result and the manual soft measuring tape measurement result is within 5 mm, the measurement is considered correct; otherwise, it is considered incorrect. A total of 40 sets of measurement data were obtained, each set including data for one foot of one person. Each set of data includes three parameters: foot length, foot width, and foot height. Of these, 20 sets of measurement data were obtained using a soft measuring tape, and 20 sets of calculated data were obtained using this invention. Comparing the measured and calculated data for each group, the difference in each parameter was within 5 mm; the calculated average difference was 2.40 mm, and the variance was 1.59. The experimental results demonstrate that this invention achieves accurate measurement of foot parameters in small samples, indicating that our system has high accuracy.

Claims

1. A three-dimensional foot measurement system based on a depth camera, comprising a depth camera and a data processing module, characterized in that, At least six images of each foot are obtained using a depth camera; the images are obtained by taking at least three images on each side of the midline, which is the line connecting the midpoint of the tip of the second toe and the midpoint of the heel; images on the same side of the midline are called ipsilateral images; images on different sides of the midline are called heteropsilateral images. The data processing module implements the following process: The images obtained for each foot are processed as follows: First, the images on the same side are registered; then the images on the opposite side are registered to obtain a reconstructed image of the foot; the parameters of the foot shape are obtained using the reconstructed image. The original image of each foot is downsampled and preprocessed, and the preprocessing includes background segmentation, discrete point filtering and image smoothing. Background segmentation includes the following steps: ① Using the RANSAC method, calculate the coefficients A, B, C, and D of the planar model corresponding to a downsampled image. The planar model is... The coordinate system of the planar model is as follows: the line connecting the midpoint of the tip of the second toe and the midpoint of the heel is the midline, the midpoint of the midline is the origin, the straight line containing the midline and pointing to the midpoint of the tip of the second toe is the X-axis, the straight line perpendicular to the X-axis and coinciding with the plane of the foot is the Y-axis, and the straight line perpendicular to the plane of the foot and pointing upward is the Z-axis. The X-axis, Y-axis and Z-axis constitute a right-handed coordinate system. ②Let the normal vector of the plane be... The plane normal vector is obtained by using the Rodriguez formula, along with the Z-axis offset D. with vector The rotation matrix R and translation vector between them ; ③ By rotation matrix R and translation vector Together they form a rigid transformation matrix The rigid transformation matrix is ​​used to perform a rigid body transformation on the downsampled image, that is, all points in the downsampled image are rotated and translated to be above the XOY plane; all points with Z coordinate values ​​less than a threshold are regarded as ground background points and are filtered out in the downsampled image; wherein, the threshold is determined according to the actual situation.

2. The three-dimensional foot measurement system according to claim 1, characterized in that, When registering images on the same side, a coarse registration process is performed first, followed by a fine registration process.

3. The three-dimensional foot measurement system according to claim 2, characterized in that, When registering images from opposite sides, a coarse registration process is performed first, followed by a fine registration process.

4. The three-dimensional foot measurement system according to claim 3, characterized in that, The six images of a certain foot are as follows: the three images taken to the left of the foot's midline are taken at angles of 45° to the left front, left center, and left back, respectively; and the three images taken to the right of the foot's midline are taken at angles of 45° to the right front, right center, and right back, respectively.

Images