Color shape change monitoring system

Abstract

To provide a color and shape change monitoring system capable of accurately monitoring the color and shape change of a human body caused by a blood flow change.SOLUTION: The image processing apparatus includes a three dimensional point cloud data acquisition unit configured to acquire three dimensional point cloud data of a measurement target portion of a human body, a color data acquisition unit configured to acquire color data of the measurement target portion, a color-shape-data generation unit configured to generate color-shape data of the measurement target portion in which the color data is associated with the three dimensional point cloud data, and a pigment component separation unit configured to separate the color-shape data into a hemoglobin pigment component, a melanin pigment component, and a shade component.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system for monitoring color and shape changes in the human body. [Background technology]

[0002] In recent years, the number of diabetes patients has been increasing. Diabetes tends to affect the feet, which are far from the heart, and if proper observation and treatment are neglected, foot ulcers can develop, often leading to foot amputation. Since prevention of diabetic foot ulcers is extremely important, the importance of continuous foot care is increasing.

[0003] A typical example of foot care is wearing shoes of the appropriate size. It is known that wearing shoes of the appropriate size reduces chafing and pressure on the feet, leading to the prevention of foot ulcers and other conditions. Traditionally, a tape measure has been used to measure the foot length, circumference, and width specified in shoe size to select shoes of the appropriate size, but measurement accuracy varies depending on the person taking the measurement. Therefore, as a technology for measuring foot shape with high accuracy, a foot shape measurement system has been developed that measures the three-dimensional shape of the foot using, for example, a light-section method using laser light (see Patent Document 1).

[0004] Peripheral arterial disease (PAD) is also known to be a predictor of foot ulcers. PAD-related color changes include a reddish hue due to capillary dilation and inflammation, and a bluish-purple discoloration due to insufficient blood supply due to arterial occlusion or ischemia. These color changes are due to changes in blood flow, and the potential for treating PAD-related foot ulcers depends on improving blood flow. Therefore, monitoring changes in foot color, i.e., foot shape and color, is important. Thus, foot care for diabetic patients requires not only accurate measurement of the three-dimensional shape of the foot to select appropriate shoe sizes, but also monitoring the color and shape of the foot and providing appropriate care to improve blood flow. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-182550 A (Page 5, Figure 1) Summary of the Invention [Problem to be solved by the invention]

[0006] However, although the foot shape measurement system in Patent Document 1 can measure the three-dimensional shape of the foot with high accuracy, it cannot monitor color and shape changes of the foot caused by changes in blood flow. Also, color and shape changes in the feet of diabetic patients are generally diagnosed based on the subjective judgment of a doctor or the like, which has the problem that the accuracy of the diagnosis depends on the skill of the doctor or the like.

[0007] The present invention has been made in light of these problems, and aims to provide a color and shape change monitoring system that can accurately monitor color and shape changes in the human body caused by changes in blood flow. [Means for solving the problem]

[0008] In order to solve the above problems, the color and shape change monitoring system of the present invention comprises: a three-dimensional point cloud data acquisition means for acquiring three-dimensional point cloud data of a measurement target portion of the human body; a color data acquisition means for acquiring color data of the measurement target portion; a color and shape data generating means for generating color and shape data of the measurement target portion in which the color data is linked to the three-dimensional point cloud data; The image processing device is characterized by comprising a pigment component separating means for separating the color and shape data into hemoglobin pigment components, melanin pigment components, and shadow components. According to this feature, by extracting the distribution of hemoglobin pigment components separated from color and shape data in which color data is linked to 3D point cloud data, it is possible to accurately monitor color and shape changes associated with changes in blood flow in measurement target areas of the human body having a 3D shape.

[0009] The pigment component separating means is characterized in that it separates the color and shape data into a hemoglobin pigment component, a melanin pigment component, and a shadow component after removing a group of white points from the color and shape data. According to this feature, the accuracy of the color and shape data can be improved by removing white outliers that occur on the surface of the color and shape data acquired from the measurement target portion.

[0010] The white point group is characterized by being a point group in which all RGB values ​​are 200 or more. According to this feature, the accuracy of the color and shape data can be further improved by reliably removing the white outliers.

[0011] The measurement target site is characterized by being the foot when weight is being applied. According to this feature, by monitoring the color shape of the foot when weight is being applied, it becomes easier to select shoes of the appropriate size.

[0012] The pigment component separating means is characterized in that it separates the color and shape data into hemoglobin pigment components, melanin pigment components, and shadow components after removing the point cloud of the floor surface from the color and shape data. According to this feature, the accuracy of the color and shape data can be improved by removing the point cloud that indicates the floor surface other than the measurement target portion.

[0013] The pigment component separation means is characterized by removing points that are farther from the center point of the color and shape data than the points on the floor, and then separating the data into hemoglobin pigment components, melanin pigment components, and shadow components. According to this feature, by removing point groups that have little relationship with the measurement target portion, the accuracy of color and shape data can be further improved. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram showing the configuration of a color and shape change monitoring system in an embodiment of the present invention. [Figure 2] 10 is a schematic diagram showing coordinate axes of color and shape data of a foot generated by a color and shape data generating means. FIG. [Figure 3] Regarding the preprocessing of color and shape data by the pigment component separation means, (a) is a diagram showing the color and shape data (point cloud before processing) generated by the color and shape data generation means, and (b) is a diagram showing the point cloud after point cloud removal processing using distance data. [Figure 4] FIG. 10 is a diagram showing a point cloud after processing in which a floor plane is removed using plane detection by RANSAC in preprocessing of color and shape data by the pigment component separation means. [Figure 5] 10 is a diagram showing unnecessary noise (point cloud) remaining below the floor plane in the point cloud after processing to remove the floor plane, in preprocessing of color and shape data by the pigment component separation means. FIG. [Figure 6] Regarding preprocessing of color and shape data by a pigment component separation means, (a) is a diagram showing a point cloud before processing to remove noise (white point cloud) using color data, and (b) is a diagram showing a point cloud after processing to remove noise (white point cloud) using color data. [Figure 7] Regarding preprocessing of color shape data by a pigment component separation means, (a) is a diagram showing a point cloud before noise removal processing using an outlier removal algorithm, and (b) is a diagram showing a point cloud after noise removal processing using an outlier removal algorithm. [Figure 8] 10 is a diagram showing a skin model assumed in a pigment component separation method based on independent component analysis for pigment component separation of color and shape data by a pigment component separation means. FIG. [Figure 9] FIG. 10 is a diagram illustrating the relationship between observed signals and independent signals in a skin model. [Figure 10] FIG. 10 is a diagram showing a flow of independent component analysis in a skin model. [Figure 11]FIG. 1A is a diagram showing the definitions of foot length and foot circumference, and FIG. 1B is a diagram showing a group of points that form the foot circumference. [Figure 12] (a) is a diagram showing the input point cloud of the foot before pigment component separation, (b) is a diagram showing the distribution of hemoglobin pigment components as the result of pigment component separation for the point cloud of the foot, (c) is a diagram showing the distribution of melanin pigment components in the same way, and (d) is a diagram showing the shadow components in the same way. [Figure 13] FIG. 1(a) is a diagram showing the distribution of hemoglobin pigment components in the feet before a foot bath, and FIG. 1(b) is a diagram showing the distribution of hemoglobin pigment components in the feet after a foot bath. [Figure 14] FIG. 1(a) is a diagram showing the distribution of hemoglobin pigment components in the foot before avascularization, and FIG. 1(b) is a diagram showing the distribution of hemoglobin pigment components in the foot after avascularization. DETAILED DESCRIPTION OF THE INVENTION

[0015] A color and shape change monitoring system according to the present invention will be described below. [Example]

[0016] A color and shape change monitoring system according to an embodiment will be described with reference to FIGS.

[0017] 1, the color and shape change monitoring system of the present invention is mainly composed of a 3D point cloud data acquisition means for acquiring 3D point cloud data of a measurement target portion of the human body, a color data acquisition means for acquiring color data of the measurement target portion, a color and shape data generation means for generating color and shape data of the measurement target portion by linking the color data acquired by the color data acquisition means to the 3D point cloud data acquired by the 3D point cloud data acquisition means, and a pigment component separation means for separating the color and shape data generated by the color and shape data generation means into hemoglobin pigment components, melanin pigment components, and shading components. Thus, the color and shape change monitoring system of the present invention measures the 3D shape of the measurement target portion (e.g., foot length and foot circumference) from the color and shape data, and extracts the distribution (more specifically, concentration distribution) of the hemoglobin pigment components separated by the pigment component separation means, thereby accurately monitoring color and shape changes associated with changes in blood flow in the measurement target portion of the human body having a 3D shape as color changes at each point constituting the 3D point cloud.

[0018] In this embodiment, the color and shape change monitoring system acquires color and shape data of a measurement target part of the human body using a smartphone (iPhone (registered trademark) 15 Pro Max).

[0019] In this embodiment, the 3D point cloud data acquisition means is a LiDAR sensor mounted on the external camera of a smartphone. The LiDAR sensor can acquire the 3D point cloud data by emitting an infrared dot pattern from a light source and acquiring the distance to the object as distance data for each dot based on the time it takes for the infrared dot pattern to reflect off the object and return.

[0020] In this embodiment, the color data acquisition means is a smartphone camera.

[0021] In this embodiment, the color and shape data generating means is an application (Scaniverse (registered trademark) manufactured by Niantic, Inc.) that generates color and shape data and is installed on a smartphone. By starting the application and capturing video of the measurement target area with the smartphone camera, color and shape data is generated that links the 3D point cloud data acquired by the LiDAR sensor with the color data acquired by the camera, i.e., the RGB values. In other words, the color and shape data is 3D color distance point cloud data. In addition, the color and shape data is acquired at regular time intervals, and the color data of each point that makes up the 3D point cloud is saved as time-series data having a predetermined time width.

[0022] In this embodiment, the pigment component separation means acquires the concentration distribution of hemoglobin pigment components by applying a pigment component separation method based on independent component analysis to the color and shape data. Independent component analysis is a multivariate analysis technique that enables estimation of the original signal from an observed signal in which multiple independent original signals are mixed with their respective weights, even when information such as the signal and its composite matrix is ​​unknown. "Independent" means that a variable is independent of other variables and its distribution is invariant. The pigment component separation method will be described in detail later.

[0023] (Color and shape data preprocessing) The color and shape data generated by the application described above contains various noises and objects other than the measurement target area, such as the floor. Therefore, the pigment component separation means performs preprocessing to remove unnecessary points other than the measurement target area from the color and shape data before separating hemoglobin pigment components and the like from the color and shape data. Note that removing unnecessary points from the color and shape data as preprocessing can provide more accurate color and shape data than removing unnecessary points as postprocessing after separating pigment components from the color and shape data. This is because removing clearly unnecessary points before the separation process results in the color and shape data being a point cloud of the measurement target area of ​​the human body, allowing for accurate separation into hemoglobin pigment components, melanin pigment components, and shadow components. The following description uses the foot as an example of the measurement target area of ​​the human body.

[0024] Specifically, as a point cloud preprocessing step for the color and shape data, first, point cloud removal is performed on the color and shape data using distance data. As shown in FIG. 2, the coordinate axes of the color and shape data generated by the application are: x-axis in the foot length direction, y-axis in the foot width direction, and z-axis in the upright direction. Furthermore, since the foot is photographed while maintaining a substantially constant distance, point clouds at a distance greater than or equal to the threshold value on the z-axis can be determined to be objects other than the foot. In this example, the x-axis is ±40 cm, the y-axis is ±15 cm, and the z-axis is ±6 cm from the origin, which is the center of photography. Furthermore, the program for removing point clouds using distance data was created using Python.

[0025] Figure 3 shows the point clouds before and after processing to remove points above a threshold using distance data. As shown in Figure 3(b), in the point cloud after processing, the area above the ankle has been removed compared to the point cloud before processing (see Figure 3(a)), and the range of the floor plane has also become narrower.

[0026] (Floor Plane Removal) Next, since the floor plane remains in the point cloud after the point cloud removal process using distance data shown in Figure 3(b), the floor plane is removed using plane detection using RANSAC (Random Sample Consensus). Figure 4 shows the point cloud after the floor plane removal process. RANSAC is a method for determining parameters by removing the influence of outliers from data that includes outliers. In this example, a plane is defined by selecting three arbitrary points, and the distance between this plane and all points is calculated. If the shortest distance is found, it is determined to be the floor plane and the corresponding point cloud is removed. The RANSAC plane detection algorithm uses functions implemented in the open source software "Open3D." Python is also used to implement Open3D.

[0027] (Noise removal below the floor plane) Next, as shown in Figure 5, in the point cloud from which the floor plane has been removed, unnecessary points (see the boxed area in Figure 5) remain as noise below the floor plane. As such, since the point cloud acquired by the LiDAR sensor may contain noise due to lighting effects, camera shake, etc., these outliers are removed.

[0028] Specifically, since the point cloud of the foot exists above the floor plane, the point cloud below the floor plane is clearly considered to be an outlier. Therefore, by removing all point clouds that make up the floor plane that are smaller than the median z coordinate plus 1 cm, all noise below the floor plane is removed.

[0029] (Noise removal using color data) Next, as shown in Figures 5 and 6(a), when 3D point cloud data of a human body is acquired using a LiDAR sensor, outliers, or so-called white point clouds, appear around the body. These white point clouds are white or a color close to white, so they can be removed by determining a threshold using the color data in the color and shape data. In this example, the thresholds for the RGB values ​​are set to 200, and point clouds with all RGB values ​​greater than or equal to 200 are removed, thereby removing the white point clouds (see Figure 6(b)). It was discovered during this research that such white point clouds can be removed. It was also discovered that white point clouds do not occur in two dimensions, but are unique to three dimensions.

[0030] (Noise removal using outlier removal algorithm) Finally, the outliers are statistically removed using the radius-based outlier filter, which is an outlier removal algorithm. The radius-based outlier filter calculates the radius of each point x in the point cloud. i For , the number of points within the radius of the sphere, which is a hyperparameter, is calculated, and if the number of points is smaller than the threshold n, then x i In this example, the radius of the sphere is set to 1 cm, the threshold value n is set to 20, and the outliers are statistically removed from the point cloud before processing (see FIG. 7(a)), thereby obtaining color and shape data from which point clouds unrelated to the foot have been removed (see FIG. 7(b)).

[0031] (Pigment component separation method) The pigment component separation means extracts the density distribution of the separated hemoglobin pigment component by applying a pigment component separation method based on independent component analysis to the color and shape data from which noise has been removed by the above-mentioned preprocessing. The method for separating skin pigment components using independent component analysis will now be described in detail.

[0032] Human skin is structured in three layers: the epidermis, dermis, and subcutaneous tissue, and contains pigments such as melanin, hemoglobin, carotene, and bilirubin. Of these pigments, melanin and hemoglobin have a significant impact on skin color, and since melanin is found in large amounts in the epidermis and hemoglobin in large amounts in the dermis, if we consider the epidermis to be the melanin layer and the dermis to be the hemoglobin layer, the two pigments can be considered to be spatially independent (see Figure 8).

[0033] As shown in Figure 8, if we assume that the boundary surface between the epidermis (melanin layer) and dermis (hemoglobin layer) in the skin model is flat, the light incident on the skin can be divided into surface reflected light that reflects from the skin surface and internal reflected light that reflects within the skin. Surface reflected light reflects the color of the light source without being affected by the skin pigment. In contrast, internal reflected light reflects the skin color because it is repeatedly absorbed and scattered within the skin. Therefore, by applying independent component analysis assuming the observed signal to be the RGB values ​​of the skin, it is possible to separate the distributions of melanin pigment components, hemoglobin pigment components, and shading components in the measurement area.

[0034] As shown in Fig. 9, the observed signal ν log is the melanin vector σ m and the hemoglobin vector σ h and a linear combination of the shading vector (1) and the bias vector e log In FIG. 9, the melanin vector σ m and the hemoglobin vector σ h The plane formed by these is defined as the skin color distribution plane.

[0035] Here, the change in the intensity of the shadow can be regarded as a vector of change in the illumination intensity. Therefore, the normal of the acquired skin color distribution plane is calculated, and the observed signal v log The distance to the skin color distribution plane of the observed signal v log By projecting onto the skin color distribution plane, the influence of the shadow component can be removed from the observed signal.

[0036] As shown in Figure 10, for the observed signal from which the shadow components have been removed, the actual melanin pigment component density vector and hemoglobin pigment component density vector are treated as independent original signals S1 and S2, and whitening is performed on the observed signals v1, v2, and v3 of each RGB channel to obtain whitened signals o1 and o2. Using independent component analysis, the melanin pigment component vector S'1 and hemoglobin pigment component vector S'2, which are independent signals, can then be estimated. Then, by projecting the melanin pigment component vector S'1 and hemoglobin pigment component vector S'2 from the skin color distribution plane from which the shadow components have been removed, the melanin pigment component density vector and hemoglobin pigment component density vector can be separated.

[0037] In this way, the pigment component separation means extracts the hemoglobin pigment component density vector separated from the color and shape data from which the shading components have been removed, thereby enabling the visualization of color and shape changes accompanying changes in blood flow in the measurement target region with even greater precision. In other words, it is possible to visualize the color changes of each point constituting the 3D point cloud based on the hemoglobin pigment component density vector.

[0038] Using the color and shape change monitoring system of this example, measurements were taken of the feet of actual subjects and color changes were monitored. The subjects were healthy individuals, and changes in blood flow in diabetic patients were simulated.

[0039] (Experiment 1: Foot Measurement) Using an application (Scaniverse) installed on a smartphone that generates color and shape data, the foot shapes of three Japanese male subjects in their twenties were measured. The shooting environment was under a white LED light source, with no obstructions within a 1m radius of the subject. To accurately measure foot length and circumference, the subjects were photographed barefoot, standing upright on a flat, level surface with their feet parallel and spaced apart and their weight evenly distributed, as defined by the JIS standard (JIS S 5037:1998).

[0040] The measurement procedure involved the photographer holding the smartphone in their hand with the application running, and recording video at a speed that allowed them to circle around the right foot in approximately 20 seconds, while maintaining a distance of approximately 10 cm from the right foot. Furthermore, to compare the accuracy of the 3D point cloud data that constitutes the color and shape data obtained by this recording, the subject's foot length and circumference were measured using a tape measure.

[0041] The analysis of the experimental results was performed within the same Python program that performed the preprocessing of the point cloud described above.To verify the accuracy of the 3D point cloud data that constitutes the color and shape data, the estimated values ​​of foot length and circumference were compared with the values ​​measured using a tape measure.

[0042] This section explains how to obtain the foot length and circumference from color and shape data that has undergone point cloud preprocessing. First, to remove areas such as the ankle that are not necessary for foot measurement, points that are 5 cm or more from the median of the floor plane derived in the preprocessing are deleted.

[0043] Next, measure the foot length and circumference. The measurement points for foot length and circumference are defined by the JIS standard (JIS S 5037:1998), where foot length is the distance from the rear end of the heel to the tip of the longest toe, and foot circumference is the length surrounding the base of the first toe and the base of the fifth toe (see Figure 11(a)).

[0044] As shown in Figure 2, if the foot length direction is defined as the x-axis and the foot width direction as the y-axis, the foot length can be obtained by calculating the difference between the point cloud with the maximum x coordinate and the point cloud with the minimum x coordinate.

[0045] The foot circumference can be obtained by deriving a plane that passes through the bases of the first and fifth toes and is parallel to the z-axis, deriving a point group close to that plane, and finding the convex hull of these point groups.

[0046] Specifically, the convex hull can be derived using a function for calculating the convex hull in scipy, a library implemented in Python. First, two points representing the bases of the first and fifth toes are selected. A plane that passes through the two selected points and is parallel to the z-axis (see Figure 2) is derived, and a point cloud that is 1 mm or less away from this plane is derived. Next, the derived plane is redefined as a new two-dimensional coordinate plane, and the derived point cloud (see the point cloud in Figure 11(b)) is projected onto this plane. The original z coordinate is redefined as a new y coordinate, and the original x and y coordinates are redefined as new x coordinates using the normal vector. Finally, the convex hull is calculated for the projected cross-section (see the solid line in Figure 11(b)).

[0047] (result) The results of a comparison between the estimated values ​​of foot length and foot circumference obtained from the 3D point cloud data of the three subjects and the actual values ​​of foot length and foot circumference measured using a tape measure are shown in Tables 1 and 2 below, respectively.

[0048] [Table 1]

[0049] [Table 2]

[0050] From the comparison results shown in Table 1, the error between the estimated foot length obtained from the created 3D point cloud data and the actual measured value was 0.24 cm for subject A, 0.10 cm for subject B, and 0.21 cm for subject C.

[0051] Furthermore, from the comparison results shown in Table 2, the error between the estimated foot circumference obtained from the created 3D point cloud data and the actual measured value was 0.27 cm for subject A, 0.36 cm for subject B, and 0.10 cm for subject C.

[0052] In this way, by comparing the estimated foot length values ​​obtained from the 3D point cloud data with the actual foot length values ​​measured using a tape measure, it was confirmed that the error in foot length was within the 0.5 cm range defined by the JIS standard for all subjects. Furthermore, it was confirmed that the error in foot circumference was within the 0.3 cm range defined by the JIS standard for subjects A and C. In other words, the color and shape data obtained by the color and shape change monitoring system of this embodiment can measure the 3D shape of the measurement target area with high accuracy.

[0053] It is presumed that the influence of outliers was the cause of the decrease in the estimation accuracy of foot circumference for subject B. In particular, the convex hull (see Figure 11(b)) used to measure foot circumference surrounds the point cloud so that all points are inside, and therefore it is presumed that it was significantly affected by these outliers.

[0054] (Experiment 2: Monitoring color-shape changes) Next, using an application (Scaniverse) installed on a smartphone that generates color and shape data, we monitored color and shape changes caused by changes in blood flow in the feet of a Japanese male subject in his 50s.The shooting environment was under a white LED light source, with the subject wearing clothing that did not compress his feet and sitting with his heels resting on a chair.

[0055] As mentioned above, the toe discoloration seen in peripheral arterial disease (PAD) is characterized by a reddish hue due to increased blood flow and a bluish-purple discoloration due to insufficient blood flow. The reddish hue due to increased blood flow was reproduced by promoting blood circulation through a warm foot bath. Specifically, the foot was photographed using the application described above before the foot bath, and then again after immersing the foot in 40°C water for approximately 5 minutes.

[0056] Regarding the abnormal color tone of the toes, the bluish-purple discoloration caused by insufficient blood flow was reproduced by performing avascularization using an aneroid sphygmomanometer (FOCAL, FC-100V). Specifically, the foot was photographed before avascularization using the above-mentioned application, and the sphygmomanometer was wrapped around the lower leg and the pressure was set to 260 mmHg. After about one minute, the foot was photographed again.

[0057] Furthermore, when the color change of the foot after foot bathing or avascularization was confirmed visually without using the color and shape change monitoring system of this example, it was confirmed that the redness of the toes was clearly stronger after the foot bath compared to before the foot bath. On the other hand, when comparing the time before and after avascularization, it was confirmed that there was almost no change in the color of the toes before and after avascularization. Thus, it is presumed that the change in blood flow caused by foot bathing is caused by changes in the external environment and is easy to distinguish visually, whereas avascularization reduces the amount of blood flow itself, and therefore does not have a visually noticeable effect on the surface of the foot.

[0058] Next, by applying a dye component separation method to the color and shape data that has undergone preprocessing of the point cloud, we can monitor subtle changes in color and shape that are difficult to distinguish with the naked eye.

[0059] The results of pigment component separation for the color and shape data of the foot are shown in Figure 12. The color and shape changes caused by changes in blood flow are visually confirmed based on the concentration distribution of the hemoglobin pigment component shown in Figure 12(b). Note that here, the concentration distribution of the hemoglobin pigment component is separated and extracted from the color and shape data from which the shadow component has been removed.

[0060] (result) As shown in Figure 13, it was visually confirmed that the concentration distribution of hemoglobin pigment components in the feet after foot bathing was clearly higher after foot bathing (see Figure 13(b)) than before foot bathing (see Figure 13(a)), especially in the toes.

[0061] Furthermore, as shown in Figure 14, it was visually confirmed that the concentration distribution of hemoglobin pigment components in the foot after avascularization was lower after avascularization (see Figure 14(b)) than before avascularization (see Figure 14(a)), especially in the toes.

[0062] These results confirmed that even for color and shape changes caused by minute changes in blood flow that are difficult to confirm directly with the naked eye, blood flow changes can be visualized with high accuracy as color and shape changes by extracting the concentration distribution of hemoglobin pigment components separated by the pigment component separation means.

[0063] As described above, the color and shape change monitoring system of this embodiment includes a LiDAR sensor as a 3D point cloud data acquisition means for acquiring 3D point cloud data of a measurement target region of the human body, a camera as a color data acquisition means for acquiring color data of the measurement target region, an application as a color and shape data generation means for generating color and shape data of the measurement target region whose color data is linked to the 3D point cloud data, and a pigment component separation means for separating the color and shape data into hemoglobin pigment components, melanin pigment components, and shading components. By extracting the distribution of the separated hemoglobin pigment components from the color and shape data, color and shape changes associated with blood flow changes in the measurement target region of the human body having a 3D shape can be accurately monitored as color changes at each point constituting the 3D point cloud. Furthermore, since the color and shape data accurately reflects the 3D shape of the measurement target region and the color data at each point constituting the 3D point cloud is stored as time-series data with a predetermined time span, color and shape changes associated with blood flow changes can be accurately monitored by visualizing the color changes at each point constituting the 3D point cloud based on the hemoglobin pigment component concentration vector.

[0064] Furthermore, while precision medical equipment such as 3D shape measuring machines that have traditionally been used to measure the 3D shape of the human body are large and expensive, the color and shape change monitoring system of this embodiment can inexpensively and easily measure foot length and circumference from color and shape data using a smartphone.

[0065] In addition, as a pre-processing step for the point cloud, the pigment component separation means removes the white point cloud from the color and shape data, and then separates hemoglobin pigment components and the like from the color and shape data.By removing white outliers that occur on the surface of the color and shape data obtained from the measurement target area, the accuracy of the color and shape data can be improved.

[0066] Furthermore, since the white point group is a point group in which all RGB values ​​are 200 or more, white outliers can be reliably removed, and the accuracy of the color and shape data can be further improved.

[0067] Furthermore, by monitoring the color and shape of the foot when weight is being applied as the measurement target area, it is possible to visualize changes in blood flow in the foot when weight is being applied, making it easier to select shoes of an appropriate size that will maintain normal blood flow in the foot.

[0068] In addition, the pigment component separation means removes the point cloud of the floor surface from the color and shape data, and then separates hemoglobin pigment components, etc. from the color and shape data.By removing the point cloud that indicates the floor surface other than the area to be measured, the accuracy of the color and shape data can be improved.

[0069] In addition, the pigment component separation means removes point groups that are farther away from the center point of the color and shape data than the point group on the floor, and then separates hemoglobin pigment components, etc. from the color and shape data.By removing point groups that have little relationship with the area to be measured, the accuracy of the color and shape data can be further improved.

[0070] In addition, the pigment component separation means can remove shading components from the pre-processed color and shape data, thereby enabling high-precision separation of hemoglobin pigment components, thereby enabling more accurate monitoring of color and shape changes associated with changes in blood flow.

[0071] Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

[0072] For example, in the above embodiment, the measurement target area was described as the foot of a human body, but this is not limited to this, and the measurement target may be other areas such as the hand, thigh, or calf of the leg.

[0073] Furthermore, in the above embodiment, a LiDAR sensor is used as the 3D point cloud data acquisition means, but this is not limiting, and other non-contact 3D shape measurement techniques using, for example, light, electromagnetic waves, sound waves, etc. may also be used. Furthermore, the 3D point cloud data acquisition means is not limited to those using passive methods such as LiDAR sensors, and may also use passive methods such as multi-view stereo measurement.

[0074] Furthermore, in the above embodiment, an application (Scaniverse) is used as the color and shape data generation means, but this is not limited to this, and any application may be used as the color and shape data generation means as long as it can generate color and shape data in which color data is linked to three-dimensional point cloud data.

[0075] Furthermore, in the above embodiment, the pigment component separation means has been described as removing shading components from pre-processed color and shape data, but this is not limiting, and the pigment component separation means may separate and extract the concentration distribution of hemoglobin component pigments without removing shading components.

[0076] Furthermore, in the above embodiment, the monitoring of color and shape changes accompanying changes in blood flow is described as being performed visually, but the monitoring of color and shape changes may also be performed by numerically analyzing changes in the concentration of hemoglobin component pigments in the point cloud, thereby enabling the detection of subtle changes that cannot be recognized visually.

[0077] In addition, in the above embodiment, a mode of preprocessing of point groups in color and shape data in which white point groups with all RGB values ​​of 200 or more are removed has been described. However, this is not limited to this, and unnecessary point groups may also be removed using color data other than white point groups.

[0078] In addition, in the above embodiment, a mode of removing the point cloud of the floor surface as pre-processing of the point cloud in the color and shape data was described, but this is not limited to this, and structures other than the human body, such as wall surfaces that have flat surfaces other than the floor surface, may also be removed as unnecessary point clouds.

[0079] In the above embodiment, the foot length and circumference defined by the JIS standard are measured, but the present invention is not limited to this. It is also possible to measure problems caused by contact between the foot and shoe, such as the instep height and the angle of the big toe relative to the base of the first toe, which is related to hallux valgus. This allows the user to select the optimal shoe size and shoe shape, thereby enhancing the effectiveness of foot care in improving blood flow. [Industrial Applicability]

[0080] The present invention has industrial applicability as a color and shape change monitoring system that can accurately monitor color and shape changes associated with changes in blood flow in a three-dimensional human body measurement site by extracting the concentration distribution of hemoglobin pigment components separated by a pigment component separation method from color and shape data of the measurement site. Furthermore, by easily obtaining color and shape data in which color data is linked to three-dimensional point cloud data using a smartphone, the present invention enables inexpensive and simple measurement of the shape of the measurement site from the color and shape data and monitoring of color and shape changes. This has a wide range of applications, not limited to foot care for diabetic patients, such as diagnosing diseases closely related to changes in blood flow and selecting appropriate-sized clothing.

Claims

1. a three-dimensional point cloud data acquisition means for acquiring three-dimensional point cloud data of a measurement target portion of a human body; a color data acquisition means for acquiring color data of the measurement target portion; a color and shape data generating means for generating color and shape data of the measurement target portion in which the color data is linked to the three-dimensional point cloud data; A color and shape change monitoring system comprising a pigment component separation means for separating the color and shape data into hemoglobin pigment components, melanin pigment components, and shadow components.

2. 2. The color and shape change monitoring system according to claim 1, wherein the pigment component separating means separates the color and shape data into hemoglobin pigment components, melanin pigment components, and shade components after removing white points from the color and shape data.

3. The color and shape change monitoring system according to claim 2 , wherein the white point cloud is a point cloud in which all RGB values ​​are 200 or more.

4. 4. The color and shape change monitoring system according to claim 1, wherein the measurement target area is a foot when weight is being applied to the foot.

5. The color and shape change monitoring system according to claim 4, characterized in that the pigment component separation means removes the floor point cloud from the color and shape data and then separates it into hemoglobin pigment components, melanin pigment components, and shadow components.

6. The color and shape change monitoring system described in claim 5, characterized in that the pigment component separation means removes points that are farther from the center point of the color and shape data than the points on the floor, and then separates the data into hemoglobin pigment components, melanin pigment components, and shadow components.

Images