Body information estimation method and body information estimation system

Abstract

To provide a body information estimation method for specifying a plurality of feature points in the body of a user on the basis of an image from two or more different directions of a measurement target site in the body of the user, and estimating measurement target site information using the plurality of feature points.SOLUTION: A body information estimation method includes: a specification process for specifying positions of a plurality of feature points in the body of a user on the basis of an image; a position specification process for specifying a specific position located in a direction from the feature points to other feature points in the body of the user on the basis of the plurality of feature points, the specific position being determined by the positional relationship between the feature points and the other feature points; and an estimation process for estimating body information related to the size or the shape of the measurement target site related to specified specific position.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a physical information estimation method. [Background technology]

[0002] There is a method for estimating the width of the front and the thickness of the sides of the torso of a subject near the navel. This method estimates the length of the area near the navel by placing a pair of hands on the area (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-198800 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the navel area of ​​the torso was determined using both hands of the subject, so there was room for improvement in terms of operability when taking images. Also, because the navel area of ​​the torso was determined by the subject, there was a problem that the position to be estimated varied depending on the subject. Furthermore, there was a problem that the measurement site was limited to the navel area of ​​the torso.

[0005] The present invention has been made in consideration of the above-mentioned problems, and relates to a physical information estimation method that identifies multiple feature points on a user's body based on images of a measurement target part on the user's body taken from two or more different directions, and estimates information about the measurement target part using the multiple feature points. [Means for solving the problem]

[0006] The present invention relates to a physical information estimation method including an acquisition step of acquiring images of a measurement target site on a user's body from two or more different directions; an identification step of identifying positions of a plurality of feature points on the user's body based on the images; a position identification step of identifying, based on the plurality of feature points, a specific position located on the user's body in a direction from a feature point to another feature point, the specific position being determined by a positional relationship between the feature point and the other feature point; and an estimation step of estimating physical information related to the size or shape of the measurement target site related to the identified specific position.

[0007] The present invention also relates to a physical information estimation system that estimates physical information of a user using a control device, including: an acquisition means that acquires images of a measurement target part on the user's body from two or more different directions; an identification means that identifies the positions of a plurality of feature points on the user's body based on the images; a position identification means that identifies, based on the plurality of feature points, a specific position on the user's body that is located in a direction from a feature point to another feature point, the specific position being determined by the positional relationship between the feature point and the other feature points; and an estimation means that estimates physical information related to the size or shape of the measurement target part related to the identified specific position. [Effects of the Invention]

[0008] According to the method provided by the present invention, it is possible to estimate physical information of a measurement target region with a simple operation by the user. [Brief explanation of the drawings]

[0009] [Figure 1] (a) is an image diagram when an image of a user is captured, and (b) is an image diagram when an image of the user is captured from a different direction. [Figure 2] (a) is an image showing feature points, and (b) is an image after segmentation processing. [Figure 3] 10 is a process flow of a physical information estimation method. [Figure 4] FIG. 10(a) is a diagram showing a specific position and an estimated position of a measurement target portion, and FIG. 10(b) is a diagram showing the estimated position of a measurement target portion. [Figure 5] FIG. 10 is a diagram showing an experimental method for determining the relationship between a feature point and its positional relationship with other feature points and a specific position, which was conducted on subjects. [Figure 6] (a) is a diagram showing a specific position and the estimated position of the measurement target area, (b) is a diagram showing the estimated position of the measurement target area, (c) is a diagram showing the estimated position of the measurement target area, and (d) is a diagram showing the estimated position of the measurement target area. [Figure 7] This is a diagram showing an ellipse whose major axis radius is 1 / 2 of the major axis length A of the measurement target area and whose minor axis radius is 1 / 2 of the minor axis length B. [Figure 8] FIG. 1 is a block diagram of a physical information estimation system. [Figure 9] FIG. 10 is a diagram showing an experimental method (another example) for determining the relationship between a feature point, its positional relationship with other feature points, and a specific position, which was conducted on a subject. [Figure 10] (a) is a diagram showing a specific position and the estimated position of the measurement target area, (b) is a diagram showing the estimated position of the measurement target area, (c) is a diagram showing the estimated position of the measurement target area, and (d) is a diagram showing the estimated position of the measurement target area. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present invention will be described below with reference to the drawings. The drawings of the present embodiments are intended to explain the technical concept, configuration, and operation of the present invention, and are not intended to specifically limit the configuration. In all drawings, similar components are denoted by similar reference numerals, and redundant descriptions will be omitted where appropriate. In this specification, a "system" includes one or more information processing devices. For example, a system can be configured by a single information processing device, or multiple information processing devices working together to perform functions such as a web server can also constitute a system. Furthermore, a "system" may include an information processing device functioning as a web server, one or more terminal devices, and one or more imaging devices and inspection devices.

[0011] An overview of the physical information estimation method according to this embodiment (hereinafter sometimes referred to as the present method) will be described. The physical information estimation method of this embodiment includes an acquisition step of acquiring images of a measurement target part on a user's body from two or more different directions; an identification step of identifying the positions of a plurality of feature points on the user's body based on the images; a position identification step of identifying, based on the plurality of feature points, a specific position on the user's body that is located in a direction from a feature point to another feature point, the specific position being determined by the positional relationship between the feature point and the other feature points; and an estimation step of estimating physical information related to the size or shape of the measurement target part associated with the identified specific position.

[0012] As shown in FIG. 1, in this method, an image of a measurement target part of a user's body (the abdomen in this embodiment) is captured in two or more different directions (for example, a front direction ( FIG. 1( a)) and a side direction ( FIG. 1( b)) using an image capture device 20. FIG. 1( a) shows that the user is captured with their arms stretched out horizontally, but the user's pose at the time of capturing the image is not limited to this. In this embodiment, in order to estimate physical information of the abdomen as the measurement target part, by capturing an image with the user's arms stretched out horizontally as shown in FIG. 1( a), it is possible to capture an image of the abdomen as the measurement target part in the side direction without the arms overlapping, as shown in FIG. 1( b). Furthermore, the user may remain wearing clothing during the image capture. As will be described in detail later, for example, thick clothing such as a down jacket is likely to result in estimation errors, so it is preferable to wear thin clothing such as a T-shirt, a jersey, or sports pants. FIG. 2 is a conceptual diagram illustrating the location of multiple feature points on a user's body based on a captured image. FIG. 2(a) illustrates the extracted skeletal points of a user extracted from a captured image. The user's skeletal points are extracted using a known skeletal information acquisition technology, such as MediaPipe Pose, a library that infers a person's posture from an image. Skeletal points are extracted as shown in FIG. 2(a). The positions and number of extracted skeletal points depend on the skeletal point acquisition technology used. Here, the following skeletal points are extracted, which are particularly used in this embodiment: the top of the head 10, both shoulders 11 (both acromions), both buttocks 12 (anterior superior iliac spine (pelvis)), heels 13, elbows 14, and knees 15. The user is then segmented using a known technology, such as Selfie Segmentation, a library that segments a specific object (e.g., a person) from an image. FIG. 2(b) illustrates the result of segmenting the acquired image and separating the person from the background. In this embodiment, MediaPipe Pose is used when extracting human skeleton points from an image, but this is not limiting and other methods such as VISION POSE and OPEN POSE may also be used. These images and the extracted skeleton points are then used as feature points, and physical information is estimated based on the feature points.

[0013] The processing flow of this method is shown in Fig. 3. This method includes an acquisition step (step S100), a determination step (step S200), a position determination step (step S300), and an estimation step (step S600).

[0014] Step S100 is an "acquisition step" that acquires images of a measurement target part of a user's body from two or more different directions. The "measurement target part on the user's body" refers to a body part whose size or shape is to be estimated. The target part can be any part that is visible from the outside, excluding the user's clothing, and can include, for example, the head, neck, chest, abdomen, buttocks, upper arms, thighs, and calves. As described in detail below, this method extracts skeletal points from an acquired image and uses them to identify the measurement target part. While feature points such as skeletal points are easy to identify from an acquired image, parts without skeletal points, such as the upper arms, thighs, calves, and abdomen, are difficult to identify. Therefore, when a measurement target part does not have skeletal points, it is necessary to identify the measurement target part by, for example, capturing an image of the abdomen with the navel visible, or capturing an image with the user placing their hand on the abdomen. However, this method can identify the measurement target part based on the positional relationship between the identified feature points and other feature points, enabling the user to estimate physical information about the measurement target part even in parts without skeletal points with simple user operations. The "images from two or more different directions" refer to images in which the user and the imaging device 20 are positioned in relatively different directions. If the user is in a fixed position, the images refer to images captured by moving the imaging device 20 in parallel (circling) around the user while maintaining a substantially constant distance from the user. Alternatively, if the position of the imaging device 20 is fixed, the images refer to images captured by the user rotating around a vertical axis while maintaining a constant distance from the imaging device 20. In either case, the images are captured from different directions of the user. Note that, because this method aims to allow a user to estimate physical information in a simple manner, it is preferable to capture images by fixing the position of the imaging device 20 and rotating around a vertical axis while maintaining a constant distance from the imaging device 20. Furthermore, the "images" may be either still images or moving images. For example, when a user takes images by themselves using a camera built into a smartphone, it is preferable to use moving images in order to easily obtain estimated abdominal information related to the size or shape of the abdomen. Furthermore, the imaging device 20 may be a general RGB camera, a monochrome camera, or a spectral camera, and there are no limitations on the performance or specifications of the imaging device 20. The imaging device 20 includes a video camera, a camera built into a smartphone, a camera built into a tablet terminal, a web camera that can be attached to a personal computer or the like by a connecting means such as a cable, and the like. "Acquire" means to acquire images, and the source of the time-series images does not matter, such as acquiring from a specified medium, acquiring via a network, or acquiring directly from a photographing device (importing directly from the photographing device).

[0015] Step S200 is an "identification step" that identifies the positions of a plurality of feature points on the user's body based on the acquired image. "Multiple feature points on the body" are points (positions) that can identify a specific position on the body, such as skeletal points such as the top of the head, neck, acromion, wrists, and ankles, points related to the positions of bones such as joints, and points related to the positions of body parts such as the corners of the eyes, the tip of the nose, and the tips of the fingers. In this embodiment, MediaPipe Pose is used to extract the following skeletal points: those used to identify height (e.g., top of the head, heels, etc.); those highly correlated with height (e.g., fingertips, shoulders, neck, pelvis, knees, heels, etc.); those that tend to differ depending on gender (e.g., pelvis, shoulders, rib cage, etc.); those related to the upper limbs (e.g., neck, shoulders, elbows, wrists, etc.); and those related to the lower limbs (e.g., buttocks, knees, ankles, heels, etc.). In this embodiment, the skeletal points used to identify height (top of the head 10, heels 13), those related to the upper limbs (shoulders 11), and those related to the lower limbs (buttocks 12) are extracted, and each of the extracted skeletal points is designated as a feature point. The coordinates (X (horizontal) and Y (vertical)) of each extracted feature point on the image are also identified. Furthermore, the user's height distance m (see FIG. 4) is calculated from the coordinates of the top of the head 10, which is one of the identified feature points, and the coordinates of the heel 13, which is also one of the feature points, and the number of pixels per predetermined unit (1 cm in this embodiment) on the image is calculated from the calculated height distance m and the user's actual height. The number of pixels per predetermined unit is used in the processing described below.

[0016] Step S300 is a "position identification step" that identifies, based on a plurality of feature points, a specific position located on the user's body in a direction from a feature point to another feature point, and determined by the positional relationship between the feature point and another feature point. "A specific position located in a direction from a feature point to another feature point on the user's body" refers to a position on the user's body that is located in a direction from a feature point to another feature point. "The direction from a feature point to another feature point" refers not only to an imaginary line connecting one feature point to another feature point, but also to a range toward the other feature point as viewed from the feature point, and up to the other feature point. "Determined by the positional relationship between a feature point and another feature point" means that it is determined based on the positions of multiple feature points and in the "direction from a feature point to another feature point," and therefore, for example, it is an interpolated position between the feature point and another feature point. Specifically, it is a predetermined position, a predetermined ratio position, a central position, or the like on a virtual line connecting the feature point to another feature point in the direction from the feature point to the other feature point. It may also be the intersection position of a virtual line at a predetermined angle from the feature point and a virtual line at a predetermined angle from the other feature point. It may also be a position on the perpendicular bisector of the virtual line connecting the feature point to the other feature point, which is a predetermined multiple of the distance between the two points. Furthermore, there may be additional feature points in addition to the feature point and the other feature point (three or more feature points), and it may be, for example, the center of gravity of all feature points or an average value obtained by weighting the coordinates of each feature point. The "specific position" determined by the feature point and its positional relationship with other feature points is a position on the user's body. "Based on the plurality of feature points" refers to the feature points described above and other feature points, and there is no limit to the number as long as it is two or more. In this embodiment, the specific position is set in the "abdomen" of the user's body. The "abdomen" is the lower half of the user's torso, and in this embodiment, it is the area near the navel in the lower half of the torso. The specific position in this embodiment will be described with reference to FIG. 4. In this embodiment, the feature point is the shoulder 11, and the other feature point is the buttocks 12. As shown in FIG. 4(a), a position 2 / 3 of the way from the shoulder 11 is determined with respect to a virtual line 21 connecting the shoulder 11 and the buttocks 12, and this position is designated as the specific position 22. This relational expression was derived as a result of extensive research into extracting various feature points from images of multiple subjects and determining the positional relationship of the extracted feature points that indicates the abdomen position. In other words, the present inventors identified multiple feature points correlated with the abdomen position and discovered a relational expression that shows the relationship between the multiple feature points. Note that, although the virtual line 21 is shown in FIG. 4(a) for ease of explanation, the virtual line 21 is not necessary because the coordinates of each acquired feature point are used. In the above, a position 2 / 3 from the shoulder 11 is determined on the imaginary line 21 connecting the shoulder 11 and the buttocks 12, and the position included in the abdomen is set as the specific position 22, but this is not limited to this, and position 22 can of course be changed appropriately to between 3 / 5 and 4 / 5 from the shoulder 11 on the imaginary line 21 connecting the shoulder 11 and the buttocks 12, and it is preferable to set it to between 2 / 3 and 3 / 4.

[0017] The measurement method used to derive the relational equation for multiple subjects will be explained using FIG. As shown in FIG. 5, the subject is asked to point to their navel with a finger or the like, and an image of this is obtained. This image is preferably the same as the image from which the user's skeletal points are extracted using MediaPipe Pose or the like. Then, of the extracted skeletal points, the percentage of the distance (on virtual line 21) along the Y axis from shoulder 11 to buttocks 12 at which the subject pointed to the navel was calculated. As a result of this measurement, the position at which the subject pointed to the navel was between 7 / 10 and 3 / 4 of the way from shoulder 11 to buttocks 12. It was also found that when the virtual line 25 connecting the buttocks 12 (left hip joint) and the buttocks 12 (right hip joint) is used as the base, the navel position is located at the vertex of the virtual equilateral triangle that does not include the virtual line 25. Therefore, if the feature point is the user's buttocks 12 (left hip joint), and the other feature point is the buttocks 12 (right hip joint), and virtual lines 26 and 27 are drawn at an angle of 60 degrees in the interpolation direction from both feature points, the intersection of these lines will be the navel position, and the position determined by this relationship is the abdomen, so it may be determined as the specific position. It should be noted that, although a plurality of feature points correlated with abdominal position are identified and a relational equation showing the relationship between the plurality of feature points is described, the information is not limited to a relational equation and may be a table showing the relationship. Any format of information may be used as long as it shows the correlation statistically obtained from feature points extracted by imaging a plurality of subjects and specific positions of the plurality of subjects (in this embodiment, the abdomens related to the specific positions). In this way, while it is easy to identify feature points such as skeletal points from an acquired image, the abdomen does not have any feature points like skeletal points, so in order to identify the abdomen, it was necessary to capture an image with the navel visible or with the user placing their hand on the abdomen. However, with this method, the abdomen can be identified from the positional relationship between the identified feature points and other feature points, so abdominal information can be estimated with a simple operation by the user, without requiring them to put on or take off excessive clothing that would expose their skin.

[0018] Step S600 is an "estimation step" in which physical information relating to the size or shape of the measurement target part relating to the specific position is estimated. The "size of the measurement target area" includes not only the perimeter (perimeter length), frontal length, lateral length, and cross-sectional area of ​​the measurement target area, but also the rank of each of the perimeter (perimeter length), frontal length, lateral length, and cross-sectional area of ​​the measurement target area (for example, for the perimeter (perimeter length), ranks are A: 60 cm or less, B: 61 cm to 75 cm, C: 76 cm to 90 cm, D: 91 cm to 105 cm, and E: 106 cm or more). Furthermore, the "shape of the measurement target area" includes terms that represent the shape, such as circle, horizontally elongated ellipse, and vertically elongated ellipse, as well as images and illustrations that show the estimated shape. Furthermore, as described above, "physical information" can be any information that represents the estimated results, such as numerical values, ranks, terms, images, and illustrations, and is not limited to specific numerical values. For example, an image in which a mark indicating the measurement target area or a perimeter line indicating the perimeter of the measurement target area is drawn on the acquired image is also physical information.

[0019] The estimation process may include an "information calculation process" that calculates a quantitative value that can be converted into physical information for each frame, and may estimate the physical information based on the calculated quantitative value. The information calculation process may also include a "smoothing process" that smoothes the calculated quantitative value for each frame using quantitative values ​​of other frames. The acquired image is a still image or a moving image of multiple frames. As shown in Figures 6(a) to 6(d), it is preferable to capture an image of 360 degrees around the user's area including the measurement target area. In this embodiment, the acquired image is a moving image of one revolution around the user, and the moving image is composed of a predetermined number of frames (360 frames in this embodiment). The "quantitative value that can be converted into physical information" is a value that indicates the size of the image of the measurement target area including a specific position in one frame, and may be the number of pixels A in Figure 4(a) or the number of pixels B in Figure 4(b), or a value obtained by replacing that number of pixels with a specified value, or any value that can be converted into physical information based on the number of pixels per cm that has been calculated in advance. "Smoothing the calculated quantitative values ​​for each frame using the quantitative values ​​of other frames" refers to a process of removing singular points and noise due to measurement errors in continuous data by averaging the quantitative values ​​for a specified frame using the quantitative values ​​(corresponding to "quantitative values ​​of other frames") of a specified number of frames before and after the frame (for example, 10 frames, 5 frames before and 5 frames after the frame). By including a smoothing process, which is a process of smoothing in this manner, it is possible to estimate the surroundings of a position including a specific position from data for one circumference of the user consisting of a specified number of acquired frames, with singular points and noise due to measurement errors removed.

[0020] The circumference and cross-sectional area of ​​the calculated measurement site may be calculated by calculating A in FIG. 4(a) from the maximum quantitative value calculated for each frame, and B in FIG. 4(b) from the minimum quantitative value calculated for each frame. Specifically, A in FIG. 4(a) (frontal length A) is estimated based on the number of pixels (number of pixels) corresponding to the maximum quantitative value and the number of pixels per cm calculated in advance. Furthermore, B in FIG. 4(b) (lateral length B) is estimated based on the number of pixels (number of pixels) corresponding to the minimum quantitative value and the number of pixels per cm calculated in advance. In this way, it is preferable to use the maximum quantitative value calculated for each frame when calculating the frontal length, and the minimum quantitative value calculated for each frame when calculating the lateral length.

[0021] Based on the calculated front length A and side length B, the cross-sectional area S of the measurement target part (abdomen) and the perimeter of the measurement target part (abdomen) are estimated as physical information as follows. <Cross-sectional area S of the part to be measured> Since the cross-sectional area parallel to the width direction of the measurement target area is elliptical, the cross-sectional area S of the abdomen is calculated assuming that the ellipse has a front length A, a major axis length A, and a side length B, a minor axis length B. Figure 7 is a diagram showing an ellipse with a major axis that is half the front length A and a minor axis that is half the side length B. The area S of the ellipse shown in Figure 7 is calculated using the following formula (1).

number

[0022] <Circumference of the part to be measured> The abdominal circumference L is the circumference of the ellipse shown in FIG. 7, and can be calculated using the front length A and the side length B using the following formula (2).

number

[0023] Under the condition of B>A, the approximation of the above formula (2) is expressed as the following formula (3).

number

[0024] Next, a method for improving the accuracy of estimating physical information will be described. One factor that may reduce the accuracy of physical information estimation is the thickness of the clothing worn by the user. This method aims to enable users to easily measure target body parts in their daily lives, so it is assumed that the user will capture images while wearing their own clothing. Since the clothing is the user's own clothing, it is expected that the thickness and material of the clothing will vary. Therefore, the estimation process preferably sets the thickness of the clothing according to the type of clothing worn by the user, and preferably also includes a process for correcting the estimated physical information according to the thickness of the clothing worn by the user. A table containing the thickness of clothing for each type of clothing (e.g., thin T-shirts, dress shirts, cut-and-sew tops) is prepared in advance, and the user is asked to specify the type of clothing to determine the thickness of the clothing. The estimated physical information is then corrected according to the determined clothing thickness. The correction method involves measuring in advance how much the estimated nude measurements should be increased for each type of clothing, and then subtracting that percentage from the estimated value. Furthermore, when measuring how much the estimated nude measurements should be increased for each type of clothing, it is preferable to measure each way the clothing is worn. The way in which clothing is worn, for example, the amount of surcharge to nude measurements varies depending on whether the same T-shirt is worn over pants or tucked into them. Therefore, it is more effective to require the user to specify the way in which the clothing is worn in addition to the type of clothing. Since it is assumed that some users wear multiple pieces of clothing, accuracy can be improved by having the user specify this if multiple pieces are worn. Instead of requiring the user to specify the type of clothing, the user may also be required to specify the fashion style being worn. The fashion style indicates the clothing and the way in which it is worn, such as a style in which a T-shirt is worn over a jersey, a style in which a shirt is worn over jeans, or a style in which a T-shirt is tucked into jeans. The user may specify the fashion style using an image depicting the fashion style. The amount of surcharge to nude measurements is then measured in advance for each fashion style. This enables corrections to be made taking into account the type, way in which it is worn, and the number of pieces of clothing.Alternatively, instead of the user specifying the type of clothing, machine learning may be used to analyze the acquired image of the user to determine the type, manner of wearing, number of pieces of clothing, or fashion style, and then a surcharge for nude measurements may be determined and corrections may be made based on the determined results. It is also preferable to have the user specify which body part is to be measured. For example, even when wearing a T-shirt, the degree of influence (surcharge) of the T-shirt on the abdomen differs from the degree of influence (surcharge) of the T-shirt on the upper arms. Therefore, it is preferable to measure the surcharge for nude measurements for each type of clothing, manner of wearing, and measurement part, and use it for corrections. In this manner, in this embodiment, highly accurate estimation can be performed even if the feature points and other feature points of the acquired image and the measurement target area are covered with clothing.

[0025] As described above, in this embodiment, the user's height distance m is calculated from the coordinates from the vertex 10 to the heel 13, and the number of pixels per predetermined unit on the image is calculated based on the user's actual height. Therefore, it is preferable to provide a step ("physical feature acquisition step") of acquiring height from the user as a physical feature other than the measurement target body part before calculating the number of pixels per predetermined unit (e.g., before step S200 in FIG. 3). In addition to height, other physical features such as weight, age, and gender may also be acquired as "physical features." By acquiring data that correlates with the positional relationship between the estimated body part, the feature point, and other feature points (e.g., a correlation between the positional relationship between the estimated body part, the feature point, and other feature points and age, or a correlation between the positional relationship between the estimated body part, the feature point, and other feature points and gender), it is possible to improve estimation accuracy. In other words, this embodiment preferably further includes a physical feature acquisition step of acquiring the user's physical features other than the measurement target body part, and the estimation step preferably uses the physical features to estimate the physical information.

[0026] <Physical Information Estimation System> FIG. 8 shows a conceptual diagram of a physical information estimation system 400 (hereinafter, sometimes referred to as "this system"). This system is composed of an acquisition unit 330, an identification unit 340, a position identification unit 350, and an estimation unit 360. It also includes an information processing terminal 300 capable of executing various processes, and the information processing terminal 300 is equipped with the acquisition unit 330, the identification unit 340, the position identification unit 350, and the estimation unit 360. The information processing terminal 300 is equipped with an input device such as a keyboard or a pointing device, a processing unit, a memory unit, etc. It is preferable that the information processing terminal 300 is equipped with a display device (display unit 380), but this may be provided external to the information processing terminal and connected via a network. The physical information estimation system 400 may also include an internal or external imaging device 20.

[0027] The acquisition unit 330 is a means for acquiring images of a measurement target part of the user's body from two or more different directions, and corresponds to an acquisition means. The identification unit 340 is a means for identifying the positions of a plurality of feature points on the user's body based on the acquired image, and corresponds to an identification means. The position specifying unit 350 is a means for specifying a specific position located on the user's body in a direction from a feature point to another feature point based on the feature point, and corresponds to a position specifying means. The estimation unit 360 corresponds to an estimation means for estimating physical information relating to the size or shape of the measurement target part relating to the identified specific position. This system is capable of executing the processing content of the physical information estimation method described above. From the above, the present invention discloses a physical information estimation system that estimates physical information of a user using a control device, including: acquisition means for acquiring images of a measurement target part on the user's body from two or more different directions; identification means for identifying the positions of a plurality of feature points on the user's body based on the images; position identification means for identifying, based on the plurality of feature points, a specific position on the user's body that is located in a direction from a feature point to another feature point, the specific position being determined by the positional relationship between the feature point and the other feature points; and estimation means for estimating physical information related to the size or shape of the measurement target part related to the identified specific position.

[0028] Furthermore, the physical information estimation system 400 may further include a body fat mass estimation unit (body fat mass estimation means) that estimates the body fat mass contained in the user's abdomen from relationship information that indicates the relationship between the estimated abdominal information and the body fat mass contained in the abdomen. In this case, the body fat mass estimation unit may be provided in the information processing terminal 300. A method for estimating body fat mass from the acquired abdominal information will be described below.

[0029] Focusing on the fact that body fat mass correlates with abdominal information, the body fat mass is estimated using relational information indicating the relationship between abdominal information estimated by this system and body fat mass. "Relationship information showing the relationship between abdominal information and the amount of body fat contained in the abdomen" is information showing the correlation between abdominal information and the amount of body fat contained in the abdomen, such as an arithmetic formula or table showing the relationship between the front length and lateral length of abdominal information and the amount of body fat, or an arithmetic formula or table showing the relationship between the circumference of abdominal information and the amount of body fat. Table 1 is an example of relationship information and is a table showing the relationship. Table 1 associates the major axis A (corresponding to "frontal length A") and the minor axis B (corresponding to "lateral length B") with the cross-sectional area S of body fat (also referred to as cross-sectional area S) within the cross-sectional area of ​​the abdomen. The table shown in Table 1 divides the expected numerical range of the frontal length and lateral length into multiple stages, and associates the cross-sectional area S of body fat with the frontal length and lateral width length of each stage. The table shown in Table 1 was created in advance from measurements of the frontal length and lateral width lengths of multiple subjects, and the cross-sectional areas S indicating the body fat mass of these multiple subjects measured using methods such as CT scans. The ranges (stages) of the frontal length and the lateral length shown in Table 1 may be determined depending on the measurement accuracy and the accuracy of the cross-sectional area of ​​the body fat mass to be estimated. [Table 1]

[0030] Table 2 is another example of relationship information, a table showing the relationship. Table 2 associates circumference L (corresponding to "abdominal circumference") with the cross-sectional area S of body fat (also called cross-sectional area S) that occupies the cross-sectional area of ​​the abdomen. The table shown in Table 2 divides the expected numerical range of circumference into multiple stages in advance, and associates the cross-sectional area S of body fat with the circumference of each stage. The table shown in Table 2 was created in advance from measurements of circumferences of multiple subjects and the cross-sectional areas S indicating the body fat mass of these multiple subjects, measured using a method such as CT scan. The ranges (stages) of circumference shown in Table 2 may be determined depending on the measurement accuracy and the accuracy of the cross-sectional area of ​​the body fat mass to be estimated. [Table 2]

[0031] As described above, this system can estimate body fat mass using relationship information indicating the relationship between the estimated abdominal information and body fat mass. While the above description has been given for a standard body type with an oval abdominal cross-section, it is known that the location of body fat affects body shape (abdominal cross-sectional shape). Specifically, the following applies: The "standard body type" has an oval abdominal cross-section, with gradually increasing abdominal thickness (lateral length). The "visceral obesity type" has a large amount of visceral fat and an abdominal cross-section that is closer to a circle than an oval. The "subcutaneous obesity type" has a large amount of subcutaneous fat, fat on the sides of the back (both sides), and different frontal lengths on the navel side and back side. The "athlete type" tends to have less body fat and a large abdominal pressure due to well-developed abdominal muscles, back muscles, and iliopsoas muscles. Because the relationship between abdominal circumference and body fat mass differs depending on the type (depending on the type), it is preferable to estimate body fat mass using the estimated abdominal circumference as well as the abdominal cross-sectional shape, which is one piece of abdominal information. In this case, the relationship between abdominal circumference and body fat mass (e.g., Table 2) is determined for each type, the type is identified based on the cross-sectional shape, and body fat mass is estimated based on the relationship between abdominal circumference and body fat mass for the identified type. This enables more accurate estimation of body fat mass based on estimated abdominal information. In the case of the athlete type, it is more preferable to correct the estimated body fat mass based on muscle mass. In this case, the circumference of a part of the body where muscle accounts for a high proportion of the cross-sectional area, such as the upper arms or thighs, may be estimated, and the estimated body fat mass may be corrected using the estimated circumference. Specifically, since it can be inferred that the larger the circumference of the upper arms or thighs, the greater the muscle mass, the relationship between the circumference and the percentage by which the estimated abdominal body fat mass should be reduced may be determined in advance, and correction may be performed based on this relationship.

[0032] As described above, the present invention has been described by showing specific embodiments, but the present invention is not limited to the above-described embodiments and includes various modifications, improvements, and other aspects as long as the object of the present invention is achieved. <Modification> In this embodiment, the number of pixels per predetermined unit on the image is calculated from the user's height distance in the image and the user's actual height, and abdominal information is estimated using the number of pixels per predetermined unit, but this is not limited to this. For example, physical information may be estimated from the length of a specific position in the image based on the distance between the image capture device 20 and the user when capturing an image of the user. Alternatively, measurements (actual lengths) of other body parts may be used instead of height. While similar results can be obtained in this way, it is preferable to use actual height considering the burden on the user when capturing an image (measuring the distance between the image capture device 20 and the user, or capturing an image at a predetermined distance, or measuring other body parts).

[0033] The feature point may be the shoulder 11, and the other feature point may be the knee 15. A diagram showing the positional relationship between the shoulder 11 and the knee 15 is shown in FIG. As shown in Fig. 9, it was found that the navel is located between 2 / 5 and 9 / 25 from the shoulder 11 with respect to the imaginary line 28 connecting the shoulder 11 and the knee 15. Therefore, the relationship found by setting the shoulder 11 as the feature point and the knee 15 as the other feature point may be used.

[0034] In this embodiment, the feature point is the user's shoulder 11 (shoulder joint) in the acquired image, and the other feature point is the user's buttocks 12 (hip joint) in the acquired image. This is because, when using known skeletal information acquisition technology, it is easy to accurately acquire the user's skeletal points, especially joints, in the acquired image, so by using joints as feature points, it is possible to accurately identify specific positions included in the abdomen. However, it is also possible to make highly accurate estimations using skeletal points other than joints.

[0035] In the present embodiment, as shown in the cases where the measurement target site (abdomen) is identified based on the positional relationship between the feature point (shoulder 11) and another feature point (buttocks 12) and the case where the measurement target site (abdomen) is identified based on the positional relationship between the feature point (shoulder 11) and another feature point (knee 15), two feature points are used to identify the measurement target site (abdomen) in both cases. However, this is not limited to this. For example, the specific position (abdomen) may be identified by averaging the results of both the positional relationship between the shoulder 11 (right shoulder) and the buttocks 12 (right buttocks) and the positional relationship between the shoulder 11 (left shoulder) and the buttocks 12 (left buttocks). That is, the specific position (abdomen) may be identified by averaging the positional relationship between the feature point on the right half of the body and another feature point and the positional relationship between the feature point on the left half of the body and another feature point. Many users have different skeletal positions (heights) between the right and left halves of their body due to lifestyle and other factors. Therefore, the measurement target site (abdomen) can be identified by averaging the results of the right and left halves of the body to improve the accuracy of physical information estimation. Furthermore, for example, the measurement target site (abdomen) may be identified based on the positional relationship between the shoulder 11 (right shoulder) and the buttocks 12 (left buttock) and the positional relationship between the shoulder 11 (left shoulder), the buttocks 12 (right buttock), and the buttocks 12 (left buttock). The abdomen is located near the navel and is therefore approximately at the center of the user's body in the left-right direction. Therefore, the specific position (abdomen) may be identified based on the intersection of the positional relationship between the shoulder 11 (right shoulder) and the buttocks 12 (left buttock) (a virtual line connecting the shoulder 11 (right shoulder) and the buttocks 12 (left buttock)) and the positional relationship between the shoulder 11 (left shoulder), the buttocks 12 (right buttock), and the buttocks 12 (left buttock) (a virtual line connecting the shoulder 11 (left shoulder), the buttocks 12 (right buttock), and the buttocks 12 (left buttock). In other words, the specific position (abdomen) may be identified based on the positional relationship between a feature point on the right half of the body and another feature point on the left half of the body.

[0036] In this embodiment, when identifying the abdomen based on the positional relationship between the feature point (shoulder 11) and another feature point (buttocks 12), a position between 3 / 5 and 4 / 5 from the shoulder 11 on the virtual line 21 connecting the feature point (shoulder 11) and another feature point (buttocks 12) is determined, and the position that is included in the abdomen is set as the specific position 22. However, this is not limited to this. For example, a Y coordinate value between 3 / 5 and 4 / 5 from the shoulder between the Y coordinate value of the feature point (shoulder 11) and the Y coordinate value of another feature point (buttocks 12) may be determined, and the position that is the Y coordinate value and is included in the abdomen may be set as the specific position 22. In either case, it is possible to estimate (abdominal information) with high accuracy. Similarly, when identifying the abdomen based on the positional relationship between the feature point (shoulder 11) and another feature point (knee 15), a position between 2 / 5 and 9 / 25 from the shoulder 11 on the virtual line 28 connecting the feature point (shoulder 11) and another feature point (knee 15) is determined, and the position that is included in the abdomen is regarded as the specific position, but this is not limited to this. For example, a Y coordinate value between 2 / 5 and 9 / 25 from the shoulder between the Y coordinate value of the feature point (shoulder 11) and the Y coordinate value of another feature point (knee 15) may be determined, and the position that is included in the abdomen and has that Y coordinate value may be regarded as the specific position. In either case, highly accurate physical information (abdominal information) can be estimated.

[0037] In this embodiment, the abdomen is used as the measurement target site and abdominal information is estimated as the physical information, but other sites may also be used. FIG. 10 shows the measurement sites on the upper arms and thighs. In FIG. 10 , specific positions are set on the upper arms and thighs of the user's body. The "upper arms" are located between the shoulders and elbows and are formed by the biceps and triceps. People who exercise regularly are interested in the size of their biceps and triceps, or in other words, the size of their upper arms, which are formed by the biceps and triceps, as an indicator of the results of their exercise. The "thighs" are located between the buttocks and knees and are formed by the rectus femoris, vastus lateralis, vastus medialis, and vastus intermedius. People who exercise regularly are interested in the size of their rectus femoris, vastus lateralis, vastus medialis, and vastus intermedius, or in other words, the size of their thighs, which are formed by the rectus femoris, vastus lateralis, vastus medialis, and vastus intermedius, as an indicator of the results of their exercise. The muscles that make up the upper arms and thighs play an important role in daily activities and also help burn fat, so they are of interest not only to people interested in improving athletic ability and maintaining their figure, but also to the elderly as a way to prevent frailty. Therefore, in this embodiment, the measurement sites are set to the "upper arms" and "thighs," which are effective locations for the upper arms and thighs and allow the user to measure certain locations using a simple method when taking measurements continuously.

[0038] FIG. 10 is a diagram showing a specific position of the upper arm (referred to as "specific position a1"). The feature point used to identify specific position a1 is the shoulder 11, and the other feature point is the elbow 14. As shown in FIG. 10(a), specific position a1 is located at 1 / 2 of the virtual line 23 connecting the shoulder 11 and the elbow 14, and the position including specific position a1 is the upper arm, which is the measurement target part in this embodiment. This is because the "upper arm" is generally located at 1 / 2 of the virtual line 23 connecting the shoulder 11 and the elbow 14. However, this is not limited to this, and the specific position can be changed as appropriate to between 1 / 3 and 3 / 4 of the virtual line 23 connecting the shoulder 11 and the elbow 14, and preferably between 2 / 5 and 2 / 3. In this way, even for the upper arm, which is a body part without a skeletal point, by identifying the specific position based on the positional relationship between the shoulder 11 and the elbow 14, it is possible to measure the upper arm at a specific position. Although the virtual line 23 is shown in FIG. 10(a) for the sake of convenience, the virtual line 23 does not need to be provided because the coordinates of each acquired feature point are used. FIG. 10 also shows a specific position on the thigh (referred to as "specific position a2"). The feature point used to identify specific position a2 is the buttocks 12, and the other feature point is the knee 15. As shown in FIG. 10(a), specific position a2 is located at 1 / 2 of the virtual line 24 connecting the buttocks 12 and the knee 15, and the position including specific position a2 is the thigh, which is the measurement target area in this embodiment. This is because the "thigh" is generally located at 1 / 2 of the virtual line 24 connecting the buttocks 12 and the knee 15. However, this is not limited to this, and the specific position can be changed as appropriate between 1 / 3 and 3 / 4 of the virtual line 24 connecting the buttocks 12 and the knee 15, and preferably between 2 / 5 and 2 / 3. In this way, even for the thigh, a body part without a skeletal point, a specific position can be identified based on the positional relationship between the buttocks 12 and the knee 15, making it possible to measure the thigh at a specific position. Although the virtual line 24 is shown in FIG. 10(a) for the sake of convenience, the virtual line 24 does not need to be provided because the coordinates of each acquired feature point are used.

[0039] The smoothing process of this method is not limited to the above. The purpose of the present invention is to allow a user to easily capture an image of themselves using a smartphone or other device and estimate their physical information. However, there is a possibility that the user may not capture the image properly, resulting in missing parts in the captured image, or that the measurement target area may overlap with other body parts and not be captured properly. Even in such cases, for areas that are generally oval-shaped, such as the abdomen, the frontal length and lateral length can be calculated using an ellipse approximation formula using the number of pixels identified from the captured image, thereby estimating the physical information of the measurement target area. Furthermore, an approximation function can be calculated using a known method, such as curve fitting, from the number of pixels of A shown in FIG. 4(a) and the number of pixels of B shown in FIG. 4(b), and the number of pixels when viewed from an imaging direction where the image is missing can be interpolated from the calculated approximation function, allowing physical information of the measurement target area to be estimated. By including such a smoothing process, it is possible to estimate physical information of the measurement target area even if there are defects in the acquired image or if it is not possible to acquire an image of the user from directly in front or directly to the side (side).

[0040] If an ellipse approximation formula is not used, such as when the cross-sectional shape of the measurement target area is not elliptical, the imaging direction cannot be estimated from the trend in changes in the number of pixels A (the number of pixels determined from an image of the user captured from the front) and the number of pixels B (the number of pixels determined from an image of the user captured from the side). Therefore, the calculated number of pixels may be stored in association with information indicating the direction in which the user was captured, and the cross-sectional shape of the measurement target area may be estimated by allocating the number of pixels to 360 degrees using the information indicating the imaging direction. In this case, the information indicating the direction in which the user was captured may be information extracted from the orientation of the user's face or hands in the acquired image. This makes it possible to estimate physical information about areas whose cross-sectional shape is not elliptical without increasing the user's workload. By including such a smoothing step, it is possible to estimate physical information regardless of the cross-sectional shape of the measurement target site.

[0041] In this embodiment, the information calculation process includes a smoothing process in which the quantitative value for each frame is smoothed using the quantitative values ​​of other frames, but this is not limiting. For example, a process of interpolating the quantitative values ​​between frames may be performed based on the quantitative values ​​for each frame.

[0042] In this embodiment, an image capturing one circumference of the user's body is used, but this is not limiting. For example, one circumference may be estimated by doubling the estimated circumference value using a quantitative value of a position including a specific position for each frame of an image capturing half the circumference. In this way, it is possible to estimate one circumference even when it is difficult to capture an image of the inside of the body, such as the thigh.

[0043] A plurality of measurement target regions may be provided, and the estimation results of one measurement target region may be used to estimate the other measurement target regions. In this way, the acquired data can be used effectively.

[0044] The present invention has been described above with reference to the embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the disclosure in this specification. Note that, in the present invention, the preferred examples of the physical information estimation device, the physical information estimation program, and the physical information estimation system are similar to those described above in relation to the physical information estimation method. [Explanation of symbols]

[0045] 10 top of head 11 Shoulders 12 Butt 13 Heel 14 Elbow 15 Knee 20 Imaging device 21 Virtual Line 22 Specific position 25 Virtual Line 26 Virtual Line 27 Virtual Line 28 Virtual Line 300 Information processing terminal 330 Acquisition Department 340 Specific part 350 Location identification part 360 Estimation Department 380 Display 400 Physical Information Estimation System

Claims

1. an acquiring step of acquiring images of a measurement target part of a user's body from two or more different directions; an identifying step of identifying positions of a plurality of feature points on the user's body based on the image; a position specifying step of specifying, based on the plurality of feature points, a specific position located on the body of the user in a direction from the feature point to another feature point, the specific position being determined by a positional relationship between the feature point and the other feature point; an estimation step of estimating physical information relating to the size or shape of the measurement target part related to the identified specific position.

2. the image is a still image or a moving image of multiple frames, The estimation step an information calculation step of calculating a quantitative value that can be converted into the physical information for each frame, The method for estimating physical information according to claim 1 , wherein the physical information is estimated based on the calculated quantitative value.

3. the information calculation step includes a smoothing step of smoothing the calculated quantitative value for each frame using the quantitative values ​​of other frames; 3. The physical information estimating method according to claim 2, wherein the estimating step estimates the physical information based on the smoothed quantitative value.

4. 3. The physical information estimation method according to claim 1, wherein the feature point is a shoulder joint of the user, and the other feature point is a hip joint of the user.

5. a physical feature acquiring step of acquiring a physical feature of the user other than the measurement target part, 3. The physical information estimating method according to claim 1, wherein the estimating step estimates the physical information using the physical characteristics as well.

6. 3. The physical information estimating method according to claim 1, wherein the estimating step corrects the physical information according to a thickness of clothing worn by the user.

7. A physical information estimation system that estimates physical information of a user using a control device, an acquisition means for acquiring images of a measurement target part of a user's body from two or more different directions; an identification means for identifying positions of a plurality of feature points on the user's body based on the image; a position specifying means for specifying a specific position located on the user's body in a direction from the feature point to another feature point, the specific position being determined by a positional relationship between the feature point and the other feature point, based on the plurality of feature points; and an estimation means for estimating physical information relating to the size or shape of the measurement target part related to the identified specific position.

Images