2d- and 3d-data fusion-based method and apparatus for measuring body circumference

Abstract

The present invention relates to a 2D- and 3D-data fusion-based method and apparatus for measuring body circumference, wherein the method is performed by the apparatus comprising: a memory; and a processor electrically connected to the memory, the method comprising the steps of: collecting a 2D image and 3D data related to at least a portion of a user's body part; detecting, from the 2D image, a pair of key points for measuring the circumference of the corresponding body part; correcting positions of the key points according to a reference direction for circumference measurement, and identifying, from among points between reference points corresponding to the corrected key points in the 3D data, two points closest to the corresponding body part as reference points; and measuring the circumference of the corresponding body part using the reference points and the 3D data.

Description

Body circumference measurement method and device based on 2D and 3D data fusion

[0001] The present invention relates to an automated body circumference measurement technology, and more particularly, to an automated body circumference measurement technology capable of extracting key points unique to a specific body part of a person within a two-dimensional image based on a posture estimation technology and a lidar sensor technology and specifying the circumference of a specific body part using three-dimensional data.

[0002] Human pose estimation technology utilizes computer vision and artificial intelligence to estimate the positions of human body parts and joints. In other words, human pose estimation aims to automatically recognize people in images or videos and accurately identify the positions of key body parts, such as the head, shoulders, elbows, wrists, hips, knees, and ankles.

[0003] LiDAR (Light Detection and Ranging) sensors are a technology that uses light to measure distance and location. Specifically, LiDAR sensors can be used to calculate the distance to an object by emitting a laser and receiving the reflected light. LiDAR sensors are particularly effective in recognizing and mapping the surrounding environment in 3D, and can be utilized in various fields such as autonomous vehicles, robots, and drones.

[0004] Body circumference measurement is widely required in real-life applications such as online clothing shopping, clothing information, and home workouts, but it presents challenges that can be difficult to solve without assistance. Therefore, in modern society, there is a growing demand for technology that allows people to automatically measure their body circumference using easily accessible mobile devices, enhancing convenience. However, currently available technologies are limited to specific areas or lack sufficient performance to satisfy the actual market.

[0005] One embodiment of the present invention provides an automated body circumference measurement technology capable of extracting key points unique to a specific body part of a person within a two-dimensional image based on a posture estimation technology and a lidar sensor technology and measuring the circumference of the specific body part using three-dimensional data.

[0006] One embodiment of the present invention provides a body circumference measurement method and device based on 2D and 3D data fusion, which can improve the accuracy of body circumference measurement by correcting the position of a reference point that serves as a body circumference measurement standard and calculating the circumference of a body part based on the corrected reference point.

[0007] Among the embodiments, a method for measuring a body circumference based on 2D and 3D data fusion is provided, which is performed in a device including a memory and a processor electrically connected to the memory, the method comprising: a step of collecting a 2D image and 3D data regarding at least a part of a body part of a user; a step of inputting the 2D image into a pre-built key point detection model to detect a pair of key points for measuring the circumference of the corresponding body part; a step of correcting the position of the key point according to a circumference measurement reference direction and identifying a reference point as two points closest to the corresponding body part among points between reference points corresponding to the corrected key points in the 3D data; and a step of measuring the circumference of the corresponding body part using the reference point and the 3D data.

[0008] The above collecting step may include a step of photographing a part of the user's body using a photographing device including a LiDAR (Light Detection and Ranging) sensor.

[0009] The above 3D data may include data regarding distance values ​​from the LiDAR sensor to points on the user's body part.

[0010] The above key point detection model can be built based on HRNet (High-Resolution Network).

[0011] The step of identifying the reference point may include a step of adjusting the coordinates of one of the pair of key points to the same value based on the coordinates of the other point along the circumference measurement reference direction so that the pair of key points are located on the same reference line defined in the circumference measurement reference direction; and a step of correcting the key points by spreading the positions of the key points by a specific distance so that the distance between the key points becomes longer than the length of the body part to be measured for circumference.

[0012] The step of identifying the above reference point may include calculating a Euclidean distance between adjacent points between the reference points, determining two points closest to the point on the corresponding body part among a plurality of points having the largest distance, and correcting them as the reference point.

[0013] The step of measuring the circumference may include a step of calculating the Euclidean distance between adjacent points of 3D points forming the circumference of the corresponding body part between the reference points to measure the front and rear circumferences of the corresponding body part, respectively; and a step of determining the circumference of the corresponding body part by adding up the front and rear circumferences.

[0014] In one embodiment, a body circumference measurement device based on 2D and 3D data fusion includes a memory; and a processor electrically connected to the memory, wherein the processor collects a 2D image and 3D data regarding at least a part of a body part of a user, detects a pair of key points for measuring the circumference of the corresponding body part from the 2D image, corrects the positions of the key points according to a circumference measurement reference direction, identifies a reference point as two points closest to the corresponding body part among points between reference points corresponding to the corrected key points among the 3D data, and measures the circumference of the corresponding body part using the reference point and the 3D data.

[0015] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment must include all or only the following effects, and therefore the scope of the disclosed technology should not be construed as being limited thereby.

[0016] A body circumference measurement method and device based on 2D and 3D data fusion according to one embodiment of the present invention can extract key points unique to a specific body part of a person within a 2D image based on posture estimation technology and lidar sensor technology and measure the circumference of a specific body part using 3D data.

[0017] A body circumference measurement method and device based on 2D and 3D data fusion according to one embodiment of the present invention can improve the accuracy of body circumference measurement by correcting the position of a reference point that serves as a body circumference measurement standard and calculating the circumference of a body part based on the corrected reference point.

[0018] Figure 1 is a drawing illustrating a measurement system according to the present invention.

[0019] Fig. 2 is a drawing explaining the system configuration of the measuring device of Fig. 1.

[0020] Figure 3 is a drawing explaining the functional configuration of the processor of Figure 2.

[0021] Figure 4 is a flowchart illustrating a body circumference measurement method based on 2D and 3D data fusion according to the present invention.

[0022] FIG. 5 is a drawing illustrating one embodiment of a 2D and 3D data collection process according to the present invention.

[0023] FIG. 6 is a drawing illustrating one embodiment of a process for detecting key points on a 2D image according to the present invention.

[0024] FIG. 7 is a drawing illustrating one embodiment of a process for measuring the circumference of a body part according to the present invention.

[0025] The description of the present invention is merely an example for structural and functional explanation, and therefore, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be modified in various ways and can take various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purposes or effects presented in the present invention do not mean that a specific embodiment must include all of them or only such effects, and therefore, the scope of the present invention should not be construed as being limited thereby.

[0026] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0027] Terms such as "first" and "second" are intended to distinguish one component from another, and the scope of the rights should not be limited by these terms. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

[0028] When a component is said to be "connected" to another component, it should be understood that while it may be directly connected to that other component, there may also be other components intervening. Conversely, when a component is said to be "directly connected" to another component, it should be understood that there are no other intervening components. Similarly, other expressions describing relationships between components, such as "between" and "directly between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0029] Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "comprises" or "have" should be understood to specify the presence of a feature, number, step, operation, component, part or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0030] For each step, the identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps. The steps may occur in a different order than stated unless the context clearly dictates a specific order. That is, the steps may occur in the same order as stated, may be performed substantially simultaneously, or may be performed in the opposite order.

[0031] The present invention can be implemented as computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Furthermore, the computer-readable recording medium can be distributed across network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner.

[0032] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted to be consistent with their meaning within the context of the relevant technology, and should not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0033] Figure 1 is a drawing illustrating a measurement system according to the present invention.

[0034] Referring to FIG. 1, the measurement system (100) may include a user terminal (110), a measurement device (130), and a database (150).

[0035] The user terminal (110) may be a computing device operated by a user that transmits data via a network or utilizes a specific service. For example, the user terminal (110) may be a terminal device capable of generating and providing images for executing the body circumference measurement method according to the present invention. For this purpose, the user terminal (110) may be implemented with a camera equipped with a LiDAR sensor.

[0036] In addition, the user terminal (110) may be implemented as a smartphone, laptop, or computer, but is not necessarily limited thereto, and may be implemented as various devices including tablet PCs, etc. The user terminal (110) may be implemented as one device constituting the measurement system (100) according to the present invention, and the measurement system (100) may be implemented in various forms depending on the subject and purpose of circumference measurement.

[0037] Additionally, the user terminal (110) can be connected to the measuring device (130) via a network, and multiple user terminals (110) can be connected to the measuring device (130) simultaneously. The user terminal can install and execute a dedicated program or application for linking with the measuring device (130).

[0038] The measuring device (130) may be implemented as a computer or server that performs a body circumference measurement method based on 2D and 3D data fusion according to the present invention. For example, the measuring device (130) may be implemented as a server that identifies a body part from image data, performs a circumference measurement operation, generates a measurement result related thereto, and provides it to a user terminal (110). Each step of the measuring method according to the present invention may be performed on the server.

[0039] In addition, the measuring device (130) can be connected to the user terminal (110) via a wired network or a wireless network such as Bluetooth, WiFi, LTE, etc., and can transmit and receive data with the user terminal (110) via the network.

[0040] Additionally, the measuring device (130) may be implemented to operate in connection with an independent external system (not shown in FIG. 1). For example, the measuring device (130) may operate in conjunction with a system that generates and provides at least one of a 2D image and 3D data regarding at least a portion of a user's body part.

[0041] The database (150) may correspond to a storage device that stores various information required during the operation of the measuring device (130). The database (150) may store 2D images and 3D data regarding body parts, and may store algorithm and model information for the body circumference measurement operation. However, the database is not necessarily limited thereto, and may store information collected or processed in various forms during the process of the measuring device (130) performing the body circumference measurement method based on 2D and 3D data fusion according to the present invention.

[0042] In addition, in FIG. 1, the user terminal (110), the measuring device (130), and the database (150) are each depicted as independent devices, but this is not necessarily limited to the above, and it goes without saying that one device may be included in the other device and implemented as a logical device.

[0043] Fig. 2 is a drawing explaining the system configuration of the measuring device of Fig. 1.

[0044] Referring to FIG. 2, the measuring device (130) may include a processor (210), a memory (230), a user input / output unit (250), and a network input / output unit (270).

[0045] The processor (210) can execute various operation procedures for the operation of the measuring device (130) according to one embodiment of the present invention, manage the memory (230) that is read or written in this process, and schedule a synchronization time between the volatile memory and the non-volatile memory in the memory (230). The processor (210) can control the overall operation of the measuring device (130), and is electrically connected to the memory (230), the user input / output unit (250), and the network input / output unit (270) to control the data flow therebetween. The processor (210) can be implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the measuring device (130).

[0046] The memory (230) may include an auxiliary memory device implemented as a non-volatile memory such as an SSD (Solid State Disk) or an HDD (Hard Disk Drive) and used to store all data required for the measuring device (130), and may include a main memory device implemented as a volatile memory such as a RAM (Random Access Memory). In addition, the memory (230) may store a set of commands that execute various operation methods of the measuring device (130) according to the present invention by being executed by an electrically connected processor (210).

[0047] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include, for example, an input device including an adapter such as a touchpad, a touch screen, a virtual keyboard, or a pointing device, and an output device including an adapter such as a monitor or a touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device that is accessed via a remote connection.

[0048] The network input / output unit (270) provides a communication environment for connecting to a user terminal (110) via a network, and may include, for example, an adapter for communication such as a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), and a Value Added Network (VAN). In addition, the network input / output unit (270) may be implemented to provide a short-range communication function such as WiFi or Bluetooth, or a wireless communication function of 4G or higher for wireless transmission of data.

[0049] Figure 3 is a drawing explaining the functional configuration of the processor of Figure 2.

[0050] Referring to FIG. 3, the measuring device (130) can perform a body circumference measurement method based on 2D and 3D data fusion according to the present invention through a processor (210). To this end, the processor (210) can include a data collection unit (310), a key point detection unit (330), a reference point identification unit (350), a body circumference measurement unit (370), and a control unit (not shown in FIG. 3).

[0051] At this time, the embodiment of the user terminal (110) according to the present invention does not have to include all of the above configurations at the same time, and may be implemented by omitting some of the above configurations or selectively including some or all of the above configurations according to each embodiment.

[0052] The data collection unit (310) can collect 2D images and 3D data regarding at least a portion of a user's body part. To this end, the data collection unit (310) can operate in conjunction with the user terminal (110), and in particular, can operate in direct conjunction with the camera of the user terminal (110). The data collection unit (310) can collect 2D images by photographing the user's body part through the camera. For example, the 2D image may correspond to an RGB image (or color image). In addition, the data collection unit (310) can collect 3D data by scanning the user's body part in 3D. At this time, the data collection unit (310) can collect the 2D images and 3D data simultaneously or separately while maintaining the same position, direction, and angle.

[0053] In one embodiment, the data collection unit (310) can capture a portion of a user's body using a photographing device including a LiDAR (Light Detection and Ranging) sensor. For example, the photographing device may be a smartphone equipped with a high-resolution camera and a LiDAR sensor. Furthermore, the photographing device may include a wearable device worn on the wrist or arm, equipped with both a camera and a sensor, and may also include specialized measuring equipment capable of measuring 3D data.

[0054] In one embodiment, the 3D data may include data regarding distance values ​​from the LiDAR sensor to points on the user's body part. In other words, the 3D data may include 3D point cloud data regarding the user's body part. While the size of the output image of a 2D camera may vary depending on the lighting conditions at the time of shooting and the distance between the camera and the subject, the LiDAR sensor may have the advantage of not being affected by such conditions since it calculates the distance to the subject using the reflection speed of light.

[0055] In this case, the 3D point cloud data collected through the camera equipped with the LiDAR sensor may include data regarding 2D coordinates of points on the user's body part, and may further include data regarding distance values ​​from the LiDAR sensor to the points on the body part. That is, the 3D point cloud data may include data regarding distances from the LiDAR sensor to all points on the user's body part within the range permitted by the resolution of the camera. Accordingly, the data collection unit (310) may calculate the x, y, and z coordinates of each point through the distance values ​​to express the position in 3D space. In addition, the 2D coordinates of the points on the body part may be respectively associated with the distance values ​​from the LiDAR sensor to the corresponding points on the body part, and the 3D point cloud data may further include information indicating the connection relationship between adjacent points.

[0056] The key point detection unit (330) can detect a pair of key points for measuring the circumference of a corresponding body part from a 2D image. Here, the key points may correspond to important points that serve as a reference for measuring the body circumference, and may be composed of a pair of points by corresponding to both ends of the body circumference. For example, the key points may correspond to characteristic points among joints, bone protrusions, and body outlines. The key point detection unit (330) can detect the key points on the 2D image using various methods. The key point detection unit (330) can basically perform the key point detection operation using a human pose estimation algorithm, but is not necessarily limited thereto.

[0057] At this time, the human posture estimation algorithm may correspond to a technology that estimates a person's posture, such as a person's joints, using a deep learning technique, and may be performed through a process of designating a unique key point for each specific body part of a person in a 2D image (or video) and learning and inferring surrounding visual features from that.

[0058] In one embodiment, the keypoint detection unit (330) can input a 2D image into a pre-built keypoint detection model to detect a pair of keypoints for measuring the circumference of a corresponding body part. Here, the keypoint detection model may correspond to a pre-built deep learning model that detects a pair of keypoints from a 2D image. The keypoint detection unit (330) can provide the 2D image as an input to the keypoint detection model and receive a predicted value regarding the positions of a pair of keypoints from the keypoint detection model as an output. The keypoint detection unit (330) can quickly and accurately detect keypoints by utilizing the keypoint detection model built by learning various body part images.

[0059] In one embodiment, a keypoint detection model can be built based on a High-Resolution Network (HRNet). HRNet is a powerful deep learning model used in various computer vision tasks, such as image segmentation, object recognition, and pose estimation. HRNet can accurately detect keypoints while maintaining high-resolution information by extracting and combining feature maps of various resolutions. It can effectively utilize information of various resolutions using a multi-stage feature fusion structure, and it can apply an efficient learning algorithm to shorten model training time and improve performance.

[0060] The reference point identification unit (350) can identify a reference point corresponding to a pair of key points among 3D data. Here, the reference point may correspond to a reference point that serves as a reference when measuring the circumference of a body part, and may correspond to point data corresponding to the key point among 3D data. That is, the key point is detected from a 2D image, while the reference point can be identified from 3D data. The reference point identification unit (350) can analyze the 3D data acquired from the LiDAR sensor to select a reference point candidate based on the connection structure of a joint or the shape of a muscle. The reference point identification unit (350) can identify an accurate reference point by matching the key point detected by the key point detection unit (330) with the reference point candidate. The reference point identification unit (350) can increase accuracy by analyzing the location, shape, and mutual relationship of the reference point candidate based on knowledge of human anatomy.

[0061] The process of measuring the circumference of the body can be performed by measuring the circumference of the front and back of the body part in half each and then adding them up. However, the accuracy of the circumference measurement can be reduced because the positions of the reference points in the front and back can be different. The reference point identification unit (350) can additionally perform an operation of correcting the position of the reference point to improve the accuracy of the body circumference measurement. The 2D image of the body part of the user to be measured for the circumference can be taken with a wall in the background, which can be a distance reference, so that a background can exist. In this case, an operation of correcting the position of the reference point can be performed by calculating the distance to the background of the 2D image, but depending on the shooting environment, the background of the 2D image may not exist as a result of taking the picture without the wall. The reference point identification unit (350) can perform an operation of adjusting the positions of key points according to the circumference measurement reference direction when there is no background surface of a 2D image, calculating the distance between points between the adjusted key points, and correcting the positions of two points closest to the body part among the positions of points having the largest distance as reference points.

[0062] More specifically, the reference point identification unit (350) adjusts the coordinates of a pair of key points detected by the key point detection unit (330) to the same value based on the coordinates of one point according to the reference direction of the circumference measurement. Here, the reference direction of the circumference measurement is defined as the direction in which the key points are spread, and may be set based on, for example, the axial direction of the body, i.e., the horizontal (horizontal) or vertical (vertical) direction. When measuring the arm circumference, the horizontal direction may be set as the reference direction in consideration of the thickness of the arm. In one embodiment, when the reference direction of the circumference measurement is vertical, the reference point identification unit (350) adjusts the x-axis coordinate value of one point to the same value based on the x-axis coordinate value of the other point based on the vertical direction. When the reference direction of the circumference measurement is horizontal, the reference point identification unit (350) adjusts the y-axis coordinate value of one point to the same value based on the y-axis coordinate value of the other point based on the horizontal direction. For example, if the detected key point coordinates class Assuming that, if we correct y2 based on y1 in the horizontal direction, the corrected coordinates are class This is done. That is, the reference point identification unit (350) can adjust the coordinates of one of the detected pair of key points to move the other point based on the coordinates of the other point so that they are located on the same reference line defined in the circumference measurement reference direction. Then, the reference point identification unit (350) spreads each key point by a specific distance so that the distance between the key points becomes greater. When the circumference measurement reference direction is horizontal, the reference point identification unit (350) spreads the x-axis coordinate value of each key point by a specific distance n, and when the circumference measurement reference direction is vertical, the y-axis coordinate value of each key point is spread by a specific distance n to move the key points to a position farther away from the body part to be measured.

[0063] The reference point identification unit (350) calculates the distance between all points between reference points corresponding to the key points that have been moved based on 3D data based on the position of the LiDAR sensor, determines a plurality of points having the largest distance among the distances between adjacent points, and determines two points that are closest to points on the body part among the determined plurality of points. The reference point identification unit (350) may determine the two newly determined points as corrected reference points. In one embodiment, the reference point identification unit (350) may calculate the distance between all points between the two separated key points as the Euclidean distance. The reference point identification unit (350) confirms the positions of the points having the largest distance and the second largest distance among the distances between all adjacent points. The distance between points within the body area to be measured circumference remains relatively constant, but the distance including a point that goes beyond the boundary of the body area increases rapidly. This is because the z value representing the distance to the body comes to represent the distance to a location that is outside the body area and farther than the body. The reference point identification unit (350) can determine the positions of the two points closest to the internal body area among the four positions of points having the largest and second largest distances among the distances between adjacent points as the positions of the circumference measurement reference points.

[0064] The body circumference measuring unit (370) can measure the circumference of a corresponding body part using a reference point and 3D data. The body circumference measuring unit (370) can receive reference point information identified from the reference point identification unit (350) and can extract an outline of a body part to be measured by analyzing 3D data acquired from a LiDAR sensor. The body circumference measuring unit (370) can perform an operation of calculating the circumference of the body part based on the outline information extracted from the reference point and 3D data. The body circumference measuring unit (370) can output the measured circumference information within the system or through a user interface.

[0065] In one embodiment, the body circumference measuring unit (370) can measure the front and rear circumferences of the body part by calculating the distance between 3D points forming the circumference of the corresponding body part between reference points, and can determine the circumference of the corresponding body part by adding up the front and rear circumferences. The body circumference measuring unit (370) can divide the body part into a front region and a rear region based on the reference point. The body circumference measuring unit (370) can extract outlines based on 3D data in the front region and the rear region and extract 3D points forming each outline.

[0066] In addition, the body circumference measuring unit (370) can calculate the distance between all extracted 3D points to derive the circumference of each region, and can determine the final circumference of the body part by adding up the circumferences of each region. Meanwhile, the body circumference measuring unit (370) can calculate the minimum / maximum value or average value between the front circumference and the rear circumference as needed, and can determine the final circumference based on this.

[0067] In one embodiment, the body circumference measuring unit (370) can calculate and add the Euclidean distance between adjacent points for 3D points forming the front or rear circumference. The body circumference measuring unit (370) can perform an operation of converting key points on a 2D image into points on 3D data to accurately calculate the circumference. For example, the body circumference measuring unit (370) can calculate two points on a 2D RGB image. Two points on a 3D point cloud can be converted into. The body circumference measurement unit (370) is the two converted points The length of a straight line can be measured by calculating the Euclidean distance. At this time, the Euclidean distance between two points can be calculated using the mathematical formula below.

[0068] [Mathematical formula]

[0069]

[0070] In one embodiment, the body circumference measuring unit (370) can estimate the circumference of the corresponding body part based on the distance between the reference point and the corresponding feature point if there are points corresponding to at least two or more feature points among the standard body parts defined in the remaining data excluding the reference point among the 3D data. That is, the body circumference measuring unit (370) can perform an operation of estimating the body circumference from the 3D data by utilizing the reference point and the feature point. For example, the standard body parts may include the head, neck, chest, waist, navel, arms, elbows, legs, knees, feet, etc., which are major parts that make up the body. The body circumference measuring unit (370) can estimate the circumference of the corresponding body part by using the first straight-line distance between the reference points and the second straight-line distances between the reference points and other feature points. That is, the body circumference measuring unit (370) can estimate the average body size of the corresponding user through the second straight-line distance, and can estimate the circumference of the corresponding body part corresponding to the first straight-line distance based on the average body size. To this end, the body circumference measurement unit (370) can perform a body part circumference estimation operation by utilizing a machine learning algorithm or further utilizing anatomical knowledge that takes into account various body types, genders, ages, etc.

[0071] The control unit (not shown in FIG. 3) controls the overall operation of the measuring device (130) and can manage the control flow or data flow between the data collection unit (310), the key point detection unit (330), the reference point identification unit (350), and the body circumference measurement unit (370).

[0072] Figure 4 is a flowchart illustrating a body circumference measurement method based on 2D and 3D data fusion according to the present invention.

[0073] Referring to FIG. 4, the measuring device (130) can collect 2D images and 3D data regarding at least a portion of a user's body part via the processor (210) (step S410). The measuring device (130) can detect a pair of key points for measuring the circumference of the corresponding body part from the 2D image via the processor (210) (step S430).

[0074] Additionally, the measuring device (130) can identify a reference point corresponding to a pair of core points among 3D data through the processor (210) (step S450). The measuring device (130) can measure the circumference of the corresponding body part using the reference point and 3D data through the processor (210) (step S470).

[0075] FIG. 5 is a drawing illustrating one embodiment of a 2D and 3D data collection process according to the present invention.

[0076] Referring to FIG. 5, the measuring device (130) can extract 2D images (RGB) and 3D Point Cloud data using a LiDAR sensor. Figure (a) of FIG. 5 may correspond to a 2D RGB image taken of an arm portion to be measured by using a portable mobile device equipped with a LiDAR sensor, figure (b) may correspond to a 3D depth image taken of the same arm portion, and figure (c) may correspond to a 3D point cloud image. The measuring device (130) can generate 3D point cloud data by adding 3D depth information to the 2D RGB image coordinates. The measuring device (130) can collect 2D images and 3D data by simultaneously taking pictures of the same body part at the same time.

[0077] FIG. 6 is a drawing illustrating one embodiment of a process for detecting key points on a 2D image according to the present invention.

[0078] Referring to FIG. 6, the measuring device (130) can perform a keypoint detection operation for measuring a body circumference on a 2D image using a keypoint detection model. To this end, the measuring device (130) can build a keypoint detection model for a 2D body image by utilizing HRNet (High-Resolution Network), which is one of the base networks for human posture estimation. The measuring device (130) can input a 3D body image into the keypoint detection model and automatically detect keypoints for measuring the circumference of the corresponding body part.

[0079] Here, HRNet can achieve outstanding performance in learning information surrounding keypoints in images by maintaining a high-resolution representation while simultaneously connecting lower-resolution networks. Furthermore, HRNet may have key advantages in the problem of detecting keypoints on the body for circumference measurement. HRNet's network structure branches into subnetworks as each step progresses, while simultaneously exchanging information with the high-resolution network, thereby maintaining high resolution throughout the entire process.

[0080] In order to measure the circumference of a specific body part, a process of first defining key points and training a key point detection model to automatically detect them can be performed. For example, in FIG. 6, a key point (610) for measuring the circumference of an arm can be defined, and a key point detection model constructed by training the key point can automatically detect a pair of key points for measuring the circumference of the arm. The measuring device (130) can obtain information for specifying the circumference by converting the coordinates of the key point (610) detected by the key point detection model into coordinates on 3D point cloud data.

[0081] FIG. 7 is a drawing illustrating one embodiment of a process for measuring the circumference of a body part according to the present invention.

[0082] Referring to FIG. 7, the measuring device (130) can derive results similar to actual measurement results through mapping between 2D and 3D data. The measuring device (130) can derive actual measurement values ​​from 3D point cloud data collected through a LiDAR sensor. The measuring device (130) can identify the 2D coordinates of each key point and the corresponding 3D point to calculate an accurate measurement value, and can perform a perimeter measurement operation through a process of converting the 2D coordinates into 3D points.

[0083] That is, the measuring device (130) can measure the actual distance between two points through a conversion operation between 2D coordinates and 3D points, and can measure the circumference of a body part based on this. Specifically, the measuring device (130) can automatically detect key points for measuring the circumference of a specific body part through a key point detection model, and set the two detected points as reference points. The measuring device (130) can correct the reference points to improve the accuracy of body circumference measurement, and can perform an operation of pairing all points between two points that become the corrected reference points, calculating the Euclidean distance between the two points, and then adding them up. The measuring device (130) can correct the positions of a pair of detected key points according to the circumference measurement reference direction, calculate the distance between points located between the reference points matching the corrected key points, and determine two points closest to the body part among the positions of a plurality of points having the largest distance as the corrected reference points. Figure 7 (a) may correspond to a key point (710) detected on a 2D image using a key point detection model, Figure 7 (b) may correspond to a key point (710') whose position is corrected according to a circumference measurement reference direction, Figure 7 (c) may correspond to all points between two points that become reference points matching the corrected key point (710') on a 3D point cloud image, and Figure 7 (d) may correspond to all points between two points that become corrected reference points. The measuring device (130) may perform the method on the front and rear regions of the body part, respectively, and then add up the perimeters of each region to calculate the entire perimeter of the body part.

[0084] For 2D images, the magnification and RGB values ​​of the image can vary depending on the lighting conditions at the time of capture and the distance between the camera and the subject. However, LiDAR sensors calculate distances based on the speed of light reflection relative to distance, allowing for accurate distance measurements unaffected by these conditions. Furthermore, when using only 2D (RGB) images, errors can occur due to radial distortion caused by the symmetrical shape of the camera lens, which causes distance distortion the farther away from the center of focus.

[0085] The present invention overcomes the limitations of existing 2D image-only methods by integrating 2D images (RGB) with 3D point cloud data from a LiDAR sensor, thereby improving model performance. Furthermore, the present invention can contribute to the revitalization of vintage industries, such as non-face-to-face health, by enabling precise dimensional measurements using 2D images and 3D fusion data.

[0086] In addition, the present invention can automatically and accurately measure body circumference by correcting the reference point even in general shooting situations without using a wall as a background.

[0087] Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0088] The best mode for carrying out the invention was discussed, including the mode for carrying out the invention.

Claims

1. Memory; and A method performed in a device including a processor electrically connected to the above memory, The above method is performed by the above processor, A step of collecting 2D images and 3D data regarding at least a part of the user's body; A step of inputting the 2D image into the established key point detection model to detect a pair of key points for measuring the circumference of the corresponding body part; A step of correcting the position of the above key point according to the circumference measurement reference direction and identifying the reference point as the two points closest to the corresponding body part among the points between the reference points corresponding to the corrected key point in the 3D data; and A step of measuring the circumference of the corresponding body part using the above reference point and the 3D data; The step of identifying the above reference point is A step of adjusting the coordinates of one of the pair of core points to the same value based on the coordinates of the other point along the circumference measurement reference direction so that the pair of core points are located on the same reference line defined in the circumference measurement reference direction; and A body circumference measurement method based on 2D and 3D data fusion, characterized in that it includes a step of correcting the key points by moving the positions of the key points apart by a specific distance so that the distance between the key points becomes longer than the length of the body part to be measured.

2. In paragraph 1, the collecting step A body circumference measurement method based on 2D and 3D data fusion, characterized in that it includes a step of photographing a part of the user's body using a photographing device including a LiDAR (Light Detection and Ranging) sensor.

3. In the second paragraph, the 3D data A body circumference measurement method based on 2D and 3D data fusion, characterized in that it includes data regarding distance values ​​from the LiDAR sensor to points on the user's body part.

4. In the first paragraph, the key point detection model A body circumference measurement method based on 2D and 3D data fusion, characterized by being built on HRNet (High-Resolution Network).

5. In the first paragraph, the step of identifying the reference point is A body circumference measurement method based on 2D and 3D data fusion, characterized in that it includes a step of calculating the Euclidean distance between adjacent points for points between the above reference points and determining the two points closest to the point on the corresponding body part among a plurality of points having the largest distance and correcting them as the reference points.

6. In the first paragraph, the step of measuring the circumference is A step of measuring the front and rear circumferences of the corresponding body part by calculating the Euclidean distance between adjacent points among 3D points forming the circumference of the corresponding body part between the reference points; and A body circumference measurement method based on 2D and 3D data fusion, characterized in that it includes a step of determining the circumference of the corresponding body part by adding the front and rear circumferences.

7. Memory; and A processor electrically connected to the above memory, The above processor Collect 2D images and 3D data about at least some parts of the user's body, A pair of key points for measuring the circumference of the corresponding body part is detected from the above 2D image, The detected pair of core points are positioned on the same reference line defined in the circumference measurement reference direction, and the coordinates of one of the pair of core points are adjusted to the same value based on the coordinates of the other point according to the circumference measurement reference direction. The position of the key points is shifted by a specific distance so that the distance between the key points becomes longer than the length of the body part to be measured, and the key points are corrected, and among the points between the reference points corresponding to the corrected key points in the 3D data, the two points closest to the corresponding body part are identified as reference points. A body circumference measuring device based on 2D and 3D data fusion, which is implemented to measure the circumference of the corresponding body part using the above reference point and the 3D data.

Images