Skeleton estimation device, skeleton estimation method, and program

Abstract

To provide a skeleton estimation device capable of accurately estimating a skeleton of an object even when part of the object is hidden in a shadow of an obstacle.SOLUTION: A skeleton estimation device includes: image acquisition means that acquires image information including an object and an obstacle; distance acquisition means that acquires distance information of the object and the obstacle; skeleton point estimation means that estimates, on the basis of the image information of the object and the obstacle, a position of each skeleton point of the object; skeleton point discrimination means that discriminates, on the basis of image information of the object and the obstacle, the estimated skeleton points into hidden skeleton points hidden by the obstacle and non-hidden skeleton points not hidden by the obstacle; distance estimation means that estimates distance information of the discriminated hidden skeleton points; and skeleton estimation means that estimates the skeleton of the object by acquiring distance information corresponding to the discriminated non-hidden skeleton points from the acquired distance information and integrating distance information of the acquired non-hidden skeleton points with the distance information of the estimated hidden skeleton points.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a skeleton estimation device, a skeleton estimation method, and a program for estimating the skeleton of an object. [Background technology]

[0002] BACKGROUND ART When a person's torso is hidden in the shadow of an obstacle, a skeleton estimation device is known that estimates the skeleton of the torso based on the positions of the person's skeletal parts such as arms and legs (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 187641 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-described skeleton estimation device can estimate the skeleton of the trunk when the hidden part is the trunk, but cannot estimate the skeleton of the arms or legs when the arms or legs are hidden.

[0005] The present disclosure has been made to solve such problems, and its main purpose is to provide a skeleton estimation device, a skeleton estimation method, and a program that can accurately estimate the skeleton of an object even when part of the object is hidden by the shadow of an obstacle. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present disclosure is to an image acquisition means for acquiring image information including an object and an obstacle that hides at least a part of the object; distance acquisition means for acquiring distance information of the object and the obstacle; a skeleton point estimation means for estimating the position of each skeleton point of the object based on the image information of the object and obstacle acquired by the image acquisition means; a skeleton point discrimination means for discriminating the plurality of skeleton points estimated by the skeleton point estimation means based on the image information of the object and the obstacle acquired by the image acquisition means into hidden skeleton points that are hidden by the obstacles and non-hidden skeleton points that are not hidden by the obstacles; distance estimation means for estimating distance information of the hidden skeleton points determined by the skeleton point determination means; a skeleton estimation means for estimating the skeleton of the object by acquiring distance information corresponding to the non-hidden skeleton points identified by the skeleton point discrimination means from the distance information acquired by the distance acquisition means, and integrating the acquired distance information of the non-hidden skeleton points with the distance information of the hidden skeleton points estimated by the distance estimation means; A skeleton estimation device comprising: is. In this aspect, The skeleton point determination means performing a segmentation process on the image information of the object and the obstacle acquired by the image acquisition means; The skeleton points included in the segmentation of the obstacle may be determined as the hidden skeleton points, and the skeleton points included in the segmentation of the object may be determined as the non-hidden skeleton points. In this aspect, The distance estimation means may estimate distance information of the hidden skeleton points determined by the skeleton point determination means based on linear interpolation processing, a three-dimensional model, or information from another sensor. In this aspect, the object is a worker performing a task, The skeleton estimation means may estimate a skeleton of the worker. In order to achieve the above object, one aspect of the present disclosure is to acquiring image information including an object and an obstacle obscuring at least a portion of the object; acquiring distance information of the object and the obstacle; estimating the position of each skeleton point of the object based on the acquired image information of the object and obstacle; a step of classifying the estimated skeleton points into hidden skeleton points that are hidden by the obstacles and non-hidden skeleton points that are not hidden by the obstacles based on the acquired image information of the object and the obstacles; estimating distance information of the determined hidden skeleton points; acquiring distance information corresponding to the determined non-hidden skeleton points from the acquired distance information, and estimating the skeleton of the object by integrating the acquired distance information of the non-hidden skeleton points and the distance information of the estimated hidden skeleton points; A skeleton estimation method including is. In order to achieve the above object, one aspect of the present disclosure is to acquiring image information including an object and an obstacle obscuring at least a portion of the object; A process of acquiring distance information of the object and the obstacle; a process of estimating the position of each skeleton point of the object based on the acquired image information of the object and obstacle; a process of classifying the estimated skeleton points into hidden skeleton points that are hidden by the obstacles and non-hidden skeleton points that are not hidden by the obstacles based on the acquired image information of the object and the obstacles; a process of estimating distance information of the determined hidden skeleton points; a process of estimating the skeleton of the object by acquiring distance information corresponding to the determined non-hidden skeleton points from the acquired distance information and integrating the acquired distance information of the non-hidden skeleton points with the distance information of the estimated hidden skeleton points; A program that causes a computer to execute is. [Effects of the Invention]

[0007] The main object of the present disclosure is to provide a skeleton estimation device, a skeleton estimation method, and a program that can accurately estimate the skeleton of an object even when part of the object is hidden in the shadow of an obstacle. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a schematic hardware configuration of a skeleton estimation device according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a schematic system configuration of a skeleton estimation device according to an embodiment of the present invention. FIG. [Figure 3] FIG. 10 is a diagram illustrating an example of a method for estimating the positions of each skeleton point of a worker. [Figure 4] FIG. 10 is a diagram illustrating an example of a segmentation process. [Figure 5] FIG. 10 is a diagram illustrating an example of a method for estimating distance information of hidden skeleton points. [Figure 6] 1 is a flowchart showing the flow of a skeleton estimation method according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present embodiment will be described below with reference to the drawings. Fig. 1 is a block diagram showing a schematic hardware configuration of a skeleton estimation device according to the present embodiment. The skeleton estimation device 1 according to the present embodiment can accurately estimate the skeleton of an object even when part of the object is hidden behind an obstacle.

[0010] The target object includes, for example, a person or animal, such as a worker performing work or an athlete playing a sport. An example in which the target object is a worker will be described below. For example, even if the worker's legs are hidden by the shadow of a dolly or the like, the skeleton estimation device 1 can accurately estimate the entire skeleton of the worker, including the legs hidden by the shadow.

[0011] The skeleton estimation device 1 has a hardware configuration of a typical computer, including, for example, a processor 11 such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), internal memory 12 such as RAM (Random Access Memory) or ROM (Read Only Memory), storage device 13 such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), input / output I / F 14 for connecting peripheral devices such as a display, and communication I / F 15 for communicating with devices external to the device.

[0012] 2 is a block diagram showing a schematic system configuration of a skeleton estimation device according to this embodiment. The skeleton estimation device 1 according to this embodiment includes an image acquisition unit 2, a distance acquisition unit 3, a skeleton point estimation unit 4, a skeleton point determination unit 5, a distance estimation unit 6, and a skeleton estimation unit 7.

[0013] The image acquisition unit 2 is a specific example of an image acquisition means. The image acquisition unit 2 acquires image information including an image of a worker and an obstacle that hides at least a part of the worker. The image acquisition unit 2 is configured as, for example, an RGB camera.

[0014] The distance acquisition unit 3 is a specific example of a distance acquisition means. The distance acquisition unit 3 acquires distance information (depth information, etc.) of the worker and obstacles. The distance acquisition unit 3 is configured, for example, as a distance measurement sensor that detects distance information. Note that the image acquisition unit 2 and the distance acquisition unit 3 may be configured integrally as an RGBD camera.

[0015] The skeleton point estimation unit 4 is a specific example of a skeleton point estimation means. The skeleton point estimation unit 4 estimates the position of each skeleton point of the worker based on the image information of the worker and obstacles acquired by the image acquisition unit 2.

[0016] For example, as shown in FIG. 3( a ), the skeleton point estimation unit 4 estimates the position (two-dimensional coordinates) of each skeleton point Pn of the worker based on the RGB information in frame n acquired by the image acquisition unit 2.

[0017] In this case, if a part of the worker (for example, the lower limbs) is hidden in the shadow of an obstacle such as a cart, as shown in Figure 3(b), the distance information (depth information) of the skeleton points of the hidden part will be smaller than the distance information of the skeleton points of the actual worker, and will differ from the actual information, as shown in Figure 3(c).

[0018] In contrast, the skeleton estimation device 1 according to this embodiment distinguishes between skeleton points hidden by obstacles (hereinafter referred to as hidden skeleton points) and skeleton points not hidden by obstacles (hereinafter referred to as non-hidden skeleton points), as will be described later. The skeleton estimation device 1 then estimates distance information of these hidden skeleton points with high accuracy using another method, which will be described later.

[0019] The skeleton point discrimination unit 5 is a specific example of a skeleton point discrimination means. Based on the image information of the worker and obstacles acquired by the image acquisition unit 2, the skeleton point discrimination unit 5 discriminates the multiple skeleton points estimated by the skeleton point estimation unit 4 into hidden skeleton points that are hidden by obstacles and non-hidden skeleton points that are not hidden by obstacles.

[0020] The skeleton point discrimination unit 5 may perform segmentation processing on the image information of the object and obstacle acquired by the image acquisition unit 2. This segmentation processing is processing to divide the image into meaningful units, and is processing to group pixels belonging to the same category into one segment.

[0021] The skeleton point discrimination unit 5 performs segmentation processing, for example, as shown in Figure 4, to determine skeleton points included in the obstacle segmentation S1 as hidden skeleton points and skeleton points included in the worker segmentation S2 as non-hidden skeleton points.

[0022] The distance estimation unit 6 is a specific example of a distance estimation means. The distance estimation unit 6 estimates distance information of the hidden skeleton points determined by the skeleton point determination unit 5.

[0023] The distance estimation unit 6 may perform linear interpolation using a Kalman filter or the like to estimate distance information of the hidden skeleton points identified by the skeleton point determination unit 5. For example, as shown in FIG. 5(a), the value of the depth (distance) Z at time tn is significantly off. Therefore, the distance estimation unit 6 may correct the value of the depth Z at time tn by linear interpolation.

[0024] The distance estimation unit 6 may estimate distance information of the hidden skeleton points identified by the skeleton point discrimination unit 5 using a three-dimensional human body model as shown in Fig. 5(b). The distance estimation unit 6 estimates the three-dimensional position where the hidden skeleton point is most likely to exist at that time based on the non-hidden skeleton points. By using a three-dimensional (3D) human body model with joint point constraints, it becomes possible to estimate, for example, that if the non-hidden upper body assumes this posture, the hidden lower body will assume this posture.

[0025] The distance estimation unit 6 may estimate the distance information of the hidden skeleton points identified by the skeleton point identification unit 5 using sensor information from another sensor. The other sensor is, for example, an inertial measurement unit (IMU) sensor. The IMU sensor is equipped with a three-axis angular velocity sensor, a three-axis acceleration sensor, a temperature sensor, and the like, and can detect three-dimensional inertial motion.

[0026] The skeleton estimation unit 7 is a specific example of a skeleton estimation means. The skeleton estimation unit 7 acquires distance information corresponding to the non-hidden skeleton points identified by the skeleton point discrimination unit 5 from the distance information acquired by the distance acquisition unit 3. The skeleton estimation unit 7 then integrates the acquired distance information of the non-hidden skeleton points with the distance information of the hidden skeleton points estimated by the distance estimation unit 6, thereby estimating the three-dimensional skeleton of the worker.

[0027] The skeleton estimation unit 7 may output the estimated three-dimensional skeleton of the worker by an output device. The output device may be, for example, a display device that displays the three-dimensional skeleton of the worker, or a printer that outputs the display result.

[0028] Next, an example of the skeleton estimation method according to the present embodiment will be described below. Fig. 6 is a flowchart showing the flow of the skeleton estimation method according to the present embodiment.

[0029] The image acquisition unit 2 acquires image information including a worker and an obstacle that hides at least a part of the worker (step S101).

[0030] The distance acquisition unit 3 acquires distance information between the worker and the obstacle (step S102).

[0031] The skeleton point estimating unit 4 estimates the position of each skeleton point of the worker based on the image information of the worker and obstacles acquired by the image acquiring unit 2 (step S103).

[0032] The skeleton point discrimination unit 5 performs segmentation processing on the image information of the worker and obstacles acquired by the image acquisition unit 2, and discriminates the multiple skeleton points estimated by the skeleton point estimation unit 4 into hidden skeleton points that are hidden by obstacles and non-hidden skeleton points that are not hidden by obstacles (step S104).

[0033] The distance estimation unit 6 estimates distance information of the hidden skeleton points determined by the skeleton point determination unit 5 using linear interpolation processing or the like (step S105).

[0034] The skeleton estimation unit 7 acquires distance information corresponding to the non-hidden skeleton points identified by the skeleton point discrimination unit 5 from the distance information acquired by the distance acquisition unit 3 (step S106). The skeleton estimation unit 7 estimates the skeleton of the worker by integrating the acquired distance information of the non-hidden skeleton points and the distance information of the hidden skeleton points estimated by the distance estimation unit 6 (step S107).

[0035] According to the skeleton estimation method using the skeleton estimation device 1 according to the present embodiment described above, the skeleton of a worker can be estimated with high accuracy even when part of the worker is hidden behind an obstacle.

[0036] Although several embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0037] The present disclosure can also be implemented by causing a processor to execute a computer program to perform the processing shown in FIG. 6, for example.

[0038] The program can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)).

[0039] The program may be provided to the computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to the computer via a wired communication path such as an electrical wire or optical fiber, or via a wireless communication path.

[0040] Each component of the skeleton estimation device 1 according to the above-described embodiment can be realized not only by a program, but also in part or in whole by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). [Explanation of symbols]

[0041] 1 Skeleton estimation device, 2 Image acquisition unit, 3 Distance acquisition unit, 4 Skeleton point estimation unit, 5 Skeleton point discrimination unit, 6 Distance estimation unit, 7 Skeleton estimation unit

Claims

1. an image acquisition means for acquiring image information including an object and an obstacle that hides at least a part of the object; distance acquisition means for acquiring distance information of the object and the obstacle; a skeleton point estimation means for estimating the position of each skeleton point of the object based on the image information of the object and obstacle acquired by the image acquisition means; a skeleton point discrimination means for discriminating the plurality of skeleton points estimated by the skeleton point estimation means based on the image information of the object and the obstacle acquired by the image acquisition means into hidden skeleton points that are hidden by the obstacles and non-hidden skeleton points that are not hidden by the obstacles; distance estimation means for estimating distance information of the hidden skeleton points determined by the skeleton point determination means; a skeleton estimation means for estimating the skeleton of the object by acquiring distance information corresponding to the non-hidden skeleton points identified by the skeleton point discrimination means from the distance information acquired by the distance acquisition means, and integrating the acquired distance information of the non-hidden skeleton points with the distance information of the hidden skeleton points estimated by the distance estimation means; A skeleton estimation device comprising:

2. The skeletal structure estimation device according to claim 1, The skeleton point determination means performing a segmentation process on the image information of the object and the obstacle acquired by the image acquisition means; determining skeleton points included in the obstacle segmentation as the hidden skeleton points and skeleton points included in the object segmentation as the non-hidden skeleton points; Skeletal hand estimation device.

3. The skeletal structure estimation device according to claim 1, the distance estimation means estimates distance information of the hidden skeleton points determined by the skeleton point determination means based on linear interpolation processing, a three-dimensional model, or information from another sensor; Skeleton estimation device.

4. acquiring image information including an object and an obstacle obscuring at least a portion of the object; acquiring distance information of the object and the obstacle; estimating the position of each skeleton point of the object based on the acquired image information of the object and obstacle; a step of classifying the estimated skeleton points into hidden skeleton points that are hidden by the obstacles and non-hidden skeleton points that are not hidden by the obstacles based on the acquired image information of the object and the obstacles; estimating distance information of the determined hidden skeleton points; acquiring distance information corresponding to the determined non-hidden skeleton points from the acquired distance information, and estimating the skeleton of the object by integrating the acquired distance information of the non-hidden skeleton points and the distance information of the estimated hidden skeleton points; A skeleton estimation method comprising:

5. acquiring image information including an object and an obstacle obscuring at least a portion of the object; A process of acquiring distance information of the object and the obstacle; a process of estimating the position of each skeleton point of the object based on the acquired image information of the object and obstacle; a process of classifying the estimated skeleton points into hidden skeleton points that are hidden by the obstacles and non-hidden skeleton points that are not hidden by the obstacles based on the acquired image information of the object and the obstacles; a process of estimating distance information of the determined hidden skeleton points; a process of estimating the skeleton of the object by acquiring distance information corresponding to the determined non-hidden skeleton points from the acquired distance information and integrating the acquired distance information of the non-hidden skeleton points with the distance information of the estimated hidden skeleton points; A program that causes a computer to execute the following.

Images