DETECTION APPARATUS, PROCESSING APPARATUS, AND PROCESSING PROGRAM

Abstract

To enable highly accurate detection of posture of an object.SOLUTION: A detection device is provided, comprising a detection unit configured to detect position information of each point on a surface of an object, and a computation unit configured to derive feature points of the object from the normal of each point according to the position information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a detection device, a processing device, a detection method, and a processing program. [Background technology]

[0002] For example, as described in Patent Document 1, a technology has been proposed in which an object is imaged using multiple imaging devices and the resulting images are input into a computer to obtain the three-dimensional shape of the object; however, it is desirable to detect the shape or orientation of all or part of the object (object) with high accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-134546 A Summary of the Invention

[0004] According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives positional information of each point on the surface of an object, and determines feature points of the object from normals at the points using the positional information; and a posture calculation unit that determines the posture of the object from the multiple feature points, wherein the posture calculation unit determines the posture of the object by fitting skeletal information to connecting lines connecting the multiple feature points. According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives point cloud data indicating positional information of each point on a surface of an object, and determines feature points of the object from normals at points on the point cloud data; a posture calculation unit that determines the posture of the object from the multiple feature points; and a model generation unit that generates model information including shape information of the object based on the point cloud data, wherein the posture calculation unit determines the posture of the object by fitting skeletal information to the multiple feature points, and the model generation unit distinguishes between a first partial point group and a second partial point group of the point cloud data based on the posture of the object determined by the posture calculation unit.

[0005] According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives position information indicating a surface position of an object and calculates feature points of the object from normals at points on the position information; and a posture calculation unit that calculates the posture of the object based on a plurality of feature points and skeletal information. According to an aspect of the present invention, there is provided a processing device comprising: a calculation unit that receives position information indicating a position of a surface of an object, and determines feature points related to the object from normals at points on the position information, and an attitude calculation unit that calculates the attitude of the object, wherein the calculation unit determines a plurality of feature points inside the object, and the attitude calculation unit calculates the attitude of the object using the plurality of feature points set inside the object. According to an aspect of the present invention, there is provided a detection device including a detection section that detects position information of each point on a surface of an object, and the above-mentioned processing device.

[0006] According to an aspect of the present invention, there is provided a processing program that causes a computer to receive positional information of each point on the surface of an object, determine feature points of the object from normals at the points based on the positional information, and determine the orientation of the object from the multiple feature points, where the process of determining the orientation of the object involves fitting skeletal information to connecting lines connecting the multiple feature points to determine the orientation of the object. According to an aspect of the present invention, there is provided a processing program that causes a computer to receive point cloud data indicating positional information of each point on the surface of an object, determine feature points related to the object from normals at points on the point cloud data, determine the orientation of the object from the plurality of feature points, and generate model information including shape information of the object based on the point cloud data, wherein in the process of determining the orientation of the object, the orientation of the object is determined by fitting skeletal information to the plurality of feature points, and in the process of generating the model information including the shape information of the object, a first partial point group and a second partial point group of the point cloud data are distinguished based on the determined orientation of the object.

[0007] According to an aspect of the present invention, there is provided a processing program that causes a computer to receive position information indicating a position of a surface of an object, determine feature points of the object from normals at points on the position information, and determine the posture of the object based on a plurality of feature points and skeletal information. According to an aspect of the present invention, there is provided a processing program that causes a computer to receive position information indicating a position of a surface of an object, determine feature points related to the object from normals at points on the position information, and calculate the orientation of the object, wherein in the process of determining the feature points, a plurality of feature points are determined within the object, and in the process of calculating the orientation of the object, the orientation of the object is calculated using the plurality of feature points set within the object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a detection device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an information acquisition unit according to the first embodiment. [Diagram 3] FIG. 4 is a diagram showing processing of a point cloud data generating unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing processing of a point cloud data generating unit according to the first embodiment. [Diagram 5] FIG. 4 is a diagram showing processing of a point cloud data generating unit according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing processing of a calculation unit according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing processing of a calculation unit according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing the processing of an attitude calculation unit according to the first embodiment. [Figure 9] FIG. 4 is a diagram showing predetermined skeletal information according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing the processing of an attitude calculation unit according to the first embodiment. [Figure 11] 4 is a flowchart showing a detection method according to the first embodiment. [Figure 12] FIG. 13 is a diagram showing a detection device according to a second embodiment. [Figure 13] FIG. 11 is a diagram showing the processing of a calculation unit according to the second embodiment. [Figure 14]FIG. 11 is a diagram showing the processing of a calculation unit according to the second embodiment. [Figure 15] 10 is a flowchart showing a detection method according to a second embodiment. [Figure 16] FIG. 13 is a diagram showing a detection device according to a third embodiment. [Figure 17] FIG. 13 is a diagram showing a detection device according to a fourth embodiment. [Figure 18] FIG. 13 is a diagram showing a detection device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described with reference to the drawings. In order to explain the embodiments, the drawings are shown with appropriate scale changes, such as enlarging or emphasizing some parts, and the size and shape of the actual product may differ. In the following description, directions may be explained using the XYZ Cartesian coordinate system shown in the drawings.

[0010] [First embodiment] A first embodiment will be described. FIG. 1 is a diagram showing an example of a detection device according to the first embodiment. The detection device 1 (detection system) is, for example, a motion capture device, a motion detection system, an exercise support system, or the like. The detection device 1 is also used for posture analysis or three-dimensional modeling, or the like. In these cases, the detection device 1 detects an object M moving in one direction or in multiple directions in a predetermined time range. The detection device 1 may detect an object M moving linearly, or may detect an object M moving in a meandering manner. That is, the movement path (for example, a trajectory) of the object M when the detection device 1 detects the object M may include a straight line, may include a curved line, or may include both a straight line and a curved line.

[0011] The object M is movable in a target area AR that is the object of detection by the detection device 1. The target area AR corresponds to, for example, an area or range of an object that can be detected by the detection device 1, or a field of view. The object M is, for example, a human body or a non-human animal, a humanoid or non-human animal-type robot, or a non-animal robot. The object M changes, for example, one or both of its posture and shape as it moves. For example, if the object M is a human body, a first part (for example, the left foot) and a second part (for example, the right foot) move alternately in the movement direction MD when it moves (for example, walking, running, exercising, operating). At this time, at least a part of the shape (for example, posture) of the object M changes.

[0012] The moving human body is, for example, a human body moving in sports such as fencing, baseball, soccer, golf, kendo, American football, ice hockey, or gymnastics. The moving human body is, for example, a human body moving in walking or posing such as running, exercising, yoga, bodybuilding, or a fashion show. The moving human body is, for example, a human body moving in games, person authentication, or work. The target M may include the moving human body and objects attached to the moving human body (for example, clothing, wearables, exercise equipment, and protective gear).

[0013] In the following, an XYZ orthogonal coordinate system shown in FIG. 1 and the like will be referred to. In this XYZ orthogonal coordinate system, the X direction is the moving direction MD of the object M, the Y direction is the vertical direction of the moving direction MD of the object M, and the Z direction is the crossing direction (orthogonal direction) that crosses (is perpendicular to) the moving direction MD of the object M and the vertical direction. When the object M is a human body, the Z direction includes the direction of the shoulder width of the human body. In addition, in the following, in each of the X, Y, and Z directions, the side indicated by the arrow is appropriately referred to as the + side (e.g., the +Z side), and the opposite side to the side indicated by the arrow is appropriately referred to as the - side (e.g., the -Z side). For example, the first part (e.g., the left foot) is on the -Z side relative to the second part (e.g., the right foot), and the second part is on the +Z side relative to the first part.

[0014] The detection device 1 includes a detection unit 10 and a processing device 100. The detection unit 10 detects position information of each point on the surface of the object M. The detection unit 10 detects point cloud data including three-dimensional coordinates of each point on the surface of the object M as the position information of the object M, as described later. In this embodiment, the movement direction MD of the object M is predetermined with respect to the object area AR. The vertical direction in the object area AR and the movement direction MD of the object M are given to the detection device 1 as known information. In the detection result (point cloud data) by the detection unit 10, the X, Y, and Z directions are set based on the above known information. The detection unit 10 includes an information acquisition unit 11 and a point cloud data generation unit 110.

[0015] The information acquisition unit 11 is, for example, at least a part of a portable device (mobile equipment). The information acquisition unit 11 may be at least a part of a stationary device. The information acquisition unit 11 may be provided inside the processing device 100. Also, the point cloud data generation unit 110 may be provided in an external device of the processing device 100 and connected to the processing device 100. For example, the detection unit 10 may be an external device of the processing device 100 and may incorporate the information acquisition unit 11 and the point cloud data generation unit 110. A part or all of the detection device 1 may be a portable device (for example, an information terminal, a smartphone, a tablet, a mobile phone with a camera, a wearable terminal). Also, a part or all of the detection device 1 may be a stationary device (for example, a fixed camera).

[0016] FIG. 2 is a diagram showing an example of an information acquisition unit according to the first embodiment. The information acquisition unit 11 detects the depth of the object M. The information acquisition unit 11 includes, for example, a depth sensor (depth camera). The information acquisition unit 11 detects the depth (distance, depth, depth) from a predetermined point to each point on the surface of an object placed in the target area AR. The predetermined point is, for example, a point at a position that serves as a reference for detection by the information acquisition unit 11 (for example, a viewpoint, a point of detection origin, a point representing the position of the information acquisition unit 11, or a pixel position of the image sensor 14 described later).

[0017] The information acquisition unit 11 includes an irradiation unit 12, an optical system 13, and an image sensor 14. The irradiation unit 12 irradiates (projects) light La (e.g., pattern light, irradiation light) onto the target area AR. The optical system 13 includes, for example, an imaging optical system (image sensing optical system). The image sensor 14 includes, for example, a CMOS image sensor or a CCD image sensor. The image sensor 14 has a plurality of pixels arranged two-dimensionally. The image sensor 14 images the target area AR via the optical system 13. The image sensor 14 detects light Lb (infrared light, return light) emitted from an object placed in the target area AR by irradiation with the light La.

[0018] The information acquisition unit 11 detects the depth from a point on the target area AR corresponding to each pixel of the image sensor 14 to each pixel of the image sensor 14 based on, for example, a pattern (e.g., intensity distribution) of light La irradiated from the irradiation unit 12 and a pattern (e.g., intensity distribution, captured image) of light Lb detected by the image sensor 14. The information acquisition unit 11 outputs a depth map (depth image, depth information, distance image) representing the distribution of depth in the target area AR as a detection result to the processing device 100. That is, the information acquisition unit 11 outputs the depth map of the object M to the processing device 100.

[0019] The information acquisition unit 11 may be a device that detects depth by a time-of-flight (TOF) method. The information acquisition unit 11 may also be a device that detects depth by a method other than the TOF method. For example, the information acquisition unit 11 may include a laser scanner (laser range finder) and detect depth by laser scanning. For example, the information acquisition unit 11 may include a phase difference sensor and detect depth by a phase difference method. For example, the information acquisition unit 11 may detect depth by a depth-from-defocus (DFD) method.

[0020] The information acquisition unit 11 may irradiate the object M with light other than infrared light (e.g., visible light) and detect the light (e.g., visible light) emitted from the object M. For example, the information acquisition unit 11 may include a stereo camera or the like and detect (image) the object M from multiple viewpoints. For example, the information acquisition unit 11 may detect the depth by triangulation using captured images of the object M captured from multiple viewpoints. For example, the information acquisition unit 11 may detect the depth by a method other than an optical method (e.g., ultrasonic scanning).

[0021] Returning to the explanation of FIG. 1, the point cloud data generation unit 110 generates point cloud data equivalent to information representing the surface position of the object M and the shape of the object M, as position information of the object M, based on the depth detected by the information acquisition unit 11. In FIG. 1, the information acquisition unit 11 is provided outside the processing device 100. Also in FIG. 1, the point cloud data generation unit 110 is provided inside the processing device 100. The processing device 100 is communicatively connected to the information acquisition unit 11. The information acquisition unit 11 outputs the detection result to the processing device 100. The processing device 100 executes processing based on the detection result output from the information acquisition unit 11.

[0022] The processing device 100 includes a point cloud data generating unit 110, a calculation unit 120, an attitude calculation unit 130, and a storage unit 140. The storage unit 140 is, for example, a non-volatile memory, a hard disk drive (HDD: Hard Disk Drive), a solid state drive (SSD: Solid State Drive), etc. The storage unit 140 stores original data to be processed by the processing device 100 and data processed (generated) by the processing device 100. The storage unit 140 stores a depth map output from the information acquisition unit 11 as original data of the point cloud data.

[0023] The point cloud data generating unit 110 executes a point cloud process to generate point cloud data as position information of the object M. The point cloud data includes three-dimensional coordinates of a plurality of points on the object M in the target area AR. In addition, the point cloud data may include three-dimensional coordinates of a plurality of points on objects (e.g., walls, floors) surrounding the object M. The point cloud data generating unit 110 reads out a depth map stored in the storage unit 140 to calculate the point cloud data of the object M. The point cloud data generating unit 110 may generate the point cloud data as model information (shape information) of the object M.

[0024] 3 to 5 are diagrams illustrating an example of processing by the point cloud data generating unit according to the first embodiment. D1 illustrated in FIG. 3 is a depth map corresponding to the detection result of the information acquiring unit 11. The depth map D1 is information (image) that represents the spatial distribution of depth measurement values ​​by the information acquiring unit 11. The depth map D1 is a grayscale image that represents the depth at each point of the target area AR by a gradation value. Parts in the depth map D1 with relatively high gradation values ​​(e.g., white parts, bright parts) are parts with relatively small depths (e.g., parts relatively close to the detection unit 10). Parts in the depth map D1 with relatively low gradation values ​​(e.g., black parts, dark parts) are parts with relatively large depths (e.g., parts relatively far from the detection unit 10).

[0025] The point cloud data generator 110 reads out the data of the depth map D1 from the storage unit 140, calculates the three-dimensional coordinates of the points in real space corresponding to each pixel based on the grayscale value (depth measurement value) of each pixel of the depth map D1, and generates the point cloud data D2 (see FIG. 4). Hereinafter, the three-dimensional coordinates of one point may be referred to as point data. The point cloud data D2 is data that groups together a plurality of point data.

[0026] The point cloud data generating unit 110 performs segmentation, pattern recognition, and the like on the point cloud data of the entire target area AR to extract point cloud data D2 of the target object M. That is, the point cloud data generating unit 110 executes an extraction process (segmentation) to extract position information of a part of an object (target object M) from position information of the object placed in the target area AR. Here, the objects placed in the target area AR include the target object M and objects around the target object M (for example, floors, walls, and objects in the background).

[0027] Prior to the above extraction process, the detection unit 10 detects position information of an object in the target area AR in each of a first state in which the target object M is not disposed in the target area AR and a second state in which the target object M is disposed in the target area AR. In the extraction process, the point cloud data generation unit 110 extracts the position information of the target object M by calculating the difference between the detection result of the detection unit 10 in the first state and the detection result of the detection unit 10 in the second state.

[0028] The first state may be a state in which the object M is placed in the target area AR in a state different from the second state. For example, the first state may be a state in which the object M is placed in a first position in the target area AR, and the second state may be a state in which the object M is placed in a second position different from the first position in the target area AR. The extraction process may be performed by a processing unit other than the point cloud data generating unit 110. Such a processing unit may be provided in the processing device 100, or may be provided in a device external to the processing device 100. The extraction process may not be performed.

[0029] The processing device 100 may also remove noise from the detection result of the detection unit 10. The noise removal process for removing noise includes, for example, a process for removing spatial noise from the depth map D1 generated by the detection unit 10 (information acquisition unit 11). For example, when the difference in depth between a first region (first pixel) and a second region (second pixel) adjacent to each other in the depth map D1 exceeds a threshold, the processing device 100 determines that the depth in the first region or the depth in the second region is noise. When the processing device 100 determines that the depth in the first region is noise, it estimates the depth of the first region by, for example, interpolation using the depths of the regions (third pixel, fourth pixel) surrounding the first region. Then, the processing device 100 removes spatial noise by updating (replacing, correcting) the depth of the first region with the estimated depth. The point cloud data generation unit 110 may generate point cloud data based on the depth map D1 from which spatial noise has been removed.

[0030] The noise removal process also includes, for example, a process of removing temporal noise (noise that changes over time) from the depth map D1 generated by the detection unit 10 (information acquisition unit 11). For example, the detection unit 10 repeats detection at a predetermined sampling frequency. Then, the processing device 100 compares the detection results (depth maps D1 generated by the information acquisition unit 11) of the detection unit 10 with different detection timings. For example, the processing device 100 calculates the amount of change in depth (for example, the amount of change over time) for each pixel between the depth map D1 in the first frame and the depth map D1 in the second frame following the first frame.

[0031] When the amount of change in depth for each pixel satisfies a predetermined condition, the processing device 100 determines that the depth of this pixel in the first frame or the depth in the second frame is noise. When the processing device 100 determines that the depth in the second frame is noise, the processing device 100 estimates the depth of the second frame by, for example, interpolation using the depth in the first frame and the depth in the third frame following the second frame. The processing device 100 removes the temporal noise by updating (replacing, correcting) the depth of the pixel corresponding to the noise with the estimated depth. The point cloud data generating unit 110 may generate point cloud data based on the depth map D1 from which the temporal noise has been removed.

[0032] The noise removal process may not include a process for removing spatial noise or a process for removing temporal noise. The noise removal process may be performed by a part other than the processing device 100 (for example, the detection unit 10). The detection device 1 may not perform the noise removal process.

[0033] The point cloud data generating unit 110 determines the normal line H at the point based on the point cloud data D2 (see FIG. 5). Here, the "point" refers to any point among the points on the surface S1 of the object M. Note that the normal line H at the point does not have to be determined by the point cloud data generating unit 110, and may be determined by a processing unit other than the point cloud data generating unit 110. The processing unit may be provided in the processing device 100, or may be provided in a device external to the processing device 100.

[0034] The point cloud data generating unit 110 selects, for example, each of the other points (D2a2, D2a3, D2a4, D2a5) existing within a range DR1 of a predetermined distance from an arbitrary point D2a1 among the points on the surface S1 of the object M in the point cloud data D2. The range DR1 indicates, for example, a range included in a sphere (or an ellipsoid) centered on the point D2a1, and may be set in advance by a user, or may be a variable range set according to the number and density of the point cloud data D2. The point cloud data generating unit 110 obtains and normalizes the vector V from the point D2a1 to each of the points (D2a2, D2a3, D2a4, D2a5). From these, the point cloud data generating unit 110 obtains a normal line H at the point D2a1. The point cloud data generating unit 110 obtains each of the normal lines H from each of the arbitrary points D2a1. When the object M is a human body, the normal H becomes a vector pointing inwardly toward the inside of the human body because the surface S1 of the human body is rounded.

[0035] Returning to the explanation of Fig. 1, the calculation unit 120 determines feature points related to the object M from a normal line H at a point (a point on the surface S1 of the object M) based on the position information detected by the detection unit 10. In this embodiment, the feature points are used to determine the posture of the object M.

[0036] FIG. 6 is a diagram showing an example of processing of the calculation unit according to the first embodiment. The calculation unit 120 selects nearby points from each point on the surface of the object M, for example, and obtains a surface formed by the selected points. The surface is obtained over the entire object M. In FIG. 6, one of the surfaces formed by the nearby points is S2. The surface formed by the nearby points may be obtained by the point cloud data generation unit 110, or may be obtained by a processing unit other than the calculation unit 120. The processing unit may be provided in the processing device 100, or may be provided in an external device to the processing device 100. When the normal line H obtained by the point cloud data generation unit 110 is extended toward the inside of the object M, and the normal line H' collides with the surface S2 (the surface facing the surface S1 of the object M), the calculation unit 120 obtains the distance between the point D2a1 and the surface S2, and obtains the midpoint of the obtained distance as the feature point Q. The feature point Q is obtained based on the normal line H toward the inside of the object M, and is therefore a point that exists inside the object M.

[0037] Fig. 7 is a diagram showing an example of processing by a calculation unit according to the first embodiment. A plurality of circles (ellipses) shown in Fig. 7 are feature points Q. Fig. 7 shows an enlarged view of the inside of the object M (more specifically, the location where the determined feature point Q exists).

[0038] 7, the feature points Q may be concentrated in a local area (range DR2). When there are multiple feature points Q included in the range DR2, the calculation unit 120 groups the multiple feature points Q into one feature point Qt. The range DR2 may be changed depending on the number of point cloud data D2 or the degree of accuracy of posture detection described later, or may be set by the user.

[0039] The calculation unit 120 may obtain the feature point Qt from, for example, the average value of the coordinates of the multiple feature points Q. Alternatively, the calculation unit 120 may obtain the feature point Qt from, for example, the median value of the coordinates of the multiple feature points Q. Note that the method for obtaining the feature point Qt is not limited to the above, and various methods may be used for combining points (feature points Q) scattered within a certain range (for example, range DR2) into one point (feature point Qt).

[0040] When the calculation unit 120 obtains the feature point Qt, the processing device 100 may not use the feature point Q included in the range DR2 at this time in the subsequent processing. When the number of feature points Q included in the range DR2 is less than a predetermined number, the calculation unit 120 may not obtain the feature point Qt in the range DR2 at this time. Furthermore, the calculation unit 120 may adopt any one (or several) of the feature points Q included in the range DR2 and not use the other feature points Q. Furthermore, the calculation unit 120 may not obtain the feature point Qt (as long as the feature point Q is used). Furthermore, information on the feature points Qt and Q obtained by the calculation unit 120 may be output to any device (an external device different from the processing device 100). The processing device 100 obtains the calculation of the feature point Qt (feature point Q) described above while changing the position inside the object M. For example, the processing device 100 calculates the feature point Qt (feature point Q) from each position while shifting the position by a predetermined distance inside the object M. That is, the processing device 100 acquires a plurality of feature points Qt (feature points Q).

[0041] Returning to the explanation of FIG. 1, the posture calculation unit 130 calculates the posture of the object M from the multiple feature points Qt (feature points Q) calculated by the calculation unit 120. As described above, the processing result of the calculation unit 120 may be only the feature points Q, may be only the feature points Qt, or may be a mixture of the feature points Qt and Q. The posture calculation unit 130 calculates the posture of the object M using these feature points Qt and Q. For example, the posture calculation unit 130 fits predetermined model information to the multiple feature points Qt and Q to calculate the posture of the object M. The predetermined model information may be stored in advance in the storage unit 140, or may be acquired from outside the processing device 100. In addition, if the object M is not a human body, the predetermined model information may be prepared accordingly.

[0042] Here, an example of a method for determining the posture of the object M will be described. FIG. 8 is a diagram showing an example of processing of the posture calculation unit according to the first embodiment. In FIG. 8, the feature points Qt, Q determined by the calculation unit 120 are shown as circles. Note that in FIG. 8, the object M is shown by a two-dot chain line for the purpose of explanation, but the object M (two-dot chain line) is not actually represented (as shown). The posture calculation unit 130 connects the multiple feature points Qt, Q determined by the calculation unit 120. FIG. 8 shows a connecting line CL connecting the feature points Qt, Q. Then, the posture calculation unit 130 may fit predetermined skeletal information (for example, skeleton information) as model information to the connecting line CL connecting the multiple feature points Qt, Q to determine the posture of the object M.

[0043] 9 is a diagram showing an example of predetermined skeleton information according to the first embodiment. That is, the posture calculation unit 130 obtains the posture of the object M using skeleton information Si, which is an example of model information. The skeleton information Si holds information on parts of the object M, such as the head, as well as the joints of the object M. As described above, when the object M is not a human body, the skeleton information Si may be prepared accordingly.

[0044] FIG. 10 is a diagram showing an example of processing of the posture calculation unit according to the first embodiment. Note that, for the sake of explanation, FIG. 10 shows a circle filled in on the connecting line CL, but this circle is not actually expressed (as shown in the figure). The posture calculation unit 130 fits the skeleton information Si to the connecting line CL as shown by the arrow in FIG. 10. The shape of the skeleton information Si obtained by fitting can be the posture of the target M. In addition, the skeleton information Si obtained by fitting and a processing result obtained by performing a predetermined process (for example, a process of adding an external shape) on the skeleton information Si may be output (displayed) to an arbitrary device (an external device different from the processing device 100). The processing result obtained by performing a predetermined process on the skeleton information Si is, for example, data including lines, points, etc. that represent the posture of the target M. Note that the detection device 1 does not need to include the posture calculation unit 130.

[0045] Next, a detection method according to an embodiment will be described based on the configuration of the above-mentioned detection device 1. Fig. 11 is a flowchart showing an example of the detection method according to the first embodiment. For the configuration of the detection device 1 and the processing by each unit, refer to Figs. 1 to 10 as appropriate.

[0046] In step S101, the detection unit 10 detects position information of each point on the surface of the object M. The detection unit 10 detects the depth of the object M (see FIG. 3) and generates point cloud data D2 as a detection result (see FIG. 4). For example, the information acquisition unit 11 detects the depth from a predetermined point (viewpoint) to each point in the object area AR. The information acquisition unit 11 stores a depth map (spatial distribution of depth) representing the detection result in the storage unit 140. The point cloud data generation unit 110 generates the point cloud data D2 based on the depth detection result. The point cloud data generation unit 110 stores the generated point cloud data D2 in the storage unit 140. The point cloud data generation unit 110 obtains a normal H at a point D2a1 on the surface S1 of the object M based on the point cloud data D2 (see FIG. 5).

[0047] In step S102, the calculation unit 120 obtains a feature point Q of the object M from the normal line H at the point D2a1 (see FIG. 6). For example, the calculation unit 120 obtains the feature point Q that is the midpoint of the distance from the point D2a1 to another surface S2 through which a normal line H' obtained by extending the normal line H passes. The calculation unit 120 may also obtain the feature point Qt from the average value or median value of the coordinates of the feature points Q included in the range DR2 (see FIG. 7).

[0048] In step S103, the orientation calculation unit 130 determines the orientation of the object M from the multiple feature points Qt, Q (see FIGS. 8, 9, and 10). For example, the orientation calculation unit 130 connects the multiple feature points Qt, Q to generate a connecting line CL (see FIG. 8). The orientation calculation unit 130 fits predetermined skeleton information Si (see FIG. 9) to the connecting line CL (see FIG. 10). As a result, the orientation calculation unit 130 determines the orientation of the object M.

[0049] As described above, the detection device 1 detects position information of each point on the surface of the object M, finds feature points Q of the object M from normals at the points based on the position information, and finds the posture of the object M from the multiple feature points Q. As a result, the detection device 1 finds the posture of the object M based on feature points Q assumed to be inside the object M, and can find the posture of the object M with high accuracy. In other words, when setting feature points on the surface of the object (for example, hands and feet that can be more distinctive than other parts), the detection device 1 can find the posture of the object M with high accuracy compared to techniques that have a large range of motion and therefore large noise.

[0050] [Second embodiment] Next, a second embodiment will be described. In this embodiment, the same components as those in the above-mentioned embodiment are given the same reference numerals, and the description thereof may be omitted or simplified. In the above-mentioned embodiment, the feature points Qt, Q are calculated based on the distance from a point to another surface through which the normal line at the point passes. In the second embodiment, a different method for calculating the feature points Qt, Q will be described.

[0051] 12 is a diagram illustrating an example of a detection device according to the second embodiment. The processing device 200 included in the detection device 1a includes a calculation unit 220 that determines a feature point Q (Qt) from a plurality of normals (normals at a point) based on the detection result of the detection unit 10.

[0052] FIG. 13 is a diagram showing an example of processing by the calculation unit according to the second embodiment. The rectangle shown in FIG. 13 represents a part of the point cloud data D2. In FIG. 13, arbitrary points on the surface of the object M are respectively designated as points D2a, D2b, and D2c. The points are respectively on the surfaces S1a, S1b, and S1c. As in the first embodiment, the calculation unit 120 obtains normals at the points D2a, D2b, and D2c. The respective normals are designated as Ha, Hb, and Hc, and the normals obtained by extending the respective normals are designated as normals Ha', Hb', and Hc'.

[0053] The calculation unit 220 obtains the feature point Q from the intersection of the multiple normal lines Ha'-Hc'. Since the feature point Q is obtained from the intersection of the multiple normal lines Ha'-Hc', it is a point that exists (is assumed to exist) inside the object M. Note that the calculation of the normal lines at the point does not have to be performed for the entire surface of the object M.

[0054] That is, in this embodiment, the feature point Q is obtained based on the intersection of multiple normal lines Ha'-Hc', and other surfaces through which the normal lines pass are not used in the processing, so it is not necessary to use points over the entire surface of the object M. In other words, in this embodiment, it is sufficient to determine at least the intersection of multiple normal lines, so normal lines from all directions on the surface of the object M are not required, and it is not necessary to use points over the entire surface of the object M. The area in which points are used may be changed depending on the type (or size, shape, etc.) of the object M, or may be set by the user.

[0055] Fig. 14 is a diagram showing an example of processing of a calculation unit according to the second embodiment. In Fig. 14, a plurality of normal lines (a plurality of extended normal lines) are represented by H'. Fig. 14 also shows an enlarged view of the vicinity of the intersection point of the plurality of normal lines H' (the inner side of the object M, where the determined feature point Q exists).

[0056] 14, the feature points Q may be concentrated in a local area (range DR3). When there are multiple feature points Q included in the range DR3, the calculation unit 220 groups the multiple feature points Q into one feature point Qt. The range DR3 may be changed depending on the number of point cloud data D2 or the degree of accuracy of posture detection, or may be set by the user.

[0057] The calculation unit 220 may obtain the feature point Qt from, for example, the average value of the coordinates of the multiple feature points Q. Alternatively, the calculation unit 220 may obtain the feature point Qt from, for example, the median value of the coordinates of the multiple feature points Q. Note that the method for obtaining the feature point Qt is not limited to the above, as in the first embodiment. When the calculation unit 220 obtains the feature point Qt, the processing device 200 may not use the feature point Q included in the range DR3 at this time in the subsequent processing.

[0058] When the number of feature points Q included in the range DR3 is less than a predetermined number, the calculation unit 220 may not calculate the feature point Qt in the range DR3 at this time. The calculation unit 220 may not calculate the feature point Qt (as long as the feature point Q is used). Information on the feature points Qt and Q calculated by the calculation unit 220 may be output to an arbitrary device (an external device different from the processing device 200).

[0059] Next, a detection method according to an embodiment will be described based on the configuration of the above-mentioned detection device 1a. Fig. 15 is a flowchart showing an example of the detection method according to the second embodiment. For the configuration of the detection device 1a and the processing by each unit, refer to Figs. 1 to 14 as appropriate.

[0060] In step S201, the detection unit 10 detects position information of each point on the surface of the object M. The detection unit 10 detects the depth of the object M (see FIG. 3) and generates point cloud data D2 as a detection result (see FIG. 4). For example, the information acquisition unit 11 detects the depth from a predetermined point (viewpoint) to each point in the object area AR. The information acquisition unit 11 stores a depth map (spatial distribution of depth) representing the detection result in the storage unit 140. The point cloud data generation unit 110 generates point cloud data D2 based on the depth detection result. The point cloud data generation unit 110 stores the generated point cloud data D2 in the storage unit 140. The point cloud data generation unit 110 calculates normals Ha, Hb, Hc at points D2a, D2b, D2c on the surfaces S1a, S1b, S1c of the object M based on the point cloud data D2.

[0061] In step S202, the calculation unit 220 obtains the feature point Q based on the intersection of a plurality of normal lines (see FIG. 13). For example, the calculation unit 220 obtains the feature point Q based on the intersection of a normal line Ha' obtained by extending the normal line Ha at the point D2a, a normal line Hb' obtained by extending the normal line Hb at the point D2b, and a normal line Hc' obtained by extending the normal line Hc at the point D2c (see FIG. 13). The calculation unit 220 may also obtain the feature point Qt from the average or median value of the coordinates of the feature point Q included in the range DR3 (see FIG. 14).

[0062] In step S203, the orientation calculation unit 130 determines the orientation of the object M from the multiple feature points Qt, Q (see FIGS. 8, 9, and 10). For example, the orientation calculation unit 130 connects the multiple feature points Qt, Q to generate a connecting line CL (see FIG. 8). The orientation calculation unit 130 fits the skeleton information Si (see FIG. 9) to the connecting line CL (see FIG. 10). As a result, the orientation calculation unit 130 determines the orientation of the object M.

[0063] As described above, the detection device 1a detects position information of each point on the surface of the object M, finds feature points Q from the intersections of multiple normals at the points based on the position information, and finds the posture of the object M from the multiple feature points Q. As a result, the detection device 1a finds feature points Q assumed to be inside the object M using points in any area, not the entire area, on the surface of the object M, and can therefore find the posture of the object M with high accuracy while reducing the processing load and increasing the processing speed.

[0064] [Third embodiment] Next, a third embodiment will be described. In this embodiment, the same components as those in the above-mentioned embodiment are given the same reference numerals, and the description thereof may be omitted or simplified. In the above-mentioned embodiment, the moving direction MD of the object M is predetermined with respect to the target area AR. In the third embodiment, the moving direction MD of the object M is detected.

[0065] FIG. 16 is a diagram showing an example of a detection device according to the third embodiment. The processing device 300 included in the detection device 1b includes a direction detection unit 350 that calculates the moving direction of the object M based on the detection result of the detection unit 10. In this embodiment, the information acquisition unit 11 repeatedly detects the object M. In addition, the point cloud data generation unit 110 generates point cloud data of the object M at each detection timing as position information. The direction detection unit 350 calculates the moving direction MD of the object M based on the time change of the position information of the object M detected by the detection unit 10. For example, the direction detection unit 350 calculates the trajectory of the object M (for example, the time history of a predetermined position of the object M) and sets the direction along the trajectory as the moving direction MD of the object M. Such a predetermined position is a position on the object M determined by default settings or user settings.

[0066] The direction detection unit 350 calculates the center of gravity of multiple points included in the point cloud data as a predetermined position of the object M. The direction detection unit 350 calculates the movement direction MD based on the time change of the calculated center of gravity. For example, the direction detection unit 350 sets the first predetermined position of the object M obtained from the detection result of the information acquisition unit 11 at a first time as the start point. Then, the direction detection unit 350 calculates a vector whose end point is the second predetermined position of the object M obtained from the detection result of the information acquisition unit 11 at a second time after the first time. Next, the direction detection unit 350 determines a direction vector (for example, a unit vector) parallel to the calculated vector as the movement direction MD of the object M at the first time.

[0067] The direction detection unit 350 may calculate the moving direction MD of the object M based on position information of the object M acquired from a global positioning system (GPS). In addition, when an acceleration sensor is provided on the object M, the direction detection unit 350 may calculate the moving direction MD of the object M based on the detection result of the acceleration sensor. For example, the object M may move accompanied by a mobile terminal (e.g., a smartphone) equipped with either or both of a receiving unit that receives information from a GPS and an acceleration sensor. In such a case, the direction detection unit 350 may calculate the moving direction MD of the object M by acquiring position information of the object M (e.g., GPS information, acceleration information) from the mobile terminal.

[0068] [Fourth embodiment] Next, a fourth embodiment will be described. In this embodiment, the same components as those in the above-mentioned embodiments are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0069] 17 is a diagram showing an example of a detection device according to the fourth embodiment. A processing device 400 included in the detection device 1c has a model generation unit 460 that generates model information of an object M. The model information is, for example, three-dimensional CG (Computer Graphics) model data, and includes shape information of the object M. As described in the first embodiment, the point cloud data generation unit 110 generates point cloud data as shape information based on the depth of the object M detected by the information acquisition unit 11. The model generation unit 460 executes surface processing to calculate surface information as shape information.

[0070] The surface information includes, for example, at least one of polygon data, vector data, and draw data. The surface information includes coordinates of a plurality of points on the surface of an object, and connection information (attribute information) between the plurality of points. The connection information includes, for example, information that associates points at both ends of a line corresponding to a ridge (e.g., an edge) of the object surface with each other. The connection information also includes, for example, information that associates a plurality of lines corresponding to the contour of the object surface with each other.

[0071] In the surface processing, the model generating unit 460 estimates a surface between a point selected from a plurality of points included in the point cloud data and a point in the vicinity of the selected point. The model generating unit 460 segments the point cloud data when estimating a surface between a point and a point in the vicinity of the point. For example, the model generating unit 460 may segment the point cloud data using the processing result of the posture calculation unit 130 when estimating. For example, when generating surface information of the left and right feet of the object M, the model generating unit 460 distinguishes the point cloud data of the left foot from the point cloud data of the right foot based on the processing result of the posture calculation unit 130. Thereby, the model generating unit 460 can avoid estimating a surface that spans the left knee and the right knee when, for example, the left knee and the right knee are close to each other (in contact), and generate highly accurate surface information.

[0072] In addition, the model generation unit 460 converts the point cloud data into polygon data having plane information between points in surface processing. The model generation unit 460 converts the point cloud data into polygon data by an algorithm using, for example, the least squares method. This algorithm may be, for example, an algorithm published in a point cloud processing library. The model generation unit 460 stores the calculated surface information in the storage unit 140.

[0073] The model information may include texture information of the object M. The model generation unit 460 may generate texture information of a surface defined by three-dimensional point coordinates and related information thereof. The texture information includes at least one of information defining characters, figures, designs, textures, patterns, and unevenness on the object surface, a specific image, and color (e.g., chromatic color, achromatic color). The model generation unit 460 may store the generated texture information in the storage unit 140.

[0074] The model information may also include spatial information of the image (e.g., lighting conditions, light source information). The light source information includes at least one item of information on the position of a light source that irradiates the object M with light, the direction in which the light source irradiates the object M with light, the wavelength of the light irradiated from the light source, and the type of the light source. The model generation unit 460 may calculate the light source information using, for example, a model that assumes Lambertian reflection, a model that includes albedo estimation, or the like. The model generation unit 460 does not need to generate either or both of the texture information and the spatial information.

[0075] [Fifth embodiment] Next, a fifth embodiment will be described. In this embodiment, the same components as those in the above-mentioned embodiments are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0076] FIG. 18 is a diagram showing an example of a detection device according to the fifth embodiment. A processing device 500 included in the detection device 1d includes a model generation unit 460 and a rendering processing unit 570. An input device 21 and a display device 22 are connected to the processing device 500. The rendering processing unit 570 includes, for example, a GPU (Graphics Processing Unit). Note that the rendering processing unit 570 may be configured such that a CPU (Central Processing Unit) and a memory execute each process according to an image processing program. The rendering processing unit 570 executes at least one process among, for example, a drawing process, a texture mapping process, and a shading process.

[0077] In the drawing process, the rendering processing unit 570 can calculate an estimated image (reconstructed image) of a shape defined in the shape information of the model information from an arbitrary viewpoint. Hereinafter, the shape indicated by the shape information may be referred to as a model shape. The rendering processing unit 570 can reconstruct a model shape (estimated image) from the model information (shape information) by the drawing process, for example. The rendering processing unit 570 stores data of the calculated estimated image in the storage unit 140, for example.

[0078] In addition, in the texture mapping process, the rendering processing unit 570 can calculate an estimated image in which an image indicated by texture information of the model information is pasted onto the surface of an object in the estimated image. The rendering processing unit 570 can calculate an estimated image in which a texture different from that of the target is pasted onto the surface of the object in the estimated image.

[0079] In the shading process, the rendering processor 570 can calculate an estimated image in which a shadow formed by a light source indicated by the light source information of the model information is added to an object in the estimated image. In addition, in the shading process, the rendering processor 570 can calculate an estimated image in which a shadow formed by an arbitrary light source is added to an object in the estimated image.

[0080] The input device 21 is used to input various information (e.g., data, commands) to the processing device 500. A user can input various information to the processing device 500 by operating the input device 21. The input device 21 includes at least one of a keyboard, a mouse, a trackball, a touch panel, and a voice input device (e.g., a microphone), for example.

[0081] The display device 22 displays the image based on the image data output from the processing device 500. For example, the processing device 500 outputs data of the estimated image generated by the rendering processing unit 570 to the display device 22. The display device 22 displays the estimated image based on the estimated image data output from the processing device 500. The display device 22 includes, for example, a liquid crystal display. The input device 21 and the display device 22 may be configured with a touch panel or the like.

[0082] Furthermore, the input device 21 and the display device 22 may be connected to the detection device according to the embodiment described above. For example, the display device 22 may display information on the feature points Qt, Q calculated by the feature point calculation unit 122 or the like. For example, the display device 22 may display skeleton information Si calculated by the posture calculation unit 130 or the like, or a processing result obtained by performing a predetermined process on the skeleton information Si (for example, data including lines, points, etc. that represent the posture of the target object M).

[0083] The detection device 1d may not include the input device 21. For example, the detection device 1d may be configured such that various commands and information are input via communication. The detection device 1d may not include the display device 22. For example, the detection device 1d may output data of an estimated image generated by a rendering process to a display device via communication, and cause the display device to display the estimated image. The rendering processing unit 570 may be provided in a device external to the processing device 500. Such an external device may be provided by cloud computing that is communicably connected to the processing device 500.

[0084] In the above-described embodiment, the processing device 100 and the like include, for example, a computer system. The processing device 100 reads a processing program stored in the storage unit 140 and executes various processes according to the read processing program. For example, such a processing program causes a computer to execute a process of determining feature points of an object from normals at points based on position information of each point on the surface of the object. This processing program may be provided by being recorded in a computer-readable storage medium (for example, a non-transitory tangible storage medium).

[0085] The technical scope is not limited to the aspects described in the above-mentioned embodiments. One or more of the requirements described in the above-mentioned embodiments may be omitted. The requirements described in the above-mentioned embodiments may be combined as appropriate. In addition, to the extent permitted by law, the disclosures of all documents cited in the above-mentioned embodiments are incorporated by reference and made part of the description in this text. [Explanation of symbols]

[0086] 1 detection device, 10 detection section, 11 information acquisition section, 100 processing section, 110 point cloud data generation section, 120 calculation section, 130 attitude calculation section, 140 storage section

Claims

1. a calculation unit that receives point cloud data indicating position information of each point on a surface of an object, and calculates feature points related to the object from normals at points on the point cloud data; a posture calculation unit that calculates a posture of the object from a plurality of the feature points; a model generation unit that generates model information including shape information of the object based on the point cloud data, the posture calculation unit obtains a posture of the object by fitting skeleton information to the plurality of feature points; The model generation unit distinguishes between a first partial point cloud and a second partial point cloud of the point cloud data based on the posture of the object calculated by the posture calculation unit.

2. The processing device described in Claim 1, wherein the model generation unit generates shape information of the first part of the object based on the first part point cloud.

3. The calculation unit determines the feature points within the object. The processing device according to claim 1 or 2.

4. the calculation unit determines the feature point from a normal at the point that is determined based on the position information between the point and each of other points that exist within a predetermined distance from the point; The processing device according to any one of claims 1 to 3.

5. The calculation unit determines the feature point based on a distance from the point to a plane through which a normal line at the point passes. The processing device according to any one of claims 1 to 4.

6. The calculation unit determines the feature point from a midpoint of a distance to a surface through which a normal line at the point passes. The processing device according to claim 5 .

7. The calculation unit calculates the feature point from an average value of coordinates of the plurality of midpoints. The processing device of claim 6.

8. The calculation unit determines the feature point from a median value of coordinates of the plurality of midpoints. The processing device of claim 6.

9. The calculation unit determines the feature points from normals at the plurality of points. The processing device according to any one of claims 1 to 4.

10. The calculation unit determines the feature point based on an intersection of normals at a plurality of the points. The processing device of claim 9.

11. the calculation unit determines the feature point from an average value of coordinates of intersections of normal lines at a plurality of the points; The processing device of claim 10.

12. the calculation unit determines the feature point from a median value of coordinates of intersections of normal lines at a plurality of the points; The processing device of claim 10.

13. The processing device according to claim 1 , wherein the orientation calculation unit calculates the orientation of the object using a connecting line that connects a plurality of the feature points.

14. A detection unit that detects position information of each point on a surface of an object; A processing device according to any one of claims 1 to 13; A detection device comprising:

15. On the computer, receiving point cloud data indicating positional information of each point on a surface of an object, and determining feature points related to the object from normals at points on the point cloud data; determining a posture of the object from a plurality of the feature points; generating model information including shape information of the object based on the point cloud data; In the process of determining the posture of the object, the posture of the object is determined by fitting skeleton information to the plurality of feature points; A processing program in which, in a process of generating model information including shape information of the object, a first partial point cloud and a second partial point cloud of the point cloud data are distinguished based on the determined posture of the object.

Images