Training device, training method, training program, behavior detection device, behavior detection method, and behavior detection program

Abstract

This training device (10) comprises: a two-dimensional projection conversion unit (12) that performs projection conversion on three-dimensional training skeleton coordinates (D11) according to a first camera parameter (P1) and generates first two-dimensional training data (D21), and performs projection conversion on the three-dimensional training skeleton coordinates (D11) according to a second camera parameter (P2) and generates second two-dimensional training data (D22); a first model generation unit (13) that generates a first training model (M1) from first two-dimensional inference skeleton coordinates (D21a); and a second model generation unit (14) that generates a second training model (M2) from second two-dimensional inference skeleton coordinates (D21b). This behavior detection device (40) calculates a two-dimensional image-derived two-dimensional inference skeleton coordinate (D42) from a two-dimensional image (D41) including an object, and detects a behavior performed by the object by using a training model selected from among the plurality of training models generated by the training device (10).

Description

Learning device, learning method, learning program, behavior detection device, behavior detection method, and behavior detection program

[0001] The present disclosure relates to a learning device, a learning method, a learning program, a behavior detection device, a behavior detection method, and a behavior detection program.

[0002] A device for supporting monitoring of a subject in bed has been proposed (see, for example, Patent Document 1). This device includes an image acquisition unit that acquires images captured by an imaging device, an information acquisition unit that acquires information related to the installation height of the imaging device, and multiple detection units (i.e., multiple modules) that correspond to the installation heights of the imaging device, respectively, for detecting the subject or the subject's condition from the acquired images. In this device, the detection unit that corresponds to the installation height of the imaging device is used to detect the subject or the subject's condition from the acquired images.

[0003] Japanese Patent No. 6729510 (see, for example, claim 1, paragraphs 0023 to 0027, and figures 1 to 3)

[0004] However, since the above-mentioned conventional device has multiple detection units corresponding to multiple installation heights, it is necessary to prepare learning data for each of the multiple detection units, which results in high costs for designing the device.

[0005] The present disclosure aims to provide a learning device, a learning method, and a learning program that enable the generation of multiple learning models for inferring behavior at low cost, as well as a behavior detection device, a behavior detection method, and a behavior detection program that accurately detect behavior using a learning model selected from multiple learning models.

[0006] The learning device of the present disclosure is a device that receives three-dimensional learning data including three-dimensional learning skeletal coordinates including three-dimensional coordinates of a target skeleton and teacher data consisting of correct answers to actions linked to the skeleton, and generates a learning model, and projects and transforms the three-dimensional learning skeletal coordinates onto a first two-dimensional plane according to predetermined first camera parameters to generate first two-dimensional learning data including first two-dimensional learning skeletal coordinates and the teacher data, and projects and transforms the three-dimensional learning skeletal coordinates onto a second two-dimensional plane according to predetermined second camera parameters to generate second two-dimensional learning data. The system is characterized by having a two-dimensional projection transformation unit that generates second two-dimensional learning data including two-dimensional skeletal coordinates for learning and the teacher data; a first model generation unit that uses the first two-dimensional learning data to generate a first learning model for inferring behavior associated with the first two-dimensional skeletal coordinates for inference from the first two-dimensional skeletal coordinates for inference; and a second model generation unit that uses the second two-dimensional learning data to generate a second learning model for inferring behavior associated with the second two-dimensional skeletal coordinates for inference from the second two-dimensional skeletal coordinates for inference.

[0007] The behavior detection device disclosed herein is a device that detects behavior performed by a subject included in a two-dimensional image acquired by a two-dimensional image acquisition unit, and is characterized by having a two-dimensional skeletal coordinate calculation unit that calculates two-dimensional skeletal coordinates for two-dimensional image-derived inference that indicate the coordinates of the subject's skeleton from the two-dimensional image, and a behavior detection unit that selects one or more learning models from the first learning model and the second learning model generated by the learning device, and detects the behavior from the two-dimensional skeletal coordinates for two-dimensional image-derived inference using the selected one or more learning models.

[0008] By using the learning device, learning method, and learning program of the present disclosure, it is possible to generate multiple learning models for inferring behavior at low cost.

[0009] Furthermore, by using the behavior detection device, behavior detection method, and behavior detection program of the present disclosure, it is possible to accurately detect behavior using a learning model selected from a plurality of learning models.

[0010] 1 is a functional block diagram illustrating an outline of the configuration of a learning device according to embodiment 1. FIG. 1 is a conceptual diagram illustrating a process of generating learning two-dimensional skeletal coordinates from learning three-dimensional skeletal coordinates according to first camera parameters. FIG. 2 is a conceptual diagram illustrating a process of generating learning two-dimensional skeletal coordinates from learning three-dimensional skeletal coordinates according to first camera parameters. FIG. 3 is a conceptual diagram illustrating a process of generating learning two-dimensional skeletal coordinates from learning three-dimensional skeletal coordinates according to second camera parameters. FIG. 4 is a conceptual diagram illustrating a process of generating learning two-dimensional skeletal coordinates from learning three-dimensional skeletal coordinates according to third camera parameters. FIG. 5 is a flowchart illustrating an outline of the operation of the learning device according to embodiment 1. FIG. 5A is a diagram illustrating a process of rotating learning two-dimensional skeletal coordinates generated from learning three-dimensional skeletal coordinates around the y-axis, FIG. 5B is a diagram illustrating a process of rotating learning two-dimensional skeletal coordinates generated from learning three-dimensional skeletal coordinates around the z-axis, and FIG. 5C is a diagram illustrating a process of rotating learning two-dimensional skeletal coordinates generated from learning three-dimensional skeletal coordinates around the x-axis. 1 is a flowchart outlining the operation of a learning device according to embodiment 1 (including rotation around the x-axis and rotation around the y-axis, but no rotation around the z-axis). (A) and (B) are diagrams illustrating a state in which two-dimensional learning skeletal coordinates generated from three-dimensional learning skeletal coordinates are rotated 90° around the z-axis. (A) and (B) are diagrams illustrating a state in which two-dimensional learning skeletal coordinates generated from three-dimensional learning skeletal coordinates are rotated 90° around the x-axis. (A) and (B) are diagrams illustrating an example of bone extension of three-dimensional learning skeletal coordinates. (A) and (B) are diagrams illustrating an interpolation process for two-dimensional learning skeletal coordinates generated from three-dimensional learning skeletal coordinates. (B) are diagrams illustrating an example of the hardware configuration of a learning device according to embodiment 1. (C) are functional block diagrams outlining the configuration of a behavior detection device according to embodiment 2. (A), (B), and (C) are conceptual diagrams illustrating two-dimensional images of a person captured by cameras with different depression angles and installation positions. (C) are conceptual diagrams illustrating two-dimensional images captured by a camera, two-dimensional skeletal coordinates for inference generated from the two-dimensional images, and behavior detection according to a first learning model. A conceptual diagram showing two-dimensional images obtained by camera photography, two-dimensional skeletal coordinates for inference generated from the two-dimensional images, and behavior detection according to a second learning model.1 is a conceptual diagram showing a two-dimensional image obtained by camera photography, two-dimensional skeletal coordinates for inference generated from the two-dimensional image, and behavior detection according to a third learning model. 2 is a flowchart showing an outline of the operation of a behavior detection device according to embodiment 2. 3 is a diagram showing an example of the hardware configuration of a behavior detection device according to embodiment 2.

[0011] A learning device, a learning method, a learning program, a behavior detection device, a behavior detection method, and a behavior detection program according to embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be combined as appropriate and each embodiment can be modified as appropriate.

[0012] 1 is a functional block diagram showing a schematic configuration of a learning device 10 according to the first embodiment. The learning device 10 is a device capable of implementing the learning method according to the first embodiment. The learning device 10 is, for example, a computer capable of executing the learning program according to the first embodiment. The learning device 10 may be a computer (i.e., a computer system) configured by cloud computing using a computer network.

[0013] The learning device 10 is a device that enables low-cost generation of multiple learning models used to infer behaviors performed by a subject. The subject is, for example, a human subject. The subject may be an animal with a skeleton (including humans and non-human animals). The subject may include a robot with joints (e.g., a humanoid robot). The behaviors performed by the subject may include the movement of the subject, the state of the subject, and the posture of the subject. Examples of behaviors include standing, walking, falling, lying, etc.

[0014] The learning device 10 can generate (including "update") multiple learning models from the three-dimensional training data D10 by using projective transformation. Projective transformation will be described below with reference to FIGS. 2 to 5. The learning device 10 includes a two-dimensional projection transformation unit 12 and multiple model generation units. The learning device 10 may also include a three-dimensional data acquisition unit 11. The three-dimensional data acquisition unit 11 may be part of the learning device 10 or may be an external device that provides data to the learning device 10. In the first embodiment, the multiple model generation units include two model generation units, i.e., a first model generation unit 13 and a second model generation unit 14. The multiple model generation units may also include three or more model generation units. In other words, the multiple learning models may include a first learning model M1, a second learning model M2, a third learning model M3, etc. The first learning model M1, the second learning model M2, and the third learning model M3 are stored in storage devices 15, 16, and 17, respectively. The storage devices 15, 16, and 17 may be the same or different storage devices, and may be part of the learning device 10 or may be storage devices in external devices.

[0015] The three-dimensional data acquisition unit 11 acquires three-dimensional training data D10. The three-dimensional data acquisition unit 11 is, for example, a detector such as a stereo camera, a TOF (Time Of Flight) camera, motion capture, or LiDAR (Light Detection And Ranging). A stereo camera is an imaging device with a distance measurement function based on the principle of trigonometry from two-eye images. A TOF camera is an imaging device with a function to measure distance using the time it takes for irradiated light to reflect and return. A motion capture is a device with a function to attach markers to a person or object and digitally record their movements. LiDAR is a device that can irradiate electromagnetic waves and measure distance based on the time until the reflected waves are detected, and speed by measuring the Doppler shift of the reflected waves. The three-dimensional training data D10 includes three-dimensional training skeletal coordinates D11 including the three-dimensional coordinates of the target's skeleton, and teacher data D12 consisting of correct answers for actions associated with the skeleton. The three-dimensional data acquisition unit 11 may also have a function for calculating the three-dimensional training skeletal coordinates using deep learning or the like from images captured by a monocular two-dimensional camera. The three-dimensional data acquisition unit 11 may also receive the three-dimensional training skeletal coordinates from a database related to skeletons stored in a storage device. The storage device is shown in FIG. 12, which will be described later.

[0016] The two-dimensional projection conversion unit 12 acquires multiple camera parameters of multiple virtual cameras, for example, from multiple virtual cameras or from a storage device (e.g., shown in FIG. 12 , described later) that pre-stores multiple camera parameters. The multiple camera parameters include information regarding the position and orientation (e.g., optical axis direction) of each of the multiple virtual cameras that generate the training three-dimensional skeleton coordinates D11. In the first embodiment, the multiple camera parameters include a predetermined first camera parameter P1 used to generate the first learning model M1 and a predetermined second camera parameter P2 used to generate the second learning model M2. Furthermore, the multiple camera parameters may include three or more predetermined camera parameters that are different from each other, and the two-dimensional projection conversion unit 12 may calculate three or more training two-dimensional skeleton coordinates using the three or more camera parameters.

[0017] When generating the two-dimensional training skeleton coordinates using perspective projection, it is desirable that each of the multiple camera parameters includes coordinates in the world coordinate system of the virtual camera to be set and the focal length of the virtual camera. Furthermore, each of the multiple camera parameters is defined as a rotation angle r about three axes consisting of the x-axis, y-axis, and z-axis with respect to the three-dimensional training skeleton coordinates D11, where the coordinate axis in the vertical direction (the direction of gravity on the Earth) is defined as the y-axis, the coordinate axis on the line obtained by projecting a straight line in the projection direction used when generating the multiple two-dimensional training skeleton coordinates (e.g., the first two-dimensional training skeleton coordinates D21a and the second two-dimensional training skeleton coordinates D21b) onto a horizontal plane that is perpendicular to the y-axis is defined as the z-axis, and the coordinate axis perpendicular to both the y-axis and the z-axis is defined as the x-axis. ｘ , r ｙ , r ｚ It is desirable to include a rotation angle (r ｙ１ ) and the rotation angle around the y-axis according to the second camera parameter P2 (r ｙ２ ) any value can be chosen.

[0018] Also, the rotation angle around the x-axis according to the first camera parameter P1 (r ｘ１ ) and a predetermined angle r that is greater than or equal to -90° and less than or equal to 0° ｘ１ｍｉｎ from a predetermined angle r that is greater than or equal to 0° and less than or equal to 90°. ｘ１ｍａｘ Within the range of r ｘ１ｍｉｎ ≦r ｘ１ ≦r ｘ１ｍａｘ Similarly, the rotation angle around the x-axis according to the second camera parameter P2 (r ｘ２ ) and a predetermined angle r that is greater than or equal to -90° and less than or equal to 0° ｘ２ｍｉｎ from a predetermined angle r that is greater than or equal to 0° and less than or equal to 90°. ｘ２ｍａｘ Within the range of r ｘ２ｍｉｎ ≦r ｘ２ ≦r ｘ２ｍａｘ It is possible to select an angle where

[0019] Furthermore, the rotation angle (r ｚ１) and a predetermined angle r that is greater than or equal to -90° and less than or equal to 0° ｚｍｉｎ１ from a predetermined angle r that is greater than or equal to 0° and less than or equal to 90°. ｚ１ｍａｘ Within the range of r ｚ１ｍｉｎ ≦r ｚ１ ≦r ｚ１ｍａｘ Similarly, the rotation angle around the z-axis according to the second camera parameter P2 (r ｚ２ ) and a predetermined angle r that is greater than or equal to -90° and less than or equal to 0° ｚ２ｍｉｎ from a predetermined angle r that is greater than or equal to 0° and less than or equal to 90°. ｚ２ｍａｘ Within the range of r ｚ２ｍｉｎ ≦r ｚ２ ≦r ｚ２ｍａｘ It is possible to select an angle where

[0020] FIG. 7A shows the three-dimensional skeletal coordinate system for learning D11 rotated around the y-axis by a rotation angle r ｙ 7B is a diagram showing a process of rotating the training 3D skeleton coordinates D11 around the z axis by a rotation angle r ｚ 7C is a diagram showing a process of rotating the training 3D skeleton coordinates D11 around the x-axis by a rotation angle r ｘ FIG. 10 is a diagram showing a process of rotating by [°].

[0021] FIG. 2 is a conceptual diagram showing the process of generating first training two-dimensional skeleton coordinates D21a from training three-dimensional skeleton coordinates D11 by parallel projection according to first camera parameters P1. In parallel projection, relative movement of each axis does not affect the shape of the projection. In parallel projection, the parameters that affect the projection are the y-axis (vertical axis), the z-axis (the direction of the line projected onto the horizontal plane (xz plane) perpendicular to the vertical axis, i.e., the x-axis), and the rotation angle of the x-axis (the axis perpendicular to the y-axis and z-axis). For ease of explanation, the z-axis is positioned so as to coincide with the line obtained by projecting the projection direction line 21 onto the horizontal plane (xz plane).

[0022] FIG. 2 shows an example in which the optical axis direction of the virtual camera 20 (first virtual camera) is the z-axis direction, i.e., r ｘ１= 0°. The two-dimensional projection transformation unit 12 performs projection transformation of the training three-dimensional skeleton coordinates D11 onto a first two-dimensional plane 22a, which is a projection plane, in accordance with the first camera parameters P1, to generate first training two-dimensional data D21 including the first training two-dimensional skeleton coordinates D21a and teacher data. The first two-dimensional plane 22a is, for example, a virtual plane perpendicular to the projection direction (straight line 21) of the virtual camera 20. The first two-dimensional plane 22a may be a non-flat surface such as a sphere, a cylinder, a paraboloid, a hyperboloid, or a hyperbolic paraboloid, or a part thereof, or a surface of a non-linear coordinate system for correcting geometric distortion.

[0023] Perspective projection can also be used for the projection transformation. Fig. 3 is a conceptual diagram showing the process of generating first training two-dimensional skeleton coordinates D21a' from training three-dimensional skeleton coordinates D11 by perspective projection according to the first camera parameters P1'. Compared to parallel projection, the difference is that the projected image changes depending on the relative positional relationship between the optical axis of the virtual camera and the training three-dimensional skeleton coordinates.

[0024] 4 is a conceptual diagram showing a process of generating second training two-dimensional skeleton coordinates D21b from training three-dimensional skeleton coordinates D11 by parallel projection in accordance with the second camera parameter P2. FIG. 4 shows an example in which the projection direction of the virtual camera 20 (second virtual camera) is inclined with respect to the z-axis direction, i.e., r ｘ２ 1 shows an example where the angle is >0°. The two-dimensional projection transformation unit 12 performs projection transformation of the training three-dimensional skeleton coordinates D11 onto a second two-dimensional surface 22b, which is a projection surface, in accordance with the second camera parameters P2, to generate second training two-dimensional data D22 including second training two-dimensional skeleton coordinates D21b and teacher data. The second two-dimensional surface 22b is, for example, a virtual plane perpendicular to the projection direction (straight line 21) of the virtual camera 20. The second two-dimensional surface 22b may be a non-flat surface such as a sphere, a cylinder, a paraboloid, a hyperboloid, or a hyperbolic paraboloid, or a part thereof, or a surface of a non-linear coordinate system for correcting geometric distortion.

[0025] 5 is a conceptual diagram showing a process of generating third training two-dimensional skeleton coordinates D21c from training three-dimensional skeleton coordinates D11 in accordance with the third camera parameter P3. FIG. 5 shows an example in which the projection direction of the virtual camera 20 is inclined with respect to the z-axis, i.e., r ｘ３ >r ｘ２ 1 shows an example in which the angle is >0°. The two-dimensional projection transformation unit 12 performs projection transformation of the training three-dimensional skeleton coordinates D11 onto a third two-dimensional surface 22c, which is a projection surface, in accordance with the third camera parameters P3, to generate third training two-dimensional data including the third training two-dimensional skeleton coordinates D21c and teacher data. The third two-dimensional surface 22c is, for example, a virtual plane perpendicular to the projection direction (straight line 21) of the virtual camera 20. The third two-dimensional surface 22c may be a non-flat surface such as a sphere, a cylinder, a paraboloid, a hyperboloid, or a hyperbolic paraboloid, or a part thereof, or a surface of a non-linear coordinate system for correcting geometric distortion.

[0026] 5 is a flowchart illustrating the operation of the learning device 10 according to embodiment 1. First, the learning device 10 acquires three-dimensional learning data D10 including three-dimensional learning skeletal coordinates D11 including the three-dimensional coordinates of the skeleton of a target and teacher data D12 including correct answers for actions associated with the skeleton (step S11).

[0027] Next, the learning device 10 generates first two-dimensional learning data D21 including first two-dimensional learning coordinates D21a and teacher data by projecting the three-dimensional learning coordinates D11 onto the first two-dimensional surface 22a according to the predetermined first camera parameters P1 (step S12). The learning device 10 also generates second two-dimensional learning data D22 including second two-dimensional learning coordinates D21b and teacher data by projecting the three-dimensional learning coordinates D11 onto the second two-dimensional surface 22b according to the predetermined second camera parameters P2 (step S12).

[0028] Next, the learning device 10 executes a learning process using the first training two-dimensional data D21 to generate (including "update") a first learning model M1 for inferring behavior associated with the first inference two-dimensional skeletal coordinates, which are new two-dimensional skeletal coordinates, from the first inference two-dimensional skeletal coordinates (step S13). The learning device 10 also executes a learning process using the second training two-dimensional data D22 to generate (including "update") a second learning model M2 for inferring behavior associated with the second inference two-dimensional skeletal coordinates, which are new two-dimensional skeletal coordinates (step S13). The first inference two-dimensional skeletal coordinates and the second inference two-dimensional skeletal coordinates are different coordinates, but may also be the same coordinates.

[0029] Next, the learning device 10 stores the generated learning models in a storage device (step S14).

[0030] The learning device 10 rotates the learning three-dimensional skeletal coordinates D11 of Figures 7(A) to (C) in various ways around the three axes of the x-axis, y-axis, and z-axis, and uses the output at that time to update multiple learning models (e.g., a first learning model M1 and a second learning model M2, etc.).

[0031] FIG. 8 is a flowchart outlining the operation of the learning device 10 according to embodiment 1 (including rotation around the x-axis and y-axis, but no rotation around the z-axis). In FIG. 8, three-dimensional learning data D10 including three-dimensional learning skeletal coordinates D11 is sequentially input to the learning device 10 (steps S101 and S102). The three-dimensional learning skeletal coordinates D11 are then rotated around three axes (steps S103 to S107, S110 to S113, and S116 to S119). The data projected onto a two-dimensional surface (projection surface) is used (steps S108 and S114), and the first learning model M1 (step S109) and the second learning model M2 (step S115) are updated. When parallel projection is used, the coordinates on the two-dimensional surface (projection surface) are the x- and y-coordinate values ​​of the rotated three-dimensional coordinates.

[0032] Rotation angle r around the y-axis ｙIt is desirable that the rotation angle r covers the entire range from 0° to 360°. This is because if the rotation around the y-axis is limited to a narrow range, it may not be possible to detect certain actions (for example, a fall on the right side may be detected, but a fall on the left side may not be detected), which would reduce the action detection performance of the inference device (described in the second embodiment). In the process of FIG. 8, the rotation angle r ｙ is changed as a parameter in increments of 10° within the range from 0° to 350° (steps S103, S118, S119).

[0033] In the process of FIG. 8, the rotation angle r around the z axis ｚ 9A and 9B are diagrams showing the state in which the training 2D skeleton coordinates generated from the training 3D skeleton coordinates are rotated around the z axis. As shown in FIG. 9A, r ｚ At r = 0°, the training 2D skeleton coordinates are in an upright posture, but ｚ At r = 90°, the training 2D skeleton coordinates are in a lying position. ｚ At r = 0°, the training 2D skeleton coordinates are in a supine position, but ｚ = -90°, the training 2D skeleton coordinates are in an upright position. ｚ In other words, the two-dimensional projection transformation unit 12 projects the three-dimensional skeleton coordinates for training to generate the first two-dimensional training data, and the rotation angle r ｚ is set within a first range from a predetermined first angle of -90° or more and 0° or less to a predetermined second angle of 0° or more and 90° or less (for example, the angle from the first angle to the second angle is set within a range of 60°), and a rotation angle r about the z axis when generating second training two-dimensional data by projectively transforming the training three-dimensional skeleton coordinates. ｚIt is desirable to set the angle within a second range from a predetermined third angle that is equal to or greater than -90° and equal to or less than 0° to a predetermined fourth angle that is equal to or greater than 0° and equal to or less than 90° (for example, the angle from the third angle to the fourth angle is set within a range of 60°).

[0034] However, in general (except in a zero-gravity environment such as outer space), people take a posture according to gravity. ｚ It is desirable to limit the rotation angle to a narrow range centered on 0°, ideally 0° from the vertical direction, and more desirably, within a narrow range such as −2° to +2° from the vertical direction. ｚ = 0°, but the rotation angle r ｚ may vary.

[0035] In the process of FIG. 8, the rotation angle r around the x-axis ｘ 9A and 9B are diagrams showing the state in which the training 2D skeleton coordinates generated from the training 3D skeleton coordinates are rotated around the x-axis. As shown in FIG. 9A, r ｘ At r = 0°, the training 2D skeleton coordinates are in an upright posture, but ｘ At r = 90°, the training 2D skeleton coordinates are in a lying position. ｚ At r = 0°, the training 2D skeleton coordinates are in a supine position, but ｘ = 90°, the training 2D skeleton coordinates are in an upright position. ｘ In other words, the two-dimensional projection transformation unit 12 limits the rotation angle r about the x-axis when generating the first two-dimensional training data by projectively transforming the three-dimensional training skeleton coordinates. ｘ is set to a third range that is larger than a predetermined fifth angle within the range of 0° to 90° and smaller than a predetermined sixth angle within the range of 0° to 90° (for example, the angle from the fifth angle to the sixth angle is set to a range of 60°), and the rotation angle r about the x-axis when generating second training two-dimensional data by projectively transforming the training three-dimensional skeleton coordinates is set to a third range that is larger than a predetermined fifth angle within the range of 0° to 90° and smaller than a predetermined sixth angle within the range of 0° to 90°.ｘ is preferably set to within a fourth range greater than a seventh predetermined angle in the range of 0° to 90° and less than an eighth predetermined angle in the range of 0° to 90° (e.g., the angle from the sixth angle to the seventh angle is set to within a range of 60°).

[0036] However, from the viewpoint of practical convenience, the rotation angle r ｘ It is desirable to make r broad. ｘ If the situation at the time of inference, i.e., the rotation angle around the x-axis at the time of inference, specifically the depression angle, which is the angle at which the camera (camera 50 in the second embodiment) looks down, differs significantly, the accuracy of the inference will decrease.

[0037] On the other hand, for surveillance cameras that monitor care recipients and surveillance cameras for surveillance purposes, the ideal installation angle for the camera varies depending on the shape of the room or the position of obstructions, and it is difficult to determine it solely based on the convenience of the fall detector as a behavior detection device. For example, in an installation situation where the ceiling is low and the depth is long, it is desirable to set the look-down angle relatively shallow (small rotation angle around the x-axis), and in an installation situation where the ceiling is high and the depth is short, it is desirable to set the look-down angle relatively deep (large rotation angle around the x-axis). In order to perform behavior detection robustly under these various conditions, the rotation angle r ｘ It is desirable to widen the scope of

[0038] To address this issue, multiple model generation units (two in FIG. 8) are prepared, and each is set to a rotation angle r around the x-axis. ｘ In FIG. 8, the range of the rotation angle r ｘ１ The angle r for the second training two-dimensional skeleton coordinates is set to three values ​​of 20°, 30°, and 40°, and is input to the second model generation unit 14. ｘ２ The rotation angle r is set to 40°, 50°, and 60°. ｘ When the range is about 30°, the phenomenon of confusion between upright and lying position mentioned above does not occur, and rather the generalization characteristics resulting from the variation in data have the effect of increasing the detection rate.

[0039] This process is performed on a large number of training three-dimensional skeletons to construct first and second training models.

[0040] Furthermore, the two-dimensional projection transformation unit 12 may generate the first two-dimensional training data D21 and the second two-dimensional training data D22 from the three-dimensional training skeletal coordinates obtained by adding a random number within a predetermined range to the three-dimensional training skeletal coordinates D11. In this case, training data for the skeletons of multiple people with different physiques can be obtained (i.e., the variation in physiques can be increased).

[0041] The two-dimensional projection transformation unit 12 may also have a function (i.e., a bone extension function) for generating first and second training two-dimensional data from three-dimensional skeletal coordinates obtained by multiplying the difference vector between the first and second skeletal points determined by the training three-dimensional skeletal coordinates by a scalar value. Bone extension may also be performed to generate first and second training two-dimensional data D21 and D22 from the training three-dimensional skeletal coordinates obtained by extending or shortening the difference vector between skeletal points determined by the training three-dimensional skeletal coordinates D11. In this case, different training two-dimensional data can be obtained with only the cost of calculation processing. FIG. 11 is an explanatory diagram showing an example of bone extension of the training three-dimensional skeletal coordinates D11. This is an example in which, instead of retaking data for a person with relatively long legs, the bones from both hips to both knees and the bones from both knees to both ankles of the training three-dimensional skeleton are each multiplied by 1.1. By calculating the coordinates of both knees and both ankles from both hips, training data for a person with long legs can be obtained without any problems.

[0042] 12 is an explanatory diagram showing the interpolation process of the training three-dimensional skeletal coordinates D11. The two-dimensional projection transformation unit 12 may generate training three-dimensional skeletal coordinates interpolated from multiple training three-dimensional skeletal coordinates at multiple times arranged in a chronological order, thereby increasing the number of training three-dimensional skeletal coordinates.

[0043] The upper part of Figure 12 shows the original data (constant speed) of the learning 3D skeletal coordinates D11. This data is based on learning 3D skeletal coordinates at three times in chronological order, and the lower part of Figure 12 shows an example of converting this data into 2 / 3 speed, i.e., slower, behavioral data. The data for the first time remains unchanged, but the learning 3D skeletal coordinates for the next time at 2 / 3 speed are obtained by interpolation between the learning 3D skeletal coordinates for the first time at constant speed and the learning 3D skeletal coordinates for the second time. Similarly, the learning 3D skeletal coordinates for the second time at 2 / 3 speed are obtained by interpolation between the learning 3D skeletal coordinates for the second time at constant speed and the learning 3D skeletal coordinates for the third time. The learning 3D skeletal coordinates for the last time at 2 / 3 speed are the same as the learning 3D skeletal data for the last time at constant speed.

[0044] In this way, by using interpolation or thinning, three-dimensional skeletal data for learning at different speeds can be obtained by calculation processing on a computer.

[0045] In addition, in behavior detection using skeletal data, skeletal data from multiple time points may be input. In such cases, data showing the same behavior performed quickly and slowly can be obtained from a single set of 3D training skeletal data.

[0046] 12 is a diagram illustrating an example of the hardware configuration of the learning device 10 according to embodiment 1. The learning device 10 includes a processor 101 such as a CPU (Central Processing Unit), a memory 102 such as a RAM (Random Access Memory), a storage device 103 such as a non-volatile storage device such as a HDD (Hard Disk Drive) or SSD (Solid State Drive), and an interface 104. These components may be configured using dedicated processing circuits.

[0047] The processor 101 can execute the learning program according to the first embodiment. The learning program is provided by being recorded on a recording medium such as an SD (Secure Digital) memory card or a USB (Universal Serial Bus) memory card, or by being downloaded via a network. The hardware configuration shown in FIG. 12 is an example, and various modifications to the hardware configuration are possible.

[0048] As described above, by using the learning device, learning method, and learning program according to the first embodiment, it is possible to generate a plurality of learning models for inferring behavior at low cost.

[0049] 14 is a functional block diagram showing a schematic configuration of a behavior detection device 40 as an inference device according to embodiment 2. The behavior detection device 40 is a device capable of implementing a behavior detection method as an inference method according to embodiment 2. The behavior detection device 40 is, for example, a computer capable of executing a behavior detection program according to embodiment 2. The behavior detection device 40 may be a computer (i.e., a computer system) configured by cloud computing using a computer network.

[0050] The behavior detection device 40 is a device that detects behaviors performed by a target included in a two-dimensional image acquired by a two-dimensional image acquisition unit 41. The target is described in the first embodiment and is, for example, a person. The two-dimensional image acquisition unit 41 includes, for example, a camera 50. The behavior detection device 40 includes a two-dimensional skeletal coordinate calculation unit 42 as a two-dimensional image-derived two-dimensional skeletal coordinate calculation unit that calculates two-dimensional skeletal coordinates for inference D42 (hereinafter also referred to as "two-dimensional skeletal coordinates for inference D42") indicating the coordinates of the target's skeleton from the two-dimensional image, and a behavior detection unit 44 that selects one or more learning models from multiple learning models (e.g., a first learning model M1 and a second learning model M2) and detects behaviors from the two-dimensional skeletal coordinates for inference D42 using the selected one or more learning models. The multiple learning models are generated by the learning device 10 according to the first embodiment. The behavior detection device 40 may use three or more learning models. Furthermore, the storage devices 15 and 16 in which the learning models are stored may be part of the behavior detection device 40. Furthermore, the entirety or part of the two-dimensional image acquisition unit 41 may be part of the behavior detection device 40 .

[0051] The behavior detection device 40 has a camera information acquisition unit 43 as a two-dimensional image acquisition unit information acquisition unit that acquires or calculates the depression angle of the optical axis when a two-dimensional image is captured by the two-dimensional image acquisition unit 41. The behavior detection unit 44 selects one or more learning models from the plurality of learning models based on the depression angle, and uses the selected one or more learning models to detect the behavior.

[0052] The camera information acquisition unit 43 can acquire the depression angle of the camera 50 from a storage device that stores the depression angle of the camera 50 (i.e., the depression angle of the shooting direction by the two-dimensional image acquisition unit 41). The camera information acquisition unit 43 may also acquire (calculate) the depression angle of the camera 50 based on the output of an acceleration sensor fixed to the camera 50 in the direction of gravitational acceleration.

[0053] 15(A), (B), and (C) are conceptual diagrams showing two-dimensional images of a person captured by a camera 50 with different depression angles and installation positions. Fig. 15(A) shows an example where the depression angle is small, Fig. 15(C) shows an example where the depression angle is large, and Fig. 15(B) shows an example where the depression angle is an intermediate value between Fig. 15(A) and Fig. 15(C).

[0054] Figure 16 is a conceptual diagram showing a two-dimensional image D41 obtained by camera photography when the camera 50 has the depression angle of Figure 15 (A), two-dimensional skeletal coordinates for inference D42 generated from the two-dimensional image D41, behavior detection according to the first learning model M1, and behavior (or behavior likelihood) D44 as the inference result.

[0055] Figure 17 is a conceptual diagram showing a two-dimensional image D41 obtained by camera photography when the camera 50 has the depression angle of Figure 15 (B), two-dimensional skeletal coordinates for inference D42 generated from the two-dimensional image D41, behavior detection according to the second learning model M2, and behavior (or behavior likelihood) D44 as the inference result.

[0056] Figure 18 is a conceptual diagram showing a two-dimensional image D41 obtained by camera photography when the camera 50 has the depression angle of Figure 15 (C), two-dimensional skeletal coordinates for inference D42 generated from the two-dimensional image D41, behavior detection according to the third learning model M3, and behavior (or behavior likelihood) D44 as the inference result.

[0057] It is desirable that the behavior detection unit 44 selects, from among a plurality of learning models (e.g., a first learning model, a second learning model, and a third learning model), a learning model whose rotation angle around the x-axis is closer to the depression angle of the camera 50. In other words, the behavior detection unit 44 selects the first learning model when the rotation angle around the x-axis of the first camera parameter is closer to the depression angle than the rotation angle around the x-axis of the second camera parameter, and selects the second learning model when the rotation angle around the x-axis of the second camera parameter is closer to the depression angle than the rotation angle around the x-axis of the first camera parameter.

[0058] Furthermore, the behavior detection device 40 may have the functions of the learning device 10 according to embodiment 1. In this case, the behavior detection device 40 further includes means for acquiring three-dimensional skeletal coordinates (not shown) (such as a stereo camera, a TOF camera, LiDAR, or a deep learning device that estimates three-dimensional skeletal coordinates from two-dimensional images) and a device for acquiring correct answers for behaviors linked to the three-dimensional skeletal coordinates (for example, a device that accepts manual input by an operator), and the learning device 10 updates multiple learning models (for example, a first learning model, a second learning model, and a third learning model).

[0059] In the behavior detection device 40 serving as an inference device, first, a two-dimensional image acquisition unit 41 acquires a two-dimensional image D41 generated by camera photography, and a two-dimensional skeletal coordinate calculation unit 42 calculates two-dimensional skeletal coordinates for inference D42 from the two-dimensional image D41. Known techniques that can be used by the two-dimensional skeletal coordinate calculation unit 42 include, for example, OpenPose, MoveNet, PoseNet, BlazePose, and OpenVino.

[0060] The two-dimensional skeleton coordinates for inference obtained in this way are used as the rotation angle r around the x-axis when the depression angle is used to construct the learning model. ｘ The model is input to a learning model that is closest to the target position (i.e., a selected learning model), and the behavior (or behavior likelihood) D44 is output. Examples of behavior likelihoods are standing, walking, falling, lying, etc. The two-dimensional skeletal coordinates for inference input to the learning model are those for a single time, but to improve accuracy, two-dimensional skeletal coordinates for inference for multiple times may be input to the learning model.

[0061] At this time, the behavior detection unit 44 selects one learning model from multiple learning models based on the depression angle of the camera 50. There are several specific methods for the camera information acquisition unit 43. The simplest method is to manually input the depression angle when installing the camera. Another method is to use an acceleration sensor fixed to the camera to acquire the direction of gravitational acceleration and calculate the depression angle. This is effective as a means of solving the errors and complexity of manual input. Another method is to estimate the depression angle from the image features of the acquired two-dimensional image, specifically, the inclination of a horizontal line on the image. This is effective in reducing costs because it does not require additional equipment such as an acceleration sensor.

[0062] r when constructing the first learning model and the second learning model ｘ Since the boundary was 40°, it is desirable to use different models depending on whether the depression angle is above or below 40°.

[0063] When using the behavior detection device 40 according to embodiment 2, the amount of data acquired remains constant even if the number of two-dimensional image acquisition units 41 is increased, and all two-dimensional image acquisition units 41 can be configured simply by performing projection calculations of two-dimensional skeletal coordinates for inference from three-dimensional skeletal coordinates for inference on a computer.

[0064] In the above explanation, an example was shown in which one learning model was selected, but the two-dimensional skeletal coordinates for inference may be input into multiple learning models, and the multiple resulting behavioral likelihoods may be used (for example, by weighted addition of the multiple behavioral likelihoods) to obtain the final behavioral likelihood.

[0065] The learning model may also be updated during inference. Specifically, even if the inference output indicates a fall, if the user cancels the alarm because they believe the fall is a false alarm, the learning model can be updated by labeling the input 2D skeleton coordinates for inference as an action other than a fall, thereby preventing false alarms in similar situations.

[0066] 19 is a flowchart schematically illustrating the operation of the behavior detection device 40 according to embodiment 2. First, the behavior detection device 40 acquires a two-dimensional image D41 generated by capturing an image of a target with the camera 50 (step S21).

[0067] Next, the behavior detection device 40 calculates two-dimensional skeletal coordinates for inference D42 indicating the coordinates of the target's skeleton from the two-dimensional image D41 (step S22).

[0068] Next, the behavior detection device 40 acquires (or calculates) the depression angle of the camera 50 (step S23). If the depression angle is equal to or less than a predetermined threshold (YES in step S24), the behavior detection device 40 performs inference using the first learning model M1 (step S25). If the depression angle is greater than the predetermined threshold (NO in step S24), the behavior detection device 40 performs inference using the second learning model M2 (step S26).

[0069] The behavior detection device 40 outputs the result of the inference (behavior or behavior likelihood) (step S27).

[0070] 20 is a diagram illustrating an example of the hardware configuration of the behavior detection device 40 according to embodiment 2. The behavior detection device 40 has a processor 201 such as a CPU, a memory 202 such as a RAM, a storage device 203 such as a non-volatile storage device such as an HDD or SSD, and an interface 204. These components may be configured using dedicated processing circuits.

[0071] The processor 201 can execute the behavior detection program according to the second embodiment. The behavior detection program is provided by being recorded on a recording medium such as an SD memory card or a USB memory card, or by being downloaded via a network. The hardware configuration shown in Fig. 20 is an example, and various modifications to the hardware configuration are possible.

[0072] As described above, by using the behavior detection device 40, the behavior detection method, and the behavior detection program according to the second embodiment, it is possible to accurately detect behavior using a learning model selected from a plurality of learning models.

[0073] REFERENCE SIGNS LIST 10 Learning device, 11 Three-dimensional data acquisition unit, 12 Two-dimensional projection transformation unit, 13 First model generation unit, 14 Second model generation unit, 20 Virtual camera, 22a First two-dimensional surface, 22b Second two-dimensional surface, 40 Behavior detection device (inference device), 41 Two-dimensional image acquisition unit, 42 Two-dimensional skeletal coordinate calculation unit (two-dimensional image-derived two-dimensional skeletal coordinate calculation unit), 43 Camera information acquisition unit (two-dimensional image acquisition unit information acquisition unit), 44 Behavior detection unit, 50 Camera, D10 Three-dimensional data for learning, D11 Three-dimensional skeletal coordinates for learning, D12 Teacher data, D21 First two-dimensional data for learning, D22 Second two-dimensional data for learning, D41 Two-dimensional image, D42 Two-dimensional skeletal coordinates for inference, P1 First camera parameters, P2 Second camera parameters, M1: first learning model; M2: second learning model; M3: third learning model.

Claims

1. A learning device that receives three-dimensional learning data including three-dimensional learning skeletal coordinates including three-dimensional coordinates of a target skeleton and teacher data consisting of correct answers for actions linked to the skeleton, and generates a learning model, comprising: a two-dimensional projection transformation unit that projects and transforms the three-dimensional learning skeletal coordinates onto a first two-dimensional plane according to predetermined first camera parameters to generate first two-dimensional learning data including first two-dimensional learning skeletal coordinates and the teacher data, and projects and transforms the three-dimensional learning skeletal coordinates onto a second two-dimensional plane according to predetermined second camera parameters to generate second two-dimensional learning data including second two-dimensional learning skeletal coordinates and the teacher data; a first model generation unit that uses the first two-dimensional learning data to generate a first learning model for inferring actions linked to the first two-dimensional inference skeletal coordinates from the first two-dimensional inference skeletal coordinates; a second model generation unit that uses the second learning two-dimensional data to generate a second learning model for inferring behavior associated with the second inference two-dimensional skeletal coordinates from the second inference two-dimensional skeletal coordinates.

2. The learning device described in claim 1, characterized in that each of the first camera parameter and the second camera parameter includes rotation angles around three axes consisting of the x-axis, the y-axis, and the z-axis, where the vertical coordinate axis is the y-axis, the coordinate axis on the line obtained by projecting a straight line in the projection direction used when generating the first learning two-dimensional skeleton coordinate and the second learning two-dimensional skeleton coordinate onto a horizontal plane that is perpendicular to the y-axis is the z-axis, and the coordinate axis perpendicular to both the y-axis and the z-axis is the x-axis.

3. The learning device described in claim 1 or 2, characterized in that the first camera parameters include coordinates in the world coordinate system of a first virtual camera and a focal length of the first virtual camera that are set when the first learning two-dimensional skeleton coordinates are generated using a perspective projection method, and the second camera parameters include coordinates in the world coordinate system of a second virtual camera and a focal length of the second virtual camera that are set when the second learning two-dimensional skeleton coordinates are generated using a perspective projection method.

4. The learning device according to claim 2, characterized in that arbitrary values ​​are set as the rotation angle around the y-axis according to the first camera parameter and the rotation angle around the y-axis according to the second camera parameter.

5. The learning device described in claim 2 or 4, characterized in that the two-dimensional projection transformation unit sets the rotation angle about the z-axis when projecting and transforming the training three-dimensional skeleton coordinates to generate the first training two-dimensional data within a first range from a predetermined first angle that is greater than or equal to -90° and less than 0° to a predetermined second angle that is greater than or equal to 0° and less than 90°, and sets the rotation angle about the z-axis when projecting and transforming the training three-dimensional skeleton coordinates to generate the second training two-dimensional data within a second range from a predetermined third angle that is greater than or equal to -90° and less than 0° to a predetermined fourth angle that is greater than or equal to 0° and less than 90°.

6. The learning device described in any one of claims 2, 4 and 5, characterized in that the two-dimensional projection transformation unit sets the rotation angle about the x-axis when projecting and transforming the training three-dimensional skeleton coordinates to generate the first training two-dimensional data within a third range that is greater than a predetermined fifth angle within the range of 0° to 90° and less than a predetermined sixth angle within the range of 0° to 90°, and sets the rotation angle about the x-axis when projecting and transforming the training three-dimensional skeleton coordinates to generate the second training two-dimensional data within a fourth range that is greater than a predetermined seventh angle within the range of 0° to 90° and less than a predetermined eighth angle within the range of 0° to 90°.

7. A learning device as described in any one of claims 1 to 6, characterized in that the two-dimensional projection transformation unit generates the first learning two-dimensional data and the second learning two-dimensional data from three-dimensional skeleton coordinates obtained by adding a random number within a predetermined range to the learning three-dimensional skeleton coordinates.

8. A learning device as described in any one of claims 1 to 7, characterized in that the two-dimensional projection transformation unit generates the first learning two-dimensional data and the second learning two-dimensional data from three-dimensional skeleton coordinates obtained by multiplying by a scalar value the difference vector between the first skeleton point and the second skeleton point determined by the learning three-dimensional skeleton coordinates.

9. A learning device as described in any one of claims 1 to 8, characterized in that the two-dimensional projection transformation unit generates training three-dimensional skeletal coordinates interpolated from a plurality of training three-dimensional skeletal coordinates at a plurality of times arranged in a chronological order, thereby increasing the number of training three-dimensional skeletal coordinates.

10. The learning device according to any one of claims 1 to 9, wherein the actions include one or more of standing upright, lying down, falling, and moving.

11. A learning device according to any one of claims 2 to 6, characterized in that the learning three-dimensional skeletal coordinates are obtained by motion capture or a stereo camera.

12. A learning method implemented by a learning device, comprising the steps of: receiving three-dimensional learning data including three-dimensional learning skeletal coordinates including three-dimensional coordinates of a skeleton of a target and teacher data consisting of correct answers for actions associated with the skeleton; projecting and transforming the three-dimensional learning skeletal coordinates onto a first two-dimensional plane in accordance with predetermined first camera parameters to generate first two-dimensional learning data including first two-dimensional learning skeletal coordinates and the teacher data, and projecting and transforming the three-dimensional learning skeletal coordinates onto a second two-dimensional plane in accordance with predetermined second camera parameters to generate second two-dimensional learning data including second two-dimensional learning skeletal coordinates and the teacher data; and using the first two-dimensional learning data to generate a first learning model for inferring an action associated with the first two-dimensional inference skeletal coordinates from the first two-dimensional inference skeletal coordinates. A learning method comprising: a step of generating a second learning model for inferring behavior associated with the second two-dimensional skeletal coordinates for inference from the second two-dimensional skeletal coordinates for inference using the second two-dimensional learning data.

13. A step of receiving three-dimensional learning data including three-dimensional learning skeletal coordinates including three-dimensional coordinates of a skeleton of a target and teacher data consisting of correct answers of actions associated with the skeleton; a step of projecting and transforming the three-dimensional learning skeletal coordinates onto a first two-dimensional plane according to a first predetermined camera parameter to generate first two-dimensional learning data including first two-dimensional learning skeletal coordinates and the teacher data, and projecting and transforming the three-dimensional learning skeletal coordinates onto a second two-dimensional plane according to a second predetermined camera parameter to generate second two-dimensional learning data including second two-dimensional learning skeletal coordinates and the teacher data; a step of generating a first learning model using the first two-dimensional learning data for inferring an action associated with the first two-dimensional inference skeletal coordinates from the first two-dimensional inference skeletal coordinates; and a step of generating a second learning model using the second two-dimensional learning data for inferring an action associated with the second two-dimensional inference skeletal coordinates from the second two-dimensional inference skeletal coordinates. A learning program characterized by causing a computer to execute the above.

14. A behavior detection device that detects behavior performed by an object included in a two-dimensional image acquired by a two-dimensional image acquisition unit, comprising: a two-dimensional skeletal coordinate calculation unit that calculates two-dimensional skeletal coordinates for two-dimensional image-derived inference indicating the coordinates of the object's skeleton from the two-dimensional image; and a behavior detection unit that selects one or more learning models from the first learning model and the second learning model generated by a learning device described in any one of claims 1 to 11, and detects the behavior from the two-dimensional skeletal coordinates for two-dimensional image-derived inference using the selected one or more learning models.

15. The behavior detection device described in claim 14, further comprising a two-dimensional image acquisition unit information acquisition unit that acquires or calculates the depression angle of the optical axis when the two-dimensional image is captured by the two-dimensional image acquisition unit, and the behavior detection unit selects the one or more learning models based on the depression angle.

16. The behavior detection device according to claim 15, characterized in that the two-dimensional image acquisition unit information acquisition unit acquires the depression angle of the two-dimensional image acquisition unit from a storage device that stores the depression angle of the optical axis when the two-dimensional image was captured.

17. The behavior detection device according to claim 15, characterized in that the two-dimensional image acquisition unit information acquisition unit acquires the depression angle based on the output of an acceleration sensor fixed to the two-dimensional image acquisition unit.

18. A behavior detection device that detects behavior performed by a target included in a two-dimensional image acquired by a two-dimensional image acquisition unit, comprising: a two-dimensional image-derived two-dimensional skeletal coordinate calculation unit that calculates two-dimensional skeletal coordinates for two-dimensional image-derived inference indicating skeletal coordinates of the target from the two-dimensional image; a behavior detection unit that selects one or more learning models from the first learning model and the second learning model generated by the learning device described in any one of claims 2, 4, 5, and 6, and detects the behavior from the two-dimensional skeletal coordinates for inference using the selected one or more learning models; and a two-dimensional image acquisition unit information acquisition unit that acquires or calculates a depression angle of the optical axis when the two-dimensional image was captured by the two-dimensional image acquisition unit, wherein the behavior detection unit selects the first learning model when the rotation angle around the x-axis of the first camera parameter is closer than the rotation angle around the x-axis of the second camera parameter, and selects the second learning model when the rotation angle around the x-axis of the second camera parameter is closer than the rotation angle around the x-axis of the first camera parameter.

19. The behavior detection device described in any one of claims 14 to 18, characterized in that the learning device updates the first learning model using the two-dimensional skeletal coordinates for inference from two-dimensional images as the first two-dimensional skeletal coordinates for learning, and updates the second learning model using the two-dimensional skeletal coordinates for inference from two-dimensional images as the second two-dimensional skeletal coordinates for learning.

20. A behavior detection method implemented by a behavior detection device that detects behavior performed by an object included in a two-dimensional image acquired by a two-dimensional image acquisition unit, comprising the steps of: calculating two-dimensional skeletal coordinates for inference indicating the coordinates of the object's skeleton from the two-dimensional image; and selecting one or more learning models from the first learning model and the second learning model generated by the learning method described in claim 12, and detecting the behavior from the two-dimensional skeletal coordinates for inference using the selected one or more learning models.

21. A behavior detection program that causes a computer that detects behavior performed by an object included in a two-dimensional image acquired by a two-dimensional image acquisition unit to execute the steps of: calculating two-dimensional skeletal coordinates for inference indicating the coordinates of the object's skeleton from the two-dimensional image; and selecting one or more learning models from the first learning model and the second learning model generated by the learning program described in claim 13, and detecting the behavior from the two-dimensional skeletal coordinates for inference using the one or more selected learning models.

Images