Human movement separation device, human movement separation method and program

Abstract

To provide a person's movement separation device and the like which can accurately separate the movement by the intention of a person from the movement of the person in a movable body.SOLUTION: A person's movement separation device 1 comprises: an image acquisition unit 21 which acquires an image in which a person is photographed by imaging the person existing in a movable body 3 by an imaging unit 10; a person's movement detection unit 25 which detects the person's movement from the image; a posture detection unit 22 which detects the posture of the person; an environment shake acquisition unit 24 which acquires environment shake information being information indicating a shake due to an environment in the horizontal direction and the vertical direction to the person; a person shake estimation unit 23 which estimates a movement amount indicating the shake of the person due to the environment on the basis of the posture of the person detected by the posture detection unit 22 and the environment shake information acquired the environment shake acquisition unit 24; and a person shake separation unit 26 which separates a movement amount indicating the shake of the person due to the environment estimated by the person shake estimation unit 23 from the person's movement detected by the person's movement detection unit 25 and calculates the person's movement from which the separated movement amount indicating the shake of the person due to the environment is removed.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a human motion separation device, a human motion separation method, and a program. [Background technology]

[0002] Patent Document 1 discloses a conventional image display device having an image memory that receives and temporarily stores image data to be displayed and a display unit that displays the image data. The image display device is provided with an imaging unit that captures an image of an object that is in a fixed position relative to the observer, and a movement vector calculation unit that calculates a movement vector that indicates the amount and direction of relative movement between the display unit and the observer based on the captured image data, and reduces or eliminates the relative movement between the display unit and the observer based on the movement vector.

[0003] Patent Document 2 discloses a conventional camera vector calculation device. The camera vector calculation device automatically extracts feature points from multiple frame images, calculates camera vectors such as highly accurate camera position and camera rotation angle from the multiple frame images, and corrects irregular blurring caused by camera shake in images acquired during video shooting based on the calculated camera vectors, thereby generating a blur-free image from a blurred image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-343622 [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-295495 Summary of the Invention [Problem to be solved by the invention]

[0005] When performing condition detection processing using human movement from captured images in a moving vehicle, the accuracy of condition detection deteriorates if the person sways due to the swaying of the moving vehicle. However, when the image display device of Patent Document 1 is applied to a moving moving vehicle to capture an image of a person riding on it, for example, a fixed object must be present inside the moving vehicle near the person. Furthermore, the image display device of Patent Document 1 has the problem of being unable to determine whether the person swayed due to the swaying of the moving vehicle or whether the person moved of their own volition.

[0006] Furthermore, when the camera vector calculation device of Patent Document 2 is applied to a moving vehicle to capture an image of a person riding on the vehicle, it is unable to accurately correct shaking if a moving object is a feature point. Also, the camera vector calculation device of Patent Document 2 has the problem of being unable to determine whether a person is shaking due to the shaking of the moving vehicle or whether the person is moving of their own volition.

[0007] Therefore, an object of the present disclosure is to provide a human movement separation device and the like that can accurately separate human movement from human intentional movement within a moving body. [Means for solving the problem]

[0008] A human movement separation device according to one aspect of the present disclosure includes an image acquisition unit that acquires an image of a person by capturing an image of a person present on the moving body using an imaging unit fixed to the moving body, a human movement detection unit that detects human movement from the image, a posture detection unit that detects the posture of the person, an environmental fluctuation acquisition unit that acquires environmental fluctuation information that is information that indicates fluctuations caused by the environment in the horizontal and vertical directions relative to the person, a human movement estimation unit that estimates a movement amount that indicates the human movement caused by the environment based on the posture of the person detected by the posture detection unit and the environmental fluctuation information acquired by the environmental fluctuation acquisition unit, and a human movement separation unit that separates the movement amount that indicates the human movement caused by the environment estimated by the human movement estimation unit from the human movement detected by the human movement detection unit, and calculates the human movement with the separated movement amount that indicates the human movement caused by the environment removed.

[0009] Furthermore, a method for separating human movement according to one aspect of the present disclosure includes obtaining an image of a person by capturing an image of a person present on the moving body using an imaging unit fixed to the moving body, detecting the movement of the person from the image, detecting the posture of the person, obtaining environmental fluctuation information that is information indicating environmental fluctuations in the horizontal and vertical directions relative to the person, estimating a movement amount indicating the environmental fluctuation of the person based on the detected posture of the person and the obtained environmental fluctuation information, separating the estimated movement amount indicating the environmental fluctuation of the person from the detected human movement, and calculating the human movement from which the separated movement amount indicating the environmental fluctuation of the person has been removed.

[0010] A program according to one aspect of the present disclosure is a program for causing a computer to execute a human movement segmentation method.

[0011] Some of these specific aspects may be realized using a system, a method, an integrated circuit, a computer program, or a computer-readable storage medium such as a CD-ROM, or may be realized using any combination of a system, a method, an integrated circuit, a computer program, and a storage medium. [Effects of the Invention]

[0012] According to the human movement separation device and the like of the present disclosure, it is possible to accurately separate movements due to a person's intention from the movements of a person within a moving body. [Brief explanation of the drawings]

[0013] [Figure 1A] FIG. 1A(a) is a side view showing a person riding on a moving object and the moving object, and FIG. 1A(b) is a top view showing the person riding on the moving object and the moving object. [Figure 1B] FIG. 1B(a) is a diagram showing the relationship between the acceleration in the x direction of the moving body and time, and FIG. 1B(b) is a diagram showing the relationship between the acceleration in the y direction of the moving body and time. [Figure 1C](a) to (c) of FIG. 1C are diagrams showing captured images of a person moving when a moving object shakes. [Figure 1D] FIG. 1D(a) is a diagram showing the amount of movement of a person's head in the x direction, and FIG. 1D(b) is a diagram showing the amount of movement of a person's head in the y direction relative to the center of the person's head. [Figure 1E] FIG. 1E shows diagrams in which (a) to (c) illustrate changes in a person's posture, and (d) of FIG. 1E illustrates the angles of the changing posture of the person. [Figure 2] FIG. 2 is a diagram showing the shaking of a moving body and the shaking of a person riding on the moving body. [Figure 3] FIG. 3 is a diagram showing the relationship between the amount of movement of a moving object in the y direction and time. [Figure 4] Figure 4(a) is a diagram showing an example of the movement of a person riding on a moving object, Figure 4(b) is a diagram showing an example of the movement of a person when the horizontal swaying component of the moving object is small, Figure 4(c) is a diagram showing an example of the movement of a person when the moving object sways in both the horizontal and vertical directions, and Figure 4(d) is a diagram showing an example of the movement of a person when the moving object sways while the person is in a more tilted position than in (a). [Figure 5] FIG. 5 is a block diagram showing a human motion separation device according to an embodiment. [Figure 6] FIG. 6(a) is a diagram showing the amount of movement of a person's head in the x direction, and FIG. 6(b) is a diagram showing the amount of movement of a person's head in the y direction. [Figure 7] FIG. 7 is a block diagram showing a human movement estimation unit. [Figure 8A] Figure 8A shows a human movement separation device, where (a) in Figure 8A is a diagram showing the amount of movement of a person in the x direction, (b) in Figure 8A is a diagram showing the amount of movement in the x direction indicating the swaying of a person due to the environment when not resisting the swaying of the moving body 3, and (c) in Figure 8A is a diagram showing the movement of the person after separation. [Figure 8B]Figure 8B is an example of a case where a person moves, and in a human movement separation device, (a) of Figure 8B is a diagram showing the amount of movement of the person in the x direction, (b) of Figure 8B is a diagram showing the amount of movement in the x direction indicating the swaying of the person due to the environment when not resisting the swaying of the moving body 3, and (c) of Figure 8B is a diagram showing the movement of the person after separation. [Figure 9] FIG. 9 is a flowchart showing the operation of the human motion separation device. [Figure 10] FIG. 10 is a block diagram showing a human movement separating device according to the first modification of the embodiment. [Figure 11] FIG. 11 is a block diagram showing a human movement separating device according to a second modification of the embodiment. [Figure 12] FIG. 12 is a block diagram showing a human movement separating device according to a third modification of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] The embodiments and the like described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0015] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, the same components are assigned the same reference numerals. Furthermore, in the following embodiments, expressions such as "approximately match" are used. For example, "approximately match" does not only mean a perfect match, but also means a substantial match, that is, including an error of, for example, a few percent. Furthermore, "approximately match" means a match within the range in which the effects of the present disclosure can be achieved. The same applies to other expressions using "approximately."

[0016] The human movement separation device, human movement separation method, and program of the present disclosure will be specifically described below with reference to the drawings.

[0017] (Embodiment 1) <Summary> First, an outline of the human movement separation device, human movement separation method, and program according to the present embodiment will be described.

[0018] The human movement separation device acquires an image of a human subject, and can estimate the human's drowsiness and the subject's concentration level from the human's facial expression and the subject's movement amount. When a moving object carrying a person is moving, the moving object may sway and the person may sway along with the movement, or the person may brace against the movement of the moving object. Therefore, in order to estimate the human's drowsiness and the subject's concentration level, it is necessary to accurately estimate the human movement that is not caused by the movement of the moving object when the moving object sways.

[0019] Therefore, the relationship between the shaking of a moving object and the movement of a person will be explained using FIGS. 1A to 1E.

[0020] FIG. 1A (a) is a side view showing a person riding on a moving object and the moving object, and FIG. 1A (b) is a top view showing the person riding on the moving object and the moving object. FIG. 1B (a) is a diagram showing the relationship between acceleration in the x direction of the moving object and time, and FIG. 1B (b) is a diagram showing the relationship between acceleration in the y direction of the moving object and time. FIG. 1C (a) to (c) are diagrams showing images of a person moving when a moving object shakes. FIG. 1C (a) illustrates the center of the person's head in the x and y directions, but this is also true for other drawings. FIG. 1D (a) is a diagram showing the amount of movement of the person's head in the x direction (hereinafter referred to as the amount of movement of the person in the x direction), and FIG. 1D (b) is a diagram showing the amount of movement of the person in the y direction relative to the center of the person's head (hereinafter referred to as the amount of movement of the person in the y direction). The amount of movement in the x direction relative to the center of the head is represented by the amount of movement of pixels in the image. 1E, (a) to (c) are diagrams showing changes in a person's posture, and (d) is a diagram showing the angle of the changing posture of the person. In this embodiment, the amount of movement of a person's head is defined as the amount of movement in the x direction and the amount of movement in the y direction relative to the center of the person's head.

[0021] First, the shaking of a moving object and the movement of a person will be explained.

[0022] In FIG. 1A, an image acquisition unit fixed to the moving body captures an image of a person riding on the moving body. For example, when the moving body moves, it sways horizontally (left and right) and vertically (up and down). In FIG. 1B (a), for example, if the moving body sways to the right and a force is applied to the human motion separator in the left direction, the acceleration sensor of the human motion separator detects acceleration in the horizontal direction. Also, as shown in FIG. 1B (b), for example, if the moving body continues to move downward and a force is continuously applied to the human motion separator in the upward direction, the acceleration sensor detects acceleration in the vertical direction.

[0023] 1B(a), when the moving object sways horizontally, the person also moves horizontally. As shown in FIG. 1C(a) to (c), the image acquisition unit continues to capture images of the person moving.

[0024] When a moving object sways in the horizontal and vertical directions, the amount of movement of a person in the x and y directions can be seen to change based on the captured images, for example, as shown in (a) and (b) of Figure 1D. In other words, as shown in Figure 1D, as the moving object sways due to movement, it can be seen that the person's head also sways in the x and y directions over time. Furthermore, as shown in (a) to (d) of Figure 1E, by detecting a person's posture from their movement based on the captured images, the change in the angle of their posture can be seen.

[0025] Next, the swaying of a moving object and a person will be explained using Figure 2.

[0026] Fig. 2 is a diagram showing the shaking of a moving body and the shaking of a person riding on the moving body, and exemplifies the moving body, the person riding on the moving body, and an imaging unit fixed to the moving body.

[0027] As shown in Figure 2, when the moving object is in its standard position, the person is standing upright along the vertical dashed line. If the moving object suddenly swings vertically upward and tilts suddenly to the left, the person's position will tilt to the right with respect to the vertical dashed line. If the moving object suddenly swings vertically downward and tilts suddenly to the right, the person's position will tilt to the left with respect to the vertical dashed line.

[0028] Next, the shaking of the moving body when the moving body is traveling will be described with reference to FIG.

[0029] 3 is a diagram showing the relationship between the amount of movement in the y direction (vertical direction) of a moving object and time, in which the solid line represents the amount of movement in the y direction and the dashed line is an approximation curve.

[0030] The swaying of a moving body is irregular due to the influence of, for example, the road environment, the track environment, wind, etc. When the moving body is a train, the swaying of the train includes irregular components of swaying caused by the influence of curvature of the rails, curvature of the wheels, sand on the rails, etc., and regular components of swaying caused by the joints of the rails.

[0031] As a regular component of shaking, when a moving body suddenly sways left and right, a person's head moves horizontally, i.e., in the x direction, like a pendulum. At this time, a person's head also moves slightly vertically, i.e., in the y direction, like a pendulum. When a moving body suddenly sways left and right, a person braces themselves against the body's swaying to avoid falling over, but it is difficult to accurately determine from the movement of the person riding on the moving body alone whether the swaying of the person is due to the body's swaying or whether the person's movement is due to their own will to resist the swaying of the body.

[0032] Next, the movement of a person riding on a moving object will be explained with reference to FIG.

[0033] Fig. 4(a) is a diagram showing an example of the movement of a person riding on a moving object, Fig. 4(b) is a diagram showing an example of the movement of a person when the horizontal swaying component of the moving object is small, Fig. 4(c) is a diagram showing an example of the movement of a person when the moving object sways in both the horizontal and vertical directions, and Fig. 4(d) is a diagram showing an example of the movement of a person when the moving object sways while the person is in a more tilted position than in Fig. 4(a). Also, the swaying of the moving object in Fig. 4(c) is a different pattern from that in Fig. 4(a) and (b).

[0034] θ1 indicates the acute angle of the straight line V along the person's posture relative to the vertical direction (the angle of the person's posture), and m x indicates the horizontal shaking force of the moving object, and m y indicates the vertical shaking force of the moving body, and m x1 is m in the direction perpendicular to the line V x indicates the components of m y1 is m in the direction perpendicular to the line V y The components of m x1 , m y1 depends on the angle θ of the person's posture.

[0035] As shown in Figure 4(a), if a person is tilted to the left at an angle θ1 relative to the vertical direction when a moving object sways, the horizontal swaying force m xand the vertical vibration force of the moving body m y The resultant component in the direction perpendicular to the line V is m x1 +m y1 This becomes:

[0036] As shown in Figure 4(b), when the shaking caused by the moving object is small, even if a person leans to the left by an angle θ1 with respect to the vertical direction, the shaking force m x is smaller than in the case of Fig. 4(a), so the composite component m x1 +m y1 is smaller than the composite component in Figure 4(a).

[0037] As shown in Figure 4(c), when a moving object sways both horizontally and vertically, if a person intentionally tilts leftward by an angle θ1 relative to the vertical, the combined component is m x1 -m y1 =0, and the force applied to the person by the shaking of the moving object is canceled out.

[0038] As shown in (d) of Figure 4, when a person is tilted to the left due to movement, if the moving object sways and the person tilts to the left by an angle θ2 with respect to the vertical direction, the composite component m x1 +m y1 is larger than in the cases of (a), (b), and (c) in Figure 4.

[0039] As described above, human movement includes environmental sway caused by the sway of the moving body and voluntary movement. In this embodiment, the amount of head movement after the separation of the human movement caused by the human's own will, that is, the human movement caused by the human's own will, is calculated, with the environmental sway caused by the sway of the moving body removed.

[0040] <Configuration and Functions> The configurations and functions of the human movement separation device 1, the human movement separation method, and the program according to this embodiment will be described.

[0041] FIG. 5 is a block diagram showing a human movement separating device 1 according to an embodiment.

[0042] As shown in FIG. 5, the human movement separation device 1 includes an image acquisition unit 21, a human movement detection unit 25, a posture detection unit 22, an environmental fluctuation acquisition unit 24, a human movement estimation unit 23, and a human movement separation unit 26.

[0043] [Image Acquisition Unit 21] The image acquisition unit 21 acquires an image of a person by capturing an image of a person present on the moving object 3 using the imaging unit 10 fixed to the moving object 3. The image acquisition unit 21 can acquire an image by having the imaging unit 10 fixed to the moving object 3 continuously capture an image of a predetermined location. The image acquisition unit 21 may be mounted on the imaging unit 10. Here, the moving object 3 is a vehicle such as an automobile or train, an airplane, a ship, etc. Furthermore, the image includes at least a person's head, a person's torso, etc., as shown in FIG. 1C, for example. The image acquisition unit 21 outputs the acquired image to the human movement detection unit 25 and the posture detection unit 22.

[0044] [Human movement detection unit 25] The human movement detection unit 25 acquires images from the image acquisition unit 21 and detects human movement from the acquired images. For example, the human movement detection unit 25 extracts a person's head for each frame of the image acquired from the image acquisition unit 21 and calculates the amount of movement of the extracted person's head.

[0045] 6(a) is a diagram showing the amount of movement of a person's head in the x direction, and FIG. 6(b) is a diagram showing the amount of movement of a person's head in the y direction. The amount of movement of a person's head is calculated as the amount of change in pixel units in an image, as shown in FIGS. 6(a) and 6(b). As shown in FIG. 5, the human movement detection unit 25 outputs the calculated amount of movement of the person's head, which is the human movement, to the human movement separation unit 26.

[0046] [Posture detection unit 22] The posture detection unit 22 detects a person's posture from the image. The posture of the person is indicated by the angle at which the person tilts relative to the vertical direction over time. For example, the posture detection unit 22 extracts the person's head and torso from each frame of the image acquired from the image acquisition unit 21 and calculates the tilt of the extracted person, i.e., the angle of the person's posture. Specifically, as shown in (a) to (c) of FIG. 1E, the posture detection unit 22 calculates the angle θ (θ1, θ2, θ3, . . . ) of the posture direction along the head and torso relative to the vertical direction. (a) to (c) of FIG. 1E show how the posture of the person changes over time. As the person shakes in this way, the angle θ of the person's posture changes as shown in (d) of FIG. 1E. In this way, as shown in FIG. 5, the posture detection unit 22 detects the person's tilt, i.e., the person's posture, from the image. The posture detection unit 22 outputs the detected angle, which is the posture of the person, to the human shaking estimation unit 23.

[0047] [Environmental Shaking Acquisition Unit 24] The environmental fluctuation acquisition unit 24 acquires environmental fluctuation information, which is information indicating fluctuations due to the environment including horizontal and vertical directions relative to the person. Specifically, the environmental fluctuation acquisition unit 24 is disposed in the moving object 3 together with the image acquisition unit 21, and acquires the environmental fluctuation information, which is information indicating the fluctuation of the moving object 3 in the horizontal direction and the fluctuation of the moving object 3 in the vertical direction, in accordance with the fluctuation of the moving object 3. The environmental fluctuation acquisition unit 24 is, for example, an acceleration sensor, a vibration meter, etc. The environmental fluctuation acquisition unit 24 outputs the acquired environmental fluctuation information to the human fluctuation estimation unit 23.

[0048] [Personal Motion Estimation Unit 23] The human motion estimation unit 23 estimates a movement amount indicating the environmentally induced motion of the person based on the person's posture detected by the posture detection unit 22 and the environmental motion information acquired by the environmental motion acquisition unit 24. That is, the human motion estimation unit 23 estimates a movement amount indicating the environmentally induced motion of the person caused by the motion of the moving body 3 based on the person's posture and the horizontal and vertical motions of the moving body 3 indicated in the environmental motion information. Specifically, as shown in FIG. 4, the human motion estimation unit 23 estimates a movement amount indicating the environmentally induced motion of the person shown in FIG. 5 by calculating, over time, a composite component of a component obtained by converting the horizontal motion component of the moving body 3 based on the environmental motion information into a direction perpendicular to the posture direction of the person according to the angle, and a component obtained by converting the vertical motion component of the moving body 3 based on the environmental motion information into a direction perpendicular to the posture direction of the person according to the angle. The movement amount indicating the environmentally induced motion of the person is indicated by the environmentally induced motion amount in the x direction and the environmentally induced motion amount in the y direction. The human movement estimation unit 23 outputs the amount of movement indicating the human movement due to the environment to the human movement separation unit 26 .

[0049] Here, the processing of the human movement estimation unit 23 will be described in more detail with reference to FIG.

[0050] FIG. 7 is a block diagram showing the human movement estimation unit 23.

[0051] The human sway estimation unit 23 includes an angle acquisition unit 33, an x-direction acceleration acquisition unit 31, a y-direction acceleration acquisition unit 32, an estimation unit , and an output unit .

[0052] First, the angle acquisition unit 33 acquires an angle indicating the person's posture from the posture detection unit 22. The angle acquisition unit 33 outputs the acquired angle of the person's posture to the estimation unit 34. Note that if the posture detection unit 22 does not calculate the angle of the person's posture, the angle acquisition unit 33 may calculate the angle of the person's posture based on the person's posture.

[0053] Furthermore, the x-direction acceleration acquisition unit 31 acquires the acceleration of the moving body 3 in the x direction indicated in the environmental fluctuation information from the environmental fluctuation acquisition unit 24. For example, the acceleration of the moving body 3 in the x direction is shown in (a) of FIG. 1B. The x-direction acceleration acquisition unit 31 outputs the acquired acceleration of the moving body 3 in the x direction to the estimation unit 34.

[0054] Furthermore, the y-direction acceleration acquisition unit 32 acquires the acceleration of the moving body 3 in the y direction indicated in the environmental fluctuation information from the environmental fluctuation acquisition unit 24. For example, the acceleration of the moving body 3 in the y direction is shown in (b) of Figure 1. The y-direction acceleration acquisition unit 32 outputs the acquired acceleration of the moving body 3 in the y direction to the estimation unit 34.

[0055] The estimation unit 34 acquires the angle of the person's posture from the angle acquisition unit 33, acquires the acceleration of the moving body 3 in the x direction from the x direction acceleration acquisition unit 31, and acquires the acceleration of the moving body 3 in the y direction from the y direction acceleration acquisition unit 32. The estimation unit 34 estimates the amount of movement indicating the swaying of the person due to the environment using a pendulum damping formula based on the acquired angle of the person's posture, the acceleration of the moving body 3 in the x direction, and the acceleration of the moving body 3 in the y direction.

[0056] The damping formula for the pendulum is given by the following equations (1) to (3).

[0057]

number

[0058]

number

[0059]

number

[0060] ax is the observed acceleration in the horizontal component.

[0061] ay is the observed acceleration in the vertical component.

[0062] θ′ is the tilt angle of the person in the previous frame where acceleration in the horizontal or vertical component was observed.

[0063] θ(s) is the tilt angle of the person in the previous frame.

[0064] A1 and A2 are coefficients determined by calibration. Calibration is performed before the estimation process. A1 and A2 are calculated from the results (horizontal or vertical component) of detecting human movement when strong shaking occurs while the person is not moving of their own volition.

[0065] t is the time since the horizontal or vertical acceleration was observed. If the horizontal or vertical acceleration is observed, t is reset to 0.

[0066] As a result, the estimation unit 34 estimates the force exerted on the person by the swaying of the moving object 3 in consideration of the posture of the person, thereby estimating the amount of movement indicating the swaying of the person due to the current environment.

[0067] The output unit 35 outputs the amount of movement estimated by the estimation unit 34, which indicates the shaking of the person due to the environment, to the human shaking separation unit 26.

[0068] The human motion estimation unit 23 may store the amount of movement indicating the human motion due to the environment in the storage unit. In this case, the amount of movement indicating the human motion due to the environment may be stored in association with environmental motion information. The human motion estimation unit 23 may estimate the amount of movement indicating the human motion due to the environment based on the amount of movement indicating the human motion due to the past environment, the human posture, and the environmental motion information.

[0069] [Personal sway separation section 26] The human movement separation unit 26 separates the human movement due to the environment estimated by the human movement estimation unit 23 from the human movement detected by the human movement detection unit 25, and calculates the human movement with the separated human movement due to the environment removed. Specifically, the human movement separation unit 26 acquires the amount of movement of the person's head in Fig. 5, which is the human movement output by the human movement detection unit 25, and the amount of movement indicating the human movement due to the environment in Fig. 5, which is output by the human movement estimation unit 23. Then, the human movement separation unit 26 calculates the separated amount of head movement, which is the amount of head movement due to the person's own will in Fig. 5, obtained by removing the amount of movement indicating the human movement due to the environment from the amount of head movement.

[0070] Here, calculation of the amount of head movement after voluntary separation will be described with reference to FIGS. 8A and 8B.

[0071] Fig. 8A shows, in the human movement separation device 1, (a) of Fig. 8A is a diagram showing the amount of movement of a person in the x direction, (b) of Fig. 8A is a diagram showing the amount of movement in the x direction indicating the swaying of a person due to the environment when not resisting the swaying of a moving object 3, and (c) of Fig. 8A is a diagram showing the movement of the person after separation. Fig. 8B shows an example of a case where a person moves, in the human movement separation device 1, (a) of Fig. 8B is a diagram showing the amount of movement of a person in the x direction, (b) of Fig. 8B is a diagram showing the amount of movement in the x direction indicating the swaying of a person due to the environment when not resisting the swaying of a moving object 3, and (c) of Fig. 8B is a diagram showing the movement of the person after separation.

[0072] When the moving object 3 sways in the x direction, the actual amount of movement of the person in the x direction (amount of head movement) is as shown in FIG. 8A(a). At this time, because the person riding on the moving object 3 does not resist the swaying of the moving object 3, the amount of movement indicating the swaying of the person due to the environment (amount of head movement due to the environment) is as shown in FIG. 8A(b). The person sway separation unit 26 separates the swaying of the person due to the environment shown in FIG. 8A(b) from the amount of movement of the person in the x direction shown in FIG. 8A(a), and calculates the movement of the person with the separated swaying of the person due to the environment removed. In other words, the difference between the amount of movement shown in FIG. 8A(a) and the amount of movement shown in FIG. 8A(b) is as shown in FIG. 8A(c). In other words, because the person does not resist the swaying of the moving object 3, the amount of head movement after separation, which is the movement of the person, is 0, as shown in FIG. 8A(c).

[0073] Next, if a person moves when the moving object 3 sways in the x direction, the actual amount of movement of the person in the x direction (amount of head movement) will be as shown in FIG. 8B(a). At this time, the amount of movement (amount of head movement due to the environment) indicating the person's swaying due to the environment when not resisting the swaying of the moving object 3 will be as shown in FIG. 8B(b). The person sway separation unit 26 separates the swaying of the person due to the environment shown in FIG. 8B(b) from the amount of movement of the person in the x direction shown in FIG. 8B(a), and calculates the movement of the person with the separated swaying of the person due to the environment removed. In other words, due to the person's movement of their own volition, the difference between the amount of movement in FIG. 8B(a) and the amount of movement in FIG. 8B(b) will be as shown in FIG. 8B(c). In other words, because the person moves of their own volition, the amount of head movement after separation, which is the movement of the person, is calculated as shown in FIG. 8B(c).

[0074] The person movement separator 26 outputs the calculated amount of head movement after separation due to the person's own will to the state detector 27.

[0075] [Status detection unit 27] A state detection unit 27 is communicatively connected to the human movement separation device 1. When the state detection unit 27 acquires the amount of head movement after separation due to the person's will from the human movement separation unit 26, the state detection unit 27 detects the person's state, such as drowsiness and concentration, based on the amount of head movement after separation due to the person's will. The state detection unit 27 outputs the state detection result to an external device. Note that the state detection unit 27 may be included as a component of the human movement separation device 1.

[0076] <Processing operation> The processing operation of the human movement separation device 1 of this embodiment will be described.

[0077] FIG. 9 is a flowchart showing the operation of the human movement separating device 1.

[0078] 9, first, the image acquisition unit 21 sequentially acquires images of a person captured by the imaging unit 10 at a predetermined number of frames per second (S11). The image acquisition unit 21 outputs the acquired images to the human movement detection unit 25 and the posture detection unit 22.

[0079] Furthermore, the environmental fluctuation acquisition unit 24 acquires environmental fluctuation information, which is information indicating fluctuations caused by the environment in the horizontal and vertical directions relative to the person (S12). The environmental fluctuation acquisition unit 24 outputs the environmental fluctuation information to the human fluctuation estimation unit 23. The processing operations of steps S11 and S12 may be reversed or may be performed simultaneously.

[0080] Next, the human movement detection unit 25 detects the amount of movement of the person's head by calculating the amount of movement of the person in the x direction and the amount of movement of the person in the y direction from the acquired image (S13). The human movement detection unit 25 outputs the amount of movement of the person's head detected as human movement to the human movement separation unit 26.

[0081] Next, the posture detection unit 22 extracts the person's head and torso from the image acquired by the image acquisition unit 21, and calculates the angle of the posture of the extracted person. Specifically, as shown in (a) to (c) of FIG. 1E, the posture detection unit 22 calculates the tilt of the posture direction along the head and torso with respect to the vertical direction, that is, the angle θ of the posture direction with respect to the vertical direction. In other words, the posture detection unit 22 detects the posture of the person by calculating the angle of the posture of the person (S14). The posture detection unit 22 outputs the detected posture of the person to the human movement estimation unit 23.

[0082] Next, the human movement estimation unit 23 estimates the amount of movement indicating the environmentally-induced human movement caused by the movement of the moving body 3, based on the posture of the person and the horizontal and vertical movement of the moving body 3 indicated in the environmental movement information (S15). The human movement estimation unit 23 outputs the amount of movement indicating the environmentally-induced human movement to the human movement separation unit 26.

[0083] Next, the human movement separator 26 acquires the amount of movement of the person's head, which is the movement of the person output by the human movement detector 25 in Fig. 5, and the amount of movement indicating the person's movement due to the environment, which is the person's movement due to the environment output by the human movement estimation unit 23. Then, the human movement separator 26 calculates the amount of head movement after separation due to the person's own will, by subtracting the amount of movement indicating the person's movement due to the environment from the amount of head movement (S16). The human movement separator 26 outputs the calculated amount of head movement after separation due to the person's own will to the state detector 27. Then, the human movement separator 1 ends the processing operation of Fig. 9.

[0084] <Action and effect> The effects of the human movement separation device 1, the human movement separation method, and the program according to this embodiment will be described.

[0085] As described above, the human movement separation device 1 according to this embodiment comprises an image acquisition unit 21 that acquires an image of a person by capturing an image of a person present on the moving body 3 using the imaging unit 10 fixed to the moving body 3, a human movement detection unit 25 that detects human movement from the image, a posture detection unit 22 that detects the posture of the person, an environmental fluctuation acquisition unit 24 that acquires environmental fluctuation information that is information that indicates fluctuations caused by the environment in the horizontal and vertical directions relative to the person, a human movement estimation unit 23 that estimates an amount of movement that indicates human movement caused by the environment (amount of head movement caused by the environment) based on the posture of the person detected by the posture detection unit 22 and the environmental fluctuation information acquired by the environmental fluctuation acquisition unit 24, and a human movement separation unit 26 that separates the amount of movement that indicates human movement caused by the environment estimated by the human movement estimation unit 23 from the human movement detected by the human movement detection unit 25, and calculates human movement with the separated amount of movement that indicates human movement caused by the environment removed.

[0086] This allows the human movement estimation unit 23 to estimate the amount of movement indicating the human movement due to the environment based on the human posture and environmental movement information. In other words, the amount of movement indicating the human movement due to the environment can be estimated taking into account not only the environmental movement but also the human posture. Furthermore, the human movement separation unit 26 can remove the amount of movement indicating the human movement due to the environment from the human movement. This makes it possible to obtain the human movement due to one's own will, that is, the amount of head movement after separation.

[0087] Therefore, the human movement separating device 1 can accurately separate the movement of a person based on the person's intention from the movement of a person within the moving object 3.

[0088] In addition, the human movement separation method according to this embodiment includes capturing an image of a person present on the moving body 3 using an imaging unit 10 fixed to the moving body 3, thereby obtaining an image of the person; detecting the movement of the person from the image; detecting the posture of the person; obtaining environmental fluctuation information, which is information indicating environmental fluctuations in the horizontal and vertical directions relative to the person; estimating the amount of movement indicating the environmental fluctuation of the person based on the detected posture of the person and the obtained environmental fluctuation information; separating the estimated amount of movement indicating the environmental fluctuation of the person from the detected human movement; and calculating the human movement with the separated amount of movement indicating the environmental fluctuation of the person removed.

[0089] This method also provides the same effects as those described above.

[0090] The program according to this embodiment is a program for causing a computer to execute a human movement segmentation method.

[0091] This program also provides the same effects as those described above.

[0092] Furthermore, in the human movement separating device 1 according to this embodiment, the posture detecting unit 22 detects the posture of a person from an image.

[0093] This allows the posture of a person to be detected with high accuracy by using an image of the person.

[0094] Furthermore, in the human movement separating device 1 according to the present embodiment, the posture of a person is the angle at which the person leans relative to the vertical direction over time.

[0095] This allows the posture of a person to be quantitatively calculated as an angle, which can be used when the human movement estimation unit 23 estimates the amount of movement indicating the human movement due to the environment.

[0096] Furthermore, in the human movement separation device 1 according to this embodiment, the human sway estimation unit 23 estimates the amount of movement indicating the sway of the person due to the environment by calculating, over time, a composite component of a component obtained by converting the horizontal sway component of the moving body 3 based on the environmental sway information into a direction perpendicular to the person's posture direction according to the angle, and a component obtained by converting the vertical sway component of the moving body 3 based on the environmental sway information into a direction perpendicular to the person's posture direction according to the angle.

[0097] This makes it possible to estimate the amount of movement that indicates the shaking of a person due to the environment, taking into account not only the shaking due to the environment but also the angle of the person's posture, thereby enabling the amount of movement that indicates the shaking of a person due to the environment to be estimated with high accuracy.

[0098] (First Modification of the Embodiment) The human motion separation device 1a of this modified example differs from the human motion separation device of the embodiment in that the posture detection unit 22 is able to detect the posture of a person without acquiring images from the image acquisition unit 21. In this modified example, the same components and functions as those of the human motion separation device of the embodiment are assigned the same reference numerals, and detailed descriptions of these components and functions will be omitted.

[0099] FIG. 10 is a block diagram showing a human movement separating device 1a according to the first modification of the embodiment.

[0100] As shown in FIG. 10, the image acquisition unit 21 in this modification outputs the acquired image only to the human movement detection unit 25.

[0101] The posture detection unit 22 is a camera sensor that can detect the posture of a person by capturing an image of the person. The posture detection unit 22 detects the posture of the person based on the captured image of the person, and outputs the detected posture of the person to the human sway estimation unit 23.

[0102] This modification also provides the same effects as the above-described embodiment.

[0103] (Modification 2 of the embodiment) The human motion separation device 1b of this modified example differs from the human motion separation device of the embodiment in that it is equipped with a first environmental fluctuation acquisition unit 24a and a second environmental fluctuation acquisition unit 24b. In this modified example, the same components and functions as those of the human motion separation device of the embodiment are assigned the same reference numerals, and detailed descriptions of these components and functions will be omitted.

[0104] FIG. 11 is a block diagram showing a human movement separating device 1b according to the second modification of the embodiment.

[0105] 11, the human movement separation device 1b in this modified example includes a plurality of environmental fluctuation acquisition units in addition to an image acquisition unit 21, a human movement detection unit 25, a posture detection unit 22, a human movement estimation unit 23, and a human movement separation unit 26. The human movement separation device 1b in this modified example illustrates a first environmental fluctuation acquisition unit 24a and a second environmental fluctuation acquisition unit 24b as the plurality of environmental fluctuation acquisition units.

[0106] The first environmental fluctuation acquisition unit 24a acquires first environmental fluctuation information, which is information indicating fluctuations caused by the environment in the up-down direction (an example of the vertical direction) and the left-right direction (an example of the horizontal direction). The first environmental fluctuation acquisition unit 24a outputs the acquired first environmental fluctuation information to the human fluctuation estimation unit 23. The first environmental fluctuation acquisition unit 24a is an example of an environmental fluctuation acquisition unit. The first environmental fluctuation information is an example of environmental fluctuation information.

[0107] The second environmental fluctuation acquisition unit 24b acquires second environmental fluctuation information, which is information indicating fluctuations due to the environment in the forward / backward direction (an example of the horizontal direction) of the moving body 3. The second environmental fluctuation acquisition unit 24b outputs the acquired second environmental fluctuation information to the human fluctuation estimation unit 23. The second environmental fluctuation acquisition unit 24b is an example of an environmental fluctuation acquisition unit. The second environmental fluctuation information is an example of environmental fluctuation information.

[0108] In this case, the image acquired by image acquisition unit 21 may be a three-dimensional image. Furthermore, human movement detection unit 25 may be able to calculate the amount of movement of the person's head in the up-down, left-right, and front-back directions. Furthermore, posture detection unit 22 may be able to detect the posture of the person in the left-right and front-back directions relative to the up-down direction.

[0109] The first environmental fluctuation acquisition unit 24a and the second environmental fluctuation acquisition unit 24b are, for example, an acceleration sensor, a vibration meter, or the like.

[0110] The human fluctuation estimation unit 23 estimates the amount of movement indicating the human fluctuation due to the environment based on the human posture detected by the posture detection unit 22, the first environmental fluctuation information acquired by the first environmental fluctuation acquisition unit 24a, and the second environmental fluctuation information acquired by the second environmental fluctuation acquisition unit 24b.

[0111] Such a human movement separating device 1b according to this modified example includes a plurality of environmental fluctuation acquisition units (first environmental fluctuation acquisition unit 24a and second environmental fluctuation acquisition unit 24b).

[0112] This allows the human movement estimation unit 23 to acquire not only environmentally induced movement in the up-down and left-right directions, but also environmentally induced movement in the front-back direction, thereby enabling more accurate estimation of the amount of movement indicating the human movement due to the environment.

[0113] This modification also provides the same effects as the above-described embodiment.

[0114] (Third Modification of the Embodiment) The human motion separation device 1c of this modified example differs from the human motion separation device of the embodiment in that it includes a posture information acquisition unit 24c. In this modified example, the same components and functions as those of the human motion separation device of the embodiment are designated by the same reference numerals, and detailed descriptions of these components and functions will be omitted.

[0115] FIG. 12 is a block diagram showing a human movement separating device 1c according to the third modification of the embodiment.

[0116] As shown in FIG. 12, the human movement separation device 1c includes an image acquisition unit 21, a human movement detection unit 25, a posture detection unit 22, a human movement estimation unit 23, an environmental movement acquisition unit 24, and a human movement separation unit 26, as well as a posture information acquisition unit 24c.

[0117] The posture information acquiring unit 24c of this modification is worn by a person to acquire posture information that indicates the posture of the person. The posture information acquiring unit 24c is, for example, an IMU (Inertial Measurement Unit) sensor including an accelerometer, a gyroscope, a geomagnetometer, etc. The posture information acquiring unit 24c outputs the acquired posture information of the person to the posture detection unit 22.

[0118] When the posture detection unit 22 acquires the posture information from the posture information acquisition unit 24c, it detects the posture of the person based on the posture information.

[0119] The human motion separating device 1c according to this modification further includes a posture information acquiring unit 24c that acquires posture information that indicates the posture of the person. Then, the posture detecting unit 22 detects the posture of the person based on the posture information.

[0120] For example, if the posture information acquisition unit 24c is attached to a person, the posture of the person can be detected easily and accurately.

[0121] This modification also provides the same effects as the above-described embodiment.

[0122] (Other variations) The human motion separation device, human motion separation method, and program according to the present disclosure have been described above based on the above-mentioned embodiments, but the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications that come to mind by those skilled in the art may also be included in the scope of the present disclosure.

[0123] For example, in the human motion separation device, human motion separation method, and program according to the above-described embodiments, the human motion is not limited to the amount of movement of the human head. The human motion may be the amount of movement of the eyes, nose, mouth, etc. in the human head, or the amount of movement of the body, neck, etc. other than the head.

[0124] Furthermore, each component of the human motion segmentation device, human motion segmentation method, and program according to the above-described embodiments may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program stored in a storage medium such as a hard disk or semiconductor memory.

[0125] Furthermore, all of the numbers used above are examples for specifically explaining the present disclosure, and the embodiments of the present disclosure are not limited to the numbers shown as examples.

[0126] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0127] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0128] In addition, this disclosure also includes forms obtained by making various modifications to the above embodiments that a person skilled in the art would think of, and forms realized by arbitrarily combining the components and functions of the embodiments within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0129] The human movement separation device of the present disclosure can be used, for example, as a device for calculating the movement of a person riding in a moving object. [Explanation of symbols]

[0130] 1, 1a, 1b, 1c Human motion isolation device 3. Mobile 10. Imaging unit 21 Image acquisition unit 22 Attitude detection unit 23 Human Motion Estimation Unit 24 Environmental vibration acquisition unit 24a First environmental fluctuation acquisition unit (environmental fluctuation acquisition unit) 24b Second environmental fluctuation acquisition unit (environmental fluctuation acquisition unit) 24c Posture information acquisition unit 25 Human movement detection unit 26 Person Shaking Separation Unit

Claims

1. an image acquisition unit that acquires an image of a person by capturing an image of a person present on the moving body using an imaging unit fixed to the moving body; a human movement detection unit that detects human movement from the image; a posture detection unit that detects the posture of a person; an environmental fluctuation acquisition unit that acquires environmental fluctuation information, which is information indicating fluctuations caused by the environment in the horizontal and vertical directions relative to a person; a human fluctuation estimation unit that estimates a movement amount indicating the human fluctuation due to the environment based on the human posture detected by the posture detection unit and the environmental fluctuation information acquired by the environmental fluctuation acquisition unit; a human movement separation unit that separates the amount of movement that indicates the human movement due to the environment estimated by the human movement estimation unit from the human movement detected by the human movement detection unit, and calculates the human movement from which the separated amount of movement that indicates the human movement due to the environment has been removed. Human movement isolation device.

2. The posture detection unit detects the posture of the person from the image. The human motion isolation device of claim 1 .

3. A device including a plurality of environmental fluctuation acquisition units that acquire environmental fluctuation information in different directions.

3. A person motion isolation device according to claim 1 or 2.

4. Furthermore, a posture information acquisition unit is provided to acquire posture information that indicates the posture of a person. The posture detection unit detects the posture of the person based on the posture information. The human motion isolation device according to any one of claims 1 to 3.

5. A person's posture is the angle at which the person leans relative to the vertical direction over time. The human motion isolation device according to any one of claims 1 to 4.

6. The human sway estimation unit estimates a movement amount indicating the sway of the person due to the environment by calculating, over time, a composite component of a component obtained by converting a horizontal sway component of the moving body based on the environmental sway information into a direction perpendicular to the posture direction of the person according to the angle, and a component obtained by converting a vertical sway component of the moving body based on the environmental sway information into a direction perpendicular to the posture direction of the person according to the angle. The human motion isolation device of claim 5.

7. capturing an image of a person present on the moving body by an imaging unit fixed to the moving body; and detecting human movement from the image; Detecting a person's posture; Acquiring environmental vibration information, which is information indicating vibrations caused by the environment in the horizontal and vertical directions relative to the person; estimating a movement amount indicating the shaking of the person due to the environment based on the detected posture of the person and the acquired environmental shaking information; and separating an amount of movement indicating the swaying of the person due to the environment estimated from the detected movement of the person, and calculating the movement of the person from which the separated amount of movement indicating the swaying of the person due to the environment has been removed. Human motion separation method.

8. A method for causing a computer to execute the human movement separation method according to claim 7. program.

Images