Image analysis device, image analysis system, image analysis method, and image analysis program

Abstract

An image analysis device (1) comprising: a time-series image data correction unit (12) which receives time-series image data (A0) and time-series position / posture information (B0) and corrects the time-series image data (A0) on the basis of the time-series position / posture information (B0) to thereby generate time-series corrected image data (C0), the time-series image data (A0) being captured by an imaging device (10) that is mobile, the time-series position / posture information (B0) being acquired by a position / posture information acquisition unit (20) that acquires position / posture information indicating the position and posture of the imaging device (10); and an image analysis unit (13) which analyzes the state of an object included in the time-series image data (A0) on the basis of the time-series corrected image data (C0) to thereby generate an image analysis result (C1) indicating the state of the object.

Description

Image analysis device, image analysis system, image analysis method, and image analysis program

[0001] The present disclosure relates to an image analysis device, an image analysis system, an image analysis method, and an image analysis program.

[0002] Conventionally, a device has been proposed that calculates the distance to an object (e.g., a pedestrian or another vehicle) based on an image of the object captured by a camera, which is an imaging device mounted on a mobile device (e.g., an automobile or a robot) (see, for example, Patent Document 1).

[0003] International Publication No. 2019 / 181284

[0004] However, the device of Patent Document 1 has a problem in that when analyzing the state of an object based on time-series image data captured by a moving imaging device, the accuracy of the analysis of the state of the object decreases.

[0005] The present disclosure has been made to solve the above-mentioned conventional problems, and aims to accurately analyze the state of an object based on time-series image data captured by a moving imaging device and position and orientation information of the imaging device.

[0006] The image analysis device of the present disclosure is characterized by comprising: a time-series image data correction unit that receives time-series image data captured by a moving imaging device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and corrects the time-series image data based on the time-series position and orientation information to generate time-series corrected image data; and an image analysis unit that analyzes the state of an object included in the time-series image data based on the time-series corrected image data to generate an image analysis result indicating the state of the object.

[0007] Another image analysis device of the present disclosure is characterized by comprising an image analysis unit that receives time-series image data captured by a moving imaging device, analyzes the state of an object contained in the time-series image data, and outputs an image analysis result indicating the state of the object; and an analysis result correction unit that receives the image analysis result and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and corrects the image analysis result based on the time-series position and orientation information, thereby generating a corrected image analysis result.

[0008] According to the present disclosure, it is possible to accurately analyze the state of an object based on time-series image data captured by a moving imaging device and position and orientation information of the imaging device.

[0009] 1 is a block diagram showing a configuration of an image analysis system (including an image analysis device) according to a first embodiment; FIG. 2 is a diagram showing an example of a hardware configuration of an image analysis device according to the first embodiment; FIG. 3 is a diagram showing another example of a hardware configuration of an image analysis device according to the first embodiment; FIG. 4 is a flowchart showing the operation of an image analysis device according to the first embodiment; FIGS. 1A and 1B are diagrams showing, for comparative example #1, how an area where people pass is photographed by a fixed camera and time-series image data when the area where people pass is photographed; FIGS. 1A and 1B are diagrams showing, for comparative example #2, how an area where people pass is photographed by a moving imaging device and time-series image data when the area where people pass is photographed; FIGS. 1A and 1B are diagrams showing, for operation example #1 of the image analysis device according to the first embodiment, how an area where people pass is photographed by a moving imaging device and time-series corrected image data when the area where people pass is photographed; FIGS. 1A and 1B are diagrams showing, for operation example #2 of the image analysis device according to the first embodiment, how an area where people pass is photographed by a moving imaging device and time-series corrected image data when the area where people pass is photographed. 1A and 1B are diagrams showing, with regard to Comparative Example #3, how an area where a plurality of moving people pass through is photographed by a moving imaging device, and time-series image data when the area where a plurality of people pass through is photographed. 1A and 1B are diagrams showing, with regard to Operation Example #3 of the image analysis device according to Embodiment 1, how an area where a plurality of moving people pass through is photographed by a moving imaging device, and time-series corrected image data when the area where a plurality of people pass through is photographed. 1A and 1B are diagrams showing, with regard to Comparative Example #4, how an area where a plurality of stationary people pass through is photographed by a moving imaging device, and time-series corrected image data when the area where a plurality of people pass through is photographed. 1A and 1B are diagrams showing, with regard to Operation Example #4 of the image analysis device according to Embodiment 1, how an area where a plurality of stationary people pass through is photographed by a moving imaging device, and time-series corrected image data when the area where a plurality of people pass through is photographed. ...C is a block diagram showing the configuration of an image analysis system (including an image analysis device) according to Embodiment 2. 1D is a flowchart showing the operation of the image analysis device according to Embodiment 2.(A), (B), and (C) are diagrams showing operation example #5 of the image analysis device of embodiment 2, illustrating how an area where moving people pass through is photographed by a moving imaging device, time-series image data when the area where people pass through is photographed, and correction of the image analysis results.

[0010] An image analysis device, an image analysis system, an image analysis method, and an image analysis program according to embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0011] <<1>> Embodiment 1 <<1-1>> Configuration Fig. 1 is a block diagram showing the configuration of an image analysis system 1a (including an image analysis device 1) according to embodiment 1. The image analysis device 1 is a device capable of implementing the image analysis method according to embodiment 1. The image analysis system 1a includes an imaging device 10 such as a camera, an image storage unit 11 serving as a storage device that temporarily stores image data captured by the imaging device 10, a position and orientation information acquisition unit 20 that acquires position and orientation information indicating the position and orientation of the imaging device 10, a position and orientation information storage unit 21 serving as a storage device that temporarily stores the position and orientation information, a time-series image data correction unit 12, and an image analysis unit 13.

[0012] The imaging device 10 is provided on a mobile object. The mobile object is, for example, a mobile robot, which is a robot with a mobile function, an automobile, an unmanned aerial vehicle (commonly called a "drone"), or other mobile device. The mobile robot is, for example, a patrol security robot, which is an autonomous vehicle that patrols a predetermined area. The imaging device 10 is preferably equipped with a function to change the shooting direction (i.e., a function to perform pan / tilt operations). The imaging device 10 may also be a device that is carried by a person and moves (for example, a wearable camera), in which case the mobile object is the person carrying the imaging device 10.

[0013] The image storage unit 11 temporarily stores image data output from the imaging device 10 and outputs time-series image data A0. The time-series image data A0 is, for example, image data for each predetermined time interval (for example, times t, t+1, t+2, ... described below). The image storage unit 11 may be part of the imaging device 10. Alternatively, the image storage unit 11 may be part of the image analysis device 1.

[0014] The position and orientation information acquisition unit 20 is a device that can detect the position and orientation of the imaging device 10. The orientation of the imaging device 10 is information (e.g., camera parameters) that indicates the imaging direction of the imaging device 10. The position and orientation information acquisition unit 20 is, for example, a Global Navigation Satellite System (GNSS) device such as a Global Positioning System (GPS) provided on a mobile object, a device with a self-position estimation function such as Simultaneous Localization and Mapping (SLAM), or a positioning device that uses a beacon. The imaging device 10 also includes a device that acquires information indicating the amount of pan, tilt, and roll when changing the imaging direction. The position and orientation information acquisition unit 20 is generally provided on a mobile object equipped with the imaging device 10, but it does not necessarily have to be provided on the mobile object.

[0015] The position and orientation information holding unit 21 temporarily stores the position and orientation information output from the position and orientation information acquisition unit 20, and outputs time-series position and orientation information B0. The time-series position and orientation information B0 is, for example, position and orientation information for each predetermined time interval (for example, times t, t+1, t+2, ... described below). The time-series position and orientation information B0 is information corresponding to the time-series image data A0. The position and orientation information holding unit 21 may be part of the position and orientation information acquisition unit 20. Alternatively, the position and orientation information holding unit 21 may be part of the image analysis device 1.

[0016] The time-series image data correction unit 12 receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by a position and orientation information acquisition unit 20 that acquires position and orientation information indicating the position and orientation of the imaging device 10, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. The time-series image data correction unit 12 determines a reference time within the time when the time-series image data A0 was captured based on the time-series position and orientation information B0, and generates time-series corrected image data C0 by changing each piece of time-series image data A0 to image data that would have been obtained if the imaging device 10 had been located in the position at the reference time. Examples of the time-series corrected image data C0 are shown in Figures 7(B), 8(B), 10(B), and 12(B), which will be described later.

[0017] The image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. The object may be, for example, a person, a crowd of people, or another mobile device (e.g., another vehicle or another mobile robot). The state of the object may be, for example, the behavior of the person, whether the person is moving or stopped, whether the crowd is stationary or moving, etc. Specifically, the state of the object indicated by the image analysis result C1 may be, for example, the behavior of the person, the movement of the crowd, the movement of another mobile device, etc. The behavior of the person may be, for example, a person running, a person standing still, a person crouching, a person falling, multiple people gathering, a person dancing, a person acting violently, or people fighting with each other. The state of the object indicated by the image analysis result C1 may be, for example, the presence of a crowd, a crowd stationary without moving, a crowd moving, etc. Furthermore, the state of the object indicated by the image analysis result C1 may be that the other mobile device is moving, that the other mobile device is stopped in the middle of the passage without moving, or that the other mobile device has collided with an object.

[0018] FIG. 2 is a diagram illustrating an example of the hardware configuration of the image analyzing device 1. As illustrated in FIG. 2, each component constituting the image analyzing device 1 is realized by, for example, a processing circuit 103. The processing circuit 103 may be dedicated hardware, or may be a circuit including a CPU (Central Processing Unit) as a processor that executes programs stored in memory. When the processing circuit 103 is dedicated hardware, the processing circuit 103 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of any of these.

[0019] FIG. 3 is a diagram illustrating another example of the hardware configuration of the image analysis device 1. As illustrated in FIG. 3, the processing circuit constituting the image analysis device 1 may be realized by a memory 102 serving as a storage device (including a storage medium) that stores a software program (e.g., the image analysis program according to the first embodiment) and a processor 101 such as a CPU that reads and executes the image processing program. In this case, the image analysis device 1 is, for example, a computer. The memory 102 is, for example, a semiconductor memory such as a RAM (Random Access Memory), a magnetic disk, or the like. The image analysis device 1 may have a configuration including the processing circuit 103 of FIG. 2 and a configuration including the processor 101 and memory 102 of FIG. 3 .

[0020] <<1-2>> Operation Figure 4 is a flowchart showing the operation of the image analysis device 1 according to embodiment 1. As shown in step S11 of Figure 4, the time-series image data correction unit 12 receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquisition unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. Next, as shown in step S12, the image analysis unit 13 analyzes the state of an object included in the time-series image data A0 based on the time-series corrected image data C0, and generates and outputs an image analysis result C1 indicating the state of the object.

[0021] 1-3 Comparative Examples #1 and #2, Operation Example #1 FIGS. 5A and 5B are diagrams illustrating, with respect to Comparative Example #1, how an area through which person 41 passes is photographed by fixed camera 32 and the time-series image data obtained when photographing the area through which person 41 passes. When moving person 41 is photographed by fixed camera 32 as shown in FIG. 5A , the image analysis device of Comparative Example #1 receives the time-series image data from fixed camera 32 and compares the time-series image data at times t, t+1, t+2, ..., obtained by photographing the same photographed area, as shown in FIG. 5B , to determine that the position of photographed person 41 has moved. In other words, when fixed camera 32 is used, each piece of time-series image data photographs the same area as shown in FIG. 5B , and therefore, the image analysis device of Comparative Example #1 can output an image analysis result indicating that "person 41 is moving" (or, if the person 41 is moving quickly, "person 41 is running") through image analysis. In other words, the image analysis device of Comparative Example #1 has the disadvantage that the distance between the fixed camera 32 and the person 41 as the target is long, but it can output highly accurate image analysis results regarding the movements of the person 41.

[0022] However, in order to expand the photographing area of ​​the imaging device 10, the imaging device 10 may be mounted on a mobile robot 31 serving as a moving body. FIGS. 6A and 6B are diagrams showing, with respect to Comparative Example #2, how the moving imaging device 10 photographs an area through which a person 41 passes and the time-series image data obtained when the area through which the person 41 passes is photographed. As shown in FIG. 6A , when a target area (e.g., including the moving person 41) is photographed by the moving imaging device 10, not only the person 41 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the photographing area also moves. Therefore, the image analysis device of Comparative Example #2, which performs image analysis using the time-series image data photographed by the imaging device 10 at times t, t+1, t+2, ..., may generate an image analysis result indicating that "the position of the photographed person has not moved," as shown in FIG. 6B . 6B , when a moving imaging device 10 is used, each piece of time-series image data captures a different area, so the image analysis device of Comparative Example #2 may output an image analysis result indicating that "person 41 is not moving" even though person 41 is moving. In other words, the image analysis device of Comparative Example #2 has low accuracy in image analysis of the state of person 41 as a target.

[0023] 7A and 7B are diagrams illustrating operation example #1 of the image analysis device 1 according to embodiment 1, showing how an area through which a person 41 passes is photographed by a moving imaging device 10, and time-series corrected image data when the area through which the person 41 passes is photographed. As shown in Fig. 7A, when a person 41 moving in approximately the same direction as the moving imaging device 10 is photographed by the moving imaging device 10, not only the person 41 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the photographed area also moves. The time-series image data corrector 12 of the image analysis device 1 according to embodiment 1 receives time-series image data A0 photographed by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquirer 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 7B , the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data and corrects image data captured at position P1 (time t+1) of the imaging device 10 to generate image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 7B , the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data and corrects image data captured at position P2 (time t+2) of the imaging device 10 to generate image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 7B, each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and image analysis can output an image analysis result C1 indicating that "the person 41 is moving" (or "the person 41 is running" if the person is moving quickly), just as in the case where a fixed camera is capturing an image of the same area. In other words, the image analysis device 1 can perform image analysis of the state of the person 41 as a target with high accuracy.The number of frames of image data to be used is not limited to three. The position of the imaging device 10 that serves as the reference for correction is not limited to position P0. The position of the imaging device 10 that serves as the reference for correction may be position P1 or position P2. Furthermore, the position of the imaging device 10 that serves as the reference for correction may be a position other than positions P0, P1, and P2 (for example, a position between positions P0 and P1, or a position between positions P1 and P2, etc.).

[0024] 7A and 7B illustrate the movement of the position of the person 41 as an example of the state of the target, but the image analyzing device 1 may also analyze the movement of the person's arms (up-down and left-right movement of the arms), the movement of the person's legs (up-down and left-right movement of the legs), etc. In this case, the image analyzing device 1 acquires information on the shooting direction of the image capturing device 10 as the position and orientation information of the image capturing device 10, and generates time-series corrected image data C0 based on the position and orientation information, thereby improving the accuracy of the image analysis.

[0025] 8A and 8B are diagrams illustrating operation example #2 of the image analysis device 1 according to embodiment 1, showing how an area through which a person 41 passes is photographed by a moving imaging device 10, and the time-series corrected image data obtained when the area through which the person passes is photographed. As shown in FIG. 8A, when the moving person 41 is photographed by an imaging device 10 moving in a direction approaching the person 41 from diagonally in front, not only the person 41 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the photographed area also moves. The time-series image data corrector 12 of the image analysis device 1 according to embodiment 1 receives time-series image data A0 photographed by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquirer 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 8(B), the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data, corrects image data captured at position P1 (time t+1) of the imaging device 10, and generates image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 8(B), the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data, corrects image data captured at position P2 (time t+2) of the imaging device 10, and generates image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. As shown in Figure 8 (B), each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and image analysis can output an image analysis result C1 that ``person 41 is moving'' (or ``person 41 is running'' if the movement is fast), just as when a fixed camera is photographing the same area.That is, the image analyzing device 1 can perform highly accurate image analysis of the state of the target person 41. In operation example #2 in Fig. 8(A), the actual shooting positions approach the person 41 in the order of positions P0, P1, and P2, so as shown in Fig. 8(B), the size of the image based on the corrected image data at position P1 is smaller than that at position P0, and the size of the image based on the corrected image data at position P2 is smaller than that at position P1.

[0026] 1-5 Comparative Example #3, Operation Example #3 FIGS. 9A and 9B are diagrams illustrating, with respect to Comparative Example #3, how an area through which multiple moving people 42 pass is captured by a moving imaging device 10, and the time-series image data obtained when the area through which the multiple people 42 pass is captured. As shown in FIG. 9A, when multiple people 42 in a moving crowd are captured by a moving imaging device 10, not only the multiple people 42 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the captured area also moves. Therefore, the image analysis device of Comparative Example #3, which performs image analysis using time-series image data captured by the imaging device 10 at times t, t+1, t+2, ..., may generate an image analysis result indicating that "the positions of the captured multiple people have not moved," as shown in FIG. 9B. 9B , when a moving imaging device 10 is used, each piece of time-series image data captures a different area, so the image analysis device of Comparative Example #3 may output an image analysis result that "a crowd of people 42 is stagnating in the aisle" through image analysis, even though the crowd of people 42 is actually moving down the aisle. In other words, the image analysis device of Comparative Example #3 has low accuracy in image analysis of the state of the target people 42.

[0027] 10A and 10B are diagrams illustrating operation example #3 of the image analyzing device 1 according to embodiment 1, showing how an area through which multiple people 42 pass is captured by a moving imaging device 10, and time-series corrected image data when the area through which the multiple people 42 pass is captured. As shown in Fig. 10A, when multiple people 42 are captured by a moving imaging device 10, not only the multiple people 42 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the captured area also moves. The time-series image data correcting unit 12 of the image analyzing device 1 according to embodiment 1 receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquiring unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 10(B), the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data, corrects image data captured at position P1 (time t+1) of the imaging device 10, and generates image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 10(B), the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data, corrects image data captured at position P2 (time t+2) of the imaging device 10, and generates image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 10B , each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and as in the case where a fixed camera is capturing an image of the same area, image analysis can output an image analysis result C1 that "multiple people 42 are moving," i.e., "there is no crowd." In other words, the image analyzing device 1 can perform image analysis of the state of multiple people 42 as targets with high accuracy.

[0028] 11A and 11B are diagrams illustrating, in Comparative Example #4, how an area where a plurality of people 42 are passing through is imaged by a moving imaging device 10, and the time-series image data obtained when the area where the plurality of people 42 are passing through is imaged. As shown in FIG. 11A, when a plurality of people 43 forming a crowd of people is imaged by the moving imaging device 10, the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the imaged area moves. Therefore, the image analysis device of Comparative Example #4, which performs image analysis using time-series image data at times t, t+1, t+2, ..., imaged by the imaging device 10, may generate an image analysis result indicating that "the positions of the plurality of people are moving," as shown in FIG. 11B. 11B , when a moving imaging device 10 is used, each piece of time-series image data captures a different area, so the image analysis device of Comparative Example #4 may output an erroneous image analysis result that "a crowd of people 43 is moving down the aisle" even though the crowd of people 43 is actually stationary in the aisle. In other words, the image analysis device of Comparative Example #4 has low accuracy in image analysis of the state of the target people 43.

[0029] 12A and 12B are diagrams illustrating operation example #4 of the image analyzing device 1 according to embodiment 1, showing a state in which a moving imaging device 10 captures an image of an area where a plurality of people 43, a crowd of people, pass through, and time-series corrected image data C0 when the area where the plurality of people 43 pass through is captured. As shown in Fig. 12A, when the moving imaging device 10 captures an image of a plurality of people 43 that are not moving, the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the captured area moves. The time-series image data correcting unit 12 of the image analyzing device 1 according to embodiment 1 receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquiring unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 12(B), the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data, corrects image data captured at position P1 (time t+1) of the imaging device 10, and generates image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 12(B), the time-series image data correction unit 12 uses image data captured at position P0 (time t) of the reference imaging device 10 as reference image data, corrects image data captured at position P2 (time t+2) of the imaging device 10, and generates image data that would have been obtained if the imaging device 10 had been captured at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 12(B), each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and as in the case where a fixed camera is photographing the same area, image analysis can output an image analysis result C1 indicating that "multiple people 43 are stationary," i.e., that "the crowd is not moving." In other words, the image analyzing device 1 can perform image analysis of the state of the multiple people 43 as targets with high accuracy.

[0030] <<1-7>> Effects According to the first embodiment, the state of a target can be analyzed accurately based on time-series image data captured by the moving imaging device 10 and the position and orientation information of the imaging device 10.

[0031] <<2>> Second Embodiment <<2-1>> Configuration Fig. 13 is a block diagram showing the configuration of an image analysis system 2a (including an image analysis device 2) according to the second embodiment. The image analysis device 2 is a device capable of implementing the image analysis method according to the second embodiment. The image analysis system 2a has an imaging device 10 such as a camera, an image storage unit 11 as a storage device that temporarily stores image data captured by the imaging device 10, an image analysis unit 14, and an analysis result correction unit 15. In the image analysis device 2 according to the second embodiment, components that are the same as or correspond to those of the image analysis device 1 according to the first embodiment are assigned the same reference numerals as those in the first embodiment.

[0032] The image analysis unit 14 receives time-series image data A0 captured by the moving imaging device 10, analyzes the state of the object contained in the time-series image data A0, and outputs an image analysis result D0 indicating the state of the object.

[0033] The analysis result correction unit 15 receives the image analysis result D0 and time-series position and orientation information B0 acquired by the position and orientation information acquisition unit 20, which acquires position and orientation information indicating the position and orientation of the image capture device 10, and generates a corrected image analysis result D1 by correcting the image analysis result D0 based on the time-series position and orientation information B0. For example, the analysis result correction unit 15 determines a reference time within the time when the time-series image data A0 was captured based on the time-series position and orientation information B0, and changes the image analysis result D0 indicating the state of the target to the image analysis result that would have been obtained if the image capture device 10 had been in the position at the reference time, thereby generating the corrected image analysis result D1.

[0034] 14 is a flowchart showing the operation of the image analysis device 2 according to embodiment 2. As shown in step S21 of FIG. 14 , the image analysis unit 14 receives time-series image data A0 captured by the moving imaging device 10, and analyzes the state of an object contained in the time-series image data A0 to generate an image analysis result D0 indicating the state of the object. Next, as shown in step S22, the analysis result correction unit 15 receives the image analysis result D0 and the time-series position and orientation information B0, and corrects the image analysis result D0 based on the time-series position and orientation information B0 to output a corrected image analysis result D1.

[0035] 15A, 15B, and 15C are diagrams illustrating operation example #5 of the image analysis device 2 according to embodiment 2, showing how an area through which a moving person 41 passes is photographed by a moving imaging device 10, time-series image data A0 obtained when the area through which the person 41 passes is photographed, and correction of the image analysis result D0. As shown in FIG. 15A, when a person 41 moving in the same direction as the imaging device 10 is photographed by the moving imaging device 10, the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the photographed area moves. As shown in FIG. 15B, the image analysis unit 14 of the image analysis device 2 according to embodiment 2 receives time-series image data A0 photographed by the moving imaging device 10, analyzes the state of the person 41 based on the time-series image data A0, and generates the image analysis result D0. 15(C), the analysis result correcting unit 15 of the image analyzing device 2 according to the second embodiment corrects the image analysis result D0 (e.g., "The person 41 is not moving.") based on the time-series position and orientation information to generate a corrected image analysis result D1 (e.g., "The person 41 is moving.") In this way, the image analyzing device 2 according to the second embodiment can perform image analysis of the state of the person 41 as a target with high accuracy.

[0036] <<2-4>> Effects According to the second embodiment, it is possible to perform image analysis of the state of an object with high accuracy based on time-series image data captured by a moving imaging device and position and orientation information of the imaging device.

[0037] 1, 2 Image analysis device, 1a, 2a Image analysis system, 10 Imaging device, 11 Image storage unit, 12 Time series image data correction unit, 13 Image analysis unit, 14 Image analysis unit, 15 Analysis result correction unit, 20 Position and orientation information acquisition unit, 21 Position and orientation information storage unit, 31 Mobile robot (moving body), 41 Person (target), 42 Multiple people (moving crowd), 43 Multiple people (stationary crowd), A0 Time series image data, B0 Time series position and orientation information, C0 Time series corrected image data, C1 Image analysis result, D0 Image analysis result, D1 Corrected image analysis result.

Claims

1. A time-series image data correction unit that receives time-series image data captured by a moving imaging device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and generates time-series corrected image data by correcting the time-series image data based on the time-series position and orientation information; An image analysis unit that generates an image analysis result indicating the state of the object by analyzing the behavior, movement, or moving or stopping state of a movable object included in the time-series image data based on the time-series corrected image data; An image analysis apparatus, characterized by comprising the above.

2. The time-series image data correction unit determines a reference time within the time when the time-series image data was captured based on the time-series position and orientation information, and changes each of the time-series image data into image data that would be obtained if the imaging device were at the position at the reference time, thereby generating the time-series corrected image data. The image analysis apparatus according to claim 1, characterized by the above.

3. An image analysis unit that receives time-series image data captured by a moving imaging device and outputs an image analysis result indicating the state of the object by analyzing the state of the object included in the time-series image data; An analysis result correction unit that receives the image analysis result and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and generates a corrected image analysis result by correcting the image analysis result based on the time-series position and orientation information; An image analysis apparatus, characterized by comprising the above.

4. The analysis result correction unit determines a reference time within the time when the time-series image data was captured based on the time-series position and orientation information, and changes the image analysis result indicating the state of the object into an image analysis result that would be obtained if the imaging device were at the position at the reference time, thereby generating a corrected image analysis result. The image analysis apparatus according to claim 3, characterized by the above.

5. The time-series position and orientation information includes position information of the imaging device and information indicating the shooting direction. The image analysis apparatus according to any one of claims 1 to 4, characterized by the above.

6. The image analysis unit determines the behavior of a person as the state of the object. The image analysis apparatus according to any one of claims 1 to 4, characterized in that...

7. The image analysis unit determines whether the person as the target is moving or stopped. The image analysis apparatus according to any one of claims 1 to 4, characterized in that...

8. The image analysis unit determines whether the crowd as the target is staying or moving. The image analysis apparatus according to any one of claims 1 to 4, characterized in that...

9. A movable imaging device, A position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, Receives time-series image data captured by the moving imaging device and time-series position and orientation information acquired by the position and orientation information acquisition unit, and generates time-series corrected image data by correcting the time-series image data based on the time-series position and orientation information. A time-series image data correction unit, An image analysis unit that generates an image analysis result indicating the state of the target by analyzing the behavior, movement, or moving or stopped state of a movable target included in the time-series image data based on the time-series corrected image data. An image analysis system, characterized by comprising the above.

10. The time-series image data correction unit determines a reference time within the time when the time-series image data was captured based on the time-series position and orientation information, and changes each of the time-series image data to the image data that would be obtained if the imaging device were at the position at the reference time, thereby generating the time-series corrected image data. The image analysis system according to claim 9, characterized in that...

11. A movable imaging device, A position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, Receives time-series image data captured by the moving imaging device, analyzes the state of the target included in the time-series image data, and outputs an image analysis result indicating the state of the target. An image analysis unit, Receives the image analysis result and the time-series position and orientation information acquired by the position and orientation information acquisition unit, and generates a corrected image analysis result by correcting the image analysis result based on the time-series position and orientation information. An analysis result correction unit, An image analysis system, characterized by comprising the above.

12. The analysis result correction unit determines a reference time within the time when the time-series image data was captured based on the time-series position and orientation information, and changes the image analysis result indicating the state of the object to the image analysis result that would be obtained if the imaging device was at the position at the reference time, thereby generating a corrected image analysis result. The image analysis system according to claim 11, characterized in that.

13. The time-series position and orientation information includes position information of the imaging device and information indicating the imaging direction. The image analysis system according to any one of claims 9 to 12, characterized in that.

14. The image analysis unit determines human behavior as the state of the object. The image analysis system according to any one of claims 9 to 12, characterized in that.

15. The image analysis unit determines whether the person as the object is moving or stopped. The image analysis system according to any one of claims 9 to 12, characterized in that.

16. The image analysis unit determines whether the crowd as the object is staying or moving. The image analysis system according to any one of claims 9 to 12, characterized in that.

17. An image analysis method executed by an image analysis device that receives time-series image data captured by a moving imaging device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, generating time-series corrected image data by correcting the time-series image data based on the time-series position and orientation information; generating an image analysis result indicating the state of the object by analyzing the behavior, movement, or moving or stopped state of a movable object included in the time-series image data based on the time-series corrected image data; An image analysis method characterized by comprising.

18. An image analysis program characterized by causing a computer as the image analysis device to execute the image analysis method according to claim 17.

19. An image analysis method executed by an image analysis device that receives time-series image data captured by a moving imaging device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, By analyzing the state of the object included in the time-series image data, outputting an image analysis result indicating the state of the object; Generating a corrected image analysis result by correcting the image analysis result based on the image analysis result and the position and orientation information of the time series; An image analysis method characterized by comprising the above.

20. An image analysis program, characterized in that a computer as the image analysis device executes the image analysis method according to claim 19.