Image analysis device, image analysis system, image analysis method, and image analysis program

Abstract

An image analysis device (1) comprising: a time-series image data correction unit (12) which receives time-series image data (A0) and time-series position / posture information (B0) and corrects the time-series image data (A0) on the basis of the time-series position / posture information (B0) to thereby generate time-series corrected image data (C0), the time-series image data (A0) being captured by an imaging device (10) that is mobile, the time-series position / posture information (B0) being acquired by a position / posture information acquisition unit (20) that acquires position / posture information indicating the position and posture of the imaging device (10); and an image analysis unit (13) which analyzes the state of an object included in the time-series image data (A0) on the basis of the time-series corrected image data (C0) to thereby generate an image analysis result (C1) indicating the state of the object.

Description

[Technical Field]

[0001] The present disclosure relates to an image analysis device, an image analysis system, an image analysis method, and an image analysis program. [Background technology]

[0002] Conventionally, a device has been proposed that calculates the distance to an object (e.g., a pedestrian or another vehicle) based on an image of the object captured by a camera, which is an imaging device mounted on a mobile device (e.g., an automobile or a robot) (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 181284 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the device of Patent Document 1 has a problem in that when analyzing the state of an object based on time-series image data captured by a moving imaging device, the accuracy of the analysis of the state of the object decreases.

[0005] The present disclosure has been made to solve the above-mentioned conventional problems, and aims to accurately analyze the state of an object based on time-series image data captured by a moving imaging device and position and orientation information of the imaging device. [Means for solving the problem]

[0006] The image analysis device of the present disclosure includes a time-series image data correction unit that receives time-series image data captured by a moving imaging device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and corrects the time-series image data based on the time-series position and orientation information to generate time-series corrected image data; and a time-series image data correction unit that corrects the time-series image data based on the time-series corrected image data to generate time-series corrected image data. Movable Target Action, movement, or movement or cessation By analyzing the state of the target The aforementioned and an image analysis unit that generates an image analysis result indicating the state.

[0007] Another image analysis device of the present disclosure receives time-series image data captured by a moving imaging device, and analyzes the state of an object included in the time-series image data to analyze the state of the object. The aforementioned and an analysis result correction unit that receives the image analysis result and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and corrects the image analysis result based on the time-series position and orientation information to generate a corrected image analysis result. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to accurately analyze the state of an object based on time-series image data captured by a moving imaging device and position and orientation information of the imaging device. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an image analysis system (including an image analysis device) according to a first embodiment. [Figure 2] 1 is a diagram illustrating an example of a hardware configuration of an image analyzing device according to a first embodiment. [Figure 3] FIG. 4 is a diagram illustrating another example of the hardware configuration of the image analyzing device according to the first embodiment. [Figure 4] 4 is a flowchart showing the operation of the image analyzing device according to the first embodiment. [Figure 5] 10A and 10B are diagrams showing, with respect to Comparative Example #1, how an area where people pass is photographed by a fixed camera, and time-series image data when the area where people pass is photographed. [Figure 6] 10A and 10B are diagrams showing, with respect to Comparative Example #2, how an area where people pass is photographed by a moving imaging device, and time-series image data when the area where people pass is photographed. [Figure 7] (A) and (B) show operation example #1 of the image analysis device of embodiment 1, showing how an area where people pass is photographed by a moving imaging device, and time-series corrected image data when the area where people pass is photographed. [Figure 8] (A) and (B) show operation example #2 of the image analysis device of embodiment 1, illustrating how an area where people pass is photographed by a moving imaging device, and the time-series corrected image data when the area where people pass is photographed. [Figure 9] (A) and (B) show comparative example #3, showing the scene where an area where multiple people pass through is photographed by a moving imaging device, and the time-series image data when the area where multiple people pass through is photographed. [Figure 10] (A) and (B) show operation example #3 of the image analysis device of embodiment 1, illustrating how an area where multiple people pass through is photographed by a moving imaging device, and the time-series corrected image data when the area where multiple people pass through is photographed. [Figure 11] (A) and (B) show comparative example #4, showing the scene where an area where multiple people are passing through is photographed by a moving imaging device, and the time-series image data when the area where multiple people are passing through is photographed. [Figure 12](A) and (B) show operation example #4 of the image analysis device of embodiment 1, illustrating how an area where multiple people are passing through is photographed by a moving imaging device, and the time-series corrected image data when the area where multiple people are passing through is photographed. [Figure 13] FIG. 10 is a block diagram showing the configuration of an image analysis system (including an image analysis device) according to a second embodiment. [Figure 14] 10 is a flowchart showing the operation of the image analyzing device according to the second embodiment. [Figure 15] (A), (B), and (C) are diagrams showing operation example #5 of the image analysis device of embodiment 2, illustrating how an area where moving people pass through is photographed by a moving imaging device, time-series image data when the area where people pass through is photographed, and correction of the image analysis results. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image analysis device, an image analysis system, an image analysis method, and an image analysis program according to embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0011] 1. First embodiment 1-1 Configuration 1 is a block diagram showing the configuration of an image analysis system 1a (including an image analysis device 1) according to Embodiment 1. The image analysis device 1 is a device capable of implementing the image analysis method according to Embodiment 1. The image analysis system 1a includes an imaging device 10 such as a camera, an image storage unit 11 serving as a storage device that temporarily stores image data captured by the imaging device 10, a position and orientation information acquisition unit 20 that acquires position and orientation information indicating the position and orientation of the imaging device 10, a position and orientation information storage unit 21 serving as a storage device that temporarily stores the position and orientation information, a time-series image data correction unit 12, and an image analysis unit 13.

[0012] The imaging device 10 is provided on a mobile object. The mobile object is, for example, a mobile robot, which is a robot with a mobile function, an automobile, an unmanned aerial vehicle (commonly called a "drone"), or other mobile device. The mobile robot is, for example, a patrol security robot, which is an autonomous vehicle that patrols a predetermined area. The imaging device 10 is preferably equipped with a function to change the shooting direction (i.e., a function to perform pan / tilt operations). The imaging device 10 may also be one that is carried by a person and moves (for example, a wearable camera), in which case the mobile object is the person carrying the imaging device 10.

[0013] The image storage unit 11 temporarily stores image data output from the imaging device 10 and outputs time-series image data A0. The time-series image data A0 is, for example, image data for each predetermined time interval (for example, times t, t+1, t+2, ... described below). The image storage unit 11 may be a part of the imaging device 10. Alternatively, the image storage unit 11 may be a part of the image analysis device 1.

[0014] The position and orientation information acquisition unit 20 is a device that can detect the position and orientation of the imaging device 10. The orientation of the imaging device 10 is information (e.g., camera parameters) that indicates the imaging direction of the imaging device 10. The position and orientation information acquisition unit 20 is, for example, a Global Navigation Satellite System (GNSS) device such as a Global Positioning System (GPS) provided in a mobile object, a device with a self-position estimation function such as Simultaneous Localization and Mapping (SLAM), or a positioning device that uses a beacon. The imaging device 10 also includes a device that acquires information indicating the amount of pan, tilt, and roll when changing the imaging direction. The position and orientation information acquisition unit 20 is generally provided in a mobile object equipped with the imaging device 10, but it does not necessarily have to be provided in the mobile object.

[0015] The position and orientation information holding unit 21 temporarily stores the position and orientation information output from the position and orientation information acquisition unit 20, and outputs time-series position and orientation information B0. The time-series position and orientation information B0 is, for example, position and orientation information for each predetermined time interval (for example, times t, t+1, t+2, ... described below). The time-series position and orientation information B0 is information corresponding to the time-series image data A0. The position and orientation information holding unit 21 may be a part of the position and orientation information acquisition unit 20. Alternatively, the position and orientation information holding unit 21 may be a part of the image analysis device 1.

[0016] The time-series image data correction unit 12 receives time-series image data A0 captured by the moving image capture device 10 and time-series position and orientation information B0 acquired by a position and orientation information acquisition unit 20 that acquires position and orientation information indicating the position and orientation of the image capture device 10, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. The time-series image data correction unit 12 determines a reference time within the time when the time-series image data A0 was captured based on the time-series position and orientation information B0, and generates time-series corrected image data C0 by changing each piece of time-series image data A0 to image data that would have been obtained if the image capture device 10 had been located in the position at the reference time. Examples of the time-series corrected image data C0 are shown in FIGS. 7(B), 8(B), 10(B), and 12(B), which will be described later.

[0017] The image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. The object may be, for example, a person, a crowd of people, or another mobile device (e.g., another vehicle or another mobile robot). The state of the object may be, for example, the behavior of the person, whether the person is moving or stopped, whether the crowd is stationary or moving, etc. Specifically, the state of the object indicated by the image analysis result C1 may be, for example, the behavior of the person, the movement of the crowd, the movement of another mobile device, etc. The behavior of the person may be, for example, a person running, a person standing still, a person crouching, a person falling, multiple people gathering, a person dancing, a person acting violently, or people fighting with each other. The state of the object indicated by the image analysis result C1 may be, for example, the presence of a crowd, a crowd stationary without moving, a crowd moving, etc. Furthermore, the state of the object indicated by the image analysis result C1 may be that the other mobile device is moving, that the other mobile device is stopped in the middle of the passage without moving, or that the other mobile device has collided with an object.

[0018] 2 is a diagram illustrating an example of the hardware configuration of the image analyzing device 1. As illustrated in FIG. 2, each component constituting the image analyzing device 1 is realized by, for example, a processing circuit 103. The processing circuit 103 may be dedicated hardware, or may be a circuit including a CPU (Central Processing Unit) as a processor that executes programs stored in memory. When the processing circuit 103 is dedicated hardware, the processing circuit 103 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of any of these.

[0019] FIG. 3 is a diagram illustrating another example of the hardware configuration of the image analyzing device 1. As illustrated in FIG. 3, the processing circuitry constituting the image analyzing device 1 may be realized by a memory 102 serving as a storage device (including a storage medium) that stores a software program (e.g., the image analysis program according to the first embodiment), and a processor 101 such as a CPU that reads and executes the image processing program. In this case, the image analyzing device 1 is, for example, a computer. The memory 102 is, for example, a semiconductor memory such as a RAM (Random Access Memory), a magnetic disk, or the like. The image analyzing device 1 may have a configuration including the processing circuitry 103 of FIG. 2 and a configuration having the processor 101 and memory 102 of FIG. 3 in combination.

[0020] 《1-2》Operation 4 is a flowchart showing the operation of the image analysis device 1 according to the first embodiment. As shown in step S11 of FIG. 4, the time-series image data correction unit 12 receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquisition unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. Next, as shown in step S12, the image analysis unit 13 analyzes the state of an object included in the time-series image data A0 based on the time-series corrected image data C0, and generates and outputs an image analysis result C1 indicating the state of the object.

[0021] 1-3 Comparative Examples #1 and #2, Operation Example #1 5(A) and 5(B) are diagrams showing, with respect to Comparative Example #1, how an area through which person 41 passes is photographed by fixed camera 32, and the time-series image data obtained when photographing the area through which person 41 passes. When moving person 41 is photographed by fixed camera 32 as shown in FIG. 5(A), the image analysis device of Comparative Example #1 that receives the time-series image data from fixed camera 32 can determine that the position of photographed person 41 has moved by comparing the time-series image data at times t, t+1, t+2, ... obtained by photographing the same photographed area, as shown in FIG. 5(B). In other words, when fixed camera 32 is used, each piece of time-series image data photographs the same area, as shown in FIG. 5(B), and therefore, the image analysis device of Comparative Example #1 can output an image analysis result that "person 41 is moving" (or "person 41 is running" if the person 41 is moving fast) through image analysis. That is, the image analysis device of Comparative Example #1 has the drawback that the distance between fixed camera 32 and person 41 as the target is long, but it can output highly accurate image analysis results regarding the movements of person 41.

[0022] However, in order to expand the photographing area of ​​the imaging device 10, the imaging device 10 may be mounted on a mobile robot 31 as a moving body. FIGS. 6A and 6B are diagrams showing, with respect to Comparative Example #2, how an area through which a person 41 passes is photographed by the moving imaging device 10 and time-series image data obtained when the area through which the person 41 passes is photographed. As shown in FIG. 6A, when a target area (e.g., including the moving person 41) is photographed by the moving imaging device 10, not only the person 41 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the photographing area also moves. Therefore, the image analysis device of Comparative Example #2, which performs image analysis using time-series image data photographed by the imaging device 10 at times t, t+1, t+2, ..., may generate an image analysis result indicating that "the position of the photographed person has not moved," as shown in FIG. 6B. In other words, when a moving imaging device 10 is used, as shown in Fig. 6(B), each piece of time-series image data captures a different area, so the image analysis device of Comparative Example #2 may output an image analysis result indicating that "person 41 is not moving" even though person 41 is moving through image analysis. In other words, the image analysis device of Comparative Example #2 has low accuracy in image analysis of the state of person 41 as a target.

[0023] 7A and 7B are diagrams illustrating operation example #1 of the image analyzing device 1 according to the first embodiment, showing how an area through which a person 41 passes is imaged by a moving imaging device 10, and time-series corrected image data when the area through which the person 41 passes is imaged. As shown in FIG. 7A, when a person 41 moving in approximately the same direction as the moving imaging device 10 is imaged by the moving imaging device 10, not only the person 41 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the imaging area also moves. The time-series image data correcting unit 12 of the image analyzing device 1 according to the first embodiment receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquiring unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 7(B), the time-series image data correction unit 12 corrects image data taken at position P1 (time t+1) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 7(B), the time-series image data correction unit 12 corrects image data taken at position P2 (time t+2) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 7(B), each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and image analysis can output an image analysis result C1 that "the person 41 is moving" (or "the person 41 is running" if the person 41 is moving fast), just as in the case where a fixed camera is photographing the same area. In other words, the image analyzing device 1 can perform image analysis of the state of the person 41 as the target with high accuracy.The number of frames of image data to be used is not limited to three. The position of the imaging device 10 that serves as the reference for correction is not limited to position P0. The position of the imaging device 10 that serves as the reference for correction may be position P1 or position P2. Furthermore, the position of the imaging device 10 that serves as the reference for correction may be a position other than positions P0, P1, and P2 (for example, a position between positions P0 and P1, or a position between positions P1 and P2, etc.).

[0024] 7(A) and 7(B) illustrate the movement of the position of person 41 as an example of the state of the target, but image analyzing device 1 may also analyze the movement of the person's arms (up-down and left-right movement of the arms), the movement of the person's legs (up-down and left-right movement of the legs), etc. In this case, image analyzing device 1 acquires information on the shooting direction of image capturing device 10 as the position and orientation information of image capturing device 10, and generates time-series corrected image data C0 based on the position and orientation information, thereby improving the accuracy of image analysis.

[0025] 《1-4》Example #2 8A and 8B are diagrams illustrating, in operation example #2 of the image analyzing device 1 according to the first embodiment, how an area through which a person 41 passes is imaged by a moving imaging device 10, and time-series corrected image data when the area through which the person passes is imaged. As shown in FIG. 8A, when the moving person 41 is imaged by an imaging device 10 moving in a direction approaching the person 41 from diagonally in front, not only the person 41 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the imaged area also moves. The time-series image data correcting unit 12 of the image analyzing device 1 according to the first embodiment receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquiring unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 8(B), the time-series image data correction unit 12 corrects image data taken at position P1 (time t+1) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 8(B), the time-series image data correction unit 12 corrects image data taken at position P2 (time t+2) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 8(B), each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and image analysis can output an image analysis result C1 indicating that "the person 41 is moving" (or "the person 41 is running" if the person is moving fast), just as in the case where a fixed camera is photographing the same area. In other words, the image analyzing device 1 can perform image analysis of the state of the person 41 as the target with high accuracy.In operation example #2 of Figure 8(A), the actual shooting positions approach person 41 in the order of positions P0, P1, and P2, so as shown in Figure 8(B), the size of the image based on the corrected image data at position P1 is smaller than that at position P0, and the size of the image based on the corrected image data at position P2 is smaller than that at position P1.

[0026] 1-5 Comparative Example #3, Operation Example #3 9A and 9B are diagrams showing, with respect to Comparative Example #3, how a moving imaging device 10 captures an image of an area through which multiple moving people 42 pass, and the time-series image data obtained when the area through which the multiple people 42 pass is captured. As shown in FIG. 9A, when a moving crowd of multiple people 42 is captured by the moving imaging device 10, not only the multiple people 42 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the captured area also moves. Therefore, the image analysis device of Comparative Example #3, which performs image analysis using time-series image data captured by the imaging device 10 at times t, t+1, t+2, ..., may generate an image analysis result indicating that "the positions of the captured multiple people have not moved," as shown in FIG. 9B. 9(B), when a moving imaging device 10 is used, each piece of time-series image data captures a different area, so the image analysis device of Comparative Example #3 may output an image analysis result that "a crowd of people 42 is stagnating in the aisle" through image analysis, even though the crowd of people 42 is actually moving down the aisle. In other words, the image analysis device of Comparative Example #3 has low accuracy in image analysis of the state of the target people 42.

[0027] 10A and 10B are diagrams illustrating, in operation example #3 of the image analyzing device 1 according to the first embodiment, how an area through which a plurality of moving people 42 pass is imaged by a moving imaging device 10, and time-series corrected image data when the area through which the plurality of people 42 pass is imaged. As shown in FIG. 10A, when the plurality of moving people 42 are imaged by the moving imaging device 10, not only the plurality of people 42 but also the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the imaged area also moves. The time-series image data correcting unit 12 of the image analyzing device 1 according to the first embodiment receives time-series image data A0 imaged by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquiring unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 10(B), the time-series image data correction unit 12 corrects image data taken at position P1 (time t+1) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 10(B), the time-series image data correction unit 12 corrects image data taken at position P2 (time t+2) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 10(B), each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and image analysis can output an image analysis result C1 that "multiple people 42 are moving," i.e., "the crowd is not stationary," just as in the case where a fixed camera is photographing the same area. In other words, the image analyzing device 1 can perform image analysis of the state of multiple people 42 as targets with high accuracy.

[0028] 1-6 Comparative Example #4, Operation Example #4 11(A) and (B) are diagrams showing, with respect to Comparative Example #4, how a moving imaging device 10 captures an image of an area where a plurality of people 42 are passing through, and the time-series image data obtained when the area where the plurality of people 42 are passing through is captured. As shown in FIG. 11(A), when a plurality of people 43 who are part of a crowd are captured by the moving imaging device 10, the position of the imaging device 10 moves in the order of positions P0, P1, P2, ..., and the captured area moves. Therefore, the image analysis device of Comparative Example #4, which performs image analysis using time-series image data captured by the imaging device 10 at times t, t+1, t+2, ..., may generate an image analysis result indicating that "the positions of a plurality of people are moving," as shown in FIG. 11(B). 11(B), when a moving imaging device 10 is used, each piece of time-series image data captures a different area, so the image analysis device of Comparative Example #4 may output an erroneous image analysis result that "a crowd of people 43 is moving down the aisle" even though the crowd of people 43 is actually stationary in the aisle. In other words, the image analysis device of Comparative Example #4 has low accuracy in image analysis of the state of the target people 43.

[0029] 12(A) and 12(B) are diagrams illustrating operation example #4 of the image analyzing device 1 according to the first embodiment, showing a state in which a moving imaging device 10 captures an image of an area where a plurality of people 43, a crowd of people, pass through, and time-series corrected image data C0 obtained when the area where the plurality of people 43 pass through, is captured. As shown in FIG. 12(A), when the moving imaging device 10 captures an image of a plurality of people 43 that are not moving, the imaging device 10 moves in this order from position P0 to position P1, P2, ..., and the captured area moves. The time-series image data correcting unit 12 of the image analyzing device 1 according to the first embodiment receives time-series image data A0 captured by the moving imaging device 10 and time-series position and orientation information B0 acquired by the position and orientation information acquiring unit 20, and corrects the time-series image data A0 based on the time-series position and orientation information B0 to generate time-series corrected image data C0. 12(B), the time-series image data correction unit 12 corrects image data taken at position P1 (time t+1) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Similarly, as shown in FIG. 12(B), the time-series image data correction unit 12 corrects image data taken at position P2 (time t+2) of the imaging device 10, using image data taken at position P0 (time t) of the reference imaging device 10 as reference image data, to generate image data that would have been obtained if the imaging device 10 had taken the image at position P0 (i.e., image data equivalent to position P0). Next, the image analysis unit 13 analyzes the state of the object included in the time-series image data A0 based on the time-series corrected image data C0, and outputs an image analysis result C1 indicating the state of the object. 12(B), each of the time-series corrected image data C0 has the same position and orientation of the imaging device 10, and image analysis can output an image analysis result C1 indicating that "multiple people 43 are staying still," i.e., "the crowd is not moving," just as in the case where a fixed camera is photographing the same area. In other words, the image analyzing device 1 can perform image analysis of the state of multiple people 43 as targets with high accuracy.

[0030] 1-7 Effects According to the first embodiment, it is possible to perform image analysis of the state of a target with high accuracy based on time-series image data captured by a moving imaging device 10 and position and orientation information of the imaging device 10.

[0031] 2. Second Embodiment 2-1 Configuration FIG. 13 is a block diagram showing the configuration of an image analysis system 2a (including an image analysis device 2) according to the second embodiment. The image analysis device 2 is a device capable of implementing the image analysis method according to the second embodiment. The image analysis system 2a has an imaging device 10 such as a camera, an image storage unit 11 as a storage device that temporarily stores image data captured by the imaging device 10, an image analysis unit 14, and an analysis result correction unit 15. In the image analysis device 2 according to the second embodiment, components that are the same as or correspond to those of the image analysis device 1 according to the first embodiment are assigned the same reference numerals as those in the first embodiment.

[0032] The image analysis unit 14 receives time-series image data A0 captured by the moving imaging device 10, analyzes the state of the object contained in the time-series image data A0, and outputs an image analysis result D0 indicating the state of the object.

[0033] The analysis result correction unit 15 receives the image analysis result D0 and time-series position and orientation information B0 acquired by the position and orientation information acquisition unit 20, which acquires position and orientation information indicating the position and orientation of the image capture device 10, and generates a corrected image analysis result D1 by correcting the image analysis result D0 based on the time-series position and orientation information B0. For example, the analysis result correction unit 15 determines a reference time within the time when the time-series image data A0 was captured based on the time-series position and orientation information B0, and changes the image analysis result D0 indicating the state of the target to the image analysis result that would have been obtained if the image capture device 10 had been in the position at the reference time, thereby generating a corrected image analysis result D1.

[0034] 《2-2》Operation Fig. 14 is a flowchart showing the operation of the image analysis device 2 according to the second embodiment. As shown in step S21 of Fig. 14, the image analysis unit 14 receives time-series image data A0 captured by the moving imaging device 10, and analyzes the state of an object contained in the time-series image data A0 to generate an image analysis result D0 indicating the state of the object. Next, as shown in step S22, the analysis result correction unit 15 receives the image analysis result D0 and the time-series position and orientation information B0, and corrects the image analysis result D0 based on the time-series position and orientation information B0 to output a corrected image analysis result D1.

[0035] 2-3 Example of operation 15(A), (B), and (C) are diagrams illustrating operation example #5 of image analysis device 2 according to embodiment 2, showing how moving imaging device 10 captures an image of an area through which moving person 41 passes, time-series image data A0 obtained when capturing the image of the area through which person 41 passes, and correction of image analysis result D0. As shown in FIG. 15(A), when moving imaging device 10 captures an image of person 41 moving in the same direction as the imaging device 10, the position of imaging device 10 moves in the order of positions P0, P1, P2, ..., and the captured area moves. As shown in FIG. 15(B), image analysis unit 14 of image analysis device 2 according to embodiment 2 receives time-series image data A0 captured by moving imaging device 10, analyzes the state of person 41 based on the time-series image data A0, and generates image analysis result D0. 15(C), the analysis result correcting unit 15 of the image analyzing device 2 according to the second embodiment corrects the image analysis result D0 (e.g., "The person 41 is not moving") based on the time-series position and orientation information to generate a corrected image analysis result D1 (e.g., "The person 41 is moving"). In this way, the image analyzing device 2 according to the second embodiment can perform image analysis of the state of the person 41 as a target with high accuracy.

[0036] 2-4 Effect According to the second embodiment, it is possible to perform image analysis of the state of an object with high accuracy based on time-series image data captured by a moving imaging device and position and orientation information of the imaging device. [Explanation of symbols]

[0037] 1, 2 Image analysis device, 1a, 2a Image analysis system, 10 Imaging device, 11 Image storage unit, 12 Time series image data correction unit, 13 Image analysis unit, 14 Image analysis unit, 15 Analysis result correction unit, 20 Position and orientation information acquisition unit, 21 Position and orientation information storage unit, 31 Mobile robot (moving body), 41 Person (target), 42 Multiple people (moving crowd), 43 Multiple people (stationary crowd), A0 Time series image data, B0 Time series position and orientation information, C0 Time series corrected image data, C1 Image analysis result, D0 Image analysis result, D1 Corrected image analysis result.

Claims

1. a time-series image data correction unit that receives time-series image data captured by a moving image capturing device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the image capturing device, and corrects the time-series image data based on the time-series position and orientation information to generate time-series corrected image data; an image analysis unit that analyzes the behavior, movement, or moving or stationary state of a movable object included in the time-series image data based on the time-series corrected image data, and generates an image analysis result indicating the state of the object; An image analysis device comprising:

2. The time-series image data correction unit determines a reference time within a time when the time-series image data was captured based on the time-series position and orientation information, and generates the time-series corrected image data by changing each piece of the time-series image data to image data that would have been obtained if the imaging device had been located at the position at the reference time.

2. The image analysis device according to claim 1.

3. an image analysis unit that receives time-series image data captured by a moving imaging device, analyzes a state of an object included in the time-series image data, and outputs an image analysis result indicating the state of the object; an analysis result correction unit that receives the image analysis result and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device, and corrects the image analysis result based on the time-series position and orientation information to generate a corrected image analysis result; An image analysis device comprising:

4. The analysis result correction unit determines a reference time within the time when the time-series image data was captured based on the time-series position and orientation information, and generates a corrected image analysis result by changing the image analysis result indicating the state of the object to an image analysis result that would have been obtained if the imaging device had been located in the position at the reference time.

4. The image analysis device according to claim 3.

5. The time-series position and orientation information includes position information of the image capturing device and information indicating the image capturing direction.

5. The image analysis device according to claim 1, wherein the image analysis device is a computer.

6. The image analysis unit determines a human behavior as the state of the object.

5. The image analysis device according to claim 1, wherein the image analysis device is a computer.

7. The image analysis unit determines whether the person as the target is moving or stationary.

5. The image analysis device according to claim 1, wherein the image analysis device is a computer.

8. The image analysis unit determines whether the target crowd is stationary or moving.

5. The image analysis device according to claim 1, wherein the image analysis device is a computer.

9. a movable imaging device; a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device; a time-series image data correction unit that receives time-series image data captured by the moving imaging device and time-series position and orientation information acquired by the position and orientation information acquisition unit, and corrects the time-series image data based on the time-series position and orientation information to generate time-series corrected image data; an image analysis unit that analyzes the behavior, movement, or moving or stationary state of a movable object included in the time-series image data based on the time-series corrected image data, and generates an image analysis result indicating the state of the object; An image analysis system comprising:

10. The time-series image data correction unit determines a reference time within a time when the time-series image data was captured based on the time-series position and orientation information, and generates the time-series corrected image data by changing each piece of the time-series image data to image data that would have been obtained if the imaging device had been located at the position at the reference time.

10. The image analysis system according to claim 9.

11. a movable imaging device; a position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the imaging device; an image analysis unit that receives time-series image data captured by the moving imaging device, analyzes a state of an object included in the time-series image data, and outputs an image analysis result indicating the state of the object; an analysis result correcting unit that receives the image analysis result and time-series position and orientation information acquired by the position and orientation information acquiring unit, and corrects the image analysis result based on the time-series position and orientation information to generate a corrected image analysis result; An image analysis system comprising:

12. The analysis result correction unit determines a reference time within the time when the time-series image data was captured based on the time-series position and orientation information, and generates a corrected image analysis result by changing the image analysis result indicating the state of the object to an image analysis result that would have been obtained if the imaging device had been located in the position at the reference time.

12. The image analysis system according to claim 11.

13. The time-series position and orientation information includes position information of the image capturing device and information indicating the image capturing direction.

13. The image analysis system according to claim 9, wherein the image analysis system is a computer-readable medium.

14. The image analysis unit determines a human behavior as the state of the object.

13. The image analysis system according to claim 9, wherein the image analysis system is a computer-readable medium.

15. The image analysis unit determines whether the person as the target is moving or stationary.

13. The image analysis system according to claim 9, wherein the image analysis system is a computer-readable medium.

16. The image analysis unit determines whether the target crowd is stationary or moving.

13. The image analysis system according to claim 9, wherein the image analysis system is a computer-readable medium.

17. An image analysis method executed by an image analysis device that receives time-series image data captured by a moving image capture device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating a position and orientation of the image capture device, generating time-series corrected image data by correcting the time-series image data based on the time-series position and orientation information; generating an image analysis result indicating the state of a movable object by analyzing the behavior, movement, or moving or stationary state of the movable object included in the time-series image data based on the time-series corrected image data; An image analysis method comprising:

18. An image analysis program causing a computer as the image analysis device to execute the image analysis method according to claim 17.

19. An image analysis method executed by an image analysis device that receives time-series image data captured by a moving image capture device and time-series position and orientation information acquired by a position and orientation information acquisition unit that acquires position and orientation information indicating a position and orientation of the image capture device, analyzing a state of an object included in the time-series image data, and outputting an image analysis result indicating the state of the object; generating a corrected image analysis result by correcting the image analysis result based on the image analysis result and the time-series position and orientation information; An image analysis method comprising:

20. 20. An image analysis program causing a computer as the image analysis device to execute the image analysis method according to claim 19.

Images