Crowd flow analysis device, crowd flow analysis method, and program

Abstract

To appropriately grasp an average flow of a crowd from a time series static images while reducing a calculation load.SOLUTION: A crowd flow analysis device includes: an image acquisition unit which acquires time series static images obtained by photographing a plurality of objects; an image dividing unit which divides each of the time series static images acquired by the image acquisition unit into a plurality of areas; an object number calculation unit which detects objects included in each of the plurality of areas divided by the image dividing unit, and calculates the number of objects included in each of the plurality of areas; a moving velocity and direction calculation unit which calculates an average moving velocity and an average moving direction of objects included in each of the plurality of areas divided by the image dividing unit; and a crowd flow calculation unit which calculates an average flow of the crowd in the entire static images on the basis of the number of objects included in each of the plurality of areas calculated by the object number calculation unit and the average moving velocity and the average moving direction of objects included in each of the plurality of area calculated by the moving velocity and direction calculation unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a crowd flow analysis device, a crowd flow analysis method, and a program. [Background technology]

[0002] In smart city projects both in Japan and overseas, images taken by cameras installed both indoors and outdoors are used to digitize and visualize the positions and movements of pedestrians, vehicles, bicycles, etc. using image analysis technology based on artificial intelligence (AI), and the data is increasingly being used for various government services and commercial purposes.

[0003] FIG. 5 is a schematic diagram showing a conventional method for converting the flow of pedestrians and other people into digital data from images captured by a camera. In conventional methods such as those shown in Figure 5, the following (1) and (2) are mainly used (the following (1) and (2) use pedestrians as examples, but the principles are the same for cars and bicycles). (1) For each still image that makes up the captured video (= a series of still images), image analysis is performed using deep learning or other techniques to detect pedestrians and calculate their coordinates in real space from their position on the screen. (2) By identifying the pedestrians detected in (1) in each still image in the time series (= identifying them as the same pedestrian), the coordinates of each pedestrian at each time are obtained, and the movement path of each pedestrian is digitized from the time change. Methods for identifying pedestrians between each image include the following (2-1) and (2-2), and (2-1) and (2-2) can also be used in combination. (2-1) Compare the positions of pedestrians between consecutive images and identify nearby pedestrians as identical. (2-2) Identify the same pedestrian in images taken at different times based on features such as each pedestrian's clothing.

[0004] The above method recognizes each individual pedestrian and digitizes their movement path, making it possible to analyze the flow of each individual pedestrian in detail. On the other hand, when using this method to analyze the flow of a crowd, where several dozen pedestrians exist in the same image, the following <1> ~ <3> For this reason, the computational load increases. <1> As the number of pedestrians in an image increases, the computational load required to detect objects using deep learning, etc., increases. <2> When using the above method (2-1) to identify pedestrians between images, in order to accurately identify the same pedestrian based on their position in a crowded area, it is necessary to shorten the time interval between still images to be analyzed, which increases the amount of calculations per target period (5 to 10 frames / second). <3> When using the method described above in (2-2) to identify pedestrians between images, in addition to detecting pedestrians, a calculation process is required to identify the same pedestrian between images from the features of each pedestrian, which significantly increases the amount of calculations. Also, in cases where pedestrians in the image are crowded together and overlap each other, making it impossible to obtain a full-body image of each pedestrian, identification based on features may be difficult.

[0005] For this reason, for example, to analyze crowd flow in real time and use it for purposes such as evacuation guidance during disasters, high-performance computers that can handle large calculation loads are required. Also, cloud servers are often used to perform such high-speed processing, but images of pedestrians and others must be transmitted over the Internet, posing a risk to the protection of personal information.

[0006] Also, a video surveillance system that directly captures the macro behavior of a group, such as traffic flow, without capturing the micro behavior of individual vehicles, has been known (see Patent Document 1). In the technology described in Patent Document 1, an inter-frame average video is obtained by performing inter-frame averaging processing on traffic surveillance video acquired by a surveillance camera, which averages the image signal for each pixel of the frame image in the most recent pre-set time (frame) in order. In addition, in the technology described in Patent Document 1, the motion field (optical flow) within the video is calculated for the inter-frame average video to obtain the trajectory motion. However, the technology described in Patent Document 1 does not take into consideration how many vehicles' movements are collectively represented by one optical flow (movement vector). Therefore, the technology described in Patent Document 1 may not be able to adequately grasp the average flow of a crowd. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2010-140371 A Summary of the Invention [Problem to be solved by the invention]

[0008] In view of the above, an object of the present invention is to provide a crowd flow analysis device, a crowd flow analysis method, and a program that are capable of appropriately grasping average crowd flow from time-series still images while reducing the calculation load. [Means for solving the problem]

[0009] One aspect of the present invention is a crowd flow analysis device that includes an image acquisition unit that acquires time-series still images in which a plurality of objects are captured; an image division unit that divides each of the time-series still images acquired by the image acquisition unit into a plurality of regions; an object number calculation unit that detects objects included in each of the plurality of regions divided by the image division unit and calculates the number of objects included in each of the plurality of regions; a movement speed direction calculation unit that calculates an average movement speed and an average movement direction of the objects included in each of the plurality of regions divided by the image division unit; and a crowd flow calculation unit that calculates an average crowd flow throughout the still images based on the number of objects included in each of the plurality of regions calculated by the object number calculation unit and the average movement speed and average movement direction of the objects included in each of the plurality of regions calculated by the movement speed direction calculation unit.

[0010] One aspect of the present invention is a crowd flow analysis method comprising: an image acquisition step of acquiring time-series still images in which a plurality of objects are photographed; an image division step of dividing each of the time-series still images acquired in the image acquisition step into a plurality of regions; an object number calculation step of detecting objects included in each of the plurality of regions divided in the image division step and calculating the number of objects included in each of the plurality of regions; a movement speed direction calculation step of calculating an average movement speed and an average movement direction of the objects included in each of the plurality of regions divided in the image division step; and a crowd flow calculation step of calculating an average crowd flow throughout the still images based on the number of objects included in each of the plurality of regions calculated in the object number calculation step and the average movement speed and average movement direction of the objects included in each of the plurality of regions calculated in the movement speed direction calculation step.

[0011] One aspect of the present invention is a program for causing a computer to execute an image acquisition step of acquiring time-series still images in which a plurality of objects are photographed; an image division step of dividing each of the time-series still images acquired in the image acquisition step into a plurality of regions; an object number calculation step of detecting objects included in each of the plurality of regions divided in the image division step and calculating the number of objects included in each of the plurality of regions; a movement speed direction calculation step of calculating an average movement speed and an average movement direction of the objects included in each of the plurality of regions divided in the image division step; and a crowd flow calculation step of calculating an average crowd flow throughout the entire still images based on the number of objects included in each of the plurality of regions calculated in the object number calculation step and the average movement speed and average movement direction of the objects included in each of the plurality of regions calculated in the movement speed direction calculation step. Effect of the Invention

[0012] According to the present invention, it is possible to provide a crowd flow analysis device, a crowd flow analysis method, and a program that can appropriately grasp the average crowd flow from time-series still images while reducing the calculation load. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of a crowd flow analysis device according to a first embodiment. [Diagram 2] 4A to 4C are diagrams for explaining an example of a plurality of regions divided by an image dividing unit. [Diagram 3] 4 is a flowchart illustrating an example of processing executed in the crowd flow analysis device of the first embodiment. [Figure 4] FIG. 11 is a diagram for explaining an example of application of average crowd flow across an entire still image calculated in the crowd flow analysis device of the first embodiment. [Diagram 5] FIG. 1 is a diagram showing a schematic diagram of a conventional method for converting the flow of pedestrians and the like into digital data from images captured by a camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of a crowd flow analysis device, a crowd flow analysis method, and a program according to the present invention will be described with reference to the drawings.

[0015] [First embodiment] FIG. 1 is a diagram showing an example of a crowd flow analysis device 1 according to the first embodiment. In the example shown in FIG. 1, the crowd flow analysis device 1 comprises an image acquisition section 11, an image division section 12, an object number calculation section 13, a movement speed direction calculation section 14, and a crowd flow calculation section 15. The image acquisition unit 11 acquires still images in time series in which a plurality of objects are captured. The image acquisition unit 11 acquires a plurality of still images captured, for example, every few seconds. The plurality of objects are included in each of the plurality of still images. The image dividing section 12 divides each of the time-series still images acquired by the image acquiring section 11 into a plurality of regions.

[0016] 2A and 2B are diagrams for explaining an example of a plurality of regions divided by the image dividing unit 12. In detail, FIG. 2A shows an example of a plurality of regions divided by the image dividing unit 12. In the example shown in FIG. 2(A), a still image acquired by the image acquisition unit 11 (more specifically, a still image captured by a camera (not shown) at time t1) is divided into 96 (=8×12) regions by the image division unit 12. In another example, the still image acquired by the image acquisition unit 11 may be divided by the image division unit 12 into a number of regions other than 96.

[0017] In the example shown in FIG. 2(A), the still image acquired by the image acquisition unit 11 includes a plurality of people (specifically, pedestrians) as a plurality of objects. In another example, the still image acquired by the image acquisition unit 11 may include a plurality of runners, etc., as the plurality of objects.

[0018] In the example shown in Figure 1, the object number calculation unit 13 detects objects contained in each of the multiple areas divided by the image division unit 12, and calculates the number of objects contained in each of the multiple areas divided by the image division unit 12. 2(A), the object number calculation unit 13 calculates, for example, "1" as the number of objects included in the first region from the left and the first region from the top. Also, the object number calculation unit 13 calculates, for example, "0" as the number of objects included in the first region from the left and the sixth region from the top.

[0019] As shown in Fig. 2(A), the number of objects included in each of the 96 regions divided by the image dividing unit 12 is not limited to "1" or "0." In the example shown in Fig. 2(A), for example, the number of objects included in the first region from the right and second region from the top is "2." 1, the movement speed / direction calculation unit 14 calculates the average movement speed and average movement direction of the object included in each of the multiple regions divided by the image division unit 12. In detail, the movement speed / direction calculation unit 14 calculates the average movement speed and average movement direction of the object included in each of the multiple regions based on the change in the feature amount in each of the multiple regions divided by the image division unit 12.

[0020] In the example shown in Figures 2(A) and 2(B), the movement speed / direction calculation unit 14 calculates the average movement speed and average movement direction of objects included in each of the 96 regions (for example, the average movement speed and average movement direction of two pedestrians included in the first region from the right and second region from the top). Fig. 2(B) shows, by vectors (arrows), the average movement speed and average movement direction of objects (pedestrians) included in each of the 96 regions calculated by the movement speed / direction calculation unit 14. In detail, Fig. 2(B) shows, by vectors, the average movement speed and average movement direction of objects included in each of the 96 regions constituting a still image captured at time t1, the average movement speed and average movement direction of objects included in each of the 96 regions constituting a still image captured at time t2 after time t1, and the average movement speed and average movement direction of objects included in each of the 96 regions constituting a still image captured at time t3 after time t2.

[0021] In the example shown in Figure 1, the crowd flow calculation unit 15 calculates the average crowd flow throughout the entire still image based on the number of objects included in each of the multiple areas calculated by the object number calculation unit 13 and the average movement speed and average movement direction of the objects included in each of the multiple areas calculated by the movement speed direction calculation unit 14.

[0022] In the example shown in Figures 2(A) and 2(B), the crowd flow calculation unit 15 calculates the average crowd flow throughout the entire still image taken at time t1 based on the number of objects contained in each of the 96 regions constituting the still image taken at time t1 and the average movement speed and average movement direction of the objects contained in each of the 96 regions constituting the still image taken at time t1. In addition, the crowd flow calculation unit 15 calculates the average crowd flow throughout the entire still image taken at time t2 based on the number of objects included in each of the 96 areas constituting the still image taken at time t2 and the average movement speed and average movement direction of the objects included in each of the 96 areas constituting the still image taken at time t2, and calculates the average crowd flow throughout the entire still image taken at time t3 based on the number of objects included in each of the 96 areas constituting the still image taken at time t3 and the average movement speed and average movement direction of the objects included in each of the 96 areas constituting the still image taken at time t3.

[0023] For example, when calculating the average crowd flow in the entire still image captured at time t3, the crowd flow calculation unit 15 makes the weight of the vector in an area where the number of objects is "1" (for example, the area first from the left and sixth from the top in Figures 2(A) and 2(B)) different from the weight of the vector in an area where the number of objects is "2" (for example, the area first from the right and second from the top in Figures 2(A) and 2(B)). In detail, the crowd flow calculation unit 15 makes the weight of the vector in an area where the number of objects is "2" greater than the weight of the vector in an area where the number of objects is "1". This makes it possible to calculate the average crowd flow more appropriately than when the number of objects in an area is not taken into consideration. Furthermore, the crowd flow calculation unit 15 can also calculate changes in the average crowd flow across all of the still images taken at time t1, the average crowd flow across all of the still images taken at time t2, and the average crowd flow across all of the still images taken at time t3.

[0024] As described above, in the crowd flow analysis device 1 of the first embodiment, the image division section 12 divides a target image (a still image in which a plurality of targets are captured) into a plurality of regions, such as a grid. The object number calculation unit 13 detects and counts objects (such as pedestrians) using still images sampled at regular time intervals. The moving speed direction calculation unit 14 detects a moving vector of a feature point in a still image by using optical flow or the like. The crowd flow calculation unit 15 calculates the average object density and flow (speed and direction) for each region by combining the results of processing by the object number calculation unit 13 and the results of processing by the movement speed direction calculation unit 14. Furthermore, the crowd flow calculation unit 15 models the crowd flow of the entire still image by integrating the average object density and flow (speed and direction) for each region.

[0025] In the example shown in FIG. 2(A), the image dividing unit 12 divides a still image into, for example, 96 regions in a grid pattern. The object number calculation unit 13 detects objects (e.g., pedestrians) as shown by rectangles in each region in FIG. 2(A) by using the above-mentioned deep learning method or other methods, and counts the number of objects in each region. Although it depends on the speed and density of the crowd flow, the object number calculation unit 13 executes this process about once every few seconds. Therefore, it is possible to reduce the detection frequency compared to the conventional method (5 to 10 frames / second) shown in FIG. 5, and the calculation load can be reduced. In addition, since only the number of objects needs to be known in the process by the object number calculation unit 13, it is not necessary to identify the objects by features or the like, and if the objects are pedestrians, only the faces of the pedestrians may be detected. When only the faces of the pedestrians are detected, the calculation load can be reduced compared to when the entire bodies of the pedestrians are detected, and even when the pedestrians are crowded together and the entire body image of the pedestrians cannot be obtained, the pedestrians can be detected with high accuracy (the number of pedestrians can be calculated with high accuracy).

[0026] Motion detection technology using optical flow as shown in FIG. 2(B) is a method for detecting "moving objects" from changes in images of a video (= still images in a time series), and is also implemented in OpenCV. The movement speed direction calculation unit 14 captures the movement of an object from changes in the feature amount of the image, so the calculation load can be reduced compared to methods for detecting objects using deep learning, etc. The movement speed direction calculation unit 14 calculates the average flow direction and speed from the feature amount within the screen in each of the multiple areas divided by the image division unit 12. Although it depends on the speed of the target crowd flow, the movement speed direction calculation unit 14 executes this process about once per second.

[0027] As described above, the object number calculation unit 13 calculates the number of objects (e.g., pedestrians, etc.) contained in each of the multiple areas, and the movement speed / direction calculation unit 14 calculates the average movement speed and average movement direction of the objects (e.g., pedestrians, etc.) contained in each of the multiple areas. Crowd flow calculation unit 15 calculates the average flow of objects included in each of the multiple regions by integrating the results of processing by object number calculation unit 13 and the results of processing by movement speed direction calculation unit 14. Furthermore, crowd flow calculation unit 15 calculates the average crowd flow for the entire still image by integrating the average flows of objects included in each of the multiple regions.

[0028] The crowd flow analysis device 1 of the first embodiment can significantly reduce the calculation load compared to existing methods for detecting the individual movement lines of objects. Therefore, the crowd flow analysis device 1 of the first embodiment can process real-time images in real time using a simpler calculation environment compared to existing methods. The data obtained by the crowd flow analysis device 1 of the first embodiment is the average crowd flow (density and speed) within the image, and is therefore not suitable for tracking the movements of individual pedestrians, but is useful for real-time analysis of large-scale crowd flow and guidance based on the results of that analysis. In fact, when estimating the number of people visiting shrines for the New Year, a method is used to calculate the total number of worshippers from visually checking the density of pedestrians and the flow speed at measurement points, and the crowd flow analysis device 1 of the first embodiment can be said to be a method in which image processing and AI are applied to this method.

[0029] FIG. 3 is a flowchart for explaining an example of the processing executed in the crowd flow analysis device 1 of the first embodiment. In the example shown in FIG. 3, in step S11, the image acquisition unit 11 acquires still images in time series in which a plurality of objects are photographed. Next, in step S12, the image dividing section 12 divides each of the time-series still images acquired in step S11 into a plurality of regions. Next, in step S13, the object number calculation unit 13 detects objects included in each of the multiple regions divided in step S12, and calculates the number of objects included in each of the multiple regions divided in step S12. In step S14, the average moving speed and the average moving direction of the object included in each of the multiple regions divided in step S12 are calculated. Next, in step S15, the crowd flow calculation unit 15 calculates the average crowd flow throughout the entire still image based on the number of objects included in each of the multiple regions calculated in step S13 and the average movement speed and average movement direction of the objects included in each of the multiple regions calculated in step S14.

[0030] 4 is a diagram for explaining an example of application of average crowd flow in an entire still image calculated by the crowd flow analysis device 1 of the first embodiment. In detail, it shows an example in which the people flow calculated by the crowd flow analysis device 1 of the first embodiment is visualized. In the example shown in Fig. 4, the crowd flow analysis device 1 of the first embodiment calculates a "small movement / stagnant" people flow based on the time-series still images (camera images) shown in the upper left of Fig. 4, and the direction and density of the people flow are visualized in a histogram. Moreover, the crowd flow analysis device 1 of the first embodiment calculates a "large movement (multi-directional)" people flow based on the time-series still images (camera images) shown in the lower left of Fig. 4, and the direction and density of the people flow are visualized in a histogram. Moreover, the crowd flow analysis device 1 of the first embodiment calculates a "large movement (one direction)" people flow based on the time-series still images (camera images) shown in the lower right of Fig. 4, and the direction and density of the people flow are visualized in a histogram.

[0031] In the crowd flow analysis device 1 of the first embodiment and its application examples, when analyzing the flow of crowds etc. by analyzing camera images, the average crowd flow (density, flow velocity, flow direction, etc.) can be digitized and visualized in a relatively simple calculation environment. Furthermore, the crowd flow analysis device 1 of the first embodiment can perform real-time analysis of crowd flow on-premise, enabling measures such as real-time guidance to be taken. Also, by processing images in real time and storing only the data, it becomes possible to process with more consideration given to personal information than methods of transferring images via the internet or recording them.

[0032] [Second embodiment] A second embodiment of the crowd flow analysis device, crowd flow analysis method, and program of the present invention will now be described. Except for the points described below, the crowd flow analysis apparatus 1 of the second embodiment is configured in the same way as the crowd flow analysis apparatus 1 of the first embodiment described above. Therefore, the crowd flow analysis apparatus 1 of the second embodiment can achieve the same effects as the crowd flow analysis apparatus 1 of the first embodiment described above, except for the points described below.

[0033] As described above, a plurality of people are included as a plurality of objects in the time-series still images acquired by the image acquisition section 11 of the crowd flow analysis device 1 of the first embodiment. On the other hand, a plurality of objects other than a plurality of people (e.g., cars, bicycles, animals, etc.) are included as a plurality of objects in the time-series still images acquired by the image acquisition section 11 of the crowd flow analysis device 1 of the second embodiment. As with the crowd flow analysis device 1 of the first embodiment, the crowd flow analysis device 1 of the second embodiment can also reduce the calculation load and appropriately grasp the average crowd flow from time-series still images.

[0034] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments and examples, and appropriate modifications can be made without departing from the spirit of the present invention. The configurations described in the above-mentioned embodiments and examples may be combined.

[0035] Incidentally, all or part of the functions of each unit of the crowd flow analysis device 1 in the above-mentioned embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and having a computer system read and execute the program recorded on this recording medium. Note that the term "computer system" here includes hardware such as the OS and peripheral devices. In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0036] 1 Crowd flow analysis device 11 Image acquisition section 12 Image division section 13 Object Count Calculation Section 14 Movement speed direction calculation section 15 Crowd Flow Calculation Unit

Claims

1. an image acquisition unit that acquires still images of a plurality of objects in a time series; an image division unit that divides each of the time-series still images acquired by the image acquisition unit into a plurality of regions; an object number calculation unit that detects objects included in each of the plurality of regions divided by the image division unit and calculates the number of objects included in each of the plurality of regions; a moving speed / direction calculation unit that calculates an average moving speed and an average moving direction of an object included in each of the plurality of regions divided by the image division unit; a crowd flow calculation unit that calculates an average crowd flow in the entire still image based on the number of objects included in each of the plurality of regions calculated by the object number calculation unit and the average movement speed and average movement direction of the objects included in each of the plurality of regions calculated by the movement speed direction calculation unit, The crowd flow analysis device, wherein the crowd flow calculation section weights the vectors of each of the divided multiple regions differently depending on the number of objects contained in each of the divided regions.

2. the plurality of objects are a plurality of pedestrians, the object number calculation unit calculates the number of pedestrians included in each of the plurality of regions divided by the image division unit by detecting only faces of pedestrians included in each of the plurality of regions.

2. The crowd flow analysis device according to claim 1.

3. the moving speed / direction calculation unit calculates an average moving speed and an average moving direction of an object included in each of the plurality of regions based on a change in a feature amount in each of the plurality of regions; 2. The crowd flow analysis device according to claim 1.

4. An image acquisition step of acquiring still images of a plurality of objects in time series; an image division step of dividing each of the time-series still images acquired in the image acquisition step into a plurality of regions; an object number calculation step of detecting objects included in each of the plurality of regions divided in the image division step, and calculating the number of objects included in each of the plurality of regions; a moving speed / direction calculation step of calculating an average moving speed and an average moving direction of an object included in each of the plurality of regions divided in the image dividing step; a crowd flow calculation step of calculating an average crowd flow in the entire still image based on the number of objects included in each of the plurality of regions calculated in the object number calculation step and the average moving speed and average moving direction of the objects included in each of the plurality of regions calculated in the moving speed direction calculation step, A crowd flow analysis method, wherein in the crowd flow calculation step, weights of vectors of each of the divided multiple regions are made different depending on the number of objects contained in each of the divided regions.

5. On the computer, An image acquisition step of acquiring still images of a plurality of objects in time series; an image division step of dividing each of the time-series still images acquired in the image acquisition step into a plurality of regions; an object number calculation step of detecting objects included in each of the plurality of regions divided in the image division step, and calculating the number of objects included in each of the plurality of regions; a moving speed / direction calculation step of calculating an average moving speed and an average moving direction of an object included in each of the plurality of regions divided in the image dividing step; and a crowd flow calculation step of calculating an average crowd flow throughout the still image based on the number of objects included in each of the plurality of regions calculated in the object number calculation step and the average movement speed and average movement direction of the objects included in each of the plurality of regions calculated in the movement speed direction calculation step, by varying the weight of the vector of each of the plurality of divided regions depending on the number of objects included in each of the plurality of regions.

Images