Crowd density prediction program, crowd density prediction method, and crowd density prediction device

Abstract

To solve a problem that technologies have been developed for calculation and monitoring of crowd density in video surveillance, for example, a technology has been developed that calculates the flow of a crowd from a surveillance image captured by a surveillance camera and estimates a crowd density outside a surveillance camera's shooting range based on the crowd flow, however, the technology for estimating the crowd density outside the surveillance camera's shooting range does not take into account, for example, crowds flowing into the shooting range from outside the shooting range, so the accuracy of estimating the crowd density is not sufficient.SOLUTION: A crowd density prediction program causes a computer to execute processing for calculating a movement vector of a person in a video captured by an imaging device based on positional information of the person, calculating second positional information of the person after a predetermined time has elapsed based on first positional information of the person and the movement vector, and calculating a crowd density of a predetermined section based on the second positional information.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present invention relates to a crowd density prediction program, a crowd density prediction method, and a crowd density prediction device for performing video surveillance.

Background Art

[0002] Currently, in video surveillance technology, techniques for calculating and monitoring crowd density have been developed. For example, by using the video obtained from a surveillance camera to calculate the number of people in a specific area, the crowd density, etc., it is used for crowd control. Note that the crowd density can be utilized not only for crowd control but also, for example, when a user who intends to visit a specific area grasps the congestion situation. Here, the crowd density, also called the cluster density, etc., is the number of people per predetermined occupied area and can be calculated by dividing the number of people in the predetermined occupied area by the area.

[0003] In addition, techniques have been developed for calculating the flow of a crowd from a surveillance image captured by a surveillance camera and estimating the crowd density outside the shooting range of the surveillance camera based on the flow of the crowd.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology for estimating the crowd density outside the shooting range of a surveillance camera, for example, crowds flowing from outside the shooting range into the shooting range are not considered, so the estimation accuracy of the crowd density cannot be said to be sufficient.

[0006] On one aspect, an object is to provide a crowd density prediction program, a crowd density prediction method, and a crowd density prediction device that improve the accuracy of predicting the crowd density from an image.

Means for Solving the Problem

[0007] In one aspect, the crowd density prediction program causes a computer to execute a process of calculating a movement vector of a person based on the position information of the person in the video captured by an imaging device, calculating the second position information of the person after a predetermined time based on the first position information of the person and the movement vector, and calculating the crowd density of a predetermined section based on the second position information.

Effect of the Invention

[0008] On one aspect, the accuracy of predicting the crowd density from an image can be improved.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0010] Hereinafter, examples of the crowd density prediction program, the crowd density prediction method, and the crowd density prediction device according to the present embodiment will be described in detail with reference to the drawings. Note that the present embodiment is not limited by this example. Also, the respective examples can be appropriately combined within a non - conflicting range.

[0011] (Overall Configuration) First, an information processing system for implementing the present embodiment will be described. FIG. 1 is a diagram showing a configuration example of the information processing system 1 according to the present embodiment. As shown in FIG. 1, the information processing system 1 is a system in which a crowd density prediction device 10, camera devices 100 - 1 to 100 - n, and information processing terminals 200 are connected to be mutually communicable via a network 50. Here, n of the camera device 100 - n is an arbitrary natural number, and the camera devices 100 - 1 to 100 - n may be collectively referred to as the "camera device 100".

[0012] The network 50 can adopt various communication networks such as the Internet and intranet, regardless of whether they are wired or wireless. Further, the network 50 is not a single network, and for example, the Internet and the intranet may be configured via a network device such as a gateway or other devices (not shown).

[0013] The crowd density prediction device 10 is an information processing device such as a desktop PC (Personal Computer), a notebook PC, or a server computer, which is managed by, for example, a monitor who performs video surveillance. Alternatively, the crowd density prediction device 10 may be a cloud computer device managed by a service provider who provides cloud computing services.

[0014] The crowd density prediction device 10 receives, for example, a video of a predetermined shooting range such as in front of a station, a park, or a footbridge, which has been shot by the camera device 100, from the camera device 100. Strictly speaking, the video is composed of a plurality of images shot by the camera device 100, that is, a series of frames of a moving picture.

[0015] Further, the crowd density prediction device 10 detects an object including a person from the video shot by the camera device 100 using, for example, an existing object (also referred to as an object) detection technique. Here, an object other than a person is an object having a size that can affect the crowd density, such as a carry bag or a baby stroller. Further, the crowd density prediction device 10 calculates, for example, the movement vectors of the detected person and objects other than the person. In order to distinguish the movement vector of a person from that of an object other than the person, the movement vector of a person may be referred to as a "walking vector". Further, the crowd density prediction device 10 calculates, for example, the future position information of a person after a predetermined time such as 5 minutes based on the current position information and movement vector of the person or the like. Then, the crowd density prediction device 10 calculates the crowd density of a predetermined section based on, for example, the future position information of a person or the like.

[0016] In FIG. 1, the crowd density prediction device 10 is shown as a single computer, but it may be a distributed computing system composed of multiple computers.

[0017] The camera device 100 is, for example, an imaging device such as a surveillance camera installed at locations where crowds may occur, such as in front of a station, a park, or a footbridge. Note that the camera device 100 may be, for example, an elevated fixed-point camera or a drone, etc., but is not limited thereto. Also, the video captured by the camera device 100 is transmitted to the crowd density prediction device 10.

[0018] The information processing terminal 200 is, for example, an information processing terminal such as a desktop PC, a notebook PC, or a mobile terminal such as a smartphone or a tablet PC used by a monitor who performs video surveillance, a user who grasps the congestion situation, etc. The monitor, etc., via the information processing terminal 200, for example, confirms the crowd density of a predetermined section displayed together with a map, etc., by the crowd density prediction device 10, and grasps the congestion situation of the predetermined section. In FIG. 1, the information processing terminal 200 is shown as a single computer, but there may be multiple ones for each user, for example.

[0019] (Functional Configuration of the Crowd Density Prediction Device 10) Next, the functional configuration of the crowd density prediction device 10 will be described. FIG. 2 is a diagram showing a configuration example of the crowd density prediction device 10 according to the present embodiment. As shown in FIG. 2, the crowd density prediction device 10 has a communication unit 11, a storage unit 12, and a control unit 20.

[0020] The communication unit 11 is a processing unit that controls communication with other devices such as the camera device 100 and the information processing terminal 200, and is, for example, a communication interface such as a network interface card.

[0021] The storage unit 12 has a function of storing various data and programs executed by the control unit 20, and is realized by a storage device such as a memory or a hard disk, for example. The storage unit 12 stores a video DB 13, an area DB 14, an object DB 15, a crowd flow DB 16, a crowd density DB 17, etc. Note that DB is an abbreviation for Data Base.

[0022] The video DB 13 stores videos captured by the camera device 100 and information related to such videos. The video captured by the camera device 100 is, for example, transmitted from the camera device 100 at any time, received by the crowd density prediction device 10, and stored in the video DB 13. Also, the information related to the video stored in the video DB 13 is, for example, the shooting date and time of the video, an identifier indicating which camera device 100 captured it, etc. FIG. 3 is a diagram showing an example of the shooting range according to this embodiment. FIG. 3 shows the shooting range 60-1 of the camera device 100-1 installed on the footbridge, the shooting range 60-2 of the camera device 100-2 installed in front of the station, and the shooting range 60-3 of the camera device 100-3 installed in the park. As shown in FIG. 3, since there may be a plurality of camera devices 100, an identifier indicating which camera device 100 captured the video may be stored in the video DB 13 together with the video.

[0023] Returning to the description of FIG. 2, the area DB 14 stores information regarding, for example, the shooting range of the camera device 100 and sections that are units for predicting the crowd density within the shooting range. FIG. 4 is a diagram showing an example of a prediction range according to the present embodiment. As shown in FIG. 4, for example, a passage or a footbridge within the shooting range 60-1 is specified as the prediction range 70, and based on the future position information of people currently in the shooting ranges 60-1 to 60-3, the crowd density of the prediction range 70 is predicted. Note that the prediction range 70 may be further divided into, for example, three sections, and each of the subdivided sections or the entire prediction range 70 may be regarded as one section, and the crowd density may be predicted in section units. Further, the area DB 14 may store the sizes of the shooting ranges 60-1 to 60-3, the prediction range 70, and each section, such as width X meters × total length Y meters. Further, the area DB 14 may store, for example, coordinate information for specifying a section from the video captured by the camera device 100.

[0024] Further, for example, the crowd density of the prediction range 70 may be predicted based on the future position information of people currently in the shooting ranges 60-2 and 60-3 as well as the shooting range 60-1. That is, for example, when calculating the crowd density, a person in the prediction range 70 may be in any of the shooting ranges 60-1 to 3 at the prediction stage. Therefore, the prediction accuracy can be improved by predicting the crowd density based on the future position information of people captured by a plurality of camera devices 100.

[0025] Returning to the description of FIG. 2, the object DB 15 stores information regarding objects such as people and luggage that can be detected from the video captured by the camera device 100. FIG. 5 is a diagram showing an example of a prediction target according to the present embodiment. As shown in FIG. 5, the prediction range 70 may include not only people but also objects (referred to as large luggage in FIG. 5) having a size that can affect the crowd density, such as a carry bag or a baby stroller. Therefore, objects other than people may be regarded as people, and the crowd density may be predicted including not only people but also objects other than people.

[0026] FIG. 6 is a diagram showing an example of data stored in the object DB 15 according to the present embodiment. As shown in FIG. 6, in the object DB 15, for example, an object number uniquely indicating an object, a detected object, a crowd count amount indicating a coefficient used for calculating the crowd density, and the like may be associated and stored. Here, as shown in FIG. 6, the crowd count amount is set to 1 for a person, for example, 0.75 for a carry bag or a baby stroller smaller than a person, and is used for counting the number of people when calculating the crowd density, so that a more accurate crowd density can be calculated. More specifically, for example, when there is one baby stroller in the prediction range 70, the number of people is counted as if there are 0.75 people, and the crowd density is calculated. Also, 0.75 of the crowd count amount is an example, and a value smaller than that may be used, or in the case of an object having a larger occupied area than a person, such as a double baby stroller, a value larger than 1 may be used.

[0027] Returning to the description of FIG. 2, the people flow DB 16 stores information regarding, for example, the position and movement vector of a person detected from the video captured by the camera device 100 for each time. Here, the movement vector will be described more specifically. FIG. 7 is a diagram showing an example of the movement vector according to the present embodiment. FIG. 7 is an example in which the position of a person moving within the shooting range of the camera device 100 every second is indicated by a circle and the movement direction is indicated by an arrow.

[0028] The method for calculating the movement vector is, for example, to obtain the positions every second until 5 seconds later, and calculate the differences in positions from the current location 1 second later, from 1 second later to 2 seconds later, ··· from 4 seconds later to 5 seconds later, that is, the vectors every second are calculated. Then, for example, in a crowded situation, since a person may stop or move repeatedly, the average of the vectors every second is calculated and used as the movement vector. Also, for example, it is assumed that a person moves while maintaining a walking vector for 5 minutes, and the position of the person 5 minutes later is calculated using the current position and the movement vector. For example, in the case of the position of the person 5 minutes later, it can be calculated by the formula "current position + (movement vector × 300 seconds)". For objects other than people, in the same way as for people, the movement vector for the object can be calculated to predict the future position. Note that the 5 seconds used for calculating the movement vector and the 5 minutes later, which is the time of the predicted destination of the position, are just examples and are not limited to these.

[0029] FIG. 8 is a diagram showing an example of data stored in the crowd flow DB 16 according to the present embodiment. As shown in FIG. 8, in the crowd flow DB 16, for example, a person number uniquely indicating a person, a current location indicating the current position of the person and camera coordinates, and camera coordinates and planar coordinates at each time may be stored in association with each other. Further, in the crowd flow DB 16, a walking vector indicating the movement vector of the person, a route direction such as a passage where the person is located, predicted coordinates five minutes after correction and its correction method, a predicted location five minutes later indicating the position where the person will be five minutes later, and the like may be stored in association with each other. Here, the current location can be determined, for example, by the installation location of the camera device 100 where the person is photographed, the camera coordinates in the shooting range of the camera device 100, or the like. The camera coordinates may be, for example, XY coordinates when the upper left of the image of the camera device 100 is the origin (0, 0). The planar coordinates may be, for example, coordinates indicating whether it is x meters to the right and y meters below from the start point, which is the upper left of the image of the camera device 100. The planar coordinates can be converted from the camera coordinates, for example, based on the actual length of one pixel in the image. More specifically, for example, the actual length of one pixel can be calculated using the number of pixels from the origin of the image to the end point at the diagonal position from the origin and the actual length from the origin of the image to the end point. For example, when the number of pixels from the origin to the end point is 500 pixels and the actual length is 62.5 meters, the actual length per pixel is 0.125 meters. Then, the planar coordinates can be converted by multiplying the actual length per pixel by the camera coordinates.

[0030] Also, the predicted coordinates five minutes before and after correction may be the camera coordinates of a person five minutes later calculated based on a movement vector corrected so as to fall within a range for predicting crowd density, for example. Details of the correction of the movement vector will be described later. Also, in the example of FIG. 8, only data regarding the position and movement vector of a person are shown in the crowd flow DB 16, but the crowd flow DB 16 may store data regarding the position and movement vector of objects other than persons, for example. In this case, for example, the person number and walking vector in the item name of the crowd flow DB 16 may be changed to an object number, a movement vector, etc., respectively.

[0031] Returning to the description of FIG. 2, the crowd density DB 17 stores information regarding, for example, the predicted crowd density and the display of the crowd density. Here, the predicted crowd density is, for example, the number of persons per predetermined occupied area calculated from the future position of a person calculated based on a movement vector.

[0032] Also, the display of the predicted crowd density is, for example, the display of the crowd density of a predetermined section displayed in conjunction with a map. FIG. 9 is a diagram showing an example of the display of the crowd density according to the present embodiment. The example of FIG. 9 is the display of the crowd density five minutes later for each of the shooting ranges 60-1 to 60-3 calculated from the position of a person five minutes later. Note that the calculated and displayed crowd density is not limited to that for each shooting range, and may be the crowd density of a passage within the shooting range or a part of a further subdivided passage.

[0033] FIG. 10 is a diagram showing another example of the display of the crowd density according to the present embodiment. The example of FIG. 10 is a representation in shades of gray of the future crowd density of the prediction range 70 shown in FIG. 4 etc. as a shade display 80. The shade display 80 of the crowd density may be, for example, a heat map that displays a higher density part darker and a lower density part lighter. Note that the shade display of the crowd density is not limited to that of the prediction range 70, and may be, for example, the crowd density for each section obtained by further subdividing the prediction range 70.

[0034] Returning to the description of FIG. 2, the information stored in the storage unit 12 is merely an example, and the storage unit 12 can store various information other than the above information.

[0035] The control unit 20 is a processing unit that controls the entire crowd density prediction device 10, and is, for example, a processor or the like. The control unit 20 includes a detection unit 21, a calculation unit 22, a correction unit 23, and a display unit 24. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0036] The detection unit 21, for example, acquires an image of a station front, a park, a footbridge, etc. taken by the camera device 100 from the video DB 13, and uses an existing object detection technique to detect people and objects other than people from the image. Here, the existing object detection techniques are, for example, YOLO (YOU Only Look Once), SSD (Single Shot Multibox Detector), RCNN (Region Based Convolutional Neural Networks), etc. Also, the detection of people by the detection unit 21 may be, for example, the detection of each person every second in a 5-second video in order to calculate a motion vector. Further, the detection unit 21 may specify, for example, the camera coordinates of feature points such as the head and feet of the detected person as the position information of the person. Alternatively, when a person or the like is detected in a region such as a bounding box, the camera coordinates of a point such as the center of the region may be specified as the position information. Also, as described with reference to FIG. 8, the camera coordinates may be converted into plane coordinates by multiplying the actual length per pixel, and the plane coordinates may be specified as the position information.

[0037] The calculation unit 22 calculates, for example, the motion vector of a person based on the position information of the person in the video taken by the camera device 100. More specifically, the calculation unit 22 calculates, for example, a vector every second based on the position information of the person detected by the detection unit 21, and calculates the average of the vectors every second, and this may be used as the motion vector, as described with reference to FIG. 7.

[0038] Further, the calculation unit 22 calculates the second position information of the person after a predetermined time, for example, based on the first position information with the current position (reference position) of the person and the calculated movement vector. Here, after the predetermined time may be, for example, 5 minutes later. Assuming that the movement vector is maintained and the person moves for 5 minutes, the calculation unit 22 calculates the second position information of the person after 5 minutes using the formula "first position information + (movement vector × 300 seconds)". Also, the movement vector used for calculating the second position information may be the movement vector corrected by the correction unit 23. That is, the calculation unit 22 calculates the second position information of the person after a predetermined time, for example, based on the first position information of the person and the corrected movement vector.

[0039] Further, the calculation unit 22 calculates the crowd density of a predetermined section, for example, based on the calculated second position information. Here, for example, when the second position information is the second position information of the person after 5 minutes, the calculated crowd density is also the crowd density after 5 minutes.

[0040] Further, the calculation unit 22 calculates the second movement vector of an object other than the person, for example, based on the position information of the object other than the person in the video captured by the camera device 100. Here, the object other than the person is an object having a size that can affect the crowd density, such as a carry bag or a baby stroller. Also, the method for calculating the second movement vector of the object other than the person is the same as the method for calculating the movement vector of the person.

[0041] Further, the calculation unit 22 calculates the fourth position information of the object other than the person after a predetermined time, for example, based on the third position information of the object other than the person and the second movement vector. Here, the method for calculating the fourth position information of the object other than the person after a predetermined time is the same as the method for calculating the second position information of the person after a predetermined time.

[0042] Further, the calculation unit 22 calculates the crowd density of a predetermined section based on, for example, the second position information, the fourth position information, and a predetermined coefficient corresponding to the object. Here, for example, when the second position information and the fourth position information are the second position information of the person after 5 minutes and the fourth position information of the object other than the person, respectively, the calculated crowd density is also the crowd density after 5 minutes. Further, the predetermined coefficient corresponding to the object is, for example, the crowd count amount shown in FIG. 6. Therefore, the calculation unit 22 counts the object other than the person as the number of people by assuming, for example, 1 person × 1 = 1 person for a person and 1 stroller × 0.75 = 0.75 person for an object other than the person, and calculates the crowd density. In this way, by calculating the crowd density including not only people but also objects other than people, a more accurate crowd density can be calculated. In particular, a stroller or a carry bag may move separately from the original person, for example, when moving together with a person, when placed in a stroller parking area, or when handed over to another person. Therefore, a more accurate crowd density can be calculated by calculating the movement vector independently of the person and calculating the crowd density.

[0043] Note that the process of calculating the crowd density based on the second position information and the fourth position information may include the process of calculating the crowd density of a predetermined section within a shooting range different from the shooting range of the camera device 100 that has shot a person or an object other than the person. That is, for example, as described with reference to FIG. 4, the crowd density of the prediction range 70 may be calculated based on the future position information of a person currently in the shooting range 60-2 or 60-3 as well as the shooting range 60-1 including the prediction range 70.

[0044] For example, when the position indicated by the second position information does not belong to (is outside) the predetermined section where the crowd density is calculated, the correction unit 23 corrects either the length or the direction of the movement vector. The process of correcting either the length or the direction of the movement vector includes, for example, a process of correcting the length of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the direction of the movement vector. Or, the process of correcting either the length or the direction of the movement vector includes a process of correcting the direction of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the length of the movement vector. The correction of the movement vector will be described more specifically with reference to FIG. 11.

[0045] FIG. 11 is a diagram showing an example of the correction of the movement vector according to the present embodiment. In FIG. 11, for example, the current position of a person or the like is represented by a circle, the vector before correction is represented by a dashed arrow, the vector after correction is represented by a solid arrow, and five sections are represented by rectangular frames. Note that the tip of the vector represented by the dashed arrow or the solid arrow in FIG. 11 is the position of a person or the like indicated by the second position information or the fourth position information after a predetermined time has elapsed, such as after 5 minutes. And, for example, assuming that there is a passage with the vertical direction as the route direction as shown in FIG. 11, since a person or the like does not move outside the passage, it is assumed that the position indicated by the second position information falls within one of the sections. Therefore, for example, if the angle formed by the route direction and the vector of a person or the like is less than a specified degree such as 30°, the correction unit 23 does not change the length of the vector, and corrects the direction of the movement vector of the person or the like, such as a, so that the direction of the vector falls within the section in the route direction. Or, for example, if the angle formed by the route direction and the vector of a person or the like is equal to or greater than the specified degree, the correction unit 23 does not change the direction of the vector, and corrects the direction of the movement vector of the person or the like, such as b, so that the length of the vector falls within the section. Then, for example, future position information of a person or the like is calculated from the corrected movement vector and the current position information, and the crowd density is calculated.

[0046] The display unit 24 displays, for example, the crowd density calculated by the calculation unit 22 and a map including a predetermined section. More specifically, as described with reference to FIGS. 9 and 10, for example, the display unit 24 displays, together with the map, the crowd density of the predetermined section and the shading display of the crowd density. Note that the display unit 24 may display, as the display of the crowd density of the predetermined section, not only the predicted crowd density such as after 5 minutes, but also the current crowd density and its shading display. Further, the display unit 24 may display, for example, the number of people in the predetermined section together with the map. This is because, for example, when the areas of the respective sections of the predetermined section are the same, it may be sufficient to display the number of people without calculating and displaying the crowd density. Furthermore, the display unit 24 may display, for example, the number of people in the predetermined section separately as the number of people and the number of objects other than people. Note that, regarding the display of the number of people in the predetermined section, not only the predicted number of people such as after 5 minutes but also the current number of people may be displayed.

[0047] (Flow of processing) Next, the flow of the crowd density calculation process executed by the crowd density prediction apparatus 10 will be described. FIG. 12 is a flowchart showing the flow of the crowd density calculation process according to the present embodiment.

[0048] First, as shown in FIG. 12, the crowd density prediction apparatus 10 acquires, for example, an image of a predetermined shooting range such as in front of a station, a park, or a footbridge captured by the camera apparatus 100 from the video DB 13 (step S101). Note that the image captured by the camera apparatus 100 is transmitted from the camera apparatus 100 to the crowd density prediction apparatus 10 at any time and stored in the video DB 13.

[0049] Next, the crowd density prediction apparatus 10 detects people from the image acquired in step S101 using, for example, an existing object detection technique, and calculates the position coordinates (corresponding to the camera coordinates in FIG. 8) of each person every second (step S102).

[0050] Next, the crowd density prediction device 10, for example, converts the coordinates of the video acquired in step S101 into planar coordinates, and obtains the coordinates of each section for calculating the crowd density (step S103). Note that the coordinates for specifying each section from the video are, for example, stored in advance in the area DB14.

[0051] Next, the crowd density prediction device 10 calculates the walking vector of each person, for example, using the position coordinates for 5 seconds of the position coordinates per second calculated in step S102 (corresponding to the planar coordinates after 〇 seconds in FIG. 8) (step S104). The walking vector calculated here may be, for example, the average of the vectors calculated every second for 5 seconds.

[0052] Next, the crowd density prediction device 10 assumes that it moves while maintaining the walking vector calculated in step S104 for 5 minutes, and calculates the predicted coordinates of each person after 5 minutes using the current coordinates of each person (corresponding to the current planar coordinates in FIG. 8) and the walking vector (step S105). The predicted coordinates after 5 minutes calculated here may be calculated, for example, by the formula "current coordinates + (walking vector × 300 seconds)".

[0053] Next, the crowd density prediction device 10 determines which section each person has moved to after 5 minutes, for example, using the predicted coordinates after 5 minutes calculated in step S105 and the coordinates of each section calculated in step S103 (step S106). Note that steps S106 to S108 are determined and processed for each person.

[0054] When the predicted coordinates after 5 minutes do not belong to any of the sections (step S107: No), the crowd density prediction device 10 corrects the walking vector, for example (step S108). The walking vector corrected here may be a vector whose length and direction are corrected so that the predicted coordinates after 5 minutes fall within any of the sections, as described using FIG. 11, for example.

[0055] On the other hand, when the predicted coordinates after five minutes belong to any of the sections (step S107: Yes), the crowd density prediction device 10 skips step S108 without correcting the walking vector, for example.

[0056] Then, for example, when the determination in step S106 and, if necessary (step S107: No), the correction of the walking vector in step S108 are performed for all the persons detected from the video for five seconds, the process proceeds to step S109.

[0057] Next, the crowd density prediction device 10 calculates the number of persons in each section, for example, based on the predicted locations of each person after five minutes (step S109). The predicted location after five minutes is the predicted coordinates of each person calculated in step S105, but when the walking vector is corrected in step S108, it is the predicted coordinates after five minutes calculated using the corrected walking vector. Also, each section may be determined based on the coordinates of each section calculated in step S103.

[0058] Next, the crowd density prediction device 10 calculates the crowd density for each section, for example, based on the area of each section calculated in advance and the number of persons in each section calculated in step S109 (step S110). The crowd density for each section calculated here may be calculated, for example, by dividing the number of persons in the section by the area of the section for each section. After the execution of step S110, the crowd density calculation process shown in FIG. 12 ends.

[0059] Next, a flow of another example of the crowd density calculation process executed by the crowd density prediction device 10 will be described. FIG. 13 is a flowchart showing a flow of another example of the crowd density calculation process according to the present embodiment. The crowd density calculation process shown in FIG. 13 is a process of calculating the crowd density including objects other than persons in addition to the crowd density calculation process shown in FIG. 12 in which the number of persons is calculated for each section and the crowd density of the person is calculated. Here, the objects other than persons are, for example, objects having a size that can affect the crowd density, such as carry bags and baby strollers.

[0060] First, as shown in FIG. 13, similar to step S101 in FIG. 12, for example, the crowd density prediction device 10 acquires, from the video DB 13, a video in which a predetermined shooting range such as in front of a station, a park, or a footbridge is shot by the camera device 100 (step S201).

[0061] Next, similar to step S102 in FIG. 12, for example, the crowd density prediction device 10 uses an existing object detection technology to detect people from the video acquired in step S201, and calculates the position coordinates of each person every second (corresponding to the planar coordinates after 〇 seconds in FIG. 8) (step S202).

[0062] Next, for example, the crowd density prediction device 10 uses an existing object detection technology to detect objects other than people from the video acquired in step S201, and calculates the position coordinates of each object every second (step S203). Note that step S203 may be executed, for example, before step S202, may be executed in parallel with step S202, or may be processed in one step as an object detection process including people.

[0063] Next, similar to step S103 in FIG. 12, for example, the crowd density prediction device 10 converts the coordinates of the video acquired in step S101 into planar coordinates, and calculates the coordinates of each section for calculating the crowd density (step S204).

[0064] Next, for example, the crowd density prediction device 10 calculates the movement vectors of each person and each object using the position coordinates for 5 seconds of the position coordinates every second calculated in steps S202 and S203 (step S205). The calculation of the movement vector of each person in step S205 is the same as that in step S104 in FIG. 12, and for each object as well, the movement vector is calculated using the position coordinates for 5 seconds of the position coordinates of each object every second.

[0065] Next, the crowd density prediction device 10 calculates predicted coordinates five minutes later for each person and each object, for example, using the current coordinates and movement vectors of each person and each object (step S206). Similar to step S105 in FIG. 12, step S206 is a process assuming that the calculated movement vector is maintained and the object moves for five minutes. The predicted coordinates five minutes later may be calculated, for example, by the formula "current coordinates + (movement vector × 300 seconds).

[0066] Next, the crowd density prediction device 10 determines which section each person and each object has moved to five minutes later, using the predicted coordinates five minutes later calculated in step S206 and the coordinates of each section calculated in step S204 (step S207). Note that steps S207 to S209 are determined and processed for each person and object other than a person.

[0067] If the predicted coordinates five minutes later do not belong to any of the sections (step S208: No), the crowd density prediction device 10 corrects the movement vector, for example (step S209). Similar to the walking vector corrected in step S108 of FIG. 12, the movement vector corrected here may be a vector whose length and direction are corrected so that the predicted coordinates five minutes later fall within one of the sections as described with reference to FIG. 11.

[0068] On the other hand, if the predicted coordinates five minutes later belong to any of the sections (step S208: Yes), the crowd density prediction device 10 skips step S209 without correcting the movement vector, for example.

[0069] And, for example, when the determination in step S207 and, if necessary (step S208: No), the correction of the movement vector in step S209 are executed for all of the persons and objects other than persons detected from the video for five seconds, the process proceeds to step S210.

[0070] Next, the crowd density prediction device 10 calculates the number of people in each section, for example, based on the predicted locations of each person and each object after five minutes (step S210). Regarding the calculation of the number of people in step S210, for example, the number of people in each section is calculated by multiplying a predetermined coefficient corresponding to the object, which is the crowd count amount shown in FIG. 6, by the number of people and objects in each section. The predicted location after five minutes is the predicted coordinates of each person and each object after five minutes calculated in step S206. However, when the movement vector is corrected in step S209, it is the predicted coordinates after five minutes calculated using the corrected movement vector. Also, each section may be determined based on the coordinates of each section calculated in step S204.

[0071] Next, similar to step S110 in FIG. 12, the crowd density prediction device 10 calculates the crowd density for each section, for example, based on the area of each section calculated in advance and the number of people in each section calculated in step S210 (step S211). After the execution of step S211, the crowd density calculation process shown in FIG. 13 ends.

[0072] (Effect) As described above, the crowd density prediction device 10 calculates the movement vector of a person based on the position information of the person in the video captured by the camera device 100, calculates the second position information of the person after a predetermined time based on the first position information of the person and the movement vector, and executes a process of calculating the crowd density of a predetermined section based on the second position information.

[0073] In this way, the crowd density prediction device 10 calculates the movement vector from the position information of the people in the video, calculates the future position coordinates of the people based on the current position and the movement vector, and calculates the crowd density of a predetermined section. Thereby, the crowd density prediction device 10 can improve the accuracy of predicting the crowd density from the video.

[0074] Further, when the position indicated by the second position information does not belong to a predetermined section, the crowd density prediction device 10 corrects either the length or the direction of the movement vector, and executes a process of calculating the second position information based on the first position information and the corrected movement vector.

[0075] Thereby, the crowd density prediction device 10 can improve the accuracy of predicting the crowd density from the video.

[0076] Further, the process of correcting either the length or the direction of the movement vector executed by the crowd density prediction device 10 includes a process of correcting the length of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the direction of the movement vector.

[0077] Thereby, the crowd density prediction device 10 can improve the accuracy of predicting the crowd density from the video.

[0078] Further, the process of correcting either the length or the direction of the movement vector executed by the crowd density prediction device 10 includes a process of correcting the direction of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the length of the movement vector.

[0079] Thereby, the crowd density prediction device 10 can improve the accuracy of predicting the crowd density from the video.

[0080] Further, the crowd density prediction device 10 calculates a second movement vector of an object based on the position information of the object other than a person in the video, and executes a process of calculating the fourth position information of the object after a predetermined time based on the third position information of the object and the second movement vector. The process of calculating the crowd density executed by the crowd density prediction device 10 includes a process of calculating the crowd density based on the second position information, the fourth position information, and a predetermined coefficient corresponding to the object.

[0081] As a result, the crowd density prediction device 10 can calculate a more accurate crowd density while improving the accuracy of predicting the crowd density from the video because it calculates the crowd density including not only people but also objects other than people.

[0082] In addition, the process of calculating the crowd density executed by the crowd density prediction device 10 includes a process of calculating the crowd density of a predetermined section within a shooting range different from the shooting range of the camera device 100 based on the second position information.

[0083] As a result, the crowd density prediction device 10 can improve the accuracy of predicting the crowd density from the video.

[0084] In addition, the crowd density prediction device 10 executes a process of displaying a map including the crowd density and the predetermined section.

[0085] As a result, the crowd density prediction device 10 can show the user the crowd density predicted from the video.

[0086] (System) The processing procedures, control procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be arbitrarily changed unless otherwise specified. Also, the specific examples, distributions, numerical values, etc. described in the embodiments are merely examples and may be arbitrarily changed.

[0087] In addition, the specific forms of the dispersion and integration of the components of each device are not limited to those shown. That is, all or part of the components may be functionally or physically dispersed and integrated in any unit according to various loads and usage situations. Furthermore, each processing function of each device may be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

[0088] (Hardware) FIG. 14 is a diagram for explaining a hardware configuration example of the crowd density prediction device 10. As shown in FIG. 14, the crowd density prediction device 10 includes a communication device 10a, a HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Further, each part shown in FIG. 14 is interconnected by a bus or the like.

[0089] The communication device 10a is a network interface card or the like and communicates with other information processing devices. The HDD 10b stores programs and databases for operating the functions shown in FIG. 2.

[0090] The processor 10d is a hardware circuit that operates a process for executing each function described in FIG. 2 and the like by reading a program for executing the same processing as each processing unit shown in FIG. 2 from the HDD 10b or the like and expanding it in the memory 10c. That is, this process executes the same functions as each processing unit included in the crowd density prediction device 10. Specifically, the processor 10d reads a program having the same functions as the detection unit 21, the calculation unit 22, the correction unit 23, and the display unit 24 and the like from the HDD 10b or the like. Then, the processor 10d executes a process for executing the same processing as the calculation unit 22 and the like.

[0091] In this way, the crowd density prediction device 10 operates as an information processing device that executes operation control processing by reading and executing a program for executing the same processing as each processing unit shown in FIG. 2. Further, the crowd density prediction device 10 can also realize the same functions as the above-described embodiments by reading a program from a recording medium by a medium reader and executing the read program. Note that the program in this other embodiment is not limited to being executed by the crowd density prediction device 10. For example, the present embodiment may be similarly applied when another information processing device executes a program, or when the crowd density prediction device 10 and another information processing device cooperate to execute a program.

[0092] In addition, a program that executes the same processing as each processing unit shown in FIG. 2 can be distributed via a network such as the Internet. Further, this program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a MO (Magneto-Optical disk), a DVD (Digital Versatile Disc), and can be executed by being read from the recording medium by a computer.

[0093] Regarding the embodiments including the above embodiments, the following additional remarks are disclosed.

[0094] (Supplementary Note 1) Based on the position information of a person in the video captured by an imaging device, calculate the movement vector of the person, Based on the first position information of the person and the movement vector, calculate the second position information of the person after a predetermined time, Calculate the crowd density of a predetermined section based on the second position information A crowd density prediction program characterized by causing a computer to execute the processing.

[0095] (Supplementary Note 2) When the position indicated by the second position information does not belong to the predetermined section, correct either the length or the direction of the movement vector, Based on the first position information and the corrected movement vector, calculate the second position information The crowd density prediction program according to Supplementary Note 1, characterized by causing the computer to execute the processing.

[0096] (Supplementary Note 3) The process of correcting either the length or the direction of the movement vector is Without changing the direction of the movement vector, correct the length of the movement vector so that the position indicated by the second position information falls within the predetermined section The crowd density prediction program according to Supplementary Note 2, characterized by including the processing.

[0097] (Appendix 4) The process of correcting either the length or the direction of the movement vector is correcting the direction of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the length of the movement vector The crowd density prediction program according to Appendix 2, characterized by including the process.

[0098] (Appendix 5) Based on the position information of an object other than the person in the video, calculate the second movement vector of the object, Based on the third position information of the object and the second movement vector, calculate the fourth position information of the object after a predetermined time Cause the computer to execute the process, The process of calculating the crowd density is Calculating the crowd density based on the second position information, the fourth position information, and a predetermined coefficient corresponding to the object The crowd density prediction program according to Appendix 1 or 2, characterized by including the process.

[0099] (Appendix 6) The process of calculating the crowd density is Calculating the crowd density of the predetermined section within a shooting range different from the shooting range of the imaging device based on the second position information The crowd density prediction program according to Appendix 1 or 2, characterized by including the process.

[0100] (Appendix 7) Displaying a map including the crowd density and the predetermined section The crowd density prediction program according to Appendix 1 or 2, characterized by causing the computer to execute the process.

[0101] (Appendix 8) Based on the position information of a person in the video captured by an imaging device, calculate the movement vector of the person, Based on the first position information of the person and the movement vector, calculate the second position information of the person after a predetermined time, Calculate the crowd density of a predetermined section based on the second position information A crowd density prediction method characterized in that a computer executes processing.

[0102] (Appendix 9) When the position indicated by the second position information does not belong to the predetermined section, correct either the length or the direction of the movement vector, Based on the first position information and the corrected movement vector, calculate the second position information A crowd density prediction method according to Appendix 8, characterized in that a computer executes processing.

[0103] (Appendix 10) The process of correcting either the length or the direction of the movement vector is Without changing the direction of the movement vector, correct the length of the movement vector so that the position indicated by the second position information falls within the predetermined section A crowd density prediction method according to Appendix 9, characterized by including processing.

[0104] (Appendix 11) The process of correcting either the length or the direction of the movement vector is Without changing the length of the movement vector, correct the direction of the movement vector so that the position indicated by the second position information falls within the predetermined section A crowd density prediction method according to Appendix 9, characterized by including processing.

[0105] (Appendix 12) Based on the position information of an object other than the person in the video, calculate a second movement vector of the object, Based on the third position information of the object and the second movement vector, calculate the fourth position information of the object after a predetermined time A computer executes processing, The process of calculating the crowd density is Based on the second position information, the fourth position information, and a predetermined coefficient corresponding to the object, calculate the crowd density A crowd density prediction method according to Appendix 8 or 9, characterized by including processing.

[0106] (Appendix 13) The process of calculating the crowd density is to calculate the crowd density of the predetermined section within a shooting range different from the shooting range of the imaging device based on the second position information. The crowd density prediction method according to Appendix 8 or 9, characterized by including the process.

[0107] (Appendix 14) To display the map including the crowd density and the predetermined section The computer executes the process. The crowd density prediction method according to Appendix 8 or 9, characterized by this.

[0108] (Appendix 15) Based on the position information of the people in the video shot by the imaging device, calculate the movement vector of the people, Based on the first position information of the people and the movement vector, calculate the second position information of the people after a predetermined time, Based on the second position information, calculate the crowd density of the predetermined section The crowd density prediction device characterized by having a control unit that executes the process.

[0109] (Appendix 16) When the position indicated by the second position information does not belong to the predetermined section, correct either the length or the direction of the movement vector, Based on the first position information and the corrected movement vector, calculate the second position information The control unit executes the process. The crowd density prediction device according to Appendix 15, characterized by this.

[0110] (Appendix 17) The process of correcting either the length or the direction of the movement vector is Without changing the direction of the movement vector, correct the length of the movement vector so that the position indicated by the second position information falls within the predetermined section The crowd density prediction device according to Appendix 16, characterized by including the process.

[0111] (Supplementary Note 18) The process of correcting either the length or the direction of the movement vector is correcting the direction of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the length of the movement vector The crowd density prediction device according to Supplementary Note 16, characterized by including the process.

[0112] (Supplementary Note 19) Based on the position information of an object other than the person in the video, calculating a second movement vector of the object, calculating fourth position information of the object after a predetermined time based on the third position information of the object and the second movement vector The control unit executes the process, The process of calculating the crowd density is calculating the crowd density based on the second position information, the fourth position information, and a predetermined coefficient corresponding to the object The crowd density prediction device according to Supplementary Note 15 or 16, characterized by including the process.

[0113] (Supplementary Note 20) The process of calculating the crowd density is calculating the crowd density of the predetermined section within a shooting range different from the shooting range of the imaging device based on the second position information The crowd density prediction device according to Supplementary Note 15 or 16, characterized by including the process.

[0114] (Supplementary Note 21) Displaying a map including the crowd density and the predetermined section The control unit of the crowd density prediction device according to Supplementary Note 15 or 16 executes the process.

[0115] (Supplementary Note 22) An information processing device including a processor, and a memory operably connected to the processor, wherein the processor calculates a movement vector of a person based on position information of the person in a video shot by an imaging device, Calculate the second position information of the person after a predetermined time based on the first position information of the person and the movement vector. Calculate the crowd density of a predetermined section based on the second position information. A crowd density prediction device characterized by executing the process.

Explanation of symbols

[0116] 1 Information processing system 10 Crowd density prediction device 10a Communication device 10b HDD 10c Processor 10d Memory 11 Communication unit 12 Storage unit 13 Video DB 14 Area DB 15 Object DB 16 Pedestrian flow DB 17 Crowd density DB 20 Control unit 21 Detection unit 22 Calculation unit 23 Correction unit 24 Display unit 50 Network 60 Shooting range 70 Prediction range 80 Shading display 100 Camera device

Claims

1. calculating a movement vector of the person based on position information of the person in a video captured by an imaging device; calculating second position information of the person after a predetermined time based on the first position information of the person and the movement vector; calculating a crowd density of a predetermined section based on the second position information; A crowd density prediction program, characterized in that a computer is caused to execute the processing.

2. when the position indicated by the second position information does not belong to the predetermined section, correcting either the length or the direction of the movement vector; calculating the second position information based on the first position information of the person and the corrected movement vector; The crowd density prediction program according to claim 1, characterized in that a computer is caused to execute the processing.

3. The process of correcting either the length or the direction of the movement vector includes a process of correcting the length of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the direction of the movement vector. The crowd density prediction program according to claim 2, characterized in that the process is included.

4. The process of correcting either the length or the direction of the movement vector includes a process of correcting the direction of the movement vector so that the position indicated by the second position information falls within the predetermined section without changing the length of the movement vector. The crowd density prediction program according to claim 2, characterized in that the process is included.

5. calculating a second movement vector of the object based on position information of an object other than the person in the video; calculating fourth position information of the object after a predetermined time based on third position information of the object and the second movement vector; Cause the computer to execute the process, The process of calculating the crowd density is, Based on the second position information, the fourth position information, and a predetermined coefficient corresponding to the object, calculate the crowd density The crowd density prediction program according to claim 1 or 2, characterized by including the process.

6. The process of calculating the crowd density is, Based on the second position information, calculate the crowd density of the predetermined section within a shooting range different from the shooting range of the imaging device The crowd density prediction program according to claim 1 or 2, characterized by including the process.

7. Cause the computer to execute the process of displaying a map including the crowd density and the predetermined section The crowd density prediction program according to claim 1 or 2, characterized by causing the computer to execute the process.

8. Based on the position information of a person in the video captured by the imaging device, calculate the movement vector of the person, Based on the first position information of the person and the movement vector, calculate the second position information of the person after a predetermined time, Based on the second position information, calculate the crowd density of a predetermined section A crowd density prediction method, characterized in that a computer executes the process.

9. Based on the position information of a person in the video captured by the imaging device, calculate the movement vector of the person, Based on the first position information of the person and the movement vector, calculate the second position information of the person after a predetermined time, Based on the second position information, calculate the crowd density of a predetermined section A crowd density prediction device, characterized by having a control unit that executes the process.

Images