Information processing device, information processing method, and program

Abstract

To grasp a state of a crowd from a video obtained by photographing the crowd.SOLUTION: An information processing device includes a first acquisition part, a second acquisition part and a generation part. The first acquisition part acquires a first monitored image photographed by a stationary camera which is positionally fixed. The second acquisition part acquires a second monitored image photographed by a mobile camera which is not positionally fixed. The generation part generates distribution information obtained by correcting a distribution of objects captured in the first monitored image on the basis of the second monitored image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to video surveillance.

Background Art

[0002] In order to monitor or analyze the state of a crowd, video captured by surveillance cameras fixed to buildings or the like on the movement path of the crowd is used. For example, Patent Document 1 discloses a technique for analyzing input surveillance video to calculate the moving direction of a crowd and controlling a surveillance device according to the calculated moving direction.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the video captured by a surveillance camera fixed to a building or the like as described above, it may be difficult to accurately monitor the state of the crowd. For example, such a surveillance camera fixed to a building often captures the crowd from a distance. In such a case, since the size of the people shown in the captured video is small, it is difficult to grasp the state of the crowd (for example, the number of people included in the crowd and their distribution).

[0005] The present invention has been made in view of the above problems. An object of the present invention is to provide a technique for grasping the state of a crowd from video capturing the crowd.

Means for Solving the Problems

[0006] This disclosure includes an information processing apparatus, an information processing method, and a program for solving the above problems.

[0007] The information processing apparatus in this disclosure is A first acquisition means for acquiring a first surveillance image captured by a fixed camera whose position is fixed, A second acquisition means for acquiring a second surveillance image captured by a mobile camera whose position is not fixed, The system includes a generation means for generating distribution information obtained by correcting the distribution of objects visible in the first monitoring image based on the second monitoring image.

[0008] The information processing method in this disclosure is: At least one computer, A first surveillance image is acquired, captured by a fixed camera whose position is fixed. A second surveillance image is obtained, captured by a mobile camera whose position is not fixed. Distribution information is generated by correcting the distribution of objects visible in the first surveillance image based on the second surveillance image. This includes the following.

[0009] The program described in this disclosure is At least one computer, A first acquisition means for acquiring a first surveillance image captured by a fixed camera whose position is fixed, A second acquisition means for acquiring a second surveillance image captured by a mobile camera whose position is not fixed. A generation means for generating distribution information obtained by correcting the distribution of objects visible in the first surveillance image based on the second surveillance image, To make it function as such. [Effects of the Invention]

[0010] According to the present invention, a technology is provided for understanding the state of a crowd from video footage of the crowd. [Brief explanation of the drawing]

[0011] The aforementioned objectives, as well as other objectives, features, and advantages, will become even clearer from the preferred embodiments described below and the accompanying drawings.

[0012] [Figure 1] It is a block diagram illustrating a monitoring information generation device according to Embodiment 1. [Figure 2] It is a diagram conceptually illustrating the operation of the monitoring information generation device of Embodiment 1. [Figure 3] It is a flowchart illustrating the flow of processing executed by the monitoring information generation device of Embodiment 1. [Figure 4] It is a diagram illustrating the hardware configuration of a computer that realizes the monitoring information generation device of Embodiment 1. [Figure 5] It is a diagram illustrating a display in which a second monitoring image is superimposed on a first monitoring image. [Figure 6] It illustrates a state in which the second monitoring images are arranged and displayed near the left end of the display screen. [Figure 7] It illustrates a state in which the first monitoring image and the second monitoring image are arranged and displayed on the display screen. [Figure 8] It is a diagram illustrating a state in which a mark indicating a moving camera is displayed on the first monitoring image. [Figure 9] It is a diagram illustrating a display when the mouse cursor is clicked in FIG. 8. [Figure 10] It is a diagram illustrating a state in which the shooting direction of the moving camera is displayed. [Figure 11] It is a first diagram illustrating an example of superimposing a display based on distribution information on the first monitoring image. [Figure 12] It is a second diagram illustrating an example of superimposing a display based on distribution information on the first monitoring image. [Figure 13] It is a diagram illustrating the overlap between the range shown in the first monitoring image and the range shown in the second monitoring image. [Figure 14] It is a block diagram illustrating a monitoring information generation device having a map information acquisition unit. [Figure 15] It is a diagram illustrating a map displayed on the display screen. [Figure 16] It is a diagram illustrating a map on which a heat map generated based on the corrected distribution information is superimposed. [Figure 17] This is a block diagram illustrating a monitoring information generation device according to Embodiment 2. [Figure 18] This figure conceptually illustrates the operation of the monitoring information generation device of Embodiment 2. [Figure 19] This is a flowchart illustrating the processing flow performed by the monitoring information generation device of Embodiment 2. [Figure 20] This diagram illustrates the optical flow calculated for the first monitoring image. [Figure 21] This diagram illustrates how the position of an object changes. [Figure 22] This is a diagram illustrating the operation of the shooting direction estimation unit. [Figure 23] This diagram illustrates a method for determining candidate shooting directions based on the direction of the sidewalk. [Figure 24] This diagram illustrates a case where the shooting ranges of a fixed camera and a mobile camera do not overlap. [Figure 25] This diagram illustrates a method for estimating the direction of movement of a crowd on a plane using map information. [Figure 26] This diagram illustrates a case where there are multiple routes for a crowd to move. [Figure 27] This is a block diagram illustrating a monitoring information generation device having a movement path information acquisition unit. [Figure 28] This diagram illustrates a method for estimating the direction of crowd movement using movement path information. [Figure 29] This diagram illustrates a method for narrowing down candidate shooting directions using an electronic compass. [Figure 30] This diagram illustrates a method for narrowing down candidate shooting directions based on the background captured in the second monitoring image. [Figure 31] This diagram illustrates a method for narrowing down candidate shooting directions based on the position of a specific background on the second monitoring image. [Figure 32] This diagram illustrates an example of how crowd flow changes near an intersection. [Figure 33]This diagram illustrates, in chronological order, the breakdown of the process by which the shooting direction estimation unit estimates the shooting direction of a moving camera. [Figure 34] This figure illustrates the change in the position of feature points in the second monitoring image. [Figure 35] This is a block diagram illustrating a shooting direction estimation device. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0014] [Embodiment 1] Figure 1 is a block diagram illustrating a monitoring information generation device 2000 according to Embodiment 1. In Figure 1, each block represents a functional unit configuration, not a hardware unit configuration.

[0015] The surveillance information generation device 2000 uses two types of surveillance images: surveillance images generated by a fixed camera and surveillance images generated by a mobile camera. A fixed camera is a camera whose position is fixed. For example, a fixed camera is a surveillance camera that is permanently installed in various places such as a wall, pillar, or ceiling. The location where the fixed camera is installed may be indoors or outdoors. Furthermore, the wall or other object on which the fixed camera is installed only needs to be fixed in position for a certain period of time and is not limited to real estate. For example, the wall or other object on which the fixed camera is installed may be a partition or pillar that is temporarily installed at an event venue, etc.

[0016] A mobile camera is a camera whose position changes. For example, a mobile camera may be worn by a person, or attached to a car, motorcycle, or aircraft. A mobile camera worn by a person may be a handheld camera (such as a video camera or a camera on a mobile device like a smartphone), or a camera fixed to the head or chest (such as a wearable camera). A camera attached to a car, motorcycle, or aircraft may be a camera installed for use as a dashcam, or a camera installed separately for surveillance purposes.

[0017] Both the mobile camera and the fixed camera record video of the area being monitored. The area to be monitored can be arbitrary. For example, the area to be monitored could be the route between an event venue and the nearest train station. The area to be monitored can be indoors or outdoors. The imaging ranges of the mobile camera and the fixed camera may or may not overlap.

[0018] Figure 2 is a diagram conceptually illustrating the operation of the surveillance information generation device 2000. The fixed camera 10 captures a crowd and generates a first surveillance image 12. Here, the crowd refers to one or more objects. The objects may be people or non-people (e.g., cars, motorcycles, animals, etc.). The mobile camera 20 captures a crowd and generates a second surveillance image 22. The crowd captured by the fixed camera 10 and the crowd captured by the mobile camera 20 may be the same crowd or different crowds.

[0019] The monitoring information generation device 2000 generates monitoring information 30 using the first monitoring image 12 and the second monitoring image 22. The monitoring information 30 is information related to the monitoring of an object. The specific contents of the monitoring information 30 and the method of generating it will be described later.

[0020] To achieve the above operation, the monitoring information generation device 2000 includes a first monitoring image acquisition unit 2020, a second monitoring image acquisition unit 2040, and a generation unit 2060. The first monitoring image acquisition unit 2020 acquires a first monitoring image 12. The second monitoring image acquisition unit 2040 acquires a second monitoring image 22. The generation unit 2060 generates monitoring information 30 using the first monitoring image 12 and the first monitoring information 14.

[0021] <Effects and Actions> According to this embodiment, crowd monitoring information is generated using monitoring images generated by the mobile camera 20 in addition to monitoring images generated by the fixed camera 10. Therefore, compared to the case where the state of the crowd must be understood using only the fixed camera 10, the state of the crowd can be understood more accurately.

[0022] The embodiment will be described in more detail below.

[0023] <Processing flow> Figure 3 is a flowchart illustrating the processing flow performed by the monitoring information generation device 2000 of Embodiment 1. The first monitoring image acquisition unit 2020 acquires the first monitoring image 12 (S102). The second monitoring image acquisition unit 2040 acquires the second monitoring image 22 (S104). The generation unit 2060 generates monitoring information 30 using the first monitoring image 12 and the second monitoring image 22 (S106).

[0024] <Example Hardware Configuration of Monitoring Information Generator 2000> Figure 4 illustrates the hardware configuration of a computer 1000 that implements the monitoring information generation device 2000 of Embodiment 1. This computer 1000 may be implemented using a dedicated device specifically designed for implementing the monitoring information generation device 2000, or it may be implemented using a general-purpose device such as a PC (Personal Computer) or a mobile terminal.

[0025] Computer 1000 has a bus 1020, a processor 1040, memory 1060, storage 1080, an input / output interface 1100, and a network interface 1120. Bus 1020 is a data transmission path for the processor 1040, memory 1060, storage 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. However, the method of connecting the processor 1040 and the others to each other is not limited to bus connection. The processor 1040 is a processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Memory 1060 is memory such as RAM (Random Access Memory) or ROM (Read Only Memory). Storage 1080 is a storage device such as a hard disk, SSD (Solid State Drive), or memory card. Alternatively, storage 1080 may also be memory such as RAM or ROM.

[0026] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, a keyboard or mouse can be connected to the input / output interface 1100.

[0027] The network interface 1120 is an interface for connecting the computer 1000 to an external device so that it can communicate with it. The network interface 1120 may be a network interface for connecting to a wired line or a network interface for connecting to a wireless line. For example, the computer 1000 that implements the monitoring information generation device 2000 is connected to the fixed camera 10 and the mobile camera 20 via a network. However, the method of connecting the computer 1000 to the fixed camera 10 and the mobile camera 20 is not limited to connection via a network. Also, the computer 1000 does not need to be connected to the fixed camera 10 and the mobile camera 20 so that it can communicate with them.

[0028] The storage 1080 stores program modules that implement each function of the monitoring information generation device 2000. The processor 1040 implements each function corresponding to each of these program modules by executing them. When the processor 1040 executes the above modules, it may either read these modules into memory 1060 before executing them, or it may execute them without reading them into memory 1060.

[0029] The hardware configuration of computer 1000 is not limited to the configuration shown in Figure 4. For example, each program module may be stored in memory 1060. In this case, computer 1000 does not need to have storage 1080.

[0030] <Details of the 1st Surveillance Image Acquisition Unit 2020> The first monitoring image acquisition unit 2020 acquires the first monitoring image 12 (S102). There are various ways in which the first monitoring image acquisition unit 2020 acquires the first monitoring image 12. For example, the first monitoring image acquisition unit 2020 receives the first monitoring image 12 transmitted from the fixed camera 10. Alternatively, the first monitoring image acquisition unit 2020 may access the fixed camera 10 and acquire the first monitoring image 12 stored in the fixed camera 10. The fixed camera 10 may store the first monitoring image 12 in a storage device provided outside the fixed camera 10. In this case, the first monitoring image acquisition unit 2020 may access this storage device and acquire the first monitoring image 12.

[0031] The first surveillance image acquisition unit 2020 may acquire the first surveillance image 12 in real time, or it may acquire the first surveillance image 12 some time after it has been generated. In the latter case, for example, the surveillance information generation device 2000 may acquire the first surveillance image 12 and the second surveillance image 22 that were taken in the past (for example, the previous day), and generate surveillance information about the past surveillance images to perform analysis of crowd behavior, etc.

[0032] <Details of the second monitoring image acquisition unit 2040> The second monitoring image acquisition unit 2040 acquires the second monitoring image 22 (S104). Here, the method by which the second monitoring image acquisition unit 2040 acquires the second monitoring image 22 is the same as the method by which the first monitoring image acquisition unit 2020 acquires the first monitoring image 12.

[0033] <Details of the generation unit 2060> The generation unit 2060 generates object monitoring information 30 using the first monitoring image 12 and the second monitoring image 22 (S106). As mentioned above, the object is not limited to people, but can be anything. What the generation unit 2060 will treat as an object may be set in advance in the generation unit 2060, stored in a storage device accessible from the generation unit 2060, or manually set when the generation unit 2060 is operating.

[0034] The monitoring information 30 generated by the generation unit 2060 is diverse. Below, specific examples of the monitoring information 30 and their respective generation methods are described.

[0035] <<Specific Example 1 of Monitoring Information 30>> The generation unit 2060 generates a display in which the second monitoring image 22 is superimposed on the first monitoring image 12 as monitoring information 30. Figure 5 is an example of a display in which the second monitoring image 22 is superimposed on the first monitoring image 12. The display screen 40 in Figure 5 shows a screen in which the second monitoring images 22-1 to 22-3 are superimposed on the first monitoring image 12.

[0036] The display screen 40 is viewed by a security guard, for example, in a security room. By viewing this display screen 40, the security guard can grasp the overall situation of the monitoring location as captured by the fixed camera 10, while also grasping the detailed situation of individual locations as captured by the mobile camera 20. As a result, security guards can grasp the state of the crowd flexibly and accurately.

[0037] The position on the first monitoring image 12 where the second monitoring image 22 is displayed is preferably the position on the first monitoring image 12 corresponding to the real-world position of the second monitoring image 22, or a position near thereto. In this case, the generation unit 2060 determines the position on which to superimpose the second monitoring image 22 using the position information of the mobile camera 20, the position information of the fixed camera 10, and camera parameters representing the orientation of the fixed camera 10. Specifically, the generation unit 2060 uses the position information and orientation of the fixed camera 10 to determine the position corresponding to the position information of the mobile camera 20 from among the locations captured in the first monitoring image 12. Here, the orientation of the fixed camera 10 includes the horizontal and vertical directions of the shooting direction of the fixed camera 10.

[0038] The location information for each camera is any information that can identify the location of that camera. For example, the location information for a camera is information that indicates the Global Positioning System (GPS) coordinates of that camera.

[0039] There are various ways in which the generation unit 2060 can acquire the location information of the mobile camera 20. For example, the location information of the mobile camera 20 is included in the metadata of the second monitoring image 22. In this case, the generation unit 2060 acquires the location information of the mobile camera 20 from the metadata of the second monitoring image 22. Alternatively, for example, the generation unit 2060 may receive location information separately transmitted by the mobile camera 20. This transmission may be performed spontaneously by the mobile camera 20 or in response to a request from the generation unit 2060.

[0040] The method by which the generation unit 2060 acquires the position information of the fixed camera 10 is the same as, for example, the method by which the generation unit 2060 acquires the position information of the mobile camera 20. Also, since the position of the fixed camera 10 is fixed, the position information of the fixed camera 10 may be stored in advance in a storage unit accessible from the generation unit 2060. Alternatively, for example, the position information of the fixed camera 10 may be manually entered into the generation unit 2060.

[0041] The position of the second monitoring image 22 on the first monitoring image 12 is not limited to a position based on the real-world position of the second monitoring image 22. For example, the second monitoring image 22 may be displayed at a predetermined position on the first monitoring image 12. Figure 6 illustrates how the second monitoring image is displayed side by side near the left edge of the display screen 40. This predetermined position may be set in advance in the generation unit 2060, or it may be stored in a storage device accessible from the generation unit 2060.

[0042] The display position of the second monitoring image 22 may be changed by user operation. For example, the monitoring information generation device 2000 accepts operations such as dragging the second monitoring image 22 with a mouse and changes the display position of the second monitoring image 22 in accordance with that operation.

[0043] Alternatively, the generation unit 2060 may display the first monitoring image 12 and the second monitoring image 22 side by side, rather than superimposing the second monitoring image 22 on the first monitoring image 12. Figure 7 illustrates how the first monitoring image 12 and the second monitoring image 22 are displayed side by side on the display screen 40.

[0044] If the display position of the second monitoring image 22 on the first monitoring image 12 is not based on the position of the mobile camera 20 in the real world, or if the second monitoring image 22 is not superimposed on the first monitoring image 12, it is preferable to make it clear which mobile camera 20 each second monitoring image 22 was captured by. In this case, for example, the generation unit 2060 displays a mark representing each mobile camera 20, as described later, at a position on the first monitoring image 12 corresponding to the position of each mobile camera 20 in the real world. The generation unit 2060 then displays information (for example, the mark number) next to each second monitoring image 22 indicating which mark each second monitoring image 22 corresponds to.

[0045] <<Specific Example 2 of Monitoring Information 30>> The generation unit 2060 generates a display in which a mark indicating the mobile camera 20 is superimposed on the first monitoring image 12. Furthermore, if this mark is selected by the user (such as a monitor), the generation unit 2060 displays the second monitoring image 22 generated by the mobile camera 20 corresponding to that mark. As a result, similar to the example 1 described above, monitoring information 30 is generated, which is a display in which the second monitoring image 22 is superimposed on the first monitoring image 12.

[0046] In this way, by displaying the second surveillance image 22 according to the selection of the monitor or other personnel, it becomes easier to grasp the overall appearance of the crowd in the first surveillance image 12, compared to the case where all second surveillance images 22 are displayed unconditionally, while also making it easier for monitors or other personnel to grasp the detailed appearance of the crowd in areas they wish to focus on.

[0047] Figure 8 illustrates how marks indicating mobile cameras 20 are displayed on the first monitoring image 12. In Figure 8, there are three mobile cameras 20-1 to 20-3 within the shooting range of the fixed camera 10, and their respective positions are represented by marks 50-1 to 3. The position of each mark 50 in Figure 8 corresponds to the real-world position of the corresponding mobile camera 20.

[0048] For example, the user selects mark 50 using the mouse. In Figure 8, the mouse cursor 60 is positioned over mark 50-3. When the user clicks the mouse cursor 60 in this state, the generation unit 2060 displays the second monitoring image 22-3 generated by the moving camera 20-3 on the display screen 40. Figure 9 is an example of the display when the mouse cursor 60 is clicked in Figure 8.

[0049] User selection of Mark 50 is not limited to mouse operation. For example, other operations include selecting Mark 50 using a touch panel or keyboard.

[0050] The display position of the mark for the mobile camera 20 shown in the first monitoring image 12 is not limited to a position corresponding to the location of the mobile camera 20 in the real world. In this respect, it is the same as the display position of the second monitoring image 22 on the first monitoring image 12 as described above.

[0051] Preferably, the display screen 40 also displays the shooting direction of each mobile camera 20. This allows the monitor or other person viewing the display screen 40 to more accurately and easily understand which location the scenery represented by the second monitoring image 22 is. Figure 10 is an example of how the shooting direction of the mobile camera 20 is displayed. In Figure 10, the direction indicated by the shooting direction 52 is the shooting direction of the mobile camera 20 corresponding to the mark 50.

[0052] Here, there are various ways in which the generation unit 2060 can determine the shooting direction of each mobile camera 20. For example, if an electronic compass is built into the mobile camera 20 or a mobile terminal integrated with the mobile camera 20, the generation unit 2060 will use the direction indicated by the output of this electronic compass as the shooting direction of the mobile camera 20. Alternatively, for example, the generation unit 2060 may use the shooting direction of the mobile camera 20 estimated by the method described in the embodiment described later.

[0053] <<Specific Example 3 of Monitoring Information 30>> The generation unit 2060 generates a display superimposed with information about the crowd onto the first monitoring image 12 as monitoring information 30. The information about the crowd is, for example, information representing the distribution of objects included in the crowd. Hereinafter, information representing the distribution of objects included in the crowd will be referred to as distribution information.

[0054] Figures 11 and 12 illustrate an example of overlaying a display based on distribution information onto the first monitoring image 12. In the display screen 40 of Figure 11, a heatmap 61 representing the distribution of people is overlaid on the first monitoring image 12. For example, this heatmap uses red to indicate areas with high population density and blue to indicate areas with low population density.

[0055] In the display screen 40 of Figure 12, areas with a high density of people are highlighted with a thick border 62. Here, in Figure 12, the first monitoring image 12 is divided into multiple sub-regions. Specifically, the first monitoring image 12 is divided into 24 sub-regions by dividing it into 6 equal parts vertically and 4 equal parts horizontally. Note that in Figure 12, dotted lines are displayed to represent each sub-region in order to facilitate understanding of the figure, but these dotted lines do not actually need to be displayed.

[0056] The generation unit 2060 generates monitoring information 30 by superimposing a frame 62 on the first monitoring image 12 to highlight areas with a high density of people. Specifically, the generation unit 2060 generates distribution information indicating the number of people in each sub-region, and designates sub-regions where this number is greater than or equal to a predetermined value as areas with a high density of people.

[0057] By superimposing crowd distribution information generated using the first monitoring image 12 and the second monitoring image 22 onto the first monitoring image 12 in this way, monitors and others can easily grasp the state of the crowd, which is difficult to understand by simply looking at the first monitoring image 12 and the second monitoring image 22.

[0058] <<<Method for generating distribution information>>> The method for generating distribution information to realize each of the above displays will now be explained. The generation unit 2060 generates distribution information using the first monitoring image 12 and the second monitoring image 22. First, the generation unit 2060 generates distribution information for the first monitoring image 12 by performing processing such as image recognition on the first monitoring image 12. Specifically, the generation unit 2060 divides the first monitoring image 12 into multiple sub-regions and calculates the number of objects in each sub-region. As a result, distribution information representing the number of objects in each sub-region of the first monitoring image 12 is generated.

[0059] Furthermore, the generation unit 2060 identifies the portion of the first monitoring image 12 that is captured in the second monitoring image 22. The generation unit 2060 corrects the number of objects shown in the identified region of the distribution information using the number of objects captured in the second monitoring image 22. The generation unit 2060 then displays the corrected distribution information superimposed on the first monitoring image 12. The number of objects captured in the second monitoring image 22 can be calculated by performing image recognition processing on the second monitoring image 22, similar to the number of objects captured in the first monitoring image 12.

[0060] Figure 13 illustrates the overlap between the area shown in the first monitoring image 12 and the area shown in the second monitoring image 22. In Figure 13, sub-region 64-1 is the aforementioned sub-region obtained by dividing the first monitoring image 12 into multiple parts. In Figure 13(a), the area 65 shown in the second monitoring image 22 is contained within one sub-region 64-1. In this case, the generation unit 2060 calculates the number of objects shown in the entire second monitoring image 22 and uses the calculated number of objects to correct the number of objects in sub-region 64-1 indicated by the distribution information. For example, the generation unit 2060 performs processes such as 1) replacing the number of objects in sub-region 64-1 indicated by the distribution information with the number of objects calculated for the second monitoring image 22, or 2) replacing the number of objects in sub-region 64-1 indicated by the distribution information with a statistical value of that number and the number of objects calculated for the second monitoring image 22. 2) The statistical values ​​are weighted averages, where the number of objects calculated for the second monitoring image 22 is weighted more heavily than the number of objects indicated by the distribution information.

[0061] In the case of Figure 13(b), the area 65 captured in the second monitoring image 22 spans two sub-regions 64-1 and 64-2. In this case, the generation unit 2060 divides the second monitoring image 22 into region A, which overlaps with sub-region 64-1, and region B, which overlaps with sub-region 64-2, and calculates the number of objects captured in each region. Then, the generation unit 2060 uses the number of objects calculated for region A to correct the number of objects in sub-region 64-1 indicated by the distribution information. Similarly, the generation unit 2060 uses the number of objects calculated for region B to correct the number of objects in sub-region 64-2 indicated by the distribution information. The method of each correction is the same as the method explained using Figure 13(a).

[0062] In this way, by correcting the object distribution information calculated for the first monitoring image 12 using the number of objects shown in the second monitoring image 22, the object distribution can be calculated more accurately. This is because the number of objects calculated for the second monitoring image 22 can be calculated more accurately than the number of objects calculated for the first monitoring image 12.

[0063] For example, as shown in Figure 11, when the fixed camera 10 is capturing a wider area from a greater distance than the mobile camera 20, the objects appear small in the first monitoring image 12, making it difficult to accurately calculate the number of objects. On the other hand, the second monitoring image 22, generated by the mobile camera 20 which is capturing the objects from a closer distance than the fixed camera 10, shows the objects larger and more clearly. Therefore, the generation unit 2060 can calculate the number of objects in the second monitoring image 22 more accurately than the number of objects in the first monitoring image 12.

[0064] Furthermore, for example, as shown in Figure 11, if the fixed camera 10 is taking pictures at an angle looking diagonally downwards from a distance, people who are close together may overlap in the image, and some people may not be captured in the first surveillance image 12. On the other hand, for example, by attaching a mobile camera 20 to a small flying object and taking pictures of a crowd from directly above, it is possible to take pictures in a way that prevents people who are close together from overlapping. Therefore, the number of objects calculated using the second surveillance image 22 generated by the mobile camera 20 taking pictures in this way will be more accurate than the number of objects calculated using the first surveillance image 12.

[0065] <<<Regarding updates to distribution information>>> Since the fixed camera 10 and the mobile camera 20 are cameras that shoot video, the first monitoring image 12 and the second monitoring image 22 are repeatedly generated. Therefore, the monitoring information generation device 2000 may repeatedly generate the distribution information described above and update the distribution information to be displayed. This makes it possible to display distribution information such as a heat map in an animated manner. The generation of distribution information may be performed using all of the first monitoring images 12 and the second monitoring images 22, or it may be performed using only some of the first monitoring images 12 and the second monitoring images 22. In the latter case, the distribution information is generated at intervals such as once per second or once every 10 seconds. The monitoring information generation device 2000 may also accept user input to instruct the generation of distribution information and update the distribution information only when such user input is received.

[0066] <<<Other ways to use distribution information>>> In the example described above, the generation unit 2060 uses a display of distribution information superimposed on the first monitoring image 12 as monitoring information 30. However, the generation unit 2060 may also use the distribution information itself as monitoring information 30. In this case, for example, the generation unit 2060 may store the distribution information in a memory device or display the distribution information on a display screen in tabular or graphical format. This distribution information can be used for crowd behavior analysis, etc. Furthermore, the monitoring information generation device 2000 may use the distribution information as shown in another specific example below.

[0067] <<Specific Example 4 of Monitoring Information 30>> The generation unit 2060 may overlay the aforementioned distribution information onto a map of the location to be monitored. In this case, the monitoring information generation device 2000 has a map information acquisition unit 2080. The map information acquisition unit 2080 acquires map information, which is information about a map of the vicinity of the location to be monitored. Figure 14 is a block diagram illustrating a monitoring information generation device 2000 having a map information acquisition unit 2080. For example, the map information shows the locations of sidewalks, roads, and buildings. For example, the map information is pre-stored in a storage device accessible from the monitoring information generation device 2000.

[0068] Figure 15 is an example of a map 200 displayed on the display screen 40. The area being monitored in Figure 15 is an indoor floor. Figure 15 shows multiple fixed cameras 10 and multiple mobile cameras 20. In Figure 15, the positions of the mobile cameras 20 are indicated by marks 50. The positions of the fixed cameras 10 and mobile cameras 20 on the map 200 can be calculated using the position information of the fixed cameras 10, the position information of the mobile cameras 20, and the position information of the locations represented by the map 200. If the arrangement of the fixed cameras 10 is fixed, their positions may be shown on the map 200 in advance.

[0069] First, the generation unit 2060 generates object distribution information within the shooting range of each fixed camera 10 using the first monitoring image 12 generated by each fixed camera 10. Furthermore, the generation unit 2060 corrects each distribution information using the number of objects calculated for the second monitoring image 22. The method for correcting the distribution information using the number of objects calculated for the second monitoring image 22 is the same as the method explained with reference to Figure 13.

[0070] The generation unit 2060 overlays the display based on the corrected distribution information onto the map 200. Figure 16 is an example of the map 200 with the heatmap generated based on the corrected distribution information overlaid on it.

[0071] As mentioned above, displaying distribution information on a map simplifies the background, making it easier for monitors and other personnel to visually grasp the distribution information. Furthermore, displaying distribution information on a map allows the crowd situation in areas monitored by multiple fixed cameras 10 to be displayed on a single screen. Therefore, monitors and other personnel can easily grasp the situation of crowds distributed over a wide area.

[0072] Furthermore, the generation unit 2060 may accept an operation from the user to select a mark 50, similar to the process described with reference to Figures 8 and 9, and display the second monitoring image 22 generated by the mobile camera 20 corresponding to the selected mark 50 on the map 200. Alternatively, the generation unit 2060 may display distribution information (information showing the distribution of objects within the shooting range of the mobile camera 20) generated using the second monitoring image 22 on the map 200, either in place of or together with the second monitoring image 22.

[0073] <Variation> The generation unit 2060 may generate the aforementioned distribution information using only the second monitoring image 22 without using the first monitoring image 12. In this case, for example, the generation unit 2060 calculates the number of objects captured in the second monitoring image 22 generated by each of the multiple mobile cameras 20, and generates the distribution information as a list of combinations of "shooting range of the second monitoring image 22 and number of objects". Then, the generation unit 2060 uses this distribution information to generate a heat map or the like, and displays it superimposed on the first monitoring image 12 or the map 200.

[0074] Alternatively, the generation unit 2060 may generate distribution information for each second monitoring image 22 in the same manner as the method used to generate distribution information for the first monitoring image 12. In this case, the generation unit 2060 generates a heatmap or the like using multiple distribution information sources and displays it superimposed on the first monitoring image 12 or the map 200.

[0075] [Embodiment 2] Figure 17 is a block diagram illustrating a monitoring information generation device 2000 according to Embodiment 2. In Figure 17, each block represents a functional unit configuration, not a hardware unit configuration.

[0076] The monitoring information generation device 2000 of Embodiment 2 has a function to estimate the shooting direction of the mobile camera 20. In the case of a fixed camera 10 installed on a wall or the like, the shooting direction of the fixed camera 10 can generally be determined by obtaining camera parameters that represent the current orientation of the camera (such as the horizontal rotation angle and the vertical rotation angle). For example, suppose there is a fixed camera 10 that is initially installed facing north, and the camera parameters obtained from that fixed camera 10 represent a horizontal rotation angle of +45 degrees. Here, the horizontal rotation angle is expressed with counterclockwise rotation as the positive direction. This point will also be the same in the following description. Then, it can be seen that the shooting direction of the fixed camera 10 is northwest, which is the direction rotated 45 degrees clockwise from north.

[0077] On the other hand, in the case of a mobile camera 20 worn by a person or other object, unlike a fixed camera 10, it is difficult to determine the shooting direction of the mobile camera 20 because it is difficult to set an initial reference orientation.

[0078] One possible solution is to equip the camera with an electronic compass to determine the shooting direction. However, with this method, if the electronic compass is inaccurate, it may not be possible to determine the correct shooting direction. Furthermore, if a high-precision electronic compass must be attached to the mobile camera 20, the manufacturing cost of the mobile camera 20 may increase. Moreover, if a high-precision electronic compass must be attached to the mobile camera 20, it becomes difficult to use general-purpose cameras such as smartphone cameras or handheld cameras as the mobile camera 20.

[0079] Therefore, the surveillance information generation device 2000 of this embodiment uses the first surveillance image 12 and the second surveillance image 22 to estimate the shooting direction of the mobile camera 20. Figure 18 is a diagram that conceptually illustrates the operation of the surveillance information generation device 2000 of Embodiment 2. The surveillance information generation device 2000 acquires the first surveillance image 12 generated by the fixed camera 10 and estimates the direction of movement of the crowd in the first surveillance image 12 (hereinafter, the first direction of movement). The surveillance information generation device 2000 acquires the second surveillance image 22 generated by the mobile camera 20 and estimates the direction of movement of the crowd in the second surveillance image 22 (hereinafter, the second direction of movement). Then, the surveillance information generation device 2000 estimates the shooting direction of the mobile camera 20 based on the direction of movement of the crowd in the first surveillance image 12 and the direction of movement of the crowd in the second surveillance image 22.

[0080] To achieve the above operation, the monitoring information generation device 2000 of Embodiment 2 further includes a first movement direction estimation unit 2100, a second movement direction estimation unit 2120, and a shooting direction estimation unit 2140. The first movement direction estimation unit 2100 estimates the movement direction of the crowd in the first monitoring image 12. The second movement direction estimation unit 2120 estimates the movement direction of the crowd in the second monitoring image 22. The shooting direction estimation unit 2140 estimates the shooting direction of the mobile camera 20 based on the first movement direction, the second movement direction, the position and orientation of the fixed camera 10, and the position of the mobile camera 20.

[0081] <Effects and Actions> According to the monitoring information generation device 2000 of this embodiment, the shooting direction of the mobile camera 20 is estimated using the first monitoring image 12 and the second monitoring image 22. Therefore, even if a device such as an electronic compass attached to the mobile camera 20 cannot accurately calculate the shooting direction of the mobile camera 20, the shooting direction of the mobile camera 20 can be accurately determined.

[0082] The shooting direction of the mobile camera 20 estimated by the monitoring information generation device 2000 can be used for processes such as visualizing the shooting direction of the mobile camera 20 as described in Embodiment 1, or mapping the area captured in the second monitoring image 22 onto the first monitoring image 12 or a map. However, the method of using the estimated shooting direction of the mobile camera 20 is arbitrary and is not limited to the method of use described in Embodiment 1.

[0083] The monitoring information generation device 2000 of this embodiment will be described in more detail below.

[0084] <Processing flow> Figure 19 is a flowchart illustrating the processing flow performed by the monitoring information generation device 2000 of Embodiment 2. The first movement direction estimation unit 2100 estimates the first movement direction using the first monitoring image 12 (S202). The second movement direction estimation unit 2120 estimates the second movement direction using the second monitoring image 22 (S204). The shooting direction estimation unit 2140 estimates the shooting direction of the mobile camera 20 based on the first movement direction, the second movement direction, the position and orientation of the fixed camera 10, and the position of the mobile camera 20 (S206).

[0085] <Details of the first movement direction estimation unit 2100> The first movement direction estimation unit 2100 estimates the first movement direction, which is the direction of movement of the crowd in the first monitoring image 12 (S202). There are various methods by which the first movement direction estimation unit 2100 estimates the first movement direction. The following describes the methods for estimating the first movement direction.

[0086] <<Method 1>> The first movement direction estimation unit 2100 calculates the optical flow of pixels and feature points contained in each of the multiple first monitoring images 12 arranged in time series. Figure 20 is an example of the optical flow calculated for the first monitoring image 12. Each arrow shown in Figure 20 represents the optical flow calculated for the first monitoring image 12.

[0087] The first movement direction estimation unit 2100 estimates the first movement direction based on the calculated optical flow. For example, the first movement direction estimation unit 2100 selects one optical flow and sets that selected optical flow as the first movement direction. For example, the first movement direction estimation unit 2100 randomly selects one optical flow.

[0088] For example, the first movement direction estimation unit 2100 statistically processes multiple calculated optical flows to calculate a single vector, and uses this vector as the first movement direction. This statistical processing is, for example, the process of calculating the average of the vectors.

[0089] Since the technique of calculating optical flow using pixels and feature points contained in an image is a known technique, a detailed explanation of this technique will be omitted.

[0090] <<Method 2>> The first movement direction estimation unit 2100 detects objects that are commonly seen in multiple first monitoring images 12 arranged in a time series, and estimates the first movement direction based on the change in the position of those objects. Figure 21 is an example of the change in the position of an object. In Figure 21, objects represented by dotted lines are seen in the t-th first monitoring image 12, and objects represented by solid lines are seen in the t+1-th first monitoring image 12. The arrows represent the change in the position of each object. The change in the position of an object is, for example, a vector connecting the centroids of multiple regions representing the same object.

[0091] Furthermore, if multiple objects are visible in the first monitoring image 12, multiple vectors representing the changes in the object's position are calculated, similar to the case where optical flow is used as described above. For example, the first movement direction estimation unit 2100 selects one object from the multiple objects and uses the vector representing the change in the position of the selected object as the first movement direction. For example, the first movement direction estimation unit 2100 randomly selects one object. Alternatively, for example, the first movement direction estimation unit 2100 selects the largest object.

[0092] For example, the first movement direction estimation unit 2100 statistically processes multiple vectors representing the changes in the positions of multiple objects to calculate a single vector, and sets this vector as the first movement direction. This statistical processing is, for example, the process of calculating the average of the vectors.

[0093] <<Method 3>> Alternatively, the first movement direction estimation unit 2100 may detect the direction of movement of the crowd based on the orientation of objects shown in the first monitoring image 12. For example, if the object is a person or an animal, the first movement direction estimation unit 2100 determines the orientation of their face or body and sets the direction in which their face or body is facing as the first direction of movement. If the object is a car, motorcycle, or flying object, the first movement direction estimation unit 2100 determines the direction of travel of the object from the shape of the object and the positions of various parts (such as bumpers and handlebars) shown in the first monitoring image 12 and sets the determined direction of travel as the first direction of movement.

[0094] <Details of the second movement direction estimation unit 2120> The method by which the second movement direction estimation unit 2120 estimates the movement direction (second movement direction) of the crowd captured in the second monitoring image 22 is the same as the method by which the first movement direction estimation unit 2100 estimates the first movement direction.

[0095] <Regarding reducing camera shake> The crowd may appear blurred in the first surveillance image 12 and the second surveillance image 22. For example, when a fixed camera 10 or a mobile camera 20 photographs a moving crowd, the crowd may appear blurred in each surveillance image. Also, for example, when the fixed camera 10 is photographed while its posture is changed, or when the mobile camera 20 is photographed while its posture and position are changed, the crowd may appear blurred in each surveillance image.

[0096] Therefore, it is preferable that the first movement direction estimation unit 2100 performs a process to reduce blur (such as so-called image stabilization) on each first monitoring image 12 before estimating the first movement direction. Similarly, it is preferable that the second movement direction estimation unit 2120 performs a process to reduce blur on each second monitoring image 22 before estimating the second movement direction.

[0097] <Details of the shooting direction estimation unit 2140> The shooting direction estimation unit 2140 estimates the shooting direction of the moving camera 20 based on the first movement direction, the second movement direction, the position and orientation of the fixed camera 10, and the position of the moving camera 20 (SS210). Figure 22 is a diagram illustrating the operation of the shooting direction estimation unit 2140. The specific process by which the shooting direction estimation unit 2140 estimates the shooting direction of the moving camera 20 will be described below using Figure 22.

[0098] First, the shooting direction estimation unit 2140 maps the first movement direction onto a plane (for example, on a map) that views the location of the monitored object in a vertical direction. In Figure 22, plane 70 is a plane that views the location of the monitored object in a vertical direction. The movement direction 80 represents the direction in which the movement direction of the crowd in the first monitoring image 12 is mapped onto plane 70. Plane 70 is a plane with north at the top.

[0099] The shooting direction estimation unit 2140 uses the movement direction 80 to calculate the movement direction (hereinafter referred to as the third movement direction) of the object captured by the moving camera 20 on the plane 70. When the shooting range of the fixed camera 10 and the shooting range of the moving camera 20 overlap, the third movement direction is the same as the movement direction 80. Figure 22, mentioned above, illustrates the case where the shooting range of the fixed camera 10 and the shooting range of the moving camera 20 overlap. The processing for the case where the shooting range of the fixed camera 10 and the shooting range of the moving camera 20 do not overlap will be described later.

[0100] The shooting direction estimation unit 2140 determines the shooting direction of the moving camera 20 to be estimated from among a plurality of candidate shooting directions (hereinafter referred to as candidate shooting directions). First, for each candidate shooting direction, the shooting direction estimation unit 2140 calculates the direction of movement of the crowd captured by the moving camera 20 (hereinafter referred to as candidate movement direction) as seen from the position of the moving camera 20 looking in that candidate shooting direction.

[0101] In Figure 22, the candidate shooting directions 92 are the eight directions: north, northeast, east, southeast, south, southwest, west, and northwest. The candidate movement directions 94-1 to 94-3 are the candidate movement directions when viewed from the position of the moving camera 20, as seen from candidate shooting directions 92-1, 92-2, and 92-3, respectively. In this example, since the crowd is hardly or not visible when viewed from the position of the moving camera 20 in candidate shooting directions other than the three mentioned above, these other candidate shooting directions have been omitted.

[0102] The shooting direction estimation unit 2140 matches each candidate movement direction with the second movement direction. In Figure 22, the second movement direction is the second movement direction 95. Specifically, the shooting direction estimation unit 2140 estimates that the candidate shooting direction 92 corresponding to the candidate movement direction 94 with the highest degree of agreement with the second movement direction 95 is the shooting direction of the moving camera 20. In Figure 22, the candidate movement direction 94 with the highest degree of agreement with the second movement direction 95 is candidate movement direction 94-2. Therefore, the shooting direction estimation unit 2140 estimates that the candidate shooting direction 92-2 corresponding to candidate movement direction 94-2 is the shooting direction of the moving camera 20.

[0103] Note that the candidate shooting directions are not limited to the eight directions mentioned above. For example, the candidate shooting directions may be the four directions of north, east, south, and west. Also, the candidate shooting directions are not limited to directions with established general names such as east, west, north, and south. For example, the shooting direction estimation unit 2140 determines the candidate shooting direction based on the direction of the sidewalk near the mobile camera 20. Figure 23 is a diagram illustrating the method of determining the candidate shooting direction based on the direction of the sidewalk. In Figure 23, the mobile camera 20 is located near the sidewalk 110. Therefore, it is thought that a monitor or other person monitoring a crowd moving on the sidewalk 110 will monitor with direction 120, which is the normal direction of the sidewalk 110, as the front direction.

[0104] For example, the shooting direction estimation unit 2140 determines each candidate shooting direction based on direction 120, which is the normal direction to the direction of the sidewalk 110. For example, the shooting direction estimation unit 2140 selects four directions as candidate shooting directions: direction 120, direction 121 obtained by rotating direction 120 by +90 degrees, direction 122 obtained by rotating direction 120 by +180 degrees, and direction 123 obtained by rotating direction 120 by +270 degrees.

[0105] Furthermore, when determining candidate shooting directions based on sidewalks or other features within the monitoring area, the shooting direction estimation unit 2140 includes the aforementioned map information acquisition unit 2080 and utilizes the map information acquired by the map information acquisition unit 2080.

[0106] <<Regarding the case where the shooting ranges of fixed camera 10 and mobile camera 20 do not overlap>> As mentioned above, the shooting direction estimation unit 2140 uses the movement direction 80 to calculate the movement direction (third movement direction) of the object captured by the moving camera 20 on the plane 70. The previous example assumed that the shooting range of the fixed camera 10 and the shooting range of the moving camera 20 overlapped. Below, we will explain the case where the shooting range of the fixed camera 10 and the shooting range of the moving camera 20 do not overlap.

[0107] If the shooting ranges of the fixed camera 10 and the mobile camera 20 do not overlap, the direction of movement of the crowd on the plane 70 as seen in the first monitoring image 12 may not be the same as the direction of movement of the crowd on the plane 70 as seen in the second monitoring image 22. Figure 24 illustrates a case where the shooting ranges of the fixed camera 10 and the mobile camera 20 do not overlap. In Figure 24, the direction in which the first direction of movement is mapped onto the plane 70 is direction of movement 80-1. In contrast, the direction of movement of the crowd on the plane 70 as seen by the mobile camera 20 is not necessarily the same as direction of movement 80-1, which is direction of movement 80-2, but could be direction of movement 80-3, for example.

[0108] For example, the shooting direction estimation unit 2140 acquires map information of the location to be monitored and calculates the direction of movement of the crowd on the plane 70 as captured by the mobile camera 20 based on that map information. Specifically, the shooting direction estimation unit 2140 uses the direction of movement of the crowd on the plane 70 as captured by the mobile camera 20, which is the direction of movement of the crowd on the plane 70 as captured by the mobile camera 20 when the mobile camera 20 is moved along a path (e.g., a sidewalk) on the map that is thought to be the path the crowd would move along.

[0109] Figure 25 is a diagram illustrating a method for estimating the direction of movement of a crowd on a plane 70 using map information. In this example, the object is a person. Also in Figure 25, the sidewalk 110 is a sidewalk where people walk. In this case, if the direction of movement 80-1, which is the direction of movement of the crowd on the plane 70 as seen in the first monitoring image 12, is moved along the sidewalk 110, the direction of movement of the crowd on the plane 70 near the mobile camera 20 becomes the direction of movement 80-3. Therefore, the shooting direction estimation unit 2140 sets the direction of movement 80-3 as the third direction of movement. More specifically, the shooting direction estimation unit 2140 sets the vector obtained by moving the direction of movement 80-1 along the line passing through the center of the sidewalk to the point where the distance between that line and the position of the mobile camera 20 is shortest as the third direction of movement.

[0110] However, if there are multiple possible routes for a crowd to move, as can be determined from the map information, it may be impossible to know which route the crowd will take. Figure 26 illustrates a case where there are multiple routes for a crowd to move. In Figure 26, since the sidewalk 110 is branched, the crowd can move in either direction 80-2 or direction 80-3.

[0111] In such cases, the shooting direction estimation unit 2140 acquires movement path information regarding the movement path of the crowd. For this purpose, the monitoring information generation device 2000 has a movement path information acquisition unit 2160. Figure 27 is a block diagram illustrating the monitoring information generation device 2000 having the movement path information acquisition unit 2160.

[0112] Movement path information indicates the direction in which a crowd will move. For example, if an event is held at an event venue, it is expected that the crowd will move from the nearest station to the venue towards the venue. In addition, when events requiring monitoring are held, it is common practice to guide the crowd along predetermined routes to the event venue in order to prevent accidents. Therefore, movement path information indicates these predetermined crowd movement routes.

[0113] Figure 28 illustrates a process for estimating the direction of movement of a crowd using movement path information. The movement path information acquired in Figure 28 is a map and information showing the movement paths on that map. In the map shown by this movement path information, the sidewalk 110 branches off, similar to Figure 26. Here, the movement path information acquired by the shooting direction estimation unit 2140 shows path 111. Therefore, the shooting direction estimation unit 2140 sets the movement direction 80-2, obtained by moving movement direction 80-1 along path 111, as the third movement direction.

[0114] It should be noted that there may be multiple pieces of information on movement routes for a particular location. For example, in the case of the event venue mentioned above, before the event starts, the crowd moves from the nearest station to the event venue, and after the event ends, the crowd moves from the event venue to the nearest station. In this case, for example, the movement route information acquisition unit 2160 acquires a combination of "the crowd's movement route and the time period in which that movement occurs" as movement route information. Here, it is assumed that the movement route information is pre-stored in a storage device accessible from the movement route information acquisition unit 2160.

[0115] Furthermore, when using movement path information, the shooting direction estimation unit 2140 does not need to use the first monitoring image 12 to estimate the shooting direction. In this case, the shooting direction estimation unit 2140 estimates the direction of movement of the crowd on the plane 70 shown in the second monitoring image 22 from the movement path of the crowd shown in the movement path information. For example, in Figure 28, the shooting direction estimation unit 2140 may not use the direction of movement 80-1, but instead estimate that the direction of movement of the crowd on the plane 70 shown in the second monitoring image 22 is the direction of movement 80-3 based on the path 111.

[0116] <<Narrowing down candidate shooting directions>> Furthermore, the shooting direction estimation unit 2140 may narrow down the candidate shooting directions before calculating candidate movement directions corresponding to the candidate shooting directions and matching them with the second movement directions. This reduces the number of matching operations, thus reducing the computational load on the shooting direction estimation unit 2140. Several methods for narrowing down candidate shooting directions are given below as examples.

[0117] <<<How to narrow down your search using an electronic compass>>> The shooting direction estimation unit 2140 uses an electronic compass to narrow down the candidate shooting directions. Specifically, the shooting direction estimation unit 2140 excludes directions from the candidate shooting directions that have a large difference in angle from the direction indicated by the electronic compass.

[0118] Figure 29 is a diagram illustrating how to narrow down candidate shooting directions using an electronic compass. In Figure 29, the candidate shooting directions are eight directions, such as north and northwest. The electronic compass is pointing northwest. In this case, due to the insufficient accuracy of the electronic compass, it is highly likely that the actual shooting direction of the mobile camera 20 is north or west. On the other hand, it is unlikely that the shooting direction of the mobile camera 20 is the opposite direction indicated by the electronic compass, such as southeast.

[0119] Therefore, the shooting direction estimation unit 2140 excludes directions such as southeast, which have a large angular difference from the direction indicated by the electronic compass, from the candidate shooting directions. Here, a predetermined number representing the number of directions to be excluded from the candidate shooting directions is set in advance. For example, if the predetermined number is 3, the shooting direction estimation unit 2140 excludes southeast, south, and east from the candidate shooting directions. Also, if the predetermined number is 5, the shooting direction estimation unit 2140 excludes southeast, south, east, southwest, and northeast from the candidate shooting directions. In Figure 29, southeast, south, and east are excluded from the candidate shooting directions.

[0120] <<<Method of narrowing down the results using the background shown in the second monitoring image 22>>> The shooting direction estimation unit 2140 narrows down candidate shooting directions based on the background visible in the second monitoring image 22. For example, the shooting direction estimation unit 2140 excludes candidate shooting directions in which the background visible in the second monitoring image 22 is not expected to be within the field of view of the moving camera 20.

[0121] Figure 30 is a diagram illustrating a method for narrowing down candidate shooting directions based on the background visible in the second surveillance image 22. Figure 30 shows a map of the area surrounding the monitored object. Here, let's assume that building 160 is visible in the second surveillance image 22. In this case, for example, if the shooting direction of the mobile camera 20 is southeast, building 160 will not be visible in the second surveillance image 22. On the other hand, if the shooting direction of the mobile camera 20 is northwest, it is thought that building 160 will be visible in the second surveillance image 22.

[0122] The shooting direction estimation unit 2140 then extracts background elements from the background visible in the second surveillance image 22, such as distinctive buildings and signs around the surveillance target. The shooting direction estimation unit 2140 then excludes candidate shooting directions that are unlikely to capture the extracted background elements, based on the relationship between the map location of the extracted background elements and the position of the mobile camera 20. Specifically, the shooting direction estimation unit 2140 excludes candidate shooting directions where the angle difference between the starting point of the mobile camera 20 and the map location of the extracted background elements is large. For example, the shooting direction estimation unit 2140 excludes a predetermined number of candidate shooting directions, similar to the exclusion of candidate shooting directions using the electronic compass described above. In Figure 30, southeast, south, and east are excluded from the candidate shooting directions.

[0123] Alternatively, for example, the shooting direction estimation unit 2140 may narrow down candidate shooting directions based on the position of the extracted background in the second monitoring image 22. Figure 31 is a diagram illustrating a method for narrowing down candidate shooting directions based on the position of a specific background on the second monitoring image 22. Figure 31(a) shows the second monitoring image 22, and Figure 31(b) shows the positional relationship between the mobile camera 20 and the building 160 on the map. In the second monitoring image 22 of Figure 31(a), the building 160 is positioned to the right of the center of the second monitoring image 22. In this case, the shooting direction of the mobile camera 20 will be a direction rotated in the positive direction from the direction (direction 170) that connects the position of the mobile camera 20 and the position of the building 160 on the map.

[0124] Therefore, the shooting direction estimation unit 2140 excludes candidate shooting directions in which the angle between the direction connecting the position of the moving camera 20 and the position of the building 160 on the map is in the range of -0 degrees to -180 degrees. For example, in the case of Figure 31(b), candidate shooting direction 174 is excluded from the candidate shooting directions, while candidate shooting direction 172 is not excluded from the candidate shooting directions.

[0125] <<Overall estimation of the shooting direction of mobile camera 20>> The shooting direction estimation unit 2140 may estimate the shooting direction of the mobile camera 20 multiple times over a predetermined period and comprehensively estimate the shooting direction of the mobile camera 20 based on these multiple estimation results. For example, the shooting direction estimation unit 2140 may calculate a comprehensive estimation result of the shooting direction of the mobile camera 20 every second. Also, suppose that the fixed camera 10 and the mobile camera 20 are cameras that capture images at a frequency of 30 frames per second (30fps). In this case, the shooting direction estimation unit 2140 estimates the shooting direction of the mobile camera 20 30 times per second by estimating the shooting direction of the mobile camera 20 each time the first monitoring image 12 and the second monitoring image 22 are generated. Then, it calculates a comprehensive estimation result based on these 30 estimation results.

[0126] Specifically, the shooting direction estimation unit 2140 statistically processes multiple estimation results calculated over a predetermined period to estimate the shooting direction of the moving camera 20. For example, the shooting direction estimation unit 2140 uses the mode of the multiple estimation results as the shooting direction of the moving camera 20. For example, suppose the breakdown of the estimation results, which were performed 30 times in one second, was "North: 20 times, Northwest: 8 times, Northeast: 2 times". In this case, the shooting direction estimation unit 2140 estimates that North, which was calculated most frequently, is the shooting direction of the moving camera 20 during that one second.

[0127] Alternatively, the shooting direction estimation unit 2140 may calculate an average value of multiple estimation results and use that average value as the shooting direction of the moving camera 20. Specifically, the shooting direction estimation unit 2140 represents each of the multiple estimated directions of the moving camera 20 calculated as a numerical value with east being +0 degrees, calculates the average value of these values, and uses this value as the shooting direction of the moving camera 20.

[0128] Furthermore, when the shooting direction estimation unit 2140 calculates an overall estimation result using the results of multiple estimations performed over a predetermined period, it may calculate the overall estimation result using only the result with the highest reliability among the multiple estimation results. For example, suppose the shooting direction estimation unit 2140 calculates an overall estimation result of the shooting direction of the moving camera 20 once per second. Also, suppose that the fixed camera 10 and the moving camera 20 are cameras that shoot at 30fps.

[0129] In this case, the shooting direction estimation unit 2140 estimates the shooting direction of the moving camera 20 30 times per second by estimating the shooting direction of the moving camera 20 each time the first monitoring image 12 and the second monitoring image 22 are generated. Next, the shooting direction estimation unit 2140 divides the 30 estimation results per second into groups of 10 that are consecutive in time series. Then, the shooting direction estimation unit 2140 uses the group with the highest confidence level from among the three groups to calculate the overall estimation result of the shooting direction of the moving camera 20 for that second.

[0130] The reliability of the above-mentioned groups is determined, for example, based on the magnitude of the variance of the estimation results within that group. If the variance of the estimation results is large, the estimated shooting direction of the mobile camera 20 is widely varied, and therefore the reliability of the estimation results is considered low. On the other hand, if the variance of the estimation results is small, the estimated shooting direction of the mobile camera 20 is widely varied, and therefore the reliability of the estimation results is considered high. For example, the shooting direction estimation unit 2140 calculates the variance of the estimation results within a group by representing each estimation result as a numerical value with east being +0 degrees. Then, the shooting direction estimation unit 2140 calculates an overall estimation result using only the estimation results included in the group whose calculated variance is below a predetermined value.

[0131] <<<Utilizing changes in crowd flow>>> In some locations, crowd flow may change periodically or irregularly. For example, near intersections, crowd flow changes with the switching of traffic lights. Figure 32 illustrates an example of how crowd flow changes near an intersection. Figure 32 shows an intersection viewed in a vertical plane. Suppose at a certain point in time, traffic light 131 for pedestrian crossing 130 is green, and traffic light 141 for pedestrian crossing 140 is red. In this case, the crowd will move in the direction of pedestrian crossing 130, for example, in direction 132. Then, suppose traffic light 131 turns red and traffic light 141 turns green. In this case, the crowd will move in the direction of pedestrian crossing 140, for example, in direction 142.

[0132] When the flow of the crowd changes in this way, the shooting direction estimation unit 2140 may estimate the shooting direction of the mobile camera 20 before and after the change, and then comprehensively estimate the shooting direction of the mobile camera 20 from the results. Specifically, the shooting direction estimation unit 2140 uses the first monitoring image 12 and the second monitoring image 22, which were taken before and after the change in the flow of the crowd, to estimate the shooting direction of the mobile camera 20 before and after the change in the flow of the crowd. Then, the shooting direction estimation unit 2140 calculates the final estimation result by statistically processing the estimated shooting direction.

[0133] For example, in the example shown in Figure 32, if the crowd is moving in direction 132, the shooting direction estimation unit 2140 estimates the shooting direction of the mobile camera 20, and the candidate directions of movement with a high degree of agreement with the second direction of movement are said to be north and northwest. Next, if the crowd is moving in direction 142, the shooting direction estimation unit 2140 estimates the shooting direction of the mobile camera 20, and the candidate directions of movement with a high degree of agreement with the second direction of movement are said to be north and northeast. In this case, the shooting direction estimation unit 2140 estimates that north, which had a high degree of agreement with the second direction of movement (the mode) in both the case where the crowd is moving in direction 132 and the case where it is moving in direction 142, is the shooting direction of the mobile camera 20.

[0134] <<Tracking of changes in the shooting direction of the mobile camera 20>> The shooting direction of the moving camera 20 may change. Therefore, it is preferable for the shooting direction estimation unit 2140 to repeatedly estimate the shooting direction of the moving camera 20. For example, the shooting direction estimation unit 2140 repeatedly estimates the shooting direction of the moving camera 20 at a frequency such as once per second or once every 10 seconds.

[0135] However, once the shooting direction estimation unit 2140 has estimated the shooting direction of the moving camera 20 using the method described above, it can then estimate the subsequent shooting direction of the moving camera 20 based on the subsequent change in the shooting direction of the moving camera 20. Specifically, the shooting direction estimation unit 2140 calculates the change in the shooting direction of the moving camera 20 and estimates the shooting direction of the moving camera 20 after the change from the calculated change and the shooting direction of the moving camera 20 estimated before the change. For example, suppose that at time t, the shooting direction estimation unit 2140 estimates that the shooting direction of the moving camera 20 is north. Then, suppose that t1 seconds later, the shooting direction estimation unit 2140 calculates that the shooting direction of the moving camera 20 has changed by +45 degrees. In this case, the shooting direction estimation unit 2140 estimates that the shooting direction of the moving camera 20 at time t+t1 is northwest.

[0136] Therefore, the shooting direction estimation unit 2140 may perform a part of the repeatedly performed process of estimating the shooting direction of the moving camera 20 by estimating the shooting direction of the moving camera 20 based on the change in the shooting direction of the moving camera 20. Hereinafter, the process of estimating the shooting direction of the moving camera 20 using the first and second movement directions will be referred to as the first estimation process, and the process of estimating the shooting direction of the moving camera 20 by calculating the change in the shooting direction of the moving camera 20 will be referred to as the second estimation process.

[0137] Figure 33 is a diagram illustrating the breakdown of the shooting direction estimation process of the moving camera 20 performed by the shooting direction estimation unit 2140 in a time series. In the example in Figure 33, the shooting direction estimation unit 2140 estimates the shooting direction of the moving camera 20 at a frequency of once per second. Here, the shooting direction estimation unit 2140 repeats the process of "performing the first estimation process once, and then performing the second estimation process nine times." Therefore, the frequency at which the first estimation process is performed is once every 10 seconds.

[0138] As mentioned above, the shooting direction estimation unit 2140 may perform multiple estimations of the shooting direction of the moving camera 20 and comprehensively estimate the shooting direction of the moving camera 20 based on the results of these multiple estimations. In this case, in Figure 33, the process of comprehensively estimating the shooting direction of the moving camera 20 is represented as a single first estimation process.

[0139] There are various methods for calculating the change in the shooting direction of the moving camera 20. For example, the change in the shooting direction of the moving camera 20 can be calculated using an acceleration sensor attached to the moving camera 20. By using an acceleration sensor attached to the moving camera 20, the relative change in the attitude of the moving camera 20 can be calculated from the change in the output of the acceleration sensor. For example, the shooting direction estimation unit 2140 calculates the change in the shooting direction of the moving camera 20 at time t+t1 from the difference between the output of the acceleration sensor when the shooting direction of the moving camera 20 at time t was estimated and the output of the acceleration sensor at time t+t1.

[0140] For example, the change in the shooting direction of the mobile camera 20 can be calculated by tracking the changes in feature points captured in the second monitoring image 22. Figure 34 illustrates the change in the position of feature points in the second monitoring image 22. Here, suppose a feature point that was captured at position 150-1 at time t moves to position 150-2 at time t+t1. Here, the horizontal distance moved is x. First, since the feature point has moved to the right, it can be seen that the shooting direction of the mobile camera 20 has changed in the + direction. Furthermore, based on the horizontal distance x to which the feature point has moved and the field of view of the mobile camera 20, the magnitude of the change in the shooting direction of the mobile camera 20 can be determined. Therefore, the shooting direction estimation unit 2140 can calculate the change in the shooting direction of the mobile camera 20 from the direction in which the shooting direction of the mobile camera 20 has changed and the magnitude of that change.

[0141] The computational complexity of the second estimation process (the process of estimating the shooting direction of the moving camera 20 by utilizing the change in the shooting direction of the moving camera 20) is smaller than that of the first estimation process (the process of estimating the shooting direction of the moving camera 20 by calculating the first and second directions of movement). Therefore, when repeatedly calculating the shooting direction of the moving camera 20, using the first and second estimation processes in combination has the effect of reducing the processing load on the monitoring information generation device 2000.

[0142] <Regarding correction of tilt in surveillance images> It is preferable that the first movement direction estimation unit 2100, the second movement direction estimation unit 2120, and the shooting direction estimation unit 2140 correct the vertical tilt of the first monitoring image 12 and the second monitoring image 22 before performing the above-described processing. For example, the vertical tilt of each image can be corrected based on the tilt of lines of buildings, etc., that are visible in the image. For example, if a building is visible in the first monitoring image 12, the first movement direction estimation unit 2100, etc., extracts the line in the height direction of the building visible in the first monitoring image 12 and corrects the vertical tilt of the first monitoring image 12 by correcting that line so that it is perpendicular to the horizontal direction of the first monitoring image 12.

[0143] Furthermore, the vertical tilt correction of the first monitoring image 12 may be performed using camera parameters that represent the vertical tilt of the fixed camera 10. Also, the vertical tilt correction of the second monitoring image 22 may be performed using the vertical tilt of the mobile camera 20, which can be calculated from the acceleration sensor attached to the mobile camera 20.

[0144] <Example Hardware Configuration of Monitoring Information Generator 2000> The monitoring information generation device 2000 of Embodiment 2 is implemented using the computer 1000, similar to Embodiment 1 (see Figure 4). In this embodiment, each program module stored in the aforementioned storage 1080 further includes programs that implement each of the functions described in this embodiment.

[0145] <Variation> A device for estimating the shooting direction of the mobile camera 20 may be provided separately from the monitoring information generation device 2000. This device will be referred to as the shooting direction estimation device 3000. Figure 35 is a block diagram illustrating the shooting direction estimation device 3000. In Figure 35, each block represents a functional unit configuration, not a hardware unit configuration.

[0146] The shooting direction estimation device 3000 includes a first monitoring image acquisition unit 2020, a second monitoring image acquisition unit 2040, a first movement direction estimation unit 2100, a second movement direction estimation unit 2120, and a shooting direction estimation unit 2140. The functions of each functional component are as described above.

[0147] The hardware configuration of the shooting direction estimation device 3000 is similar to that of the monitoring information generation device 2000, as shown in Figure 4, for example. The storage of the computer that implements the shooting direction estimation device 3000 stores program modules for realizing the functions of each functional component shown in Figure 35.

[0148] Furthermore, if a shooting direction estimation device 3000 is provided independently of the monitoring information generation device 2000, the generation unit 2060 of the monitoring information generation device 2000 acquires and uses the shooting direction of the mobile camera 20 from the shooting direction estimation device 3000.

[0149] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and various other configurations can also be adopted.

[0150] Examples of reference formats are provided below. 1. A first acquisition means for acquiring a first surveillance image captured by a fixed camera, which is a camera whose position is fixed, A second acquisition means for acquiring a second surveillance image captured by a mobile camera, which is a camera whose position is not fixed, A monitoring information generation device having generation means for generating object monitoring information using the first monitoring image and the second monitoring image. 2. The monitoring information generating device according to 1, wherein the generating means generates a display in which the second monitoring image is superimposed on the first monitoring image as monitoring information. 3. The monitoring information generation device according to 2., wherein the generation means superimposes the second monitoring image onto the position on the first monitoring image corresponding to the real-world position of the mobile camera. 4. The generating means is A mark representing the mobile camera is displayed on the first surveillance image. The monitoring information generating device according to 2. or 3., which, when the operation to select the mark is performed, generates a display as monitoring information in which the second monitoring image generated by the mobile camera corresponding to the mark is superimposed on the first monitoring image. 5. The generating means is Distribution information representing the distribution of objects captured in the first monitoring image is generated. The number of objects shown in the second surveillance image is calculated, The monitoring information generation device according to 1., which corrects the distribution information using the number of objects captured in the second monitoring image and generates the corrected distribution information as the monitoring information. 6. The monitoring information generation device according to 5, wherein the generation means generates a display in which the corrected distribution information is superimposed on the first monitoring image. 7. The system has means for acquiring map information to acquire map information of the location to be monitored, The monitoring information generation device according to 5., wherein the generation means generates a display in which the corrected distribution information is superimposed on a map represented by the map information. 8. The generating means is Using the shooting direction of the mobile camera, the shooting range of the mobile camera within the first monitoring image is calculated. A monitoring information generating device according to any one of 5. to 7., which corrects the number or distribution of objects within the shooting range of the mobile camera, as indicated by the distribution information, using the number of objects captured in the second monitoring image. 9. The monitoring information generating device according to any one of 1 to 8, wherein the generating means superimposes an indication of the shooting direction of the mobile camera onto the first monitoring image. 10. A first movement direction estimation means for estimating a first movement direction, which is the direction of movement of an object in the first monitoring image, A second movement direction estimation means for estimating a second movement direction, which is the direction of movement of an object in the second monitoring image, The system includes a shooting direction estimation means for estimating the shooting direction of the moving camera based on the first direction of movement, the second direction of movement, the position and orientation of the fixed camera, and the position of the moving camera. The monitoring information generating device according to 8. or 9., wherein the generating means uses the estimated shooting direction of the mobile camera as the shooting direction of the mobile camera. 11. The shooting direction estimation means is: Using the position and orientation of the fixed camera and the first direction of movement, a third direction of movement is calculated, which is the direction of movement of the object shown in the second monitoring image on a plane obtained by viewing the location of the monitored object in a vertical plane. When viewing an object moving in the third direction of movement from each of the multiple candidate shooting directions at the position of the aforementioned moving camera, the multiple candidate movement directions of the object are calculated. The monitoring information generating device according to 10., which estimates that the candidate movement direction that has the highest degree of agreement with the second movement direction is the shooting direction of the moving camera. 12. The surveillance information generation device according to 11, wherein the shooting direction estimation means calculates the direction of movement of an object captured in the first surveillance image on the plane using the position and orientation of the fixed camera and the first direction of movement, and sets the calculated direction of movement as the third direction of movement. 13. The system has means for acquiring movement path information that shows the movement path of an object, The aforementioned shooting direction estimation means is Using the position and orientation of the fixed camera and the first direction of movement, the direction of movement of the object captured in the first monitoring image on the plane is calculated. The monitoring information generating device according to 11., wherein the position obtained by moving the calculated direction of movement along the movement path shown in the movement path information toward the vicinity of the moving camera is defined as the third direction of movement. 14. A first movement direction estimation means for estimating the first movement direction, which is the direction of movement of an object in a first surveillance image captured by a fixed camera, which is a camera whose position is fixed. A second movement direction estimation means for estimating the second movement direction, which is the direction of movement of an object in a second surveillance image captured by a mobile camera, which is a camera whose position is not fixed, A shooting direction estimation device having shooting direction estimation means for estimating the shooting direction of the moving camera based on the first direction of movement, the second direction of movement, the position and orientation of the fixed camera, and the position of the moving camera. 15. The shooting direction estimation means is Using the position and orientation of the fixed camera and the first direction of movement, a third direction of movement is calculated, which is the direction of movement of the object shown in the second monitoring image on a plane obtained by viewing the location of the monitored object in a vertical plane. When viewing an object moving in the third direction of movement from each of the multiple candidate shooting directions at the position of the aforementioned moving camera, the multiple candidate movement directions of the object are calculated. The shooting direction estimation device according to 14., which estimates that the candidate movement direction that has the highest degree of agreement with the second movement direction is the shooting direction of the moving camera. 16. The shooting direction estimation device according to 15, wherein the shooting direction estimation means calculates the direction of movement of an object captured in the first monitoring image on the plane using the position and orientation of the fixed camera and the first direction of movement, and sets the calculated direction of movement as the third direction of movement. 17. The system has means for acquiring movement path information that shows the movement path of an object, The aforementioned shooting direction estimation means is Using the position and orientation of the fixed camera and the first direction of movement, the direction of movement of the object captured in the first monitoring image on the plane is calculated. The shooting direction estimation device according to 15., wherein the position obtained by moving the calculated direction of movement along the movement path shown in the movement path information toward the vicinity of the moving camera is defined as the third direction of movement. 18. A method for generating monitoring information performed by a computer, A first acquisition step involves acquiring a first surveillance image captured by a fixed camera, which is a camera whose position is fixed. A second acquisition step involves acquiring a second surveillance image captured by a mobile camera, which is a camera whose position is not fixed. A method for generating monitoring information, comprising: a generation step of generating monitoring information for an object using the first monitoring image and the second monitoring image. 19. The monitoring information generation method according to 18, wherein the generation step generates a display in which the second monitoring image is superimposed on the first monitoring image as monitoring information. 20. The method for generating surveillance information according to 19, wherein the generation step involves superimposing the second surveillance image onto the position on the first surveillance image that corresponds to the real-world position of the mobile camera. 21. The generation step is: A mark representing the mobile camera is displayed on the first surveillance image. The monitoring information generation method according to 19. or 20., wherein when the operation to select the mark is performed, a display is generated as monitoring information, in which the second monitoring image generated by the mobile camera corresponding to the mark is superimposed on the first monitoring image. 22. The generation step is: Distribution information representing the distribution of objects captured in the first monitoring image is generated. The number of objects shown in the second surveillance image is calculated, The monitoring information generation method according to 18., wherein the distribution information is corrected using the number of objects captured in the second monitoring image, and the corrected distribution information is generated as the monitoring information. 23. The monitoring information generation method according to 22, wherein the generation step generates a display in which the corrected distribution information is superimposed on the first monitoring image. 24. The system includes a map information acquisition step to acquire map information of the location to be monitored, The monitoring information generation method according to 22, wherein the generation step generates a display in which the corrected distribution information is superimposed on a map represented by the map information. 25. The generation step is: Using the shooting direction of the mobile camera, the shooting range of the mobile camera within the first monitoring image is calculated. A method for generating surveillance information according to any one of 22 to 24, wherein the number or distribution of objects within the shooting range of the mobile camera, as indicated by the distribution information, is corrected using the number of objects captured in the second surveillance image. 26. The method for generating surveillance information according to any one of 18 to 25, wherein the generation step involves superimposing an indication of the shooting direction of the mobile camera onto the first surveillance image. 27. A first movement direction estimation step for estimating a first movement direction, which is the direction of movement of an object in the first monitoring image, A second movement direction estimation step for estimating a second movement direction, which is the direction of movement of an object in the second monitoring image, The system includes a shooting direction estimation step which estimates the shooting direction of the moving camera based on the first direction of movement, the second direction of movement, the position and orientation of the fixed camera, and the position of the moving camera, The method for generating surveillance information according to 25. or 26., wherein the generation step uses the estimated shooting direction of the mobile camera as the shooting direction of the mobile camera. 28. The above-mentioned step of estimating the shooting direction is: Using the position and orientation of the fixed camera and the first direction of movement, a third direction of movement is calculated, which is the direction of movement of the object shown in the second monitoring image on a plane obtained by viewing the location of the monitored object in a vertical plane. When viewing an object moving in the third direction of movement from each of the multiple candidate shooting directions at the position of the aforementioned moving camera, the multiple candidate movement directions of the object are calculated. The monitoring information generation method according to 27, wherein the candidate movement direction that has the highest degree of agreement with the second movement direction is estimated to be the shooting direction of the moving camera. 29. The method for generating surveillance information according to 28, wherein the shooting direction estimation step involves calculating the direction of movement of an object captured in the first surveillance image on the plane using the position and orientation of the fixed camera and the first direction of movement, and setting the calculated direction of movement as the third direction of movement. 30. The process includes a step of obtaining movement path information that shows the movement path of an object, The aforementioned step of estimating the shooting direction is: Using the position and orientation of the fixed camera and the first direction of movement, the direction of movement of the object captured in the first monitoring image on the plane is calculated. The monitoring information generation method according to 28., wherein the position obtained by moving the calculated direction of movement along the movement path shown in the movement path information toward the vicinity of the moving camera is defined as the third direction of movement. 31. A method for estimating the direction of photography performed by a computer, A first movement direction estimation step, which estimates the first movement direction, which is the direction of movement of an object in a first surveillance image captured by a fixed camera, which is a camera whose position is fixed. A second movement direction estimation step, which estimates the second movement direction, which is the direction of movement of an object in a second surveillance image captured by a mobile camera, which is a camera whose position is not fixed; A method for estimating a shooting direction, comprising: a shooting direction estimation step of estimating the shooting direction of the moving camera based on the first direction of movement, the second direction of movement, the position and orientation of the fixed camera, and the position of the moving camera. 32. The above shooting direction estimation step is: Using the position and orientation of the fixed camera and the first direction of movement, a third direction of movement is calculated, which is the direction of movement of the object shown in the second monitoring image on a plane obtained by viewing the location of the monitored object in a vertical plane. When viewing an object moving in the third direction of movement from each of the multiple candidate shooting directions at the position of the aforementioned moving camera, the multiple candidate movement directions of the object are calculated. The method for estimating a shooting direction according to 31, wherein the candidate movement direction that has the highest degree of agreement with the second movement direction is estimated to be the shooting direction of the moving camera. 33. The method for estimating the shooting direction according to 32, wherein the shooting direction estimation step involves calculating the direction of movement of an object captured in the first monitoring image on the plane using the position and orientation of the fixed camera and the first direction of movement, and the calculated direction of movement is defined as the third direction of movement. 34. The process includes a step to obtain movement path information that shows the movement path of an object, The aforementioned step of estimating the shooting direction is: Using the position and orientation of the fixed camera and the first direction of movement, the direction of movement of the object captured in the first monitoring image on the plane is calculated. The method for estimating the shooting direction according to 32, wherein the position obtained by moving the calculated direction of movement along the movement path shown in the movement path information toward the vicinity of the moving camera is defined as the third direction of movement. 35. A program that causes a computer to perform each of the steps described in any one of items 18 through 34.

[0151] This application claims priority based on Japanese Patent Application No. 2015-172082, filed on September 1, 2015, and incorporates all of its disclosures herein.

Claims

1. A first acquisition means for acquiring multiple first surveillance images captured by multiple fixed cameras whose positions are fixed, A second acquisition means for acquiring a second surveillance image captured by a mobile camera attached to an aircraft and not fixed in position, The system includes a generation means for generating a plurality of distribution information obtained by correcting the distribution of objects visible in the plurality of first monitoring images based on the second monitoring image, The generation means generates a display on which the plurality of distribution information is superimposed on the map, The aforementioned multiple distribution information is a heat map showing the degree of density of the objects. Information processing device.

2. The information processing apparatus according to claim 1, wherein the generation means generates the distribution information by correcting the distribution of objects shown in the first monitoring image using the number of objects shown in the second monitoring image.

3. The generating means is Using the shooting direction of the mobile camera, the shooting range of the mobile camera within the first monitoring image is calculated. The information processing apparatus according to claim 1 or 2, wherein the distribution information is generated by correcting the distribution of objects in the shooting range of the mobile camera, among the distribution of objects visible in the first monitoring image, based on the second monitoring image.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the object is a person.

5. At least one computer, Multiple first surveillance images are acquired, captured by multiple fixed cameras whose positions are fixed. A second surveillance image is obtained, captured by a mobile camera attached to the aircraft but not fixed in position. Multiple distribution information is generated by correcting the distribution of objects visible in the multiple first monitoring images based on the second monitoring image. A display is generated in which the above-mentioned distribution information is superimposed on the map. This includes, The aforementioned multiple distribution information is a heat map showing the degree of density of the objects. Information processing methods.

6. At least one computer, A first acquisition means for acquiring multiple first surveillance images captured by multiple fixed cameras whose positions are fixed, A second acquisition means for acquiring a second surveillance image captured by a mobile camera attached to an aircraft and not fixed in position. A generation means for generating a plurality of distribution information obtained by correcting the distribution of objects visible in the plurality of first monitoring images based on the second monitoring image, To make it function as, The generation means generates a display on which the plurality of distribution information is superimposed on the map, The aforementioned multiple distribution information is a heat map showing the degree of density of the objects. program.

Images