Information processing device, control method, and program

Abstract

This technology allows sensors used to detect moving objects to be positioned more freely. [Solution] The information processing device uses the sensor 10 to detect the moving object 20 and determines the direction (first direction 12) from the sensor 10 to the moving object 20. The information processing device moves the optical axis direction 34 of the camera 30 along the first direction 12 and causes the camera 30 to take images. As a result, the camera 30 generates multiple captured images 32, each capturing an image in a different direction. The information processing device 2000 detects the moving object 20 by performing image analysis on the captured images 32.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method, and a program.

Background Art

[0002] Cameras are used to monitor moving objects. For example, Patent Document 1 discloses a monitoring camera device that monitors a moving object using a wide-angle camera and a telephoto camera arranged adjacent to each other. This monitoring camera device detects a moving object using an image generated by the wide-angle camera, and changes the direction of the imaging optical axis of the telephoto camera based on the position of the moving object in the image. By doing so, the moving object is captured at the center of the imaging range of the telephoto camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the monitoring camera device of Patent Document 1, it is premised that the telephoto camera is placed near the wide-angle camera, that is, the telephoto camera and the wide-angle camera can image in substantially the same direction. Because of this premise, by directing the imaging optical axis of the telephoto camera in substantially the same direction as the direction from the wide-angle camera toward the moving object, the moving object can be imaged with the telephoto camera. Therefore, with the technology of Patent Document 1, the telephoto camera and the wide-angle camera cannot be provided at positions separated from each other.

[0005] The present invention has been made in view of this problem, and one of its objects is to provide a technology that can arrange sensors used for detecting a moving object at more flexible positions.

Means for Solving the Problems

[0006] The information processing device of the present invention includes: 1) a first detection means for detecting a moving object using a sensor and identifying a first direction from the sensor toward the moving object; 2) a control means for causing the first camera to take an image while moving the optical axis direction of the first camera along the first direction; and 3) a second detection means for detecting a moving object from a first image generated by the first camera.

[0007] The control method of the present invention is performed by a computer. The control method includes: 1) a first detection step of detecting a moving object using a sensor and identifying a first direction from the sensor toward the moving object; 2) a control step of causing the first camera to take an image while moving the optical axis direction of the first camera along the first direction; and 3) a second detection step of detecting a moving object from a first image generated by the first camera.

[0008] The program of the present invention causes a computer to execute each step of the control method of the present invention. [Effects of the Invention]

[0009] According to the present invention, a technology is provided that allows sensors used for detecting moving objects to be placed at more flexible positions. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating the overview of the information processing device of Embodiment 1. [Figure 2] This diagram illustrates the functional configuration of the information processing device in Embodiment 1. [Figure 3] This diagram illustrates a computer used to implement an information processing device. [Figure 4] This is a flowchart illustrating the processing flow performed by the information processing device of Embodiment 1. [Figure 5] This diagram illustrates a case where a second camera is used as a sensor. [Figure 6] This diagram illustrates a case where the system accepts user input to specify the detection area. [Figure 7] This is a diagram illustrating the detection space. [Figure 8] This diagram illustrates how the first direction is updated. [Figure 9] This is a plan view of the area where the moving object is being monitored, viewed vertically. [Figure 10] This is a side view of the area where the moving object is being monitored. [Figure 11] This figure illustrates the functional configuration of the information processing device in Embodiment 2. [Figure 12] This diagram illustrates the functional configuration of the information processing device in Embodiment 3. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted as appropriate. Unless otherwise specified, in each block diagram, each block represents a functional unit configuration, not a hardware unit configuration.

[0012] [Embodiment 1] <Overview> Figure 1 is a diagram illustrating the overview of the information processing device of Embodiment 1 (the information processing device 2000 shown in Figure 2). The operation of the information processing device 2000 described below is an example to facilitate understanding of the information processing device 2000, and the operation of the information processing device 2000 is not limited to the example below. Details and variations of the operation of the information processing device 2000 will be described later.

[0013] The information processing apparatus 2000 detects the moving object 20 by using the sensor 10, and identifies the direction (the first direction 12 in FIG. 1) from the sensor 10 toward the moving object 20. The information processing apparatus 2000 causes the camera 30 to perform imaging while moving the optical axis direction 34 of the camera 30 along the first direction 12 (that is, toward the moving object 20 from the sensor 10). As a result, a plurality of imaging images 32 obtained by imaging in different directions are generated by the camera 30. The information processing apparatus 2000 detects the moving object 20 by analyzing the imaging images 32.

[0014] According to this method, even if the sensor 10 and the camera 30 are not installed in the vicinity of each other, the camera 30 can detect the moving object 20. This is because since the moving object 20 exists on the first direction 12, by repeating the imaging by the camera 30 while moving the optical axis direction 34 of the camera 30 along the first direction 12, the moving object 20 can be included in one of the imaging images 32. Therefore, the information processing apparatus 2000 of the present embodiment has an advantage that the sensor 10 and the moving object 20 do not have to be installed in the vicinity of each other, and they can be arranged at more flexible positions.

[0015] Hereinafter, the information processing apparatus 2000 of the present embodiment will be described in more detail.

[0016] <Example of the functional configuration of the information processing apparatus 2000> FIG. 2 is a diagram illustrating the functional configuration of the information processing apparatus 2000 according to Embodiment 1. The information processing apparatus 2000 includes a first detection unit 2020, a control unit 2040, and a second detection unit 2060. The first detection unit 2020 detects the moving object 20 by using the sensor 10, and identifies the first direction 12 from the sensor 10 toward the moving object 20. The control unit 2040 causes the camera 30 to perform imaging while moving the optical axis direction 34 of the camera 30 along the first direction 12. The second detection unit 2060 detects the moving object 20 from the imaging images 32 generated by the camera 30.

[0017] <Hardware configuration of the information processing apparatus 2000> Each functional component of the information processing device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and programs that control them). The following will further explain the case where each functional component of the information processing device 2000 is implemented by a combination of hardware and software.

[0018] Figure 3 illustrates a computer 1000 for implementing the information processing device 2000. Computer 1000 is any computer. For example, computer 1000 may be a personal computer (PC), a server machine, a tablet terminal, or a smartphone. Computer 1000 may be a dedicated computer designed to implement the information processing device 2000, or it may be a general-purpose computer.

[0019] Computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. Bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. However, the method of connecting the processor 1040 and the other components is not limited to bus connection. The processor 1040 is a variety of processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). Memory 1060 is a main memory device implemented using RAM (Random Access Memory), etc. Storage device 1080 is an auxiliary storage device implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0020] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1100.

[0021] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1120 may connect to the network via a wireless connection or a wired connection.

[0022] For example, the network interface 1120 is connected to the sensor 10 and camera 30 via the network. Sensor 10 is any sensor capable of detecting the moving object 20. As will be described in more detail later, for example, sensor 10 can be a camera, acoustic sensor, or radio wave sensor. Camera 30 is any imaging device that takes images and generates captured images. For example, camera 30 is a video camera that generates video data by taking images periodically. In this case, the captured images 32 are each frame that makes up the video data.

[0023] The storage device 1080 stores program modules that realize each functional component of the information processing device 2000. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to each program module.

[0024] <Processing flow> Figure 4 is a flowchart illustrating the processing flow performed by the information processing device 2000 of Embodiment 1. The first detection unit 2020 detects the moving object 20 using the sensor 10 (S102). The first detection unit 2020 identifies a first direction 12 from the sensor 10 toward the moving object 20 (S104).

[0025] S106 to S116 is a loop process that is repeated until a predetermined condition is met. If the predetermined condition is met, the process in Figure 4 ends. On the other hand, if the predetermined condition is not met, the process in Figure 4 proceeds to S108.

[0026] The control unit 2040 moves the optical axis direction 34 of the camera 30 along the first direction 12 (S108). The control unit 2040 causes the camera 30 to take an image (S110). The second detection unit 2060 performs object detection processing on the captured image 32 generated by the camera 30 (S112). If an object is detected from the captured image 32 (S114: YES), the second detection unit 2060 detects the detected object as a moving object 20 (S118). Then the process in Figure 4 ends. On the other hand, if no object is detected from the captured image 32 (S114: NO), the process in Figure 4 proceeds to S106.

[0027] There are various predetermined conditions for terminating loop processing A. For example, one predetermined condition is that the camera 30 has finished imaging the entire range that can be imaged along the first direction 12. More specifically, one predetermined condition is that the imaging direction of the camera 30 has become parallel to the first direction 12.

[0028] Note that the processing flow executed by the information processing device 2000 is not limited to the flow shown in Figure 4. For example, the control unit 2040 may perform object detection processing on each of the generated captured images after the camera 30 has taken multiple images. In other words, the object detection processing for each of the multiple captured images generated by the camera 30 may be executed together after loop processing A.

[0029] Furthermore, the control unit 2040 may not unconditionally treat the object detected in S112 as a mobile body 20, but rather treat the object as a mobile body 20 only if it satisfies predetermined conditions. This point will be discussed later.

[0030] <Detection of a moving object 20 using sensor 10> The first detection unit 2020 detects the moving object 20 using the sensor 10. The method of detecting the moving object 20 using the sensor 10 depends on the type of sensor used. Below, examples of methods for detecting the moving object 20 are given for each of the main types of sensors that can be used as sensor 10.

[0031] <<Cases where the camera is used>> For example, a camera can be used as sensor 10. Here, the camera used as sensor 10 is referred to as the second camera. Figure 5 is an example of a case in which the second camera is used as sensor 10. In Figure 5, the second camera is represented by reference numeral 40. Hereafter, the image captured by the second camera 40 will be referred to as the second captured image 42.

[0032] The second camera 40 captures images with a wider field of view than camera 30. In other words, the second camera 40 captures images with a shorter focal length than camera 30. For example, a telephoto camera could be used for camera 30, and a wide-angle camera for the second camera 40. By using these two cameras, it becomes possible to first capture a wide area with the wide-angle camera (camera 30) to quickly determine the presence and direction of the moving object 20, and then capture a larger image of the moving object 20 with the telephoto camera (second camera 40) to gain a detailed understanding of the moving object 20.

[0033] The first detection unit 2020 detects the moving object 20 from the second captured image 42 generated by the second camera 40. Here, the first detection unit 2020 may detect any object included in the second captured image 42 as the moving object 20 (see Figure 3), or it may detect an object included in the second captured image 42 that satisfies predetermined conditions as the moving object 20. In the former case, the first detection unit 2020 extracts, for example, the foreground region from the second captured image 42 and identifies the object represented by that foreground region as the moving object 20. Here, existing techniques can be used for extracting the foreground region from the image.

[0034] On the other hand, let's say we want to detect an object that meets certain conditions as a moving object 20. In this case, for example, the feature quantities of the object to be detected as a moving object 20 are predetermined. The first detection unit 2020 detects an image region having these feature quantities from the second captured image 42, and the object represented by the detected image region is designated as a moving object 20. The object to be detected as a moving object 20 is, for example, a flying object such as an airplane, a drone, or a bird. Existing technologies can be used for the technology to determine the feature quantities of the object to be detected and for the technology to detect an object having those feature quantities from an image.

[0035] In addition, for example, the first detection unit 2020 may detect only moving objects among the objects detected in the second captured image 42 as moving objects 20. For example, if the control unit 2040 detects an object in a certain second captured image 42, it will track that object for each subsequent second captured image 42 to determine whether the object is moving. If it determines that the object is moving, the control unit 2040 will treat that object as a moving object 20. On the other hand, if it determines that the object is not moving, the control unit 2040 will not treat that object as a moving object 20. Existing technologies can be used for determining whether an object detected from an image is moving.

[0036] The first detection unit 2020 may target the entire image area of ​​the second captured image 42 for detection of the moving object 20, or it may target only a portion of the image area within the second captured image 42 for detection of the moving object 20. In the latter case, the image area targeted for detection of the moving object 20 is called the detection area.

[0037] For example, the information processing device 2000 accepts an input operation from the user to specify a detection area. Figure 6 illustrates a case in which the information processing device 2000 accepts a specification of a detection area from the user. In Figure 6, the touch panel 50 displays a video 52 consisting of a second captured image 42 captured by the second camera 40. The user finds an object that they want the information processing device 2000 to detect as a mobile object 20 by looking at the video 52. Once the user finds the object they want to detect as a mobile object 20, they touch the position on the touch panel 50 where that object is displayed. The first detection unit 2020 identifies the image position in the second captured image 42 that corresponds to this touched position and performs object detection in the image area of ​​the second captured image 42 based on that image position. The first detection unit 2020 then treats the detected object as a mobile object 20. For example, in Figure 6, a rectangular area 56 centered on the image position touched by the user's finger 54 is treated as the detection area. As a result, the drone 58 is detected as a mobile object 20.

[0038] There are various ways for a user to specify the detection area. For example, as described above, the user specifies a position on the second captured image 42. The first detection unit 2020 treats an image area of ​​a predetermined size and shape, determined based on the specified position, as the detection area. Existing technologies can be used to determine an image area of ​​a predetermined size and shape based on the specified position.

[0039] The user may specify a region rather than a position on the second captured image 42. In this case, the region specified by the user will be treated as the detection region. For example, drag or slide operations can be used to specify the region.

[0040] <<Cases where acoustic sensors are used>> For example, an acoustic sensor can be used as sensor 10. This acoustic sensor uses a microphone array in which multiple microphones are arranged three-dimensionally. Here, the arrangement of multiple microphones three-dimensionally means that there is no single plane that passes through all the microphones that make up the microphone array.

[0041] When the same sound is detected by multiple microphones arranged in a three-dimensional manner, the direction from the microphone array towards the source of that sound can be determined. Therefore, when a sound with predetermined characteristics is detected by multiple microphones constituting the acoustic sensor, the first detection unit 2020 detects the source of that sound as a moving object 20.

[0042] The "predetermined characteristics" described above can encompass a variety of features. For example, a predetermined characteristic may be represented by the magnitude of sound pressure. In this case, the first detection unit 2020 determines whether the sound pressure of the sound detected by the acoustic sensor is above a predetermined value, and if it is above the predetermined value, it treats the sound source as a mobile object 20. That is, it treats any sound source that emits a sound louder than a preset magnitude as a mobile object 20.

[0043] In addition, for example, a predetermined characteristic is represented by the frequency spectrum of the sound emitted by the mobile object 20 to be detected. In this case, the first detection unit 2020 analyzes the frequency spectrum of the sound detected by the acoustic sensor, and if the sound has a frequency spectrum that matches or is similar to the predetermined characteristic, it treats the source of that sound as the mobile object 20. When detecting an airplane as the mobile object 20, the predetermined characteristic is, for example, the frequency spectrum of the airplane's engine sound. When detecting a drone as the mobile object 20, the predetermined characteristic is, for example, the frequency spectrum of the drone's rotor rotation sound (hereinafter referred to as rotor sound).

[0044] Note that the frequency spectrum of an airplane engine sound varies depending on the aircraft model. The specified characteristics described above may be defined as a fairly broad range of frequency spectra to include the frequency spectra of various airplane engine sounds, or they may be represented as the frequency spectrum of the engine sound of a specific aircraft model. In the latter case, only aircraft of that specific model will be detected as moving objects 20.

[0045] Similarly, the frequency spectrum of a drone's rotor sound also differs depending on the drone model. Therefore, the predetermined feature may be defined as a range of a fairly broad frequency spectrum that includes the frequency spectra of rotor sounds from various drones, or it may be represented as the frequency spectrum of the rotor sound from a specific drone model. In the latter case, only drones of that specific model will be detected as the moving object 20.

[0046] Here, the characteristics of the sound emitted by the moving object to be detected as the moving object 20, that is, the information indicating the predetermined characteristics described above, is called sound characteristic information. The sound characteristic information is stored in a storage device accessible from the first detection unit 2020. The first detection unit 2020 acquires the sound characteristic information and determines whether or not a sound having the predetermined characteristics shown in the sound characteristic information has been detected by the acoustic sensor. If the first detection unit 2020 has detected a sound having the predetermined characteristics by the acoustic sensor, it identifies the direction from the sensor 10 toward the source of that sound as the first direction 12.

[0047] <<Cases utilizing radio wave sensors>> For example, a radio wave sensor can be used as sensor 10. This radio wave sensor uses a receiver array in which multiple radio wave receivers are arranged three-dimensionally. Here, the arrangement of multiple radio wave receivers three-dimensionally means that there is no single plane that passes through all the receivers constituting the receiver array. When the first detection unit 2020 detects radio waves having predetermined characteristics by the multiple receivers constituting the radio wave sensor, it treats the source of those radio waves as a moving object 20.

[0048] The above-mentioned predetermined characteristics can encompass a variety of features. For example, the predetermined characteristics can be represented by the amplitude of the radio waves. In this case, the first detection unit 2020 determines whether the amplitude of the radio waves detected by the radio wave sensor is greater than or equal to a predetermined value, and if it is greater than or equal to the predetermined value, it treats the source of those radio waves as a moving object 20. That is, it treats a source that emits radio waves with an amplitude greater than a preset value as a moving object 20.

[0049] In addition, for example, a predetermined characteristic is represented by the frequency spectrum of the radio waves emitted by the moving object 20 to be detected. In this case, the first detection unit 2020 analyzes the frequency spectrum of the radio waves detected by the radio wave sensor, and if the radio waves have a frequency spectrum that matches or is similar to the predetermined characteristic, it treats the source of the sound as the moving object 20. In this case, the frequency spectrum of the radio waves emitted by the moving object 20 to be detected is known in advance, and information representing that frequency spectrum is stored in the memory device.

[0050] In addition, for example, a predetermined characteristic can be represented by data carried on the radio waves emitted by the mobile object 20 to be detected. When the mobile object 20 transmits data carried on radio waves, the receiver can demodulate the radio waves and extract the data to understand its content. For example, suppose the mobile object 20 transmits an Ethernet® frame wirelessly. In this case, the source MAC (Media Access Control) address of this Ethernet® frame indicates the MAC address of the network interface that the mobile object 20 has. Therefore, the first detection unit 2020 can demodulate the radio waves received by the radio wave sensor to extract the Ethernet® frame, and by looking at the source MAC address of that Ethernet® frame, it can obtain information about the source of the radio waves.

[0051] For example, the first detection unit 2020 identifies the source of the data obtained from the radio waves received by the radio wave sensor, and detects the source as a mobile object 20 if it satisfies predetermined conditions. For example, the predetermined condition is that "the source is an object with a specific source address." In this case, the first detection unit 2020 detects only objects with a specific address as mobile objects 20. This makes it possible to detect only objects with a specific address as mobile objects 20.

[0052] Other conditions include, for example, that "the source belongs to a specified group." For example, MAC addresses generally include a vendor code. Therefore, this vendor code can be used to identify the manufacturer of the radio wave source. Thus, it becomes possible to detect only mobile devices 20 manufactured by a specific manufacturer.

[0053] Here, the information that exhibits the predetermined characteristics described above is called characteristic information. The characteristic information is stored in a storage device accessible from the first detection unit 2020. The first detection unit 2020 acquires the characteristic information and determines whether or not a sound having the predetermined characteristics shown in the characteristic information has been detected by the acoustic sensor, thereby detecting a moving object 20 that emits a sound having the predetermined characteristics.

[0054] Here, the characteristics of the radio waves emitted by the moving object to be detected as the moving object 20, that is, the information indicating the predetermined characteristics described above, is called radio wave characteristic information. The radio wave characteristic information is stored in a memory device accessible from the first detection unit 2020. The first detection unit 2020 acquires the radio wave characteristic information and determines whether or not a radio wave having the predetermined characteristics shown in the radio wave characteristic information has been detected by the radio wave sensor. If a radio wave having the predetermined characteristics has been detected by the radio wave sensor, the first detection unit 2020 identifies the direction from the sensor 10 toward the source of that radio wave as the first direction 12.

[0055] <Identification of the first direction 12> The first detection unit 2020 uses the sensor 10 to determine the first direction 12. The first direction 12 is the direction from the sensor 10 towards the moving object 20. For example, when the second camera 40 is used as the sensor 10, the first detection unit 2020 determines the first direction 12 based on various setting parameters of the camera 30 (such as imaging direction and focal length) and the position of the moving object 20 in the second captured image 42. Here, existing technologies can be used for the technique of determining the direction from the camera towards the object based on the camera setting parameters and the position of the object in the captured image.

[0056] When the aforementioned acoustic sensor is used as sensor 10, for example, the control unit 2040 determines the direction from the acoustic sensor toward the mobile body 20 based on the difference in the timing at which each microphone detects the sound emitted from the mobile body 20. Here, existing technologies can be used for the technique of determining the direction toward the sound source of a sound based on the difference in the timing at which each microphone, which is arranged in three dimensions, detects a certain sound. In addition, when the aforementioned radio wave sensor is used as sensor 10, the first direction 12 can be determined in the same way as when an acoustic sensor is used.

[0057] <Control of camera 30: S106, S108> The control unit 2040 causes the camera 30 to take multiple images while moving the optical axis direction 34 of the camera 30 along the first direction 12 (S106, S108). Here, the control unit 2040 may change the optical axis direction 34 of the camera 30 by changing the orientation of the camera 30, or it may change the optical axis direction 34 of the camera 30 without changing the orientation of the camera 30. In the former case, for example, the control unit 2040 moves the optical axis direction 34 of the camera 30 by changing the orientation of the stand, such as a dimmer stand, on which the camera 30 is mounted. In the latter case, for example, an optical element (such as a mirror) is placed on the optical axis of the camera 30, and the control unit 2040 moves the optical axis direction 34 of the camera 30 by controlling the orientation of this optical element.

[0058] The initial position of the camera 30 in the first direction 12 through which the optical axis direction 34 passes may be determined statically or dynamically. In the former case, for example, the starting point of the first direction 12 is set as the initial position.

[0059] On the other hand, in the latter case, for example, the spatial range for detecting the moving object 20 using the information processing device 2000 (hereinafter referred to as the detection space) is predetermined, and the position where the first direction 12 enters the detection space is set as the initial position. Figure 7 is a diagram illustrating the detection space. In Figure 7, the detection space is represented by reference numeral 60. The position where the first direction 12 enters the detection space 60 is position 62. Therefore, the control unit 2040 sets the initial position on the first direction 12 through which the optical axis direction 34 of the camera 30 passes as position 62. Note that the position on the first direction 12 where the first direction 12 exits the detection space 60 is position 64. Therefore, the control unit 2040 may move the optical axis direction 34 of the camera 30 from position 62 to position 64.

[0060] In addition, the control unit 2040 may determine the initial position based on the detection result of the sensor 10. For example, suppose the sensor 10 is a camera. In this case, the control unit 2040 obtains information representing the size of the moving object 20 in real space by identifying the type of moving object 20 included in the captured image 32. The control unit 2040 estimates the distance from the sensor 10 to the moving object 20 based on the size of the moving object 20 in real space and the size of the moving object 20 in the captured image 32. The control unit 2040 then sets the position at the estimated distance from the sensor 10 along the first direction 12 as the initial position on the first direction 12 through which the optical axis direction 34 of the camera 30 passes. Existing techniques can be used to estimate the distance from the camera to the object based on the object's size in real space and its size in the captured image.

[0061] There are various ways to have the camera 30 take multiple images. For example, a video camera that takes images periodically at a predetermined interval (e.g., 30 fps (frames per second)) can be used as the camera 30. The camera 30 may start taking images in response to control by the control unit 2040, or it may take images continuously. In the latter case, the second detection unit 2060 uses the images 32 generated by the camera 30 that were generated after the control unit 2040 started controlling the camera 30's imaging direction.

[0062] Alternatively, for example, the control unit 2040 may transmit a control signal instructing the camera 30 to take an image at each timing when it is instructed to take an image. In this case, the camera 30 takes an image when it receives the control signal.

[0063] <<Regarding the focus of Camera 30>> The control unit 2040 may set the focal length of the camera 30 so that the focal point of the camera 30 is located on the first direction 12. Specifically, the control unit 2040 calculates the distance from the camera 30 to the intersection of the imaging direction of the camera 30 and the first direction 12, and sets that distance as the focal length of the camera 30. The control unit 2040 also moves the optical axis direction of the camera 30 and changes the focal length of the camera 30.

[0064] By setting the focus of the camera 30 to the first direction 12 in this way, the camera 30 can capture images of the moving object 20 in focus. Therefore, the second detection unit 2060 can more easily detect the moving object 20 from the captured image 32.

[0065] Furthermore, even if an object other than the moving object 20 exists in the optical axis direction of the camera 30, that object will be captured in an out-of-focus state. Therefore, by detecting only the objects in focus among the objects included in the captured image 32 as the moving object 20, false detection of the moving object 20 can be prevented. In other words, it is possible to prevent objects other than those detected as the moving object 20 by the sensor 10 from being mistakenly detected as the moving object 20 by the second detection unit 2060. Note that existing technologies can be used to determine whether an object in the image is in focus or not.

[0066] <Detection of moving object 20 from captured image 32: S112, S114, S116> The second detection unit 2060 detects objects from the captured image 32, thereby detecting the moving object 20 from the captured image 32 (S112, S114, S116). For example, the second detection unit 2060 analyzes multiple captured images 32 in order from the earliest generation time, and treats the first object detected as the captured image 32. In this way, if multiple objects exist in the first direction 12, the object closest to the sensor 10 is detected as the moving object 20.

[0067] However, the second detection unit 2060 may use the detection results from the first detection unit 2020 to determine whether the object detected from the captured image 32 is the same as the moving object 20 detected by the first detection unit 2020. For example, if the sensor 10 is the second camera 40, the first detection unit 2020 calculates the feature quantities of the moving object 20 detected from the second captured image 42. These feature quantities allow for a general understanding of the characteristics of the moving object 20, such as whether it is a bird, an airplane, or a drone.

[0068] The second detection unit 2060 then determines whether the object detected from the captured image 32 has the feature quantities of the moving object 20 calculated by the first detection unit 2020. If the object detected from the captured image 32 has these feature quantities, the second detection unit 2060 detects that object as the moving object 20. On the other hand, if the object detected from the captured image 32 does not have these feature quantities, the second detection unit 2060 determines that the object is not the moving object 20.

[0069] As mentioned above, there are cases where the type of moving object 20 to be detected by the sensor 10 is predetermined. For example, if the sensor 10 is an acoustic sensor and detects the sound of the rotor of a specific model of drone, the moving object 20 detected by the sensor 10 will be that specific model of drone. In this case, the second detection unit 2060 may limit the objects it detects from the captured image 32 to that specific model of drone. In this case, for example, a feature quantity that is common to objects classified as a type to be detected using the sensor 10 is predetermined. The second detection unit 2060 detects an object as a moving object 20 if the object detected from the captured image 32 has this feature quantity. On the other hand, if the object detected from the captured image 32 does not have this feature quantity, the second detection unit 2060 determines that the object is not a moving object 20.

[0070] In addition, for example, as mentioned above, if the focus of the camera 30 is controlled so that the focus of the camera 30 is located on the first direction 12, the second detection unit 2060 may detect only those objects that are in focus among the objects detected from the captured image 32 as moving objects 20. Here, existing technologies can be used for determining whether or not an object included in the image is in focus.

[0071] <Update of the first direction 12> Here, even while the optical axis direction 34 of the camera 30 is moving along the first direction 12, the moving object 20 can be detected using the sensor 10. When the position of the moving object 20 detected by the sensor 10 changes, the first direction 12 changes accordingly. In this case, the control unit 2040 may update the first direction 12 based on the detection result from the sensor 10.

[0072] Figure 8 illustrates how the first direction 12 is updated. The first direction 12-1 is the first direction identified by the first detection unit 2020 based on the detection result of the sensor 10 at time t1. From time t1 onward, the control unit 2040 moves the optical axis direction 34 of the camera 30 along the first direction 12-1, causing the camera 30 to take images.

[0073] Subsequently, the first detection unit 2020 identifies the first direction 12-2 based on the detection result of the sensor 10 at time t2. Therefore, the control unit 2040 moves the optical axis direction 34 of the camera 30 along the first direction 12-2, rather than the first direction 12-1, from time t2 onward. Similarly, the first direction 12-3 is identified based on the detection result of the sensor 10 at time t3, and the optical axis direction 34 of the camera 30 moves along the first direction 12-3 from time t3 onward.

[0074] By updating the first direction 12 while the camera 30 is moving along the first direction 12, the camera 30 can detect the moving object 20 while the optical axis direction 34 of the camera 30 is moving along the first direction 12, taking into account the change in the position of the moving object 20. Therefore, the moving object 20 can be detected more reliably.

[0075] <Regarding the use of multiple sensors 10 and cameras 30> The above explanation used the case where there is one sensor 10 and one camera 30 as examples. However, there may be multiple sensors 10 and cameras 30. Below, we will provide specific examples of cases where multiple sensors 10 and cameras 30 are provided.

[0076] Figures 9 and 10 illustrate use cases in which multiple sensors 10 and cameras 30 are installed. Figure 9 is a vertical plan view of the area monitored by the mobile body 20. Figure 10 is a side view of the area monitored by the mobile body 20. In this use case, the mobile body 20 is a drone 120. The drone 120 flies between building 100 and building 110. Its flight path is set within the monitoring area 130.

[0077] The monitoring area 130 is equipped with eight sensors 10 and two cameras 30. Camera 30-1 is positioned at the center of sensors 10-1 to 10-4. Meanwhile, camera 30-2 is positioned at the center of sensors 10-5 to 10-7. The dotted rectangles within the monitoring area 130 represent the range in which each sensor 10 detects the drone 120.

[0078] The first detection unit 2020 detects the drone 120 using sensors 10-1 to 10-8. If the drone 120 is detected by any of sensors 10-1 to 10-4, the control unit 2040 controls camera 30-1 to generate an image 32 and detects the drone 120 from that image 32. On the other hand, if the drone 120 is detected by any of sensors 10-5 to 10-7, the control unit 2040 controls camera 30-2 to generate an image 32 and detects the drone 120 from that image 32.

[0079] [Embodiment 2] Figure 11 is a diagram illustrating the functional configuration of the information processing device 2000 of Embodiment 2. Except for the matters described below, the information processing device 2000 of Embodiment 2 has the same functions as the information processing device 2000 of Embodiment 1.

[0080] The information processing device 2000 of Embodiment 2 has an output unit 2080. The output unit 2080 outputs an image 32 in which the moving object 20 has been detected by the second detection unit 2060. There are various ways to output the image 32. For example, the output unit 2080 can store the image 32 in which the moving object 20 has been detected by outputting it to a storage device. Alternatively, for example, the output unit 2080 can display the image 32 in which the moving object 20 has been detected by outputting it to a display device. Alternatively, for example, the output unit 2080 may transmit the image 32 in which the moving object 20 has been detected to a terminal other than the information processing device 2000. For example, the image 32 in which the moving object 20 has been detected may be transmitted to a terminal used by a monitor who is monitoring the moving object (hereinafter referred to as the monitoring terminal). The image 32 received by the monitoring terminal is displayed on a display device provided in the monitoring terminal, for example.

[0081] Here, the information processing device 2000 may have a function to make the camera 30 track the moving object 20 so that the camera 30 continues to capture images of the moving object 20 after the second detection unit 2060 has detected the moving object 20. In this case, each captured image 32 generated after the second detection unit 2060 has detected the moving object 20 will include the moving object 20. That is, the camera 30 will generate video data that includes the moving object 20. In this case, the output unit 2080 may output video data that includes not only the captured image 32 in which the moving object 20 was detected by the second detection unit 2060, but also the captured images 32 generated thereafter. Existing technologies can be used for the technique of making the camera track a specific object.

[0082] <Example of hardware configuration> The hardware configuration of the computer implementing the information processing device 2000 of Embodiment 2 is shown, for example, in Figure 3, similar to Embodiment 1. However, the storage device 1080 of the computer 1000 implementing the information processing device 2000 of this embodiment further stores program modules that realize the functions of the information processing device 2000 of this embodiment.

[0083] <Effects and Effects> According to the information processing device 2000 of this embodiment, an image 32 in which the moving object 20 is detected is output. Here, since imaging by the camera 30 can be performed by moving it along the optical axis direction 34 along the first direction 12 identified using the sensor 10, the imaging range of the camera 30 can be narrowed to some extent. For example, as mentioned above, a telephoto camera can be used as the camera 30. Therefore, according to the information processing device 2000, an image 32 in which the moving object 20 is captured at a large size can be obtained, and the appearance of the moving object 20 can be grasped in detail using the image 32.

[0084] [Embodiment 3] Figure 12 is a diagram illustrating the functional configuration of the information processing device 2000 of Embodiment 3. Except for the matters described below, the information processing device 2000 of Embodiment 3 has the same functions as the information processing device 2000 of Embodiment 1 or 2.

[0085] The information processing device 2000 of Embodiment 2 has a calculation unit 2100. The calculation unit 2100 has the function of calculating the three-dimensional position of the moving object 20 detected by the second detection unit 2060. According to the information processing device 2000, the moving object 20 is detected by the sensor 10 and the camera 30, which are installed at different positions. Therefore, the three-dimensional position of the moving object 20 is determined as the intersection of the direction from the sensor 10 toward the moving object 20 (first direction 12) and the direction from the camera 30 toward the moving object 20 (hereinafter referred to as the second direction). The calculation unit 2100 then calculates the intersection of the first direction 12 and the second direction and sets that intersection as the three-dimensional position of the moving object 20.

[0086] A three-dimensional position can be represented, for example, as a combination of GPS coordinates and altitude. Specifically, the x and y coordinates are the x and y coordinates of the GPS coordinate system, respectively, and the z coordinate is the altitude. Alternatively, for example, the three-dimensional position of the moving object 20 may be represented as a relative position based on the positions of the sensor 10 and the camera 30.

[0087] Furthermore, it is preferable that the three-dimensional coordinates of the moving object 20 calculated by the calculation unit 2100 are output in the same manner as the method used by the output unit 2080 to detect the captured image 32 in which the moving object 20 is detected. For example, when displaying the captured image 32 on a display device, it is preferable to superimpose the information of the three-dimensional position of the moving object 20 onto the vicinity of the moving object 20 on the captured image 32.

[0088] <Example of hardware configuration> The hardware configuration of the computer implementing the information processing device 2000 of Embodiment 3 is shown, for example, in Figure 3, similar to Embodiment 1. However, the storage device 1080 of the computer 1000 implementing the information processing device 2000 of this embodiment further stores program modules that realize the functions of the information processing device 2000 of this embodiment.

[0089] <Effects and Effects> According to the information processing device 2000 of this embodiment, the three-dimensional position of the moving object 20 can be determined based on the direction toward the moving object 20 from the sensors 10 and camera 30, which are positioned at different locations. The three-dimensional position calculated in this way can then be used for various purposes such as monitoring and tracking the moving object 20.

[0090] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and configurations combining the above embodiments or various other configurations can also be adopted.

[0091] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0092] Examples of reference formats are provided below. 1. A first detection means that detects a moving object using a sensor and identifies a first direction from the sensor toward the moving object, A control means for causing the first camera to perform imaging while moving the optical axis direction of the first camera along the first direction, An information processing apparatus having a second detection means for detecting the moving object from a first captured image generated by the first camera. 2. The information processing apparatus according to 1, wherein the second detection means treats the moving object detected from the first image that is generated earliest among the first image images in which the moving object is detected as the moving object detected by the sensor. 3. The sensor is a second camera with a shorter focal length than the first camera. The information processing apparatus according to 1. or 2., wherein the first detection means detects the moving object by performing image analysis on the captured image generated by the second camera. 4. The sensor is an acoustic sensor, The information processing apparatus according to 1. or 2., wherein the first detection means acquires information representing the characteristics of a sound emitted by a moving object to be detected, and when a sound having the characteristics is detected by the acoustic sensor, it identifies the direction from the acoustic sensor toward the source of the sound as the first direction. 5. The aforementioned sensor is a radio wave sensor, The information processing device according to 1. or 2., wherein the first detection means acquires information representing the characteristics of radio waves emitted by a moving object to be detected, and when radio waves having the characteristics are detected by the radio wave sensor, it identifies the direction from the radio wave sensor toward the source of the radio waves as the first direction. 6. The information processing apparatus according to any one of 1 to 5, wherein the control means determines the initial position of the imaging direction of the first camera based on the size of the moving object detected by the sensor, and moves the imaging direction of the first camera from the initial position along the first direction. 7. The information processing apparatus according to any one of 1 to 6, wherein the control means sets the focal length of the first camera such that the focal point of the first camera is located in the first direction. 8. The first detection means repeatedly identifies the first direction, The information processing apparatus according to any one of 1 to 7, wherein the control means moves the optical axis direction of the first camera along the newly identified first direction when the optical axis direction of the first camera is being moved along the first direction. 9. An information processing apparatus according to any one of 1 to 8, comprising output means for outputting the first captured image in which the moving object has been detected. 10. An information processing apparatus according to any one of 1 to 9, comprising a calculation means for calculating the three-dimensional position of a moving object based on the first direction and the optical axis direction of the first camera when the moving object is detected.

[0093] 11. A control method performed by a computer, A first detection step involves using a sensor to detect a moving object and determining a first direction from the sensor toward the moving object, A control step of causing the first camera to perform imaging while moving the optical axis direction of the first camera along the first direction, A control method comprising: a second detection step of detecting the moving object from a first image captured by the first camera. 12. The control method according to 11, wherein, in the second detection step, the moving object detected from the first image image generated earliest among the first image images in which the moving object is detected is treated as the moving object detected by the sensor. 13. The sensor is a second camera with a shorter focal length than the first camera. The control method according to 11. or 12., wherein in the first detection step, the moving object is detected by performing image analysis on the captured image generated by the second camera. 14. The sensor is an acoustic sensor, The control method according to 11. or 12., wherein in the first detection step, information representing the characteristics of the sound emitted by the moving object to be detected is acquired, and when a sound having the characteristics is detected by the acoustic sensor, the direction from the acoustic sensor toward the source of the sound is identified as the first direction. 15. The aforementioned sensor is a radio wave sensor, The control method according to 11. or 12., wherein in the first detection step, information representing the characteristics of radio waves emitted by the moving object to be detected is acquired, and when radio waves having the said characteristics are detected by the radio wave sensor, the direction from the radio wave sensor toward the source of those radio waves is identified as the first direction. 16. The control method according to any one of 11 to 15, wherein in the control step, the initial position of the imaging direction of the first camera is determined based on the size of the moving object detected by the sensor, and the imaging direction of the first camera is moved from the initial position along the first direction. 17. The control method according to any one of 11 to 16, wherein in the control step, the focal length of the first camera is set such that the focal point of the first camera is located in the first direction. 18. In the first detection step, the first direction is repeatedly identified, The control method according to any one of 11 to 17, wherein, in the control step, if the optical axis direction of the first camera is moved along the first direction, the optical axis direction of the first camera is moved along the newly identified first direction. 19. A control method according to any one of 11 to 18, comprising an output step of outputting the first image captured in which the moving object was detected. 20. A control method according to any one of 11 to 19, comprising a calculation step of calculating the three-dimensional position of the moving object based on the first direction and the optical axis direction of the first camera when the moving object is detected.

[0094] 21. A program that causes a computer to perform each step of the control method described in any one of items 11 through 20. [Explanation of Symbols]

[0095] 10 sensors 12 1st direction 20 Mobile Units 30 Cameras 32 Acquired Images 34 Optical axis direction 40. Second camera 42 Second image 50 Touch Panel 52 Videos 56 areas 58 Drones 60 detection space 62 positions 64 positions 100 Building 110 Building 120 Drones 130 surveillance areas 1000 calculator 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface 2000 Information Processing Equipment 2020 First Detection Unit 2040 Control Unit 2060 Second detection unit 2080 Output Section 2100 Calculation Unit

Claims

1. A first detection means that detects a moving object using a sensor and identifies a first direction from the sensor toward the moving object, A control means for causing the first camera to perform imaging while moving the optical axis direction of the first camera along the first direction, An information processing apparatus having a second detection means for detecting the moving object from a first captured image generated by the first camera.

2. The information processing apparatus according to claim 1, wherein the second detection means treats the moving object detected from the first image that is generated earliest among the first image images in which the moving object is detected as the moving object detected by the sensor.

3. The sensor is a second camera with a shorter focal length than the first camera. The information processing apparatus according to claim 1 or 2, wherein the first detection means detects the moving object by performing image analysis on the captured image generated by the second camera.

4. The aforementioned sensor is an acoustic sensor, The information processing apparatus according to claim 1 or 2, wherein the first detection means acquires information representing the characteristics of a sound emitted by a moving object to be detected, and when a sound having the characteristics is detected by the acoustic sensor, it identifies the direction from the acoustic sensor toward the source of the sound as the first direction.

5. The aforementioned sensor is a radio wave sensor. The information processing apparatus according to claim 1 or 2, wherein the first detection means acquires information representing the characteristics of radio waves emitted by a moving object to be detected, and when radio waves having the characteristics are detected by the radio wave sensor, the direction from the radio wave sensor toward the source of the radio waves is identified as the first direction.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the control means determines an initial position of the imaging direction of the first camera based on the size of the moving object detected by the sensor, and moves the imaging direction of the first camera from the initial position along the first direction.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the control means sets the focal length of the first camera such that the focal point of the first camera is located in the first direction.

8. The first detection means repeatedly identifies the first direction, The information processing apparatus according to any one of claims 1 to 7, wherein the control means moves the optical axis direction of the first camera along the newly identified first direction when the optical axis direction of the first camera is being moved along the first direction.

9. The information processing apparatus according to any one of claims 1 to 8, further comprising output means for outputting the first image captured in which the moving object was detected.

10. The information processing apparatus according to any one of claims 1 to 9, comprising calculation means for calculating the three-dimensional position of the moving object based on the first direction and the optical axis direction of the first camera when the moving object is detected.

11. A control method performed by a computer, A first detection step involves using a sensor to detect a moving object and determining a first direction from the sensor toward the moving object, A control step of causing the first camera to perform imaging while moving the optical axis direction of the first camera along the first direction, A control method comprising: a second detection step of detecting the moving object from a first image captured by the first camera.

12. A program that causes a computer to execute each step of the control method described in claim 11.

Images