Information processing device, information processing method, and information processing program

Abstract

Provided are an information processing device, an information processing method, and an information processing program capable of utilizing position information on a capturing target associated with a captured image and position information acquired by a terminal device.　A management device (11) acquires: first position information on a capturing target position associated with an image captured by a monitoring camera (10) and a turn mechanism (16); and second position information on a terminal device (100) acquired by the terminal device (100) within a capturing region of the monitoring camera (10) and the turn mechanism (16). The management device (11) then generates first data on the basis of the acquired first position information and second position information.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] Patent Literature 1 describes a wide-area surveillance system that generates suspicious person information, including notification destinations, based on location information of a surveillance camera linked to a captured image and the latitude and longitude of a user's mobile device. Patent Literature 2 describes an information processing system in which historical data of location information of a mobile device is stored in a database server, and in a process of obtaining an activity history of the mobile device's location information from a communication terminal, map information corresponding to the historical data of location information is obtained from a map search service, the historical data is superimposed on the map information, and map information is generated and output to the communication terminal. Patent Literature 3 describes a control device for an imaging device that takes into account errors in latitude and longitude values ​​and identifies a range containing true latitude and longitude values ​​based on actual latitude and longitude values.

[0003] Japanese Patent Publication No. 2014-078150 Japanese Patent Publication No. 2013-246570 Japanese Patent Publication No. 2019-124858

[0004] One embodiment of the technology disclosed herein provides an information processing device, an information processing method, and an information processing program that can utilize location information of an image capture target associated with a captured image and location information obtained by a terminal device.

[0005] (1) An information processing device having a processor, wherein the processor acquires first position information of an imaging target position associated with an image captured by an imaging system and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system, and generates first data based on the first position information and the second position information.

[0006] (2) The information processing device according to (1), wherein the terminal device is a terminal device possessed by a moving object included in the captured image.

[0007] (3) The information processing device according to (1) or (2), wherein the first data includes instruction information for specifying the terminal device as a transmission destination.

[0008] (4) An information processing device according to (3), wherein the processor acquires the second location information for a plurality of terminal devices, and generates the instruction information whose destination is a terminal device among the plurality of terminal devices whose second location information is included in a range based on the first location information.

[0009] (5) The information processing device according to (4), in which the processor sets the range based on the first position information and zoom information of the imaging system.

[0010] (6) The information processing device according to (4) or (5), wherein the processor sets the range based on the first position information and the number of moving objects having the terminal device detected from the captured image.

[0011] (7) An information processing device according to any one of (3) to (6), wherein the processor, when receiving information from the terminal device indicating an abnormality in the owner of the terminal device, controls the display device to display the captured image showing the area of ​​the captured area indicated by the second location information.

[0012] (8) The information processing device according to (1) or (2), wherein the first data is information relating to a movement history of a mobile object having the terminal device.

[0013] (9) The information processing device according to (8), wherein the processor generates an image as information relating to the movement history by superimposing an image showing the movement history of the moving object on the captured image based on the first position information and the second position information.

[0014] (10) An information processing device according to any one of (1) to (9), wherein the processor controls a display device to display a wide-area image showing the imaging area and a detailed image showing a range of the imaging area narrower than the wide-area image at a higher magnification than the wide-area image.

[0015] (11) An information processing device according to any one of (1) to (10), wherein the imaging system includes an imaging device that obtains the captured image by imaging and a rotation mechanism that rotates the imaging device, and the first position information is associated with information indicating the rotation state of the imaging device by the rotation mechanism when the captured image is obtained by the imaging system.

[0016] (12) An information processing method, in which a processor of an information processing device acquires first position information of an imaging target position associated with an image captured by an imaging system and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system, and generates first data based on the first position information and the second position information.

[0017] (13) An information processing program for causing a processor of an information processing device to execute a process of acquiring first position information of an imaging target position associated with an image captured by an imaging system and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system, and generating first data based on the first position information and the second position information.

[0018] An optical observation device according to one aspect of the present invention includes the above-described projection device.

[0019] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program that can utilize location information of an image capture target associated with a captured image and location information obtained by a terminal device.

[0020] 10 and 11. FIG. 11 is a diagram showing an example of an imaging system 1 equipped with a control device of this embodiment. FIG. 12 is a diagram showing an example of rotation of the monitoring camera 10 in the pitch direction by the rotation mechanism 16. FIG. 13 is a diagram showing an example of rotation of the monitoring camera 10 in the yaw direction by the rotation mechanism 16. FIG. 14 is a block diagram showing an example of the configuration of the optical system and electrical system of the monitoring camera 10. FIG. 15 is a diagram showing an example of the configuration of the rotation mechanism 16 and the electrical system of the management device 11. FIG. 16 is a diagram showing an example of an image displayed by the management device 11. FIG. 17 is a diagram showing an example of the hardware configuration of a terminal device carried by a worker in the monitored area E1. FIG. 18 is a flowchart showing an example of processing by the management device 11. FIG. 19 is a diagram showing a first modified example of the imaging system 1. FIG. 19 is a diagram showing an example of the configuration of the electrical system of the management device 11 shown in FIG. 12. FIG. 19 is a diagram showing an example of an image displayed by the management device 11 in the configuration shown in FIGS. 9 and 10. FIG. 20 is a diagram showing a second modified example of the imaging system 1. FIG. 21 is a diagram showing an example of the configuration of the electrical system of the management device 11 shown in FIG. 12. FIG. 22 is a diagram showing an example of processing to check for unwell persons, injured persons, etc. 11 is a diagram showing an example of an image displayed by the management device 11 when the imaging system 1 of the second embodiment has the configuration shown in Fig. 9 and Fig. 10. Fig. 12 is a diagram showing an example of a manner in which an information processing program of an operation control example is installed into the control device 60 of the management device 11 from a storage medium storing the information processing program.

[0021] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0022] (Embodiment 1) <Imaging system of embodiment> Fig. 1 is a diagram showing an example of an imaging system 1 equipped with a control device of this embodiment. As shown in Fig. 1 as an example, the imaging system 1 includes a surveillance camera 10, a management device 11, and a rotation mechanism 16. The surveillance camera 10 and the rotation mechanism 16 are an example of an imaging system in the present invention. The management device 11 is an example of an information processing device in the present invention.

[0023] The surveillance camera 10 is installed on a pillar, a wall, or a part of a building (e.g., a rooftop) indoors or outdoors via a swivel mechanism 16, and captures an image of a subject. The surveillance camera 10 transmits the captured image and imaging information related to the image to the management device 11 via a communication line 12.

[0024] The management device 11 includes a display 13a, a keyboard 13b, a mouse 13c, and a secondary storage device 14. Examples of the display 13a include a liquid crystal display, a plasma display, an organic EL (Electro-Luminescence) display, and a CRT (Cathode Ray Tube) display. The display 13a is an example of a display device of the present invention.

[0025] An example of the secondary storage device 14 is a hard disk drive (HDD). The secondary storage device 14 is not limited to an HDD, and may be any non-volatile memory such as a flash memory, a solid state drive (SSD), or an electrically erasable and programmable read-only memory (EEPROM).

[0026] The management device 11 receives the captured images and imaging information transmitted from the monitoring camera 10 , and displays the received captured images and imaging information on the display 13 a and stores them in the secondary storage device 14 .

[0027] The management device 11 performs imaging control to control imaging by the surveillance camera 10. For example, the management device 11 performs imaging control by communicating with the surveillance camera 10 via the communication line 12. The imaging control is a control to set imaging parameters for the surveillance camera 10 to capture images and to cause the surveillance camera 10 to capture images. The imaging parameters include parameters related to exposure and parameters for the zoom position.

[0028] The management device 11 also controls the rotation mechanism 16 to control (pan and tilt) the imaging direction of the surveillance camera 10. For example, the management device 11 sets the rotation direction, rotation amount, rotation speed, etc. of the surveillance camera 10 in response to operations on the keyboard 13b or mouse 13c, or touch operations on the screen of the display 13a.

[0029] <Rotation of surveillance camera 10 by rotation mechanism 16> Fig. 2 is a diagram showing an example of rotation of surveillance camera 10 in the pitch direction by rotation mechanism 16. Fig. 3 is a diagram showing an example of rotation of surveillance camera 10 in the yaw direction by rotation mechanism 16. The surveillance camera 10 is attached to the rotation mechanism 16. The rotation mechanism 16 enables the surveillance camera 10 to rotate.

[0030] Specifically, as shown in Fig. 2 as an example, the rotation mechanism 16 is a two-axis rotation mechanism that can rotate the surveillance camera 10 in a rotation direction (pitch direction) that intersects with the yaw direction and has the pitch axis PA as the central axis, and in a rotation direction (yaw direction) that has the yaw axis YA as the central axis, as shown in Fig. 3 as an example. Note that, although the rotation mechanism 16 according to the present embodiment is an example of a two-axis rotation mechanism, the technology of the present disclosure is not limited to this, and may be a three-axis rotation mechanism or a single-axis rotation mechanism.

[0031] <Configuration of Optical System and Electrical System of Surveillance Camera 10> FIG. 4 is a block diagram showing an example of the configuration of the optical system and electrical system of the surveillance camera 10. As shown in FIG. 4, the surveillance camera 10 includes an optical system 15 and an image sensor 25. The image sensor 25 is located downstream of the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis OA of the optical system 15 from the target subject side (object side) to the light receiving surface 25A side (image side) of the image sensor 25. The lens group 15B includes an anti-vibration lens 15B1, a focus lens (not shown), a zoom lens 15B2, and the like. The zoom lens 15B2 is supported by a lens actuator 21 (described later) so as to be movable along the optical axis OA. The anti-vibration lens 15B1 is supported by a lens actuator 17 (described later) so as to be movable in a direction perpendicular to the optical axis OA.

[0032] By increasing the focal length with the zoom lens 15B2, the surveillance camera 10 becomes telephoto, thereby narrowing the angle of view (narrowing the imaging range). By decreasing the focal length with the zoom lens 15B2, the surveillance camera 10 becomes wide-angle, thereby widening the angle of view (widening the imaging range).

[0033] The optical system 15 may include various lenses (not shown) in addition to the objective lens 15A and the lens group 15B. The optical system 15 may also include an aperture. The positions of the lenses, lens group, and aperture included in the optical system 15 are not limited, and the technology of the present disclosure is valid even if they are located at positions different from those shown in FIG. 4 .

[0034] The vibration-proof lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0035] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force to the vibration-proof lens 15B1 that fluctuates in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 fluctuates in a direction perpendicular to the optical axis OA.

[0036] The lens actuator 21 applies a force to the zoom lens 15B2 to move it along the optical axis OA of the optical system 15. The lens actuator 21 is controlled by the lens driver 28. When the lens actuator 21 is driven under the control of the lens driver 28, the position of the zoom lens 15B2 moves along the optical axis OA. When the position of the zoom lens 15B2 moves along the optical axis OA, the focal length of the surveillance camera 10 changes.

[0037] In addition, if the outline of the captured image is, for example, a rectangle with a short side in the pitch axis PA direction and a long side in the yaw axis YA direction, the angle of view in the pitch axis PA direction is narrower than the angle of view in the yaw axis YA direction and is narrower than the angle of view of the diagonal.

[0038] By the optical system 15 configured in this manner, light representing the imaging area is focused on the light receiving surface 25A of the imaging element 25, and the imaging area is imaged by the imaging element 25.

[0039] Incidentally, the vibrations that are applied to the surveillance camera 10 include, if outdoors, vibrations caused by passing cars, vibrations caused by wind, vibrations caused by road construction, etc., and if indoors, vibrations caused by the operation of an air conditioner, vibrations caused by people entering and exiting, etc. Therefore, the surveillance camera 10 shakes due to the vibrations that are applied to the surveillance camera 10 (hereinafter also simply referred to as "vibrations").

[0040] In this embodiment, "shake" refers to a phenomenon in which the image of a target subject on the light-receiving surface 25A of the image sensor 25 in the surveillance camera 10 fluctuates due to a change in the positional relationship between the optical axis OA and the light-receiving surface 25A. In other words, "shake" can also be said to be a phenomenon in which the optical axis OA tilts due to vibrations applied to the surveillance camera 10, causing the optical image formed on the light-receiving surface 25A to fluctuate. The fluctuation of the optical axis OA means, for example, that the optical axis OA tilts with respect to a reference axis (e.g., the optical axis OA before the shake occurs). Hereinafter, shake caused by vibrations will also be simply referred to as "shake."

[0041] Shake is included in the captured image as a noise component and affects the image quality of the captured image. Therefore, in order to remove the noise component included in the captured image due to shake, the surveillance camera 10 is equipped with a lens-side shake correction mechanism 29, an image sensor-side shake correction mechanism 45, and an electronic shake correction unit 33, which are used to correct shake.

[0042] The lens-side shake correction mechanism 29 and the image sensor-side shake correction mechanism 45 are mechanical shake correction mechanisms that apply power generated by a drive source such as a motor (for example, a voice coil motor) to a shake correction element (for example, the vibration-proof lens 15B1 and / or the image sensor 25) to move the shake correction element in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake.

[0043] Specifically, the lens-side image stabilization mechanism 29 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image stabilization lens 15B1, thereby moving the image stabilization lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The image sensor-side image stabilization mechanism 45 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 25, thereby moving the image sensor 25 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The electronic image stabilization unit 33 corrects shake by performing image processing on the captured image based on the amount of shake. In other words, the image stabilization unit (image stabilization component) performs shake correction mechanically or electronically using a hardware and / or software configuration. Here, mechanical shake correction refers to shake correction achieved by mechanically moving shake correction elements such as the anti-vibration lens 15B1 and / or the image sensor 25 using power generated by a driving source such as a motor (e.g., a voice coil motor), and electronic shake correction refers to shake correction achieved, for example, by image processing performed by a processor.

[0044] As an example, as shown in FIG. 4, the lens-side shake correction mechanism 29 includes an anti-vibration lens 15B1, a lens actuator 17, an OIS driver 23, and a position sensor 39.

[0045] Various well-known methods can be used to correct shake using the lens-side shake correction mechanism 29. In this embodiment, the method used is to correct shake by moving the vibration-proof lens 15B1 based on the amount of shake detected by a shake amount detection sensor 40 (described later). Specifically, shake correction is performed by moving the vibration-proof lens 15B1 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0046] A lens actuator 17 is attached to the vibration-proof lens 15B1. The lens actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the vibration-proof lens 15B1 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the lens actuator 17 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0047] The lens actuator 17 is controlled by an OIS driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 is mechanically changed within a two-dimensional plane perpendicular to the optical axis OA.

[0048] The position sensor 39 detects the current position of the vibration-proof lens 15B1 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 39. Here, the current position of the vibration-proof lens 15B1 refers to the current position within a two-dimensional plane of the vibration-proof lens. The vibration-proof lens two-dimensional plane refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 39, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0049] The lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 along at least one of the pitch axis PA direction and the yaw axis YA direction within the range that is actually captured. In other words, the lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 within the two-dimensional vibration-proof lens plane by an amount corresponding to the amount of shake.

[0050] The image sensor side shake correction mechanism 45 includes the image sensor 25 , a BIS (Body Image Stabilizer) driver 22 , an image sensor actuator 27 , and a position sensor 47 .

[0051] As with the shake correction method used by the lens-side shake correction mechanism 29, various well-known methods can be used to correct shake using the image sensor-side shake correction mechanism 45. In this embodiment, the shake correction method used is to correct shake by moving the image sensor 25 based on the amount of shake detected by the shake amount detection sensor 40. Specifically, shake correction is performed by moving the image sensor 25 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0052] An imaging element actuator 27 is attached to the imaging element 25. The imaging element actuator 27 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the imaging element 25 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the imaging element actuator 27 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be applied instead of the voice coil motor.

[0053] The imaging element actuator 27 is controlled by the BIS driver 22. When the imaging element actuator 27 is driven under the control of the BIS driver 22, the position of the imaging element 25 is mechanically moved in a direction perpendicular to the optical axis OA.

[0054] The position sensor 47 detects the current position of the image sensor 25 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 47. Here, the current position of the image sensor 25 refers to the current position within a two-dimensional plane of the image sensor. The two-dimensional plane of the image sensor refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 47, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0055] The surveillance camera 10 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake amount detection sensor 40, and a UI (User Interface) device 43. The computer 19 includes a memory 35, a storage 36, and a CPU (Central Processing Unit) 37.

[0056] The image sensor 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage 36, CPU 37, shake amount detection sensor 40, and UI device 43 are connected to a bus 38. The OIS driver 23 is also connected to the bus 38. Note that, although one bus is shown as the bus 38 in the example shown in Fig. 4 for convenience of illustration, multiple buses may be used. The bus 38 may be a serial bus or a parallel bus such as a data bus, an address bus, and a control bus.

[0057] The memory 35 temporarily stores various information and is used as a work memory. An example of the memory 35 is a random access memory (RAM), but other types of storage devices may be used. The storage 36 stores various programs for the surveillance camera 10. The CPU 37 reads the various programs from the storage 36 and executes them on the memory 35 to control the entire surveillance camera 10. Examples of the storage 36 include a flash memory, an SSD, an EEPROM, and an HDD. Furthermore, various types of non-volatile memory, such as a magnetoresistive memory or a ferroelectric memory, may be used instead of or in combination with the flash memory.

[0058] The image sensor 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 25 captures an image of a target subject at a predetermined frame rate under the direction of the CPU 37. The "predetermined frame rate" here refers to, for example, several tens to several hundreds of frames per second. The image sensor 25 itself may incorporate a control device (image sensor control device), in which case the image sensor control device performs detailed control of the image sensor 25 in accordance with the image capture instructions output by the CPU 37. The image sensor 25 may also capture an image of a target subject at a predetermined frame rate under the direction of the DSP 31, in which case the image sensor control device performs detailed control of the image sensor 25 in accordance with the image capture instructions output by the DSP 31. The DSP 31 is sometimes referred to as an ISP (Image Signal Processor).

[0059] The light receiving surface 25A of the image sensor 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the image sensor 25, each photosensitive pixel is exposed to light, and photoelectric conversion is performed for each photosensitive pixel. The electric charge obtained by photoelectric conversion for each photosensitive pixel is an analog image signal representing the target subject. Here, a plurality of photoelectric conversion elements (e.g., photoelectric conversion elements with color filters) sensitive to visible light are used as the photosensitive pixels. In the image sensor 25, a plurality of photoelectric conversion elements are used, including a photoelectric conversion element sensitive to R (red) light (e.g., a photoelectric conversion element with an R filter corresponding to R), a photoelectric conversion element sensitive to G (green) light (e.g., a photoelectric conversion element with a G filter corresponding to G), and a photoelectric conversion element sensitive to B (blue) light (e.g., a photoelectric conversion element with a B filter corresponding to B). The surveillance camera 10 uses these photosensitive pixels to capture images based on visible light (e.g., light on the short wavelength side of approximately 700 nanometers or less). However, this embodiment is not limited to this, and images may be captured based on infrared light (e.g., light on the long wavelength side of approximately 700 nanometers). In this case, a plurality of photoelectric conversion elements sensitive to infrared light may be used as the photosensitive pixels. In particular, for capturing images of short-wavelength infrared (SWIR), an InGaAs sensor and / or a type-II quantum well (T2SL; Simulation of Type-II Quantum Well) sensor may be used, for example.

[0060] The image sensor 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog image signal to generate a digital image, which is a digital image signal. The image sensor 25 is connected to the DSP 31 via a bus 38, and outputs the generated digital image to the DSP 31 via the bus 38 in frame units.

[0061] Note that, although a CMOS image sensor is described here as an example of the image sensor 25, the technology of the present disclosure is not limited thereto, and a CCD (Charge Coupled Device) image sensor may also be used as the image sensor 25. In this case, the image sensor 25 is connected to the bus 38 via an AFE (Analog Front End) (not shown) that has a built-in CCD driver. The AFE generates a digital image by performing signal processing such as A / D conversion on the analog image signal obtained by the image sensor 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, the CCD driver may also be provided separately.

[0062] The DSP 31 performs various types of digital signal processing on the digital image. The various types of digital signal processing refer to, for example, demosaic processing, noise removal processing, gradation correction processing, color correction processing, etc. The DSP 31 outputs the digital image after digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0063] The shake amount detection sensor 40 is a device including, for example, a gyro sensor, and detects the amount of shake of the surveillance camera 10. In other words, the shake amount detection sensor 40 detects the amount of shake in each of a pair of axial directions. The gyro sensor detects the amount of rotational shake around each of the pitch axis PA, the yaw axis YA, and the roll axis RA (an axis parallel to the optical axis OA) (see FIG. 1 ). The shake amount detection sensor 40 detects the amount of shake of the surveillance camera 10 by converting the amount of rotational shake around the pitch axis PA and the amount of rotational shake around the yaw axis YA detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA.

[0064] Here, a gyro sensor is used as an example of the shake amount detection sensor 40, but this is merely an example, and the shake amount detection sensor 40 may also be an acceleration sensor. The acceleration sensor detects the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 outputs the detected amount of shake to the CPU 37.

[0065] Furthermore, although an example in which the amount of shake is detected by a physical sensor, namely, the shake amount detection sensor 40, is given here, the technology of the present disclosure is not limited to this. For example, a motion vector obtained by comparing successive captured images stored in the image memory 32 in chronological order may be used as the amount of shake. Furthermore, the amount of shake that is ultimately used may be derived based on the amount of shake detected by the physical sensor and the motion vector obtained by image processing.

[0066] The CPU 37 acquires the amount of shake detected by the shake amount detection sensor 40, and controls the lens-side shake compensation mechanism 29, the image sensor-side shake compensation mechanism 45, and the electronic shake compensation unit 33 based on the acquired amount of shake. The amount of shake detected by the shake amount detection sensor 40 is used for shake compensation by each of the lens-side shake compensation mechanism 29 and the electronic shake compensation unit 33.

[0067] The electronic shake correction unit 33 is a device including an ASIC (Application Specific Integrated Circuit). The electronic shake correction unit 33 corrects shake by performing image processing on the captured image in the image memory 32 based on the amount of shake detected by the shake amount detection sensor 40.

[0068] Note that, although a device including an ASIC is exemplified as the electronic shake correction unit 33 here, the technology of the present disclosure is not limited to this and may be, for example, a device including a field programmable gate array (FPGA) or a programmable logic device (PLD). Furthermore, for example, the electronic shake correction unit 33 may be a device including multiple ASICs, FPGAs, and PLDs. Furthermore, a computer including a CPU, storage, and memory may be employed as the electronic shake correction unit 33. There may be a single CPU or multiple CPUs. Furthermore, the electronic shake correction unit 33 may be realized by a combination of hardware and software configurations.

[0069] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information between the monitoring camera 10 and the management device 11 via a network. This network may be, for example, a wide area network (WAN) such as the Internet, or a local area network (LAN). The communication I / F 34 performs communication between the monitoring camera 10 and the management device 11.

[0070] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key or a touch panel, and receives various instructions from the user. The CPU 37 acquires the various instructions received by the reception device 43A and operates in accordance with the acquired instructions.

[0071] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions accepted by the accepting device 43A, captured images, and the like.

[0072] <Configuration of Electrical Systems of Swivel Mechanism 16 and Management Device 11> Fig. 5 is a diagram showing an example of the configuration of the electrical systems of the swing mechanism 16 and the management device 11. As shown in Fig. 5 as an example, the swing mechanism 16 includes a yaw axis swing mechanism 71, a pitch axis swing mechanism 72, a motor 73, a motor 74, a driver 75, a driver 76, and communication I / Fs 79 and 80.

[0073] The yaw axis rotation mechanism 71 rotates the surveillance camera 10 in the yaw direction. The motor 73 generates power by being driven under the control of a driver 75. The yaw axis rotation mechanism 71 rotates the surveillance camera 10 in the yaw direction by receiving the power generated by the motor 73. The pitch axis rotation mechanism 72 rotates the surveillance camera 10 in the pitch direction. The motor 74 generates power by being driven under the control of a driver 76. The pitch axis rotation mechanism 72 rotates the surveillance camera 10 in the pitch direction by receiving the power generated by the motor 74.

[0074] The communication I / Fs 79 and 80 are, for example, network interfaces, and control the transmission of various information between the management device 11 and the rotation mechanism 16 via a network. This network is, for example, a WAN such as the Internet or a LAN. The communication I / Fs 79 and 80 perform communication between the rotation mechanism 16 and the management device 11.

[0075] 5, the management device 11 includes a display 13a, a secondary storage device 14, a control device 60, a receiving device 62, and communication I / Fs 66, 67, and 68. The control device 60 includes a CPU 60A, a storage 60B, and a memory 60C. The CPU 60A is an example of a processor according to the present invention.

[0076] The accepting device 62, the display 13a, the secondary storage device 14, the CPU 60A, the storage 60B, the memory 60C, and the communication I / F 66 are each connected to a bus 70. In the example shown in Fig. 5, for convenience of illustration, one bus is shown as the bus 70, but multiple buses may be used. The bus 70 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0077] The memory 60C temporarily stores various information and is used as a work memory. An example of the memory 60C is a RAM, but other types of storage devices may be used. The storage 60B stores various programs for the management device 11 (hereinafter simply referred to as "management device programs").

[0078] The CPU 60A reads out the management device program from the storage 60B and executes the read management device program on the memory 60C to control the entire management device 11. The management device program includes the information processing program of the present invention.

[0079] The communication I / F 66 is, for example, a network interface. The communication I / F 66 is communicatively connected to the communication I / F 34 of the surveillance camera 10 via a network, and controls the transmission of various information between the communication I / F 66 and the surveillance camera 10. The communication I / Fs 67 and 68 are, for example, network interfaces. The communication I / F 67 is communicatively connected to the communication I / F 79 of the rotation mechanism 16 via a network, and controls the transmission of various information between the communication I / F 67 and the yaw axis rotation mechanism 71. The communication I / F 68 is communicatively connected to the communication I / F 80 of the rotation mechanism 16 via a network, and controls the transmission of various information between the communication I / F 67 and the pitch axis rotation mechanism 72.

[0080] The CPU 60A receives captured images, captured image information, and the like from the surveillance camera 10 via the communication I / F 66 and the communication I / F 34 .

[0081] The CPU 60A controls the driver 75 and motor 73 of the turning mechanism 16 via the communication I / F 67 and communication I / F 79, thereby controlling the turning operation of the yaw axis turning mechanism 71. The CPU 60A also controls the driver 76 and motor 74 of the turning mechanism 16 via the communication I / F 68 and communication I / F 80, thereby controlling the turning operation of the pitch axis turning mechanism 72.

[0082] The reception device 62 is, for example, the keyboard 13b, the mouse 13c, or the touch panel of the display 13a, and receives various instructions from the user. The CPU 60A acquires the various instructions received by the reception device 62 and operates in accordance with the acquired instructions. For example, when the reception device 62 receives processing content for the surveillance camera 10 and / or the rotation mechanism 16, the CPU 60A operates the surveillance camera 10 and / or the rotation mechanism 16 in accordance with the instruction content received by the reception device 62.

[0083] The display 13a displays various types of information under the control of the CPU 60A. Examples of the various types of information displayed on the display 13a include the contents of various instructions accepted by the acceptance device 62, and captured images and imaging information received by the communication I / F 66. The CPU 60A causes the display 13a to display the contents of various instructions accepted by the acceptance device 62, and captured images and imaging information received by the communication I / F 66.

[0084] The secondary storage device 14 is, for example, a non-volatile memory, and stores various types of information under the control of the CPU 60 A. Examples of the various types of information stored in the secondary storage device 14 include captured images and imaging information received by the communication I / F 66. The CPU 60 A stores the captured images and imaging information received by the communication I / F 66 in the secondary storage device 14.

[0085] The communication I / F 69 is, for example, a network interface. A plurality of workers are present in the area to be monitored (imaged) by the imaging system 1 (hereinafter referred to as the "monitored area"), and each worker carries a terminal device such as a smartphone (see, for example, FIG. 7). The communication I / F 69 communicates directly or indirectly with the terminal device carried by each worker in the monitored area via a network. This network is, for example, a WAN or LAN. The monitored area is, for example, a place where dangerous work is performed by a plurality of workers, and one example is a construction site.

[0086] 6 is a diagram showing an example of an image displayed by the management device 11. The management device 11 can display, for example, a wide-area image 90 and a detailed image 91 to a user of the management device 11 (e.g., a person monitoring the monitored area) on the display 13a. In this example, the angle of view of the surveillance camera 10 is set to an angle of view that can capture only a portion of the monitored area.

[0087] The wide-area image 90 is a pseudo-wide-angle image showing the entire monitored area E1, which is generated by the management device 11 controlling the monitoring camera 10 and the rotation mechanism 16 to cause the monitoring camera 10 to capture images of each area of ​​the monitored area E1 multiple times and synthesizing (connecting) the captured images. This series of image capture control and generation of the wide-area image 90 is performed periodically, for example, at a predetermined time every day (for example, 7:00 a.m.).

[0088] The detailed image 91 is an image that represents a partial area e1 of the monitored area E1 in real time, generated from the most recent image information captured by the monitoring camera 10.

[0089] The wide-area image 90 and the detailed image 91 may be displayed side by side at the same time, or may be displayed by switching between them in response to an operation by the user of the management device 11, for example.

[0090] An area designation cursor 90a is included in the wide-area image 90. The user of the management apparatus 11 can change the position and size of the area designation cursor 90a by operating the reception device 62.

[0091] For example, the memory 60C or secondary storage device 14 of the management device 11 stores correspondence information that uniquely associates the coordinates of the wide-area image 90 with the latitude and longitude (longitude and latitude) of the position corresponding to those coordinates in the monitored area E1, and the control parameters (pan and tilt control values ​​of the surveillance camera 10) of the rotation mechanism 16 for capturing images using the surveillance camera 10 centered on the position corresponding to those coordinates in the monitored area E1.

[0092] For example, when generating the wide-area image 90 described above, the management device 11 derives a correspondence between the coordinates of the wide-area image 90 and the control parameters of the rotation mechanism 16. Furthermore, for example, the management device 11 adjusts the control parameters of the rotation mechanism 16 for multiple positions included in the monitored area E1 and having known longitude and latitude so that the monitoring camera 10 can capture images centered on those positions, and derives a correspondence between the control parameters of the rotation mechanism 16 and the longitude and latitude by associating the adjusted control parameters with the (known) longitude and latitude of those positions. This makes it possible to generate correspondence information that associates the coordinates of the wide-area image 90 with the control parameters of the rotation mechanism 16 and the longitude and latitude.

[0093] When the area designation cursor 90a is set by a user operation, the management device 11 acquires, from the correspondence information, the control parameters of the rotation mechanism 16 that correspond to the coordinates of the center of the area in the wide-area image 90 designated by the area designation cursor 90a, and sets the acquired control parameters to the rotation mechanism 16. As a result, a detailed image 91 is displayed that represents the area designated by the user of the management device 11 with the area designation cursor 90a within the monitored area E1.

[0094] That is, the user of the management device 11 can view the entire monitored area E1 through the wide-area image 90. Furthermore, if the user of the management device 11 wants to view a partial area e1 of the monitored area E1 in detail, the user can view a detailed image 91 that shows the partial area e1 in detail by setting an area designation cursor 90a on the portion of the partial area e1 in the wide-area image 90. In the example shown in Fig. 6, the detailed image 91 shows a vehicle V1 and workers W1 to W3.

[0095] In this way, by using real-time imaging information obtained by the surveillance camera 10 and a pseudo-wide-angle image generated by combining the imaging information obtained by having the surveillance camera 10 capture images of each area of ​​the monitored area E1, it is possible to display both a wide-area image 90 and a detailed image 91 using a single set of surveillance camera 10 and rotation mechanism 16.

[0096] <Hardware Configuration of Terminal Device Carried by Workers in Monitored Area E1> Figure 7 is a diagram showing an example of the hardware configuration of a terminal device carried by a worker in the monitored area E1. Each worker (e.g., workers W1 to W3) in the monitored area E1 carries, for example, the terminal device 100 shown in Figure 7. The terminal device 100 includes a processor 101, a memory 102, a communication interface 103, a GNSS (Global Navigation Satellite System) unit 104, and a user interface 105. The processor 101, the memory 102, the communication interface 103, the GNSS unit 104, and the user interface 105 are connected by, for example, a bus 109.

[0097] The processor 101 is a circuit that performs signal processing, and is, for example, a CPU that performs overall control of the terminal device 100. The processor 101 may be realized by other digital circuits such as an FPGA or a DSP. The processor 101 may also be realized by combining multiple digital circuits.

[0098] The memory 102 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM. The main memory is used as a work area for the processor 101. The auxiliary memory is, for example, a non-volatile memory such as a magnetic disk, an optical disk, or a flash memory. Various programs that operate the terminal device 100 are stored in the auxiliary memory. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 101.

[0099] The auxiliary memory may also include a portable memory that is removable from the terminal device 100. Portable memories include memory cards such as a USB (Universal Serial Bus) flash drive and an SD (Secure Digital) memory card, and external hard disk drives.

[0100] The communication interface 103 is a communication interface that performs wireless communication with the outside of the terminal device 100. For example, the communication interface 103 connects to the Internet via a mobile communication network, thereby indirectly communicating with the management device 11. The communication interface 103 is controlled by the processor 101.

[0101] The GNSS unit 104 is a satellite positioning system such as a GPS (Global Positioning System), and acquires position information (longitude and latitude) of the terminal device 100. The GNSS unit 104 is controlled by the processor 101.

[0102] The user interface 105 includes, for example, an input device that accepts operation input from the user and an output device that outputs information to the user. The input device can be realized by, for example, keys (e.g., a keyboard) or a remote control. The output device can be realized by, for example, a display or a speaker. The input device and the output device may also be realized by, for example, a touch panel. The user interface 105 is controlled by the processor 101.

[0103] <Processing by Management Device 11> Fig. 8 is a flowchart showing an example of processing by the management device 11. As described above, the management device 11 stores correspondence information that associates the coordinates of the wide-area image 90 with the control parameters of the rotation mechanism 16 and longitude and latitude. In a state where the detailed image 91 is displayed, the management device 11 executes, for example, the processing shown in Fig. 8 in response to an instruction information transmission operation from the user.

[0104] First, the management device 11 acquires the latitude and longitude corresponding to the current detailed image 91 (step S11). The latitude and longitude corresponding to this detailed image 91 is position information of the imaging target (partial area e1) associated with the image (detailed image 91) captured by the imaging system (monitoring camera 10 and rotation mechanism 16), and is an example of the first position information of the present invention. The latitude and longitude corresponding to the detailed image 91 are, for example, the latitude and longitude of the point appearing at the center of the detailed image 91. For example, based on the above-mentioned correspondence information, the management device 11 acquires the latitude and longitude corresponding to the current control parameters of the rotation mechanism 16 as the latitude and longitude corresponding to the current detailed image 91. The longitude of the latitude and longitude acquired in step S11 is designated a, and the latitude of the latitude and longitude acquired in step S11 is designated b.

[0105] Next, the management device 11 acquires the position information of the terminal device 100 of each worker in the monitored area E1 (step S12). This position information is the position information of the terminal device 100 obtained by the terminal device 100 in the imaging area (monitored area E1) of the imaging system (monitoring camera 10 and swivel mechanism 16), and is an example of the second position information of the present invention.

[0106] For example, the terminal device 100 of each worker in the monitored area E1 repeatedly transmits the position information of the terminal device 100 acquired by the GNSS unit 104 of the terminal device 100 to the management device 11. In response to this, in step S12, the management device 11 acquires the latest position information from the received position information for each of the terminal devices 100 of each worker in the monitored area E1.

[0107] Alternatively, in step S12, the management device 11 may send a request signal to the terminal device 100 of each worker in the monitored area E1 requesting the transmission of location information, and obtain the location information sent from the terminal device 100 in response to the request signal.

[0108] Next, the management device 11 determines the zoom position (focal length or zoom magnification) of the surveillance camera 10 based on the control parameters set for the surveillance camera 10, and sets Δa and Δb according to the determined zoom position (step S13). For example, the management device 11 sets Δa and Δb to be larger as the zoom position becomes wider.

[0109] Next, based on the longitude and latitude (a, b) acquired in step S11 and the currently set values ​​Δa and Δb, the management device 11 extracts, from among the terminal devices 100 of each worker in the monitored area E1, terminal devices 100 whose longitude and latitude indicated by the location information acquired in step S12 are within the range of (a±Δa, b±Δb) (step S14). The range (a±Δa, b±Δb) is a rectangular range with longitude ranging from a-Δa to a+Δa and latitude ranging from b-Δb to b+Δb. Note that the range used as the criterion in step S14 is not limited to the rectangular range (a±Δa, b±Δb) and may be a range of other shapes, such as a circular range with a radius Δ centered on the longitude and latitude (a, b).

[0110] Next, the management device 11 determines whether the number of terminal devices 100 extracted in step S14 (extracted number) is within a predetermined appropriate range (step S15). The appropriate range is, for example, a range set in advance. If the extracted number is not within the appropriate range (step S15: No), the management device 11 changes Δa and Δb (step S16) and returns to step S14.

[0111] In step S16, the management device 11 notifies the user of the number of extracted devices via the display 13a or the like, and changes Δa and Δb by accepting an instruction from the user to increase or decrease Δa and Δb. As an example, if the appropriate range is a range of 1 or more devices and the number of extracted devices is 0, the management device 11 notifies the user that the number of extracted devices is 0 and accepts an instruction from the reception device 62 as to how much to increase Δa and Δb. Furthermore, in step S16, the management device 11 may notify the user of not only the number of extracted devices but also information such as the name, affiliation, and email address of the owner of the terminal device 100 extracted in step S14. Furthermore, in step S16, the management device 11 may increase Δa and Δb if the number of extracted devices is below the appropriate range, and decrease Δa and Δb if the number of extracted devices is above the appropriate range, without accepting an instruction from the user.

[0112] In step S15, if the extracted number of vehicles is within the appropriate range (step S15: Yes), the management device 11 receives from the user of the management device 11 the setting of instructions to be given to the owner of the terminal device 100 extracted in step S14 (step S17). The instructions may be text information, an image, a voice, or a combination of these. As an example, the instructions may be a combination of the detailed image 91 (real-time image) and the text information input by the user in step S15 (e.g., a message such as "Be careful as a vehicle nearby is moving").

[0113] Furthermore, in step S17, the management device 11 may notify the user of the default instruction content. Furthermore, in step S17, the management device 11 may display information such as the name, affiliation, and email address of the owner of the terminal device 100 extracted in step S14, and input controls such as check boxes (all "on" by default) that allow the owner to specify whether or not to send the information by turning them on or off. Terminal devices 100 for which this check box is set to "off" are excluded from the destinations of the instruction information in step S18, which will be described later.

[0114] Next, the management device 11 transmits instruction information of the instruction content received in step S17 to the terminal device 100 extracted in step S14 (step S18), thereby completing the series of processes. This instruction information is an example of the first data of the present invention. As an example, the management device 11 generates an e-mail with the e-mail address of the terminal device 100 extracted in step S14 set as the destination, and the instruction content received in step S17 set in the subject and body of the e-mail, and sends the generated e-mail via the communication I / F 69.

[0115] The terminal device 100 that has received the instruction information transmitted in step S18 reproduces (for example, displays) the instruction content of the received instruction information.

[0116] Although the case where the appropriate range in step S15 of FIG. 8 is set in advance has been described, the appropriate range may also be set based on, for example, the detailed image 91. For example, in the process of FIG. 8, the management device 11 may detect the number of workers appearing in the detailed image 91 through image recognition processing based on the detailed image 91, and set the appropriate range in step S15 based on the detected number of workers. As an example, if the detected number of workers is N, the management device 11 sets the range from N-ΔNa to N+Δb as the appropriate range. However, if N-ΔNa is less than 0, the management device 11 sets the range from 0 to N+Δb as the appropriate range. The values ​​of ΔNa and ΔNb are set in advance.

[0117] For example, a user of the management device 11 can set the area designation cursor 90a in the wide-area image 90 to a location where danger occurs (for example, a location where vehicle V1 is moving) and perform an instruction information transmission operation, thereby notifying workers (for example, workers W1 to W3) in the vicinity of that location of instructions urging them to be careful or evacuate.

[0118] In this way, the management device 11 of embodiment 1 acquires first position information of the imaging target (partial area e1) associated with the captured image (detailed image 91) by the imaging system (e.g., the surveillance camera 10 and the rotation mechanism 16), and second position information of the terminal device (e.g., the terminal device 100 possessed by a mobile object such as workers W1 to W3 included in the captured image) within the imaging area (monitored area E1) of the imaging system, and generates instruction information (first data) in which a transmission destination is set based on the acquired first position information and second position information.

[0119] Specifically, the management device 11 acquires the longitude and latitude (second location information) for multiple terminal devices 100 present in the monitored area E1, extracts the terminal devices 100 whose acquired longitude and latitude are within a range based on the longitude and latitude (first location information) corresponding to the detailed image 91, and generates and transmits instruction information (first data) with the extracted terminal devices 100 as the destination.

[0120] This makes it possible to efficiently transmit instruction information to people (e.g., workers W1 to W3) who are in a specific area (e.g., partial area e1) of the monitored area E1 by utilizing the first location information of the imaged object (partial area e1) associated with the captured image (detailed image 91) and the second location information obtained by the terminal device 100.

[0121] Furthermore, the management device 11 may set the range of terminal devices 100 to be extracted as destinations based on zoom information of the imaging system (e.g., surveillance camera 10) in addition to the longitude and latitude (first location information) corresponding to the detailed image 91. For example, the management device 11 sets a wider range of terminal devices 100 to be extracted as destinations (e.g., larger Δa and Δb described above) the wider the zoom position of the surveillance camera 10. This allows the terminal devices 100 located in a range corresponding to the width of the range viewed by the user of the management device 11 in the detailed image 91 to be set as destinations, thereby making it possible for the user of the management device 11 to easily transmit instruction information to the intended worker (e.g., workers W1 to W3).

[0122] Furthermore, the management device 11 may set the range of terminal devices 100 to be extracted as destinations based on the number of mobile objects (e.g., workers W1 to W3) carrying terminal devices 100 detected from the detailed image 91 (captured image) in addition to the longitude and latitude (first location information) corresponding to the detailed image 91. For example, the management device 11 sets the appropriate range based on the number of workers detected from the detailed image 91, and performs processing to change the range of terminal devices 100 to be extracted as destinations (e.g., the above-mentioned Δa and Δb) so that the extracted range of the terminal devices 100 becomes the appropriate range. In this way, if there is a large difference between the number of extracted terminal devices 100 and the number of workers appearing in the detailed image 91 viewed by the user of the management device 11 when the instruction information was transmitted, the difference can be reduced.

[0123] <First Modification of Imaging System 1> Fig. 9 is a diagram showing a first modification of the imaging system 1. As shown in Fig. 9, the imaging system 1 may include a surveillance camera 10a in addition to the configuration shown in Fig. 1. The surveillance camera 10a has a wider angle of view than the surveillance camera 10 and is installed so as to be able to capture an image of the entire monitoring target area E1. The configuration of the surveillance camera 10a is similar to that of the surveillance camera 10 shown in Fig. 4, for example, but differs in that the optical system 15 has wider-angle optical characteristics.

[0124] Fig. 10 is a diagram showing an example of the configuration of the electrical system of the management device 11 shown in Fig. 9. In the imaging system 1 shown in Fig. 9, the surveillance camera 10a includes a communication I / F 34a similar to the communication I / F 34 of the surveillance camera 10. The communication I / F 34a performs communication between the surveillance camera 10a and the management device 11.

[0125] The communication I / F 66 of the management device 11 is communicatively connected to the communication I / F 34 of the surveillance camera 10 as well as the communication I / F 34a of the surveillance camera 10a, and controls the transmission of various information between the surveillance cameras 10 and 10a.

[0126] Fig. 11 is a diagram showing an example of an image displayed by the management device 11 in the configuration shown in Fig. 9 and Fig. 10. In the configuration shown in Fig. 9 and Fig. 10, the management device 11 displays an image based on imaging information obtained from the monitoring camera 10a as a wide-area image 90 representing the monitored area E1. That is, the wide-area image 90 in this case is not a pseudo-wide-angle image generated by combining the imaging information obtained by having the monitoring camera 10 capture images of each area of ​​the monitored area E1, but is a wide-angle image based on a single piece of imaging information obtained by the wide-angle monitoring camera 10a.

[0127] In this case, the wide-area image 90 may be a non-real-time image obtained by periodic imaging as described above, or a real-time image obtained from the most recent imaging information obtained by imaging by the surveillance camera 10a.

[0128] In this case, management device 11 also stores the above-mentioned correspondence information that uniquely associates coordinates of the wide area image 90 with the control parameters and longitude and latitude of the rotation mechanism 16. In this case, the coordinates of the wide area image 90 that correspond to the control parameters and longitude and latitude of the rotation mechanism 16 are derived, for example, by a user of management device 11 specifying corresponding coordinates in the wide area image 90 for multiple positions included in the monitored area E1 and with known longitude and latitude. Management device 11 uses this correspondence information to execute the processing shown in FIG. 8.

[0129] <Second Modification of Imaging System 1> Fig. 12 is a diagram showing a second modification of the imaging system 1. As shown in Fig. 12, the imaging system 1 may have a configuration in which the surveillance camera 10 and the rotation mechanism 16 are omitted from the configuration shown in Fig. 9.

[0130] Fig. 13 is a diagram showing an example of the configuration of the electrical system of the management device 11 shown in Fig. 12. In the imaging system 1 shown in Fig. 12, the management device 11 has a configuration obtained by omitting the communication I / F 67 and the communication I / F 68 from the configuration shown in Fig. 10. The communication I / F 66 of the management device 11 is communicably connected to the communication I / F 34a of the surveillance camera 10a, and controls the transmission of various information between the management device 11 and the surveillance camera 10a.

[0131] 12 and 13, the images displayed by the management device 11 are similar to, for example, the wide-area image 90 and detailed image 91 shown in Fig. 11. However, the management device 11 displays, as the detailed image 91 representing the partial area e1, a digitally zoomed image obtained by cutting out and enlarging an area designated by an area designation cursor 90a in the wide-area image 90. In this case, the wide-area image 90 is a real-time image obtained from the most recent imaging information captured by, for example, the surveillance camera 10a.

[0132] In this case, the management device 11 stores correspondence information that uniquely associates coordinates of the wide-area image 90 with longitude and latitude. That is, the correspondence information in this case does not require the control parameters of the rotation mechanism 16. In this case, the coordinates of the wide-area image 90 that correspond to the longitude and latitude are derived, for example, by a user of the management device 11 specifying corresponding coordinates in the wide-area image 90 for multiple positions that are included in the monitored area E1 and have known longitude and latitude.

[0133] The management device 11 uses this correspondence information to execute the processing shown in Fig. 8. In this case, however, in step S11, the management device 11 acquires the longitude and latitude corresponding to the detailed image 91 based on the coordinates of the digitally zoomed area of ​​the wide-area image 90 and the correspondence information. In step S13, the management device 11 also sets Δa and Δb according to the amount of digital zoom of the detailed image 91.

[0134] 14 is a diagram showing an example of a process for confirming whether a person is unwell, injured, etc. The imaging system 1 can also be applied to a system that can confirm whether a worker has collapsed or is unable to move due to heat stroke or injury in a monitored area E1 (e.g., a construction site).

[0135] For example, each of the terminal devices 100 detects an abnormality in the worker carrying the terminal device 100. The abnormality in the worker is detected based on at least one of the following: a state in which the longitude and latitude acquired by the GNSS unit 104 included in the terminal device 100 does not fluctuate for a certain period of time or more; a state in which the terminal device 100 remains stationary for a certain period of time or more as detected by an acceleration sensor included in the terminal device 100; or an abnormal value in the worker's biological information measured by a wearable device that is capable of communicating with the terminal device 100 and worn by the worker.

[0136] In this case, the terminal device 100 transmits to the management device 11, together with longitude and latitude information acquired by the GNSS unit 104 provided in the terminal device 100, abnormality detection information indicating that an abnormality in the worker has been detected. Upon receiving the abnormality detection information and the longitude and latitude information, the management device 11 displays a detailed image 91 of an area in the wide-area image 90 that corresponds to the longitude and latitude information. This makes it possible to automatically display a detailed image 91 indicating the position of the worker when an abnormality in the worker is detected by the terminal device 100. This allows the user of the management device 11 to quickly check the condition of the worker in whom an abnormality has been detected.

[0137] 14 , worker W2 has collapsed in partial area e1, and the terminal device 100 carried by worker W2 transmits abnormality detection information and latitude and longitude information to the management device 11. In this case, the management device 11 displays a detailed image 91 of partial area e1 based on the latitude and longitude information received together with the abnormality detection information. This allows the user of the management device 11 to quickly confirm that worker W2 has collapsed.

[0138] Furthermore, by performing the above-mentioned instruction information transmission operation while the detailed image 91 showing the worker W2 is displayed, the user of the management device 11 can transmit instruction information to inquire about the situation to the worker W2, or transmit instruction information to instruct other workers W1 and W2 around the worker W2 to rescue, etc.

[0139] <Modifications of Instruction Information> Although the instruction information, which is an example of the first data, is described as being an email, the instruction information is not limited to an email and may be various types of message information such as a message sent via SMS (Short Message Service) or a messenger app.

[0140] (Embodiment 2) Embodiment 2 will be described with respect to differences from embodiment 1. The imaging system 1 of embodiment 2 has a configuration similar to that of the imaging system 1 shown in, for example, Figures 1 to 5, and can display a wide-area image 90 and a detailed image 91 on the display 13a in the same manner as the example shown in Figure 6.

[0141] 15 is a flowchart showing an example of processing by the management device 11 of the second embodiment. The management device 11 of the second embodiment stores a wide-area image 90, which is a pseudo-wide-angle image, in the secondary storage device 14 in association with the time of generation of the wide-area image 90. For example, the management device 11 generates the wide-area image 90 at a predetermined time every day, and stores the generated wide-area image 90 in the secondary storage device 14 in association with the date and time of generation.

[0142] The management device 11 of the second embodiment executes, for example, the process shown in FIG. 15 in response to an operation from a user.

[0143] First, the management device 11 acquires a wide-area image 90 (step S21). This wide-area image 90 is, for example, a pseudo-wide-angle image generated by causing the monitoring camera 10 to capture images of each area of ​​the monitored area E1 and combining the captured images.

[0144] Next, the management device 11 acquires a history of location information of the terminal device 100 of the worker in the monitored area E1 (step S22). The history of location information of the terminal device 100 is, for example, the longitude and latitude of the terminal device 100 at each of a plurality of times. For example, the terminal device 100 of the worker in the monitored area E1 repeatedly transmits its own location information acquired by its own GNSS unit 104 to the management device 11. In response to this, the management device 11 accumulates the location information received from the terminal device 100 of the worker in the monitored area E1, and thereby acquires the history of location information of the terminal device 100 in step S12.

[0145] Alternatively, in step S22, the management device 11 may send a request signal to the terminal device 100 of the worker in the monitored area E1 requesting the transmission of a history of location information, and obtain the history of location information sent from the terminal device 100 in response to the request signal.

[0146] Next, the management device 11 superimposes and displays a movement history image indicating the history of the location information of the terminal device 100 acquired in step S22 on the wide-area image 90 acquired in step S21 (step S23), and ends the series of processes. This movement history image is an example of information regarding the movement history of the mobile object in the present invention.

[0147] 15 may be all workers who are present in the monitored area E1 and carry the terminal device 100, or may be some of these workers (e.g., worker W1). The worker who is the target of the process shown in FIG. 15 may be designated by the user of the management device 11.

[0148] <Display of wide-area image 90 with movement history image superimposed> Figure 16 is a diagram showing an example of the display of the wide-area image 90 with a movement history image superimposed. In the example of Figure 16, a case will be described in which the process shown in Figure 15 is performed for worker W1. In step S23 shown in Figure 15, the management device 11 displays, on the display 13a, the wide-area image 90 with a movement history image 161 superimposed thereon, such as the wide-area image 90 shown in Figure 16. The movement history image 161 is an image showing the history of location information acquired from the terminal device 100 carried by worker W1.

[0149] For example, the management device 11 identifies each coordinate in the wide area image 90 to which the longitude and latitude correspond for each time indicated by the history of position information, based on the above-mentioned correspondence information that associates coordinates of the wide area image 90 with longitude and latitude. Then, the management device 11 displays the movement history image 161 superimposed on the wide area image 90 by drawing lines connecting each of the identified coordinates in the chronological order of the history of position information.

[0150] The wide-area image 90 on which the movement history image 161 is superimposed may be the latest wide-area image 90 (latest image) among the wide-area images 90 stored by the management device 11, or it may be a wide-area image 90 (time history corresponding image) among the wide-area images 90 stored by the management device 11 that corresponds to the time of history (for example, the first or last time) indicated by the movement history image 161.

[0151] Furthermore, in response to an operation by the user, the management device 11 may display a detailed image 91 on which the movement history image 161 is superimposed, similar to the wide-area image 90. In this case, the detailed image 91 may be a digitally zoomed image obtained by cutting out and enlarging an area designated by the area designation cursor 90a in the wide-area image 90 on which the movement history image 161 is superimposed (pseudo-wide-angle image mode), or may be an image on which the movement history image 161 is superimposed on a real-time image based on an image captured by controlling the rotation mechanism 16 and the monitoring camera 10 to capture an image of the area designated by the area designation cursor 90a (real-time video mode).

[0152] The detailed image 91 may also be a pseudo-wide-angle image generated by having the surveillance camera 10 capture images of partial areas of an area wider than the area specified by the area designation cursor 90a and combining the captured image information. This makes it possible to display the detailed image 91 representing an area wider than the angle of view of the surveillance camera 10 (but narrower than the wide-area image 90). This prevents a situation in which the angle of view of the surveillance camera 10 is too narrow to display the entire movement history image 161 in the detailed image 91. The management device 11 may also set the above-mentioned "area wider than the specified area and including the specified area" so as to include each longitude and latitude position included in the location information history.

[0153] The management device 11 may also accept from the user a setting for a period (e.g., a start time and an end time) of the history of location information to be displayed. For example, the management device 11 accepts the setting for the period of the history of location information to be displayed by displaying a time bar on the display 13a and accepting a setting for the pointer position of the time bar via the accepting device 62. In this case, in step S22 shown in FIG. 15 , the management device 11 acquires the history of location information for the period specified by the user.

[0154] 15 is performed on worker W1 in the above example, but the management device 11 may also receive a selection of workers who are the targets of the processing in FIG. 15 from among the workers who are present in the monitored area E1 and who are carrying the terminal device 100. This worker selection may be performed by individually specifying the worker, or by specifying the group (for example, company or department) to which the worker belongs.

[0155] In this way, the management device 11 of embodiment 2 acquires first position information of the imaging target (monitored area E1) associated with the image (wide-area image 90) captured by the imaging system (e.g., the surveillance camera 10 and the rotation mechanism 16), and second position information of a terminal device (e.g., the terminal device 100 possessed by a moving object such as workers W1 to W3 included in the captured image) within the imaging area (monitored area E1) of the imaging system, and generates an image (first data) obtained by superimposing a movement history image 161 showing the movement history of the worker on at least one of the wide-area image 90 and the detailed image 91 based on the acquired first position information and second position information.

[0156] This makes it possible to display an image that allows easy confirmation of the movement history of the worker in the monitored area E1 by utilizing the first position information of the imaged target (monitored area E1) associated with the captured image (wide-area image 90) and the second position information obtained by the terminal device 100. This makes it easy to, for example, identify whether the worker has entered a dangerous location, determine whether the worker has acted in accordance with predetermined procedures, and analyze the cause of an accident if one occurs.

[0157] <First Modification of the Imaging System 1 of the Second Embodiment> The imaging system 1 of the second embodiment may have the configuration shown in FIGS. 9 and 10, for example.

[0158] Fig. 17 is a diagram showing an example of an image displayed by the management device 11 when the imaging system 1 of the second embodiment has the configuration shown in Fig. 9 and Fig. 10. In the configuration shown in Fig. 9 and Fig. 10, the management device 11 of the second embodiment displays an image based on imaging information obtained from the monitoring camera 10a as a wide-area image 90 representing the monitored area E1, similar to the example of Fig. 11.

[0159] In this case as well, the management device 11 stores the above-mentioned correspondence information that associates the coordinates of the wide-area image 90 with the longitude and latitude. The management device 11 uses this correspondence information to execute the process shown in FIG. 15 .

[0160] <Modification 2 of Imaging System 1 of Embodiment 2> The imaging system 1 of Embodiment 2 may have the configuration shown in, for example, FIGS. 12 and 13. In this case, the images displayed by the management device 11 are similar to the wide-area image 90 and detailed image 91 shown in, for example, FIG. 17. However, the management device 11 displays, as the detailed image 91 representing the partial area e1, a digitally zoomed image obtained by cutting out and enlarging an area specified by the area specifying cursor 90a in the wide-area image 90. In this case, the management device 11 also stores the above-mentioned correspondence information that associates the coordinates of the wide-area image 90 with longitude and latitude. The management device 11 uses this correspondence information to execute the processing shown in FIG. 15.

[0161] <Modified example of mobile body> In the first and second embodiments, a person (workers W1 to W3) was used as an example of a mobile body having the terminal device 100, but the mobile body having the terminal device 100 is not limited to a person, and may be any mobile body that moves together with the terminal device 100, such as a vehicle (for example, vehicle V1) equipped with the terminal device 100.

[0162] <Storage medium for information processing program> In each of the above operation control examples, an example has been described in which the information processing program of each embodiment is stored in the storage 60B of the management device 11 and the CPU 60A of the management device 11 executes the information processing program in the memory 60C, but the technology of the present disclosure is not limited to this.

[0163] 18 is a diagram showing an example of how an information processing program of an operational control example is installed from a storage medium storing the information processing program into the control device 60 of the management device 11. As an example, as shown in FIG. 18, the information processing program 221 may be stored in a storage medium 220, which is a non-transitory storage medium. In the example shown in FIG. 18, the information processing program 221 stored in the storage medium 220 is installed into the control device 60, and the CPU 60A executes the above-described processes in accordance with the information processing program 221.

[0164] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0165] This application is based on a Japanese patent application (Patent Application No. 2022-088615) filed on May 31, 2022, the contents of which are incorporated herein by reference.

[0166] 1 Imaging system 10, 10a Surveillance camera 11 Management device 12 Communication line 13a, 43B Display 13b Keyboard 13c Mouse 14 Secondary storage device 15 Optical system 15B Lens group 15B1 Anti-vibration lens 15B2 Zoom lens 16 Swivel mechanism 17, 21 Lens actuator 19 Computer 22, 23, 75, 76 Driver 22 BIS driver 23 OIS driver 25 Imaging element 25A Light receiving surface 27 Imaging element actuator 28 Lens driver 29, 45 Correction mechanism 31 DSP 32 Image memory 33 Correction unit 34, 34a, 66 to 69, 79, 80 Communication I / F 35, 60C, 102 Memory 36, 60B Storage 37, 60A CPU 38, 70, 109 Bus 39, 47 Position sensor 40 Quantity detection sensor 43 UI device 43A, 62 Reception device 60 Control device 71 Yaw axis rotation mechanism 72 Pitch axis rotation mechanism 73, 74 Motor 90 Wide area image 90a Area designation cursor 91 Detailed image 100 Terminal device 101 Processor 103 Communication interface 104 GNSS unit 105 User interface 161 Movement history image 220 Storage medium 221 Information processing program E1 Monitoring target area e1 Partial area V1 Vehicle W1 to W3 Workers

Claims

1. An information processing device including a processor, The processor, acquiring first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a plurality of terminal devices obtained by the plurality of terminal devices in an imaging area of ​​the imaging system; deriving a range based on the first location information, extracting, from among the plurality of terminal devices, a terminal device whose second location information is included in the derived range, and generating instruction information whose destination is the extracted terminal device; Information processing device.

2. 2. The information processing device according to claim 1, the terminal device is a terminal device owned by a moving object included in the captured image, Information processing device.

3. An information processing device according to claim 1, The processor sets the range based on the first position information and zoom information of the imaging system. Information processing device.

4. An information processing device according to claim 1, The processor sets the range based on the first position information and the number of moving objects having the terminal devices detected from the captured image. Information processing device.

5. An information processing device according to claim 1, The processor, and performing control to display, on a display device, a wide-area image showing the imaging area and a detailed image showing a range of the imaging area narrower than the wide-area image at a higher magnification than the wide-area image. Information processing device.

6. A processor of an information processing device, acquiring first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a plurality of terminal devices obtained by the plurality of terminal devices in an imaging area of ​​the imaging system; deriving a range based on the first location information, extracting, from among the plurality of terminal devices, a terminal device whose second location information is included in the derived range, and generating instruction information whose destination is the extracted terminal device; Information processing methods.

7. A processor of an information processing device, acquiring first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a plurality of terminal devices obtained by the plurality of terminal devices in an imaging area of ​​the imaging system; deriving a range based on the first location information, extracting, from among the plurality of terminal devices, a terminal device whose second location information is included in the derived range, and generating instruction information whose destination is the extracted terminal device; An information processing program for executing processing.

8. An information processing device having a processor, The processor, Acquire first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system; generating first data including instruction information for designating the terminal device as a transmission destination based on the first location information and the second location information; When receiving information indicating an abnormality of the holder of the terminal device from the terminal device, control is performed to display the captured image indicating the area of ​​the captured area indicated by the second position information on a display device. Information processing device.

9. A processor of an information processing device, Acquire first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system; generating first data including instruction information for designating the terminal device as a transmission destination based on the first location information and the second location information; When receiving information indicating an abnormality of the holder of the terminal device from the terminal device, control is performed to display the captured image indicating the area of ​​the captured area indicated by the second position information on a display device. Information processing methods.

10. A processor of an information processing device, Acquire first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system; generating first data including instruction information for designating the terminal device as a transmission destination based on the first location information and the second location information; When receiving information indicating an abnormality of the holder of the terminal device from the terminal device, control is performed to display the captured image indicating the area of ​​the captured area indicated by the second position information on a display device. An information processing program for executing processing.

11. An information processing device having a processor, The processor, Acquire first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system; generating an image, which is obtained by superimposing an image showing a movement history of a mobile object having the terminal device on the captured image based on the first location information and the second location information, as information regarding the movement history of the mobile object; Information processing device.

12. A processor of an information processing device, Acquire first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system; generating an image, which is obtained by superimposing an image showing a movement history of a mobile object having the terminal device on the captured image based on the first location information and the second location information, as information regarding the movement history of the mobile object; Information processing methods.

13. A processor of an information processing device, Acquire first position information of an imaging target position associated with an image captured by an imaging system, and second position information of a terminal device obtained by the terminal device within an imaging area of ​​the imaging system; generating an image, which is obtained by superimposing an image showing a movement history of a mobile object having the terminal device on the captured image based on the first location information and the second location information, as information regarding the movement history of the mobile object; An information processing program for executing processing.

14. An information processing device having a processor, The processor, acquire first position information of an imaging target position associated with an imaging image captured by an imaging system including an imaging device that obtains an imaging image by imaging and a rotation mechanism that rotates the imaging device, the first position information being associated with information indicating a rotation state of the imaging device by the rotation mechanism when the imaging image was obtained by the imaging system, and second position information of the terminal device obtained by a terminal device in an imaging area of ​​the imaging system; generating first data based on the first location information and the second location information; Information processing device.

15. A processor of an information processing device, acquire first position information of an imaging target position associated with an imaging image captured by an imaging system including an imaging device that obtains an imaging image by imaging and a rotation mechanism that rotates the imaging device, the first position information being associated with information indicating a rotation state of the imaging device by the rotation mechanism when the imaging image was obtained by the imaging system, and second position information of the terminal device obtained by a terminal device in an imaging area of ​​the imaging system; generating first data based on the first location information and the second location information; Information processing methods.

16. A processor of an information processing device, acquire first position information of an imaging target position associated with an imaging image captured by an imaging system including an imaging device that obtains an imaging image by imaging and a rotation mechanism that rotates the imaging device, the first position information being associated with information indicating a rotation state of the imaging device by the rotation mechanism when the imaging image was obtained by the imaging system, and second position information of the terminal device obtained by a terminal device in an imaging area of ​​the imaging system; generating first data based on the first location information and the second location information; An information processing program for executing processing.