Information processing device, information processing method, and information processing program

Abstract

Provided are an information processing device, an information processing method, and an information processing program with which it is possible to use position information of an imaging subject associated with a captured image and position information obtained by a terminal device. A management device (11) acquires: first position information relating to a position of an imaging target associated with an image captured by a surveillance camera (10) and a swing mechanism (16); and second position information of the terminal device (100) obtained by monitoring the terminal device (100) in an imaging region of the camera (10) and the swing mechanism (16). The management device (11) then generates first data on the basis of the acquired first position information and second position information.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method and an information processing program. Background Art

[0002] Patent document 1 describes a wide-area monitoring system that generates suspicious person information including a notification destination based on the location information of a monitoring camera associated with a photographic image and the longitude and latitude of a user's mobile terminal. Patent document 2 describes an information processing system in which the history data of the location information of a mobile terminal is stored in a database server, and in the process of obtaining the action history of the location information of the mobile terminal from a communication terminal, map information corresponding to the history data of the location information is obtained from a map search service, and map information is generated in which the history data is superimposed on the map information, and output to the communication terminal. Patent document 3 describes a control device for a camera device that takes into account the error of the latitude and longitude values ​​and determines a range including the true latitude and longitude values ​​based on the actual latitude and longitude values.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2014-078150

[0006] Patent Document 2: Japanese Patent Application Publication No. 2013-246570

[0007] Patent Document 3: Japanese Patent Application Publication No. 2019-124858 Summary of the invention

[0008] One embodiment of the technology according to the present invention provides an information processing device, an information processing method, and an information processing program that can use position information of a photographed object associated with a photographed image and position information obtained by a terminal device.

[0009] Means for solving technical problems

[0010] (1) An information processing device comprising a processor, wherein:

[0011] The above processor performs the following processing:

[0012] Acquire first position information of a position of an image capturing object corresponding to an image captured by the image capturing system and second position information of the terminal device obtained by the terminal device in the image capturing area of ​​the image capturing system; and

[0013] The first data is generated based on the first position information and the second position information.

[0014] (2) The information processing device according to (1), wherein:

[0015] The terminal device is a terminal device possessed by the moving object included in the captured image.

[0016] (3) The information processing device according to (1) or (2), wherein:

[0017] The first data includes instruction information for setting the terminal device as a transmission destination.

[0018] (4) The information processing device according to (3), wherein:

[0019] The above processor performs the following processing:

[0020] Regarding the acquisition of the second location information by a plurality of terminal devices; and

[0021] The instruction information is generated to set the terminal device whose second location information is included in the range based on the first location information among the plurality of terminal devices as the transmission destination.

[0022] (5) The information processing device according to (4), wherein:

[0023] The processor sets the range based on the first position information and zoom information of the imaging system.

[0024] (6) The information processing device according to (4) or (5), wherein:

[0025] The processor sets the range based on the first position information and the number of moving objects having the terminal device detected from the camera image.

[0026] (7) The information processing device according to any one of (3) to (6), wherein:

[0027] The above processor performs the following control:

[0028] When information indicating abnormality of the holder of the terminal device is received from the terminal device, the captured image indicating the area indicated by the second position information in the imaging area is displayed on a display device.

[0029] (8) The information processing device according to (1) or (2), wherein:

[0030] The first data is information related to the movement history of the mobile body having the terminal device.

[0031] (9) The information processing device according to (8), wherein:

[0032] The above processor performs the following processing:

[0033] Based on the first position information and the second position information, an image obtained by superimposing an image showing the movement history of the moving object on the captured image is generated as information related to the movement history.

[0034] (10) The information processing device according to any one of (1) to (9), wherein:

[0035] The above processor performs the following control:

[0036] A wide area image showing the imaging area and a detailed image showing a narrower range in the imaging area than the wide area image at a higher magnification than the wide area image are displayed on a display device.

[0037] (11) The information processing device according to any one of (1) to (10), wherein:

[0038] The above-mentioned camera system includes a camera device that obtains the above-mentioned camera image by shooting and a rotating mechanism that rotates the above-mentioned camera device. The above-mentioned first position information is correspondingly associated with information representing the rotation state of the above-mentioned camera device based on the above-mentioned rotating mechanism when the above-mentioned camera image is obtained by the above-mentioned camera system.

[0039] (12) An information processing method, wherein a processor of an information processing device performs the following processing:

[0040] Acquire first position information of a position of an image capturing object corresponding to an image captured by the image capturing system and second position information of the terminal device obtained by the terminal device in the image capturing area of ​​the image capturing system; and

[0041] The first data is generated based on the first position information and the second position information.

[0042] (13) An information processing program for causing a processor of an information processing device to perform the following processing:

[0043] Acquire first position information of a position of an image capturing object corresponding to an image captured by the image capturing system and second position information of the terminal device obtained by the terminal device in the image capturing area of ​​the image capturing system; and

[0044] The first data is generated based on the first position information and the second position information.

[0045] An optical observation device according to one aspect of the present invention includes the above-mentioned projection device.

[0046] Effects of the Invention

[0047] According to the present invention, it is possible to provide an information processing apparatus, an information processing method, and an information processing program that utilize position information of a photographed object associated with a photographed image and position information obtained by a terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a diagram showing an example of an imaging system 1 equipped with a control device according to the present embodiment.

[0049] Figure 2 FIG. 1 is a diagram showing an example of rotation of the monitoring camera 10 in the pitch direction by the rotation mechanism 16 .

[0050] Figure 3 FIG. 1 is a diagram showing an example of the rotation of the monitoring camera 10 in the yaw direction by the rotation mechanism 16 .

[0051] Figure 4 It is a block diagram showing an example of the configuration of the optical system and the electrical system of the monitoring camera 10 .

[0052] Figure 5 It is a diagram showing an example of the configuration of the electric system of the turning mechanism 16 and the management device 11 .

[0053] Figure 6 This is a diagram showing an example of an image displayed on the management device 11.

[0054] Figure 7 This is a diagram showing an example of the hardware configuration of a terminal device owned by a worker in the monitoring target area E1.

[0055] Figure 8 This is a flowchart showing an example of processing performed by the management device 11.

[0056] Fig. 9 It is a diagram showing a first modification of the imaging system 1 .

[0057] Fig.10 Yes means Fig. 9 A diagram showing an example of the configuration of an electrical system of the management device 11 shown.

[0058] Fig.11 It means in Fig. 9 , Fig.10 A diagram showing an example of an image displayed by the management device 11 in the structure shown.

[0059] Fig.12 It is a diagram showing a second modification of the imaging system 1 .

[0060] Fig.13 Yes means Fig.12A diagram showing an example of the configuration of an electrical system of the management device 11 shown.

[0061] Fig.14 This is a diagram showing an example of a process for confirming a person who is unwell or injured.

[0062] Fig.15 This is a flowchart showing an example of processing performed by the management device 11 according to the second embodiment.

[0063] Fig.16 1 is a diagram showing an example of display of wide area image 90 on which the movement history image is superimposed.

[0064] Fig.17 This means that the camera system 1 of the second embodiment is set to Fig. 9 , Fig.10 FIG. 1 is a diagram showing an example of an image displayed by the management device 11 in the case of the structure shown.

[0065] Fig.18 This is a diagram showing an example of a method of installing an information processing program of an action control example from a storage medium storing the information processing program in the control device 60 of the management device 11 . DETAILED DESCRIPTION

[0066] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0067] (Implementation Method 1)

[0068] <Image capture system according to the embodiment>

[0069] Figure 1 FIG. 1 is a diagram showing an example of an imaging system 1 equipped with a control device according to the present embodiment. Figure 1 As shown, as an example, the camera system 1 includes a monitoring camera 10, a management device 11, and a rotation mechanism 16. The monitoring camera 10 and the rotation mechanism 16 are examples of the camera system in the present invention. The management device 11 is an example of the information processing device in the present invention.

[0070] The monitoring camera 10 is installed on a pillar, a wall, or a part of a building (e.g., a roof) indoors or outdoors via a rotating mechanism 16, and shoots a subject. The monitoring camera 10 transmits a shooting image obtained by shooting and shooting information related to the shooting to the management device 11 via a communication line 12.

[0071] The management device 11 includes a display 13a, a keyboard 13b, a mouse 13c, and a secondary storage device 14. Examples of the display 13a include a liquid crystal display, a plasma display, an organic EL (Electro-Luminescence) display, a CRT (Cathode Ray Tube) display, etc. The display 13a is an example of a display device of the present invention.

[0072] An example of the secondary storage device 14 is a hard disk drive (HDD). The secondary storage device 14 is not limited to a HDD, and may be any nonvolatile memory such as a flash memory, a solid state drive (SSD), or an electrically erasable and programmable read only memory (EEPROM).

[0073] The management device 11 receives the captured images or captured information transmitted from the monitoring camera 10 , and displays the received captured images or captured information on the display 13 a or stores them in the secondary storage device 14 .

[0074] The management device 11 performs imaging control for controlling imaging by the monitoring camera 10. For example, the management device 11 performs imaging control by communicating with the monitoring camera 10 via the communication line 12. The imaging control is a control for setting imaging parameters for the monitoring camera 10 to perform imaging in the monitoring camera 10 and causing the monitoring camera 10 to perform imaging. The imaging parameters include parameters related to exposure and parameters of zoom position, etc.

[0075] Furthermore, the management device 11 controls the rotation mechanism 16 to control the camera direction (panning or tilting) of the monitoring camera 10. For example, the management device 11 sets the rotation direction, rotation amount, rotation speed, etc. of the monitoring camera 10 according to the operation of the keyboard 13b or the mouse 13c or the touch operation on the screen of the display 13a.

[0076] <Rotation of the surveillance camera 10 by the rotation mechanism 16>

[0077] Figure 2 FIG. 1 is a diagram showing an example of rotation of the monitoring camera 10 in the pitch direction by the rotation mechanism 16 . Figure 3 1 is a diagram showing an example of the yaw direction rotation of the monitoring camera 10 by the rotation mechanism 16. The monitoring camera 10 is mounted on the rotation mechanism 16. The rotation mechanism 16 can rotate the monitoring camera 10.

[0078] Specifically, the swivel mechanism 16 is a device that enables the surveillance camera 10 to be swivel-rotated as an example. Figure 2 The rotation direction (pitch direction) intersecting the yaw direction and centered on the pitch axis PA is shown as an example. Figure 3 The double-axis rotating mechanism shown rotates in the rotation direction (yaw direction) with the yaw axis YA as the central axis. In addition, in the rotating mechanism 16 involved in this embodiment, an example of a double-axis rotating mechanism is shown, but the technology of the present invention is not limited to this, and it can also be a three-axis rotating mechanism or a single-axis rotating mechanism.

[0079] <Configuration of the Optical System and Electrical System of the Monitoring Camera 10>

[0080] Figure 4 1 is a block diagram showing an example of the structure of the optical system and the electrical system of the monitoring camera 10. Figure 4 As shown, as an example, the surveillance camera 10 includes an optical system 15 and an imaging element 25. The imaging element 25 is located at the rear section of the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in the order of the objective lens 15A and the lens group 15B along the optical axis OA of the optical system 15 from the object subject side (object side) to the light receiving surface 25A side (image side) of the imaging element 25. The lens group 15B includes an anti-vibration lens 15B1, a focusing lens (not shown) and a zoom lens 15B2. The zoom lens 15B2 is supported so as to be movable along the optical axis OA by a lens actuator 21 described later. The anti-vibration lens 15B1 is supported so as to be movable in a direction orthogonal to the optical axis OA by a lens actuator 17 described later.

[0081] By extending the focal length by the zoom lens 15B2, the monitoring camera 10 becomes telephoto, so the angle of view becomes smaller (the imaging range becomes narrower). By shortening the focal length by the zoom lens 15B2, the angle of view becomes larger (the imaging range becomes wider).

[0082] In addition, the optical system 15 may include various lenses not shown in the figure in addition to the objective lens 15A and the lens group 15B. In addition, the optical system 15 may include an aperture. The positions of the lenses, lens groups, and apertures included in the optical system 15 are not limited. Figure 4 The technology of the present invention is also applicable to positions different from the positions shown.

[0083] The anti-vibration lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0084] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force to the anti-vibration lens 15B1 to change the position in a direction perpendicular to the optical axis of the anti-vibration lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. By driving the lens actuator 17 under the control of the OIS driver 23, the position of the anti-vibration lens 15B1 changes in a direction perpendicular to the optical axis OA.

[0085] The lens actuator 21 applies a force to the zoom lens 15B2 for moving along the optical axis OA of the optical system 15. The lens actuator 21 is controlled by the lens driver 28. The position of the zoom lens 15B2 moves along the optical axis OA by driving the lens actuator 21 under the control of the lens driver 28. By the position of the zoom lens 15B2 moving along the optical axis OA, the focal length of the monitoring camera 10 changes.

[0086] In addition, when the outline of the captured image is, for example, a rectangle having short sides in the pitch axis PA direction and long sides in the yaw axis YA direction, the field angle in the pitch axis PA direction is narrower than the field angle in the yaw axis YA direction and narrower than the diagonal field angle.

[0087] By the optical system 15 configured in this way, light representing the imaging area is imaged on the light receiving surface 25A of the imaging element 25 , and the imaging area is captured by the imaging element 25 .

[0088] The vibrations applied to the surveillance camera 10 include, for example, the vibrations caused by the passage of cars, the vibrations caused by wind, and the vibrations caused by road construction outdoors, and for example, the vibrations caused by the operation of air conditioners and the vibrations caused by the coming and going of people indoors. Therefore, the surveillance camera 10 generates jitters due to the vibrations applied to the surveillance camera 10 (hereinafter, also simply referred to as "vibrations").

[0089] In addition, in the present embodiment, "jitter" refers to a phenomenon in which the image of the object subject on the light receiving surface 25A of the imaging element 25 in the monitoring camera 10 changes due to a change in the positional relationship between the optical axis OA and the light receiving surface 25A. In other words, "jitter" can also be said to be a phenomenon in which the optical axis OA is tilted due to the vibration applied to the monitoring camera 10, thereby changing the optical image obtained by imaging on the light receiving surface 25A. The change of the optical axis OA refers to, for example, the tilt of the optical axis OA relative to the reference axis (for example, the optical axis OA before the jitter occurs). Hereinafter, jitter caused by vibration is also referred to as "jitter".

[0090] The jitter is included in the captured image as a noise component, which affects the image quality of the captured image. Therefore, in order to remove the noise component included in the captured image due to the jitter, the monitoring camera 10 includes a lens-side jitter correction mechanism 29, an imaging element-side jitter correction mechanism 45, and an electronic jitter correction unit 33 for correcting the jitter.

[0091] The lens-side shake correction mechanism 29 and the imaging element-side shake correction mechanism 45 are mechanical shake correction mechanisms. The mechanical shake correction mechanism is a mechanism that corrects shake by applying power generated by a driving source such as a motor (for example, a voice coil motor) to a shake correction element (for example, the anti-vibration lens 15B1 and / or the imaging element 25) to move the shake correction element in a direction perpendicular to the optical axis of the imaging optical system.

[0092] Specifically, the lens-side shake correction mechanism 29 is a mechanism that corrects shake by applying power generated by a driving source such as a motor (e.g., a voice coil motor) to the anti-vibration lens 15B1, thereby moving the anti-vibration lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system. The imaging element-side shake correction mechanism 45 is a mechanism that corrects shake by applying power generated by a driving source such as a motor (e.g., a voice coil motor) to the imaging element 25, thereby moving the imaging element 25 in a direction perpendicular to the optical axis of the imaging optical system. The electronic shake correction unit 33 corrects shake by performing image processing on the captured image according to the amount of shake. That is, the shake correction unit (shake correction component) corrects shake mechanically or electronically through a hardware structure and / or a software structure. Here, mechanical shake correction refers to the correction of shake achieved by mechanically moving shake correction elements such as the anti-vibration lens 15B1 and / or the imaging element 25 using power generated by a driving source such as a motor (e.g., a voice coil motor), and electronic shake correction refers to the correction of shake achieved, for example, by performing image processing using a processor.

[0093] like Figure 4 As shown, the lens-side image stabilization mechanism 29 includes, as an example, an anti-vibration lens 15B1 , a lens actuator 17 , an OIS driver 23 , and a position sensor 39 .

[0094] As a method for correcting the vibration by the lens-side vibration correction mechanism 29, various known methods can be adopted. In the present embodiment, as a method for correcting the vibration, a method for correcting the vibration is adopted by moving the vibration-proof lens 15B1 according to the vibration amount detected by the vibration amount detection sensor 40 (described later). Specifically, the vibration is corrected by moving the vibration-proof lens 15B1 in the direction to cancel the vibration only by the amount to cancel the vibration.

[0095] The anti-vibration lens 15B1 is provided with a lens actuator 17. The lens actuator 17 is a displacement mechanism equipped with a voice coil motor, and drives the voice coil motor to move the anti-vibration lens 15B1 in a direction perpendicular to the optical axis of the anti-vibration lens 15B1. In addition, here, as the lens actuator 17, a displacement mechanism equipped with a voice coil motor is used, but the technology of the present invention is not limited to this, and other power sources such as a stepping motor or a piezoelectric element can also be used instead of the voice coil motor.

[0096] The lens actuator 17 is controlled by the OIS driver 23. By driving the lens actuator 17 under the control of the OIS driver 23, the position of the anti-vibration lens 15B1 is mechanically varied within a two-dimensional plane perpendicular to the optical axis OA.

[0097] The position sensor 39 detects the current position of the anti-vibration lens 15B1, and outputs a position signal indicating the detected current position. Here, as an example of the position sensor 39, a device including a Hall element is used. Here, the current position of the anti-vibration lens 15B1 refers to the current position within the two-dimensional plane of the anti-vibration lens. The two-dimensional plane of the anti-vibration lens refers to a two-dimensional plane perpendicular to the optical axis of the anti-vibration lens 15B1. In addition, in the present embodiment, as an example of the position sensor 39, a device including a Hall element is used, but the technology of the present invention is not limited to this, and a magnetic sensor or an optical sensor may also be used instead of the Hall element.

[0098] The lens-side shake correction mechanism 29 corrects the shake by moving the anti-vibration lens 15B1 in at least one of the pitch axis PA direction and the yaw axis YA direction within the actual shooting range. That is, the lens-side shake correction mechanism 29 corrects the shake by moving the anti-vibration lens 15B1 within the two-dimensional plane of the anti-vibration lens by an amount of movement corresponding to the amount of shake.

[0099] The imaging element side shake correction mechanism 45 includes the imaging element 25 , a BIS (Body Image Stabilizer) driver 22 , an imaging element actuator 27 , and a position sensor 47 .

[0100] As with the method of correcting the shake by the lens-side shake correction mechanism 29, various well-known methods can be used for the method of correcting the shake by the imaging element-side shake correction mechanism 45. In the present embodiment, as the method of correcting the shake, a method of correcting the shake by moving the imaging element 25 according to the shake amount detected by the shake amount detection sensor 40 is adopted. Specifically, the shake is corrected by moving the imaging element 25 in the direction to eliminate the shake only by the amount to eliminate the shake.

[0101] An imaging element actuator 27 is mounted on the imaging element 25. The imaging element actuator 27 is a displacement mechanism equipped with a voice coil motor, and drives the voice coil motor to move the imaging element 25 in a vertical direction relative to the optical axis of the anti-vibration lens 15B1. In addition, here, as the imaging element actuator 27, a displacement mechanism equipped with a voice coil motor is used, but the technology of the present invention is not limited to this, and other power sources such as a stepping motor or a piezoelectric element can also be used instead of the voice coil motor.

[0102] The imaging element actuator 27 is controlled by the BIS driver 22. By driving the imaging element actuator 27 under the control of the BIS driver 22, the position of the imaging element 25 is mechanically changed in a direction perpendicular to the optical axis OA.

[0103] The position sensor 47 detects the current position of the imaging element 25 and outputs a position signal indicating the detected current position. Here, as an example of the position sensor 47, a device including a Hall element is used. Here, the current position of the imaging element 25 refers to the current position within the two-dimensional plane of the imaging element. The two-dimensional plane of the imaging element refers to a two-dimensional plane perpendicular to the optical axis of the anti-vibration lens 15B1. In addition, in the present embodiment, as an example of the position sensor 47, a device including a Hall element is used, but the technology of the present invention is not limited to this, and a magnetic sensor or an optical sensor may also be used instead of the Hall element.

[0104] The monitoring camera 10 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake amount detection sensor 40, and a UI (User Interface) system device 43. The computer 19 includes a memory 35, a storage device 36, and a CPU (Central Processing Unit) 37.

[0105] The imaging element 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage device 36, CPU 37, shake amount detection sensor 40 and UI system device 43 are connected to the bus 38. In addition, the OIS driver 23 is also connected to the bus 38. Figure 4 In the example shown, for convenience of illustration, one bus is shown as the bus 38, but a plurality of buses may be used. The bus 38 may be a serial bus or a parallel bus such as a data bus, an address bus, and a control bus.

[0106] The memory 35 temporarily stores various information and is used as a working memory. As an example of the memory 35, a RAM (Random Access Memory) can be cited, but it is not limited to this, and other types of storage devices can also be used. Various programs for the monitoring camera 10 are stored in the storage device 36. The CPU 37 controls the entire monitoring camera 10 by reading various programs from the storage device 36 and executing the read various programs on the memory 35. As the storage device 36, for example, a flash memory, an SSD, an EEPROM, or an HDD can be cited. In addition, for example, various non-volatile memories such as a magnetoresistive memory, a ferroelectric memory, etc. can also be used instead of a flash memory or in combination with a flash memory.

[0107] The imaging element 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging element 25 captures the target subject at a predetermined frame rate under the instruction of the CPU 37. The "predetermined frame rate" mentioned here refers to, for example, tens of frames per second to hundreds of frames per second. In addition, a control device (imaging element control device) may be built into the imaging element 25 itself. In this case, the imaging element control device performs detailed control inside the imaging element 25 according to the imaging instruction output by the CPU 37. Furthermore, the imaging element 25 may also capture the target subject at a predetermined frame rate under the instruction of the DSP 31. In this case, the imaging element control device performs detailed control inside the imaging element 25 according to the imaging instruction output by the DSP 31. In addition, the DSP 31 is sometimes also referred to as an ISP (Image Signal Processor).

[0108] The light receiving surface 25A of the imaging element 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the imaging element 25, each photosensitive pixel is exposed, and photoelectric conversion is performed for each photosensitive pixel. The charge obtained by photoelectric conversion for each photosensitive pixel is an analog imaging signal representing the object. Here, as a plurality of photosensitive pixels, a plurality of photoelectric conversion elements having sensitivity to visible light (as an example, a photoelectric conversion element having a color filter) are used. In the imaging element 25, as a plurality of photoelectric conversion elements, a photoelectric conversion element having sensitivity to R (red) light (for example, a photoelectric conversion element having an R color filter corresponding to R), a photoelectric conversion element having sensitivity to G (green) light (for example, a photoelectric conversion element having a G color filter corresponding to G), and a photoelectric conversion element having sensitivity to B (blue) light (for example, a photoelectric conversion element having a B color filter corresponding to B) are used. In the monitoring camera 10, imaging based on visible light (for example, light on the short wavelength side of about 700 nanometers or less) is performed by using these photosensitive pixels. However, the present embodiment is not limited thereto, and shooting based on infrared light (for example, light with a longer wavelength than about 700 nanometers) can also be performed. In this case, as a plurality of photosensitive pixels, a plurality of photoelectric conversion elements having sensitivity to infrared light can be used. In particular, for SWIR (Short-wavelength infrared) shooting, for example, an InGaAs sensor and / or a Type-II quantum well (T2SL; Simulation of Type-IIQuantum Well) sensor can be used.

[0109] The imaging element 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog imaging signal to generate a digital image as a digital imaging signal. The imaging element 25 is connected to the DSP 31 via the bus 38 and outputs the generated digital image to the DSP 31 via the bus 38 in units of frames.

[0110] In addition, here, as an example of the imaging element 25, a CMOS image sensor is cited for explanation, but the technology of the present invention is not limited to this, and a CCD (Charge Coupled Device) image sensor can also be applied as the imaging element 25. In this case, the imaging element 25 is connected to the bus 38 via an AFE (Analog Front End) not shown in the figure built into the CCD driver, and the AFE generates a digital image by performing signal processing such as A / D conversion on the analog camera signal obtained by the imaging element 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, a CCD driver can also be provided separately.

[0111] The DSP 31 performs various digital signal processing on the digital image. The various digital signal processing refers to, for example, demosaic processing, noise removal processing, grayscale correction processing, and color correction processing. The DSP 31 outputs the digital image after the digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0112] The shake amount detection sensor 40 is a device including a gyro sensor, for example, and detects the shake amount of the monitoring camera 10. In other words, the shake amount detection sensor 40 detects the shake amount in each of a pair of axial directions. The gyro sensor detects the respective axes (reference numerals 14 and 15 ) around the pitch axis PA, the yaw axis YA, and the roll axis RA (axes parallel to the optical axis OA). Figure 1 The shake amount detection sensor 40 detects the shake amount of the monitoring camera 10 by converting the amount of rotational shake around the pitch axis PA and the amount of rotational shake around the yaw axis YA detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA.

[0113] Here, a gyro sensor is cited as an example of the shake amount detection sensor 40, but this is only an example, and the shake amount detection sensor 40 may also be an acceleration sensor. The acceleration sensor detects the shake amount in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 outputs the detected shake amount to the CPU 37.

[0114] Here, an example of detecting the amount of shake by the physical sensor of the shake amount detection sensor 40 is given, but the technology of the present invention is not limited to this. For example, a motion vector obtained by comparing the captured images before and after the time series stored in the image memory 32 can be used as the amount of shake. In addition, the amount of shake to be used finally can be derived from the amount of shake detected by the physical sensor and the motion vector obtained by image processing.

[0115] The CPU 37 acquires the amount of shake detected by the shake amount detection sensor 40, and controls the lens-side shake correction mechanism 29, the imaging element-side shake correction mechanism 45, and the electronic shake correction section 33 according to the acquired shake amount. The amount of shake detected by the shake amount detection sensor 40 is used for correction based on the shake of each of the lens-side shake correction mechanism 29 and the electronic shake correction section 33.

[0116] The electronic image stabilization section 33 is a device including an ASIC (Application Specific Integrated Circuit) and performs image processing on the captured image in the image memory 32 according to the image stabilization amount detected by the image stabilization amount detection sensor 40 to correct the image stabilization.

[0117] In addition, here, as the electronic jitter correction unit 33, a device including an ASIC is exemplified, but the technology of the present invention is not limited to this. For example, it can be a device including an FPGA (Field Programmable Gate Array) or a PLD (Programmable Logic Device). And, for example, the electronic jitter correction unit 33 can be a device including multiple devices among ASIC, FPGA and PLD. And, as the electronic jitter correction unit 33, a computer including a CPU, a storage device and a memory can also be used. The CPU can be single or multiple. And the electronic jitter correction unit 33 can be implemented by a combination of a hardware structure and a software structure.

[0118] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information with the management device 11 via a network. The network is, for example, a WAN (Wide Area Network) such as the Internet or a LAN (Local Area Network). The communication I / F 34 performs communication between the monitoring camera 10 and the management device 11.

[0119] The UI system device 43 includes a receiving device 43A and a display 43B. The receiving device 43A is, for example, a hardware key or a touch panel, and receives various instructions from the user. The CPU 37 acquires various instructions received by the receiving device 43A and operates according to the acquired instructions.

[0120] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions received by the receiving device 43A and captured images.

[0121] <Configuration of the electrical system of the slewing mechanism 16 and the management device 11>

[0122] Figure 5 1 is a diagram showing an example of the structure of the electrical system of the rotary mechanism 16 and the management device 11. Figure 5 As shown, the slewing mechanism 16 includes a yaw axis slewing mechanism 71 , a pitch axis slewing mechanism 72 , a motor 73 , a motor 74 , a driver 75 , a driver 76 , and communication I / Fs 79 and 80 , as an example.

[0123] The yaw axis rotation mechanism 71 rotates the monitoring camera 10 in the yaw direction. The motor 73 generates power by being driven under the control of the driver 75. The yaw axis rotation mechanism 71 rotates the monitoring camera 10 in the yaw direction by receiving the power generated by the motor 73. The pitch axis rotation mechanism 72 rotates the monitoring camera 10 in the pitch direction. The motor 74 generates power by being driven under the control of the driver 76. The pitch axis rotation mechanism 72 rotates the monitoring camera 10 in the pitch direction by receiving the power generated by the motor 74.

[0124] The communication I / Fs 79 and 80 are, for example, network interfaces, and control the transmission of various information with the management device 11 via a network. The network is, for example, a WAN such as the Internet or a LAN. The communication I / Fs 79 and 80 perform communication between the rotation mechanism 16 and the management device 11.

[0125] like Figure 5 As shown, as an example, the management device 11 includes a display 13a, a secondary storage device 14, a control device 60, a receiving device 62, and communication I / Fs 66, 67, and 68. The control device 60 includes a CPU 60A, a storage device 60B, and a memory 60C. The CPU 60A is an example of a processor in the present invention.

[0126] The receiving device 62, the display 13a, the secondary storage device 14, the CPU 60A, the storage device 60B, the memory 60C and the communication I / F 66 are connected to the bus 70. Figure 5 In the example shown, for the convenience of illustration, one bus is shown as the bus 70, but a plurality of buses may be used. The bus 70 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, and the like.

[0127] The memory 60C temporarily stores various information and is used as a working memory. As an example of the memory 60C, a RAM can be cited, but it is not limited to this, and other types of storage devices can also be used. Various programs for the management device 11 (hereinafter referred to as "management device programs") are stored in the storage device 60B.

[0128] The CPU 60A reads out the management device program from the storage device 60B and executes the read management device program on the memory 60C to control the entire management device 11. The management device program includes the information processing program in the present invention.

[0129] The communication I / F 66 is, for example, a network interface. The communication I / F 66 is communicably connected to the communication I / F 34 of the monitoring camera 10 via the network, and performs transmission control of various information with the monitoring camera 10. The communication I / Fs 67 and 68 are, for example, network interfaces. The communication I / F 67 is communicably connected to the communication I / F 79 of the rotation mechanism 16 via the network, and performs transmission control of various information with the yaw axis rotation mechanism 71. The communication I / F 68 is communicably connected to the communication I / F 80 of the rotation mechanism 16 via the network, and performs transmission control of various information with the pitch axis rotation mechanism 72.

[0130] The CPU 60A receives captured images, captured image information, and the like from the monitoring camera 10 via the communication I / F 66 and the communication I / F 34 .

[0131] The CPU 60A controls the rotation of the yaw axis rotation mechanism 71 by controlling the driver 75 and the motor 73 of the rotation mechanism 16 via the communication I / F 67 and the communication I / F 79. The CPU 60A controls the rotation of the pitch axis rotation mechanism 72 by controlling the driver 76 and the motor 74 of the rotation mechanism 16 via the communication I / F 68 and the communication I / F 80.

[0132] The receiving device 62 is, for example, a keyboard 13b, a mouse 13c, and a touch panel of the display 13a, and receives various instructions from the user. The CPU 60A obtains various instructions received by the receiving device 62 and operates according to the instructions obtained. For example, when the receiving device 62 receives the processing content of the monitoring camera 10 and / or the rotating mechanism 16, the CPU 60A operates the monitoring camera 10 and / or the rotating mechanism 16 according to the instruction content received by the receiving device 62.

[0133] The display 13a displays various information under the control of the CPU 60A. Examples of the various information displayed on the display 13a include the contents of various instructions received by the receiving device 62 and the captured images or captured information received by the communication I / F 66. The CPU 60A displays the contents of various instructions received by the receiving device 62 and the captured images or captured information received by the communication I / F 66 on the display 13a.

[0134] The secondary storage device 14 is, for example, a nonvolatile memory, and stores various information under the control of the CPU 60A. Examples of the various information stored in the secondary storage device 14 include a captured image or captured image information received by the communication I / F 66. The CPU 60A stores the captured image or captured image information received by the communication I / F 66 in the secondary storage device 14.

[0135] The communication I / F 69 is, for example, a network interface. In the area of ​​the monitoring object (camera object) based on the camera system 1 (hereinafter referred to as the "monitoring object area"), there are a plurality of staff members, each of whom holds a terminal device such as a smartphone (for example, a reference Figure 7 The communication I / F 69 communicates directly or indirectly with the terminal devices held by each worker in the monitoring target area via a network. The network is, for example, a WAN or a LAN. The monitoring target area is, for example, a location where multiple workers perform dangerous operations, such as a construction site.

[0136] <Image displayed by the management device 11>

[0137] Figure 6 1 is a diagram showing an example of an image displayed by the management device 11. The management device 11 can display, for example, a wide-area image 90 and a detailed image 91 to a user of the management device 11 (for example, a supervisor of the monitoring target area) via the display 13a. In this example, the viewing angle of the monitoring camera 10 is a viewing angle capable of capturing only a partial area of ​​the monitoring target area.

[0138] The wide-area image 90 is a pseudo wide-angle image showing the entirety of the monitoring target area E1, which is generated by controlling the monitoring camera 10 and the slewing mechanism 16 by the management device 11 to make the monitoring camera 10 capture each area of ​​the monitoring target area E1 a plurality of times and synthesizing (connecting) each imaging information obtained by the capturing. This series of imaging control and generation of the wide-area image 90 is performed regularly, for example, at a predetermined time (for example, 7 a.m.) every day.

[0139] The detailed image 91 is an image that is generated based on the latest imaging information acquired by the monitoring camera 10 and that shows a partial area e1 of the monitoring target area E1 in real time.

[0140] The wide-area image 90 and the detailed image 91 may be displayed in parallel at the same time, for example, or may be displayed by switching between them in accordance with an operation by a user of the management device 11 or the like.

[0141] The wide area image 90 includes an area designation cursor 90a. The user of the management device 11 can change the position or size of the area designation cursor 90a by operating the receiving device 62.

[0142] For example, the memory 60C or the secondary storage device 14 of the management device 11 stores corresponding information that uniquely establishes a corresponding association between the coordinates of the wide-area image 90, the longitude and latitude (longitude and latitude) of the position corresponding to the coordinates in the monitoring object area E1, and the control parameters of the rotating mechanism 16 (the control value of the pan or tilt of the monitoring camera 10) for shooting with the position corresponding to the coordinates in the monitoring object area E1 as the center.

[0143] For example, when generating the wide area image 90, the management device 11 derives the correspondence between the coordinates of the wide area image 90 and the control parameters of the slewing mechanism 16. Furthermore, for example, regarding a plurality of positions included in the monitoring target area E1 and with known longitudes and latitudes, the management device 11 adjusts the control parameters of the slewing mechanism 16 so that the monitoring camera 10 can capture the position as the center, and derives the correspondence between the control parameters of the slewing mechanism 16 and the longitudes and latitudes by establishing a correspondence association between the adjusted control parameters and the longitudes and latitudes of the positions (known). Thus, it is possible to generate correspondence information in which the coordinates of the wide area image 90, the control parameters of the slewing mechanism 16, and the longitudes and latitudes are established in a correspondence association.

[0144] When the area designation cursor 90a is set by the operation from the user, the management device 11 acquires the control parameters of the rotation mechanism 16 corresponding to the coordinates of the center of the area designated by the area designation cursor 90a in the wide area image 90 from the corresponding information, and sets the acquired control parameters in the rotation mechanism 16. Thus, a detailed image 91 showing the area designated by the user of the management device 11 using the area designation cursor 90a in the monitoring target area E1 is displayed.

[0145] That is, the user of the management device 11 can observe the entire monitoring target area E1 through the wide area image 90. Furthermore, when the user of the management device 11 wants to observe the local area e1 of the monitoring target area E1 in detail, the user can observe the detailed image 91 showing the local area e1 in detail by setting the area designation cursor 90a on a part of the local area e1 in the wide area image 90. Figure 6 In the illustrated example, the existing vehicle V1 and workers W1 to W3 are shown in the detailed image 91 .

[0146] In this way, by using the real-time video information obtained by the monitoring camera 10 and the pseudo wide-angle image generated by synthesizing the video information obtained by making the monitoring camera 10 photograph each area of ​​the monitoring object area E1, it is possible to display both the wide-area image 90 and the detailed image 91 through a set of monitoring cameras 10 and a rotating mechanism 16.

[0147] <Hardware configuration of terminal devices held by staff in monitoring target area E1>

[0148] Figure 7 1 is a diagram showing an example of the hardware configuration of a terminal device held by a worker in the monitoring target area E1. Each worker in the monitoring target area E1 (for example, workers W1 to W3) holds, for example, Figure 7 The terminal device 100 shown in FIG. 1 includes a processor 101 , a memory 102 , a communication interface 103 , a GNSS (Global Navigation Satellite System) unit 104 , and a user interface 105 . The processor 101 , the memory 102 , the communication interface 103 , the GNSS unit 104 , and the user interface 105 are connected via a bus 109 , for example.

[0149] The processor 101 is a circuit for performing signal processing, for example, a CPU for controlling the entire terminal device 100. In addition, the processor 101 may also be implemented by other digital circuits such as FPGA or DSP. Furthermore, the processor 101 may be implemented by combining multiple digital circuits.

[0150] The memory 102 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM. The main memory is used as a working area for the processor 101. The auxiliary memory is, for example, a non-volatile memory such as a magnetic disk, an optical disk, a flash memory, etc. Various programs for operating the terminal device 100 are stored in the auxiliary memory. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 101.

[0151] Furthermore, the auxiliary memory may include a portable memory that can be removed from the terminal device 100. The portable memory includes a USB (Universal Serial Bus) flash drive, a memory card such as an SD (Secure Digital) memory card, an external hard disk drive, and the like.

[0152] The communication interface 103 is a communication interface for wireless communication with the outside of the terminal device 100. For example, the communication interface 103 is connected to the Internet via a mobile communication network to indirectly communicate with the management device 11. The communication interface 103 is controlled by the processor 101.

[0153] The GNSS unit 104 is, for example, a satellite positioning system such as GPS (Global Positioning System), and acquires location information (latitude and longitude) of the terminal device 100. The GNSS unit 104 is controlled by the processor 101.

[0154] The user interface 105 includes, for example, an input device for receiving an operation input from a user or an output device for outputting information to the user. The input device can be implemented, for example, by a key (e.g., a keyboard) or a remote controller. The output device can be implemented, for example, by a display or a speaker. In addition, the input device and the output device can be implemented by a touch panel. The user interface 105 is controlled by the processor 101.

[0155] <Processing by the management device 11>

[0156] Figure 8 1 is a flowchart showing an example of processing performed by the management device 11. As described above, the management device 11 stores the corresponding information that associates the coordinates of the wide-area image 90, the control parameters of the rotation mechanism 16, and the longitude and latitude. The management device 11 sends an operation, such as executing a command information from a user, while displaying the detailed image 91. Figure 8 Processing shown.

[0157] First, the management device 11 obtains the longitude and latitude corresponding to the current detailed image 91 (step S11). The longitude and latitude corresponding to the detailed image 91 is the position information of the camera object (local area e1) correspondingly associated with the camera image (detailed image 91) taken by the camera system (monitoring camera 10 and rotating mechanism 16), and is an example of the first position information of the present invention. The longitude and latitude corresponding to the detailed image 91 is, for example, the longitude and latitude of the location shown in the center of the detailed image 91. For example, the management device 11 obtains the longitude and latitude corresponding to the current control parameters of the rotating mechanism 16 as the longitude and latitude corresponding to the current detailed image 91 based on the above-mentioned corresponding information. The longitude and latitude obtained by step S11 is set to a, and the latitude and latitude obtained by step S11 is set to b.

[0158] Next, the management device 11 obtains the position information of the terminal device 100 of each staff member in the monitoring target area E1 (step S12). This position information is the position information of the terminal device 100 obtained by the terminal device 100 in the imaging area (monitoring target area E1) of the imaging system (monitoring camera 10 and slewing mechanism 16), and is an example of the second position information of the present invention.

[0159] For example, the terminal device 100 of each staff member in the monitoring target area E1 repeatedly sends the location information of the terminal device 100 obtained by the GNSS unit 104 of the terminal device 100 to the management device 11. In response, the management device 11 obtains the latest location information among the received location information for the terminal device 100 of each staff member in the monitoring target area E1 in step S12.

[0160] Alternatively, in step S12 , the management device 11 may send a request signal for requesting the terminal device 100 of each worker in the monitoring target area E1 to send the position information, and obtain the position information sent from the terminal device 100 in response to the request signal.

[0161] Next, the management device 11 determines the zoom position (focal length or zoom ratio) of the monitoring camera 10 according to the control parameters set in the monitoring camera 10, and sets Δa and Δb corresponding to the determined zoom position (step S13). For example, at a wide-angle zoom position, the management device 11 sets Δa and Δb to be larger.

[0162] Next, the management device 11 extracts the terminal devices 100 whose longitude and latitude are within the range of (a±Δa, b±Δb) indicated by the location information obtained in step S12 from the terminal devices 100 of each staff member in the monitoring target area E1 based on the longitude and latitude (a, b) obtained in step S11 and the currently set Δa and Δb (step S14). The range of (a±Δa, b±Δb) refers to a rectangular range with longitude ranging from a-Δa to a+Δa and latitude ranging from b-Δb to b+Δb. In addition, the range that serves as the judgment basis of step S14 is not limited to the rectangular range (a±Δa, b±Δb), and may be, for example, a circular range with a radius Δ centered on the longitude and latitude (a, b) or other shape ranges.

[0163] Next, the management device 11 determines whether the number of terminal devices 100 extracted in step S14 (the number of extracted devices) is within a predetermined appropriate range (step S15). The appropriate range is, for example, a pre-set range. If the number of extracted devices is not within the appropriate range (step S15: No), the management device 11 changes Δa and Δb (step S16) and returns to step S14.

[0164] In step S16, the management device 11 notifies the user of the number of extracted units through the display 13a, etc., receives an increase or decrease instruction of Δa and Δb from the user, and changes Δa and Δb. As an example, when the appropriate range is within 1 unit or more and the number of extracted units is 0, the management device 11 notifies the user of the intention to extract the number of units to 0, and receives an instruction of how much Δa and Δb should be increased by the receiving device 62. In addition, in step S16, the management device 11 may notify the user of not only the number of extracted units, but also the name, affiliation, email address, etc. of the holder of the terminal device 100 extracted in step S14. In addition, the management device 11 may perform the following processing: in step S16, without receiving an instruction from the user, when the number of extracted units is less than the appropriate range, Δa and Δb are increased, and when the number of extracted units is more than the appropriate range, Δa and Δb are decreased.

[0165] In step S15, if the number of extracted devices is within the appropriate range (step S15: Yes), the management device 11 receives the setting of the instruction content for the holder of the terminal device 100 extracted in step S14 from the user of the management device 11 (step S17). The instruction content is text information, image, sound, or a combination thereof. As an example, the instruction content is a combination of the detailed image 91 (real-time image) and the text information input by the user in step S15 (for example, a message such as "The nearby vehicle will move, please pay attention").

[0166] Furthermore, in step S17, the management device 11 may notify the user of the default instruction content. Furthermore, in step S17, the management device 11 may display the name, affiliation, email address, and other information of the holder of the terminal device 100 extracted in step S14 and input controls such as check boxes (by default, "on" for all persons) that allow these holders to specify whether to send in an on / off manner. The terminal device 100 for which the check box is set to "off" is excluded from the destination of the instruction information in step S18 described later.

[0167] Next, the management device 11 sends the instruction information of the instruction content received in step S17 to the terminal device 100 extracted in step S14 (step S18), and ends a series of processing. The instruction information is an example of the first data of the present invention. As an example, the management device 11 sets the mail address of the terminal device 100 extracted in step S14 as the destination, generates an email with the instruction content received in step S17 set as the subject (Subject) or the body, and sends the generated email via the communication I / F 69.

[0168] The terminal device 100 that receives the indication information sent in step S18 plays (eg displays) the indication content of the received indication information.

[0169] About the pre-set Figure 8 The case of the appropriate range in step S15 is described above, but the appropriate range may be set based on the detailed image 91, for example. Figure 8 In the process of the detailed image 91, the number of staff members shown in the detailed image 91 is detected by image recognition processing based on the detailed image 91, and the appropriate range in step S15 is set according to the detected number of people. As an example, if the detected number of people is set to N, the management device 11 sets the range of N-ΔNa to N+Δb as the appropriate range. In the case where N-ΔNa is less than 0, the management device 11 sets the range of 0 to N+Δb as the appropriate range. The values ​​of ΔNa and ΔNb are set in advance.

[0170] For example, a user of the management device 11 can perform an instruction information sending operation by setting the area designation cursor 90a in the wide-area image 90 to a location where danger occurs (e.g., the location where the vehicle V1 moves), thereby notifying staff members located near the location (e.g., staff members W1 to W3) of instruction contents urging them to pay attention or retreat.

[0171] In this way, the management device 11 of implementation mode 1 obtains the first position information of the camera object (local area e1) corresponding to the camera image (detailed image 91) based on the camera system (for example, the monitoring camera 10 and the rotating mechanism 16) and the second position information of the terminal device obtained by the terminal device in the camera area (monitoring object area E1) of the camera system (for example, the terminal device 100 possessed by the mobile body such as the staff members W1 to W3 included in the camera image), and generates the indication information (first data) with the sending destination set based on the obtained first position information and second position information.

[0172] Specifically, the management device 11 obtains the longitude and latitude (second position information) of multiple terminal devices 100 existing in the monitoring object area E1, extracts the terminal devices 100 whose acquired longitude and latitude are included in the range based on the longitude and latitude (first position information) corresponding to the detailed image 91, and generates and sends indication information (first data) with the extracted terminal devices 100 as the sending destination.

[0173] Thus, it is possible to utilize the first position information of the camera object (local area e1) corresponding to the camera image (detailed image 91) and the second position information obtained through the terminal device 100 to efficiently send instruction information to people (for example, staff members W1 to W3) located in a specific area (for example, local area e1) in the monitored object area E1.

[0174] Furthermore, the management device 11 can set the range of the terminal devices 100 extracted as the destination according to the zoom information of the camera system (e.g., the monitoring camera 10) in addition to the longitude and latitude corresponding to the detailed image 91 (the first position information). For example, the wider the zoom position of the monitoring camera 10 is, the wider the range of the terminal devices 100 extracted as the destination is set by the management device 11 (e.g., the larger the above-mentioned Δa and Δb are set). Thus, the user of the management device 11 can set the terminal devices 100 located in the range corresponding to the width of the range observed by the detailed image 91 as the destination, and thus can easily send instruction information to the staff members desired by the user of the management device 11 (e.g., the staff members W1 to W3).

[0175] Furthermore, the management device 11 can set the range of the terminal devices 100 extracted as the transmission destination according to the number of mobile bodies (e.g., workers W1 to W3) having the terminal devices 100 detected from the detailed image 91 (camera image) in addition to the longitude and latitude corresponding to the detailed image 91 (first position information). For example, the management device 11 performs the following processing: the above-mentioned appropriate range is set according to the number of workers detected from the detailed image 91, and the range of the terminal devices 100 extracted as the transmission destination (e.g., the above-mentioned Δa, Δb) is changed so that the extraction range of the terminal devices 100 becomes the appropriate range. Thus, in the case where there is a large difference between the number of extracted terminal devices 100 and the number of workers shown in the detailed image 91 observed by the user of the management device 11 when transmitting the instruction information, the difference can be reduced.

[0176] <Modification 1 of the imaging system 1>

[0177] Fig. 9 FIG. 1 is a diagram showing a modified example 1 of the imaging system 1. Fig. 9 As shown, the camera system 1 has Figure 1 In addition to the structure shown in the figure, a monitoring camera 10a may be included. The monitoring camera 10a is a camera with a wider field of view than the monitoring camera 10, and is set in a manner that can capture the entire monitoring target area E1. The structure of the monitoring camera 10a is, for example, similar to Figure 4 The structure of the surveillance camera 10 shown is the same, but the difference is that the optical system 15 has wider-angle optical characteristics.

[0178] Fig.10 Yes means Fig. 9 FIG. 1 is a diagram showing an example of a configuration of an electrical system of the management device 11. Fig. 9 In the imaging system 1 shown, the monitoring camera 10a includes a communication I / F 34a which is the same as the communication I / F 34 of the monitoring camera 10. The communication I / F 34a performs communication between the monitoring camera 10a and the management device 11.

[0179] The communication I / F 66 of the management device 11 is communicatively connected to the communication I / F 34 a of the monitoring camera 10 in addition to the communication I / F 34 of the monitoring camera 10 , and controls transmission of various information between the monitoring cameras 10 and 10 a .

[0180] Fig.11 It means in Fig. 9 , Fig.10 FIG. 1 is a diagram showing an example of an image displayed by the management device 11 in the structure shown. Fig. 9 , Fig.10In the illustrated configuration, the management device 11 displays an image based on the imaging information obtained from the monitoring camera 10a as the wide-area image 90 representing the monitoring target area E1. That is, the wide-area image 90 in this case is not a pseudo wide-angle image generated by synthesizing the imaging information obtained by causing the monitoring camera 10 to capture the respective areas of the monitoring target area E1, but is a wide-angle image based on single imaging information obtained by the wide-angle monitoring camera 10a.

[0181] In this case, the wide area image 90 may be a non-real-time image obtained by periodic imaging as described above, or may be a real-time image obtained from the latest imaging information obtained by imaging by the monitoring camera 10 a .

[0182] In this case, the management device 11 stores the correspondence information that uniquely associates the coordinates of the wide-area image 90, the control parameters of the slewing mechanism 16, and the longitude and latitude. In this case, among the coordinates of the wide-area image 90 corresponding to the control parameters and the longitude and latitude of the slewing mechanism 16, for a plurality of positions included in the monitoring target area E1 and having known longitude and latitude, the user of the management device 11 can, for example, derive the corresponding coordinates by specifying the corresponding coordinates in the wide-area image 90. The management device 11 uses the correspondence information to execute Figure 8 Processing shown.

[0183] <Modification 2 of the imaging system 1>

[0184] Fig.12 FIG. 2 is a diagram showing a second modification of the imaging system 1. Fig.12 As shown, the camera system 1 can be set to Fig. 9 In the illustrated structure, the structures of the monitoring camera 10 and the rotating mechanism 16 are omitted.

[0185] Fig.13 Yes means Fig.12 FIG. 1 is a diagram showing an example of a configuration of an electrical system of the management device 11. Fig.12 In the camera system 1 shown in FIG. 1 , the management device 11 becomes Fig.10 The communication I / F 67 and the communication I / F 68 are omitted in the illustrated configuration. The communication I / F 66 of the management device 11 is communicatively connected to the communication I / F 34a of the monitoring camera 10a, and performs transmission control of various information with the monitoring camera 10a.

[0186] exist Fig.12 , Fig.13 In the structure shown, the image displayed by the management device 11 is, for example, Fig.11The wide-area image 90 and the detailed image 91 shown are the same. The management device 11 displays a digital zoom image of the area designated by the area designation cursor 90a in the wide-area image 90 cut out and enlarged as the detailed image 91 showing the local area e1. In this case, the wide-area image 90 is, for example, a real-time image obtained from the latest imaging information obtained by the monitoring camera 10a.

[0187] In this case, the management device 11 stores the correspondence information that uniquely associates the coordinates of the wide-area image 90 with the longitude and latitude. That is, the correspondence information in this case does not require the control parameters of the slewing mechanism 16. In this case, among the coordinates of the wide-area image 90 corresponding to the longitude and latitude, the corresponding coordinates in the wide-area image 90 are derived by, for example, specifying the corresponding coordinates in the wide-area image 90 for a plurality of positions whose longitude and latitude are known and are included in the monitoring target area E1 by the user of the management device 11.

[0188] The management device 11 uses this correspondence information to execute Figure 8 In this case, in step S11, the management device 11 obtains the longitude and latitude corresponding to the detailed image 91 based on the coordinates and corresponding information of the digitally zoomed area in the wide-area image 90. And in step S13, the management device 11 sets Δa and Δb corresponding to the digital zoom amount of the detailed image 91.

[0189] <Confirmation and handling of those who are unwell or injured>

[0190] Fig.14 1 is a diagram showing an example of a process for identifying a sick or injured person, etc. The imaging system 1 can also be applied to a system that can identify a worker who has collapsed or is unable to move due to heat stroke or injury, etc. in a monitoring target area E1 (for example, a construction site).

[0191] For example, the terminal device 100 detects abnormalities of the staff holding the terminal device 100. For example, the detection of abnormalities of the staff is performed based on at least one of the following situations: the state where the longitude and latitude acquired by the GNSS unit 104 of the terminal device 100 do not change for a certain period of time, or the stationary state of the terminal device 100 detected by the acceleration sensor of the terminal device 100 continues for a certain period of time, or the biometric information of the staff measured by a wearable device that can communicate with the terminal device 100 and is worn by the staff is an abnormal value.

[0192] In this case, the terminal device 100 sends the abnormality detection information indicating that the abnormality of the staff member is detected to the management device 11 together with the latitude and longitude information obtained by the GNSS unit 104 of the terminal device 100. If the management device 11 receives the abnormality detection information and the latitude and longitude information, it displays the detailed image 91 of the area corresponding to the latitude and longitude in the wide-area image 90. Thus, when the abnormality of the staff member is detected by the terminal device 100, the detailed image 91 indicating the position of the staff member can be automatically displayed. Therefore, the user of the management device 11 can quickly confirm the status of the staff member whose abnormality is detected.

[0193] exist Fig.14 In the state shown, the worker W2 falls in the local area e1, and the terminal device 100 held by the worker W2 sends the abnormality detection information and the longitude and latitude information to the management device 11. In this case, the management device 11 displays a detailed image 91 of the local area e1 based on the longitude and latitude information received together with the abnormality detection information. Thus, the user of the management device 11 can quickly confirm the state of the worker W2 falling.

[0194] In addition, the user of the management device 11 can send instruction information to inquire about the status of the staff member W2, or send instruction information to instruct other staff members W1 and W2 around the staff member W2 to provide rescue, etc. by performing the above-mentioned instruction information sending operation while displaying the detailed image 91 of the staff member W2.

[0195] <Modification example of instruction information>

[0196] Although an e-mail has been described as the instruction information as an example of the first data, the instruction information is not limited to e-mail, and may be various message information such as SMS (Short Message Service) or a message based on Feishuxun software.

[0197] (Implementation Method 2)

[0198] Regarding the second embodiment, the differences from the first embodiment will be described. The imaging system 1 of the second embodiment is, for example, Figure 1 to Figure 5 The camera system 1 shown has the same structure as Figure 6 Similarly to the illustrated example, the wide-area image 90 and the detailed image 91 can be displayed on the display 13 a .

[0199] <Processing by the Management Device 11 of Embodiment 2>

[0200] Fig.151 is a flowchart showing an example of processing performed by the management device 11 of Embodiment 2. The management device 11 of Embodiment 2 stores the wide-area image 90, which is a pseudo wide-angle image, in a corresponding relationship with the time when the wide-area image 90 was generated, in the secondary storage device 14. For example, the management device 11 generates the wide-area image 90 at a predetermined time every day, and stores the generated wide-area image 90 in the secondary storage device 14 in a corresponding relationship with the date and time when the wide-area image 90 was generated.

[0201] The management device 11 of the second embodiment performs, for example, Fig.15 Processing shown.

[0202] First, the management device 11 acquires the wide-area image 90 (step S21 ). The wide-area image 90 is, for example, a pseudo wide-angle image generated by imaging each area of ​​the monitoring target area E1 with the monitoring camera 10 and using imaging information obtained by imaging.

[0203] Next, the management device 11 obtains the history of the location information of the terminal device 100 of the staff in the monitoring target area E1 (step S22). The history of the location information of the terminal device 100 is, for example, the longitude and latitude of the terminal device 100 at each of multiple time points. For example, the terminal device 100 of the staff in the monitoring target area E1 repeatedly sends the location information of the device itself obtained by the GNSS unit 104 of the device itself to the management device 11. In contrast, the management device 11 accumulates the location information received from the terminal device 100 of the staff in the monitoring target area E1, thereby obtaining the history of the location information of the terminal device 100 in step S12.

[0204] Alternatively, in step S22 , the management device 11 may send a request signal for requesting the terminal device 100 of the staff in the monitoring target area E1 to send the history of the location information, and obtain the history of the location information sent from the terminal device 100 in response to the request signal.

[0205] Next, the management device 11 superimposes and displays a movement history image showing the history of the position information of the terminal device 100 obtained in step S22 on the wide area image 90 obtained in step S21 (step S23), and ends a series of processing. The movement history image is an example of information related to the movement history of the mobile object in the present invention.

[0206] In addition, become Fig.15 The workers to be processed may be all workers who are present in the monitoring target area E1 and have the terminal device 100, or may be a part of these workers (for example, worker W1). Fig.15The staff member who is the processing target can be specified by the user of the management device 11.

[0207] <Display of wide area image 90 with movement history image superimposed>

[0208] Fig.16 FIG. 1 is a diagram showing an example of display of a wide area image 90 on which a movement history image is superimposed. Fig.16 In the example, for worker W1 Fig.15 The processing situation shown is explained below. Fig.15 In step S23 shown in FIG. 1 , the management device 11 displays the wide-area image 90 (for example, as shown in FIG. 1 ) superimposed with the movement history image 161 on the display 13a. Fig.16 The movement history image 161 is an image showing the history of position information acquired from the terminal device 100 held by the worker W1.

[0209] For example, the management device 11 determines the coordinates corresponding to the longitude and latitude at each time indicated by the history of the position information in the wide area image 90 based on the above-mentioned correspondence information that establishes a correspondence between the coordinates of the wide area image 90 and the longitude and latitude. Then, the management device 11 displays the movement history image 161 on the wide area image 90 in an overlapping manner by drawing a line so as to connect the determined coordinates in the time sequence of the history of the position information.

[0210] The wide area image 90 overlapping the movement history image 161 can be the latest wide area image 90 (latest image) among the wide area images 90 saved by the management device 11, or it can be the wide area image 90 (time-corresponding image) among the wide area images 90 saved by the management device 11 that corresponds to the moment (for example, the first or last moment) of the history represented by the movement history image 161.

[0211] Furthermore, the management device 11 can display the detailed image 91 superimposed with the movement history image 161 in the same manner as the wide-area image 90, according to an operation by the user or the like. The detailed image 91 in this case can be a digital zoom image obtained by cutting and enlarging the area designated by the area designation cursor 90a in the wide-area image 90 superimposed with the movement history image 161 (pseudo wide-angle image mode), or can be an image obtained by superimposing the movement history image 161 on a real-time image of a camera image obtained by controlling the rotation mechanism 16 and the monitoring camera 10 to capture the area designated by the area designation cursor 90a (real-time image mode).

[0212] Furthermore, the detailed image 91 may be a pseudo wide-angle image generated by synthesizing each piece of imaging information obtained by shooting and capturing each part of the area wider than the designated area by the monitoring camera 10, including the designated area based on the area designation cursor 90a. Thus, it is possible to display the detailed image 91 representing a range wider than the field of view of the monitoring camera 10 (but narrower than the wide-area image 90). Therefore, it is possible to suppress the situation where the field of view of the monitoring camera 10 is too narrow and the movement history image 161 cannot be fully displayed on the detailed image 91. Furthermore, the management device 11 may set "an area that includes the designated area and is wider than the designated area" so that the positions of each longitude and latitude in the history of the position information are included.

[0213] Furthermore, the management device 11 may receive the setting of the history period (e.g., the start time and the end time) of the location information of the display object from the user. For example, the management device 11 displays a time bar on the display 13a, and receives the setting of the pointer position of the time bar through the receiving device 62, thereby receiving the setting of the history period of the location information of the display object. In this case, Fig.15 In step S22 shown, the management device 11 acquires the history of position information within a period specified by the user.

[0214] Furthermore, the staff member W1 is subjected to Fig.15 However, the management device 11 may receive the information of the staff who are present in the monitoring target area E1 and have the terminal device 100. Fig.15 The staff member who is the object of the processing can be selected. The staff member can be selected by specifying the staff member individually or by specifying the group (such as a company or department) to which the staff member belongs.

[0215] In this way, the management device 11 of embodiment 2 obtains the first position information of the camera object (monitoring object area E1) corresponding to the camera image (wide-area image 90) based on the camera system (for example, the monitoring camera 10 and the rotating mechanism 16) and the second position information of the terminal device obtained by the terminal device in the camera area (monitoring object area E1) of the camera system (for example, the terminal device 100 possessed by the mobile body such as the staff W1~W3 included in the camera image), and generates an image (first data) based on the acquired first position information and second position information, in which the movement history image 161 representing the movement history of the staff is superimposed on at least one of the wide-area image 90 and the detailed image 91.

[0216] Thus, an image that can easily confirm the movement history of the worker in the monitoring target area E1 can be displayed using the first position information of the camera object (monitoring target area E1) associated with the camera image (wide-area image 90) and the second position information obtained by the terminal device 100. Therefore, for example, it is easy to determine whether the worker has entered a dangerous position, determine whether the worker has acted in accordance with the prescribed order, or analyze the cause when an accident has occurred.

[0217] <Modification 1 of the imaging system 1 according to the second embodiment>

[0218] The imaging system 1 of the second embodiment may be, for example, Fig. 9 , Fig.10 The structure shown.

[0219] Fig.17 This means that the camera system 1 of the second embodiment is set to Fig. 9 , Fig.10 FIG. 1 is a diagram showing an example of an image displayed by the management device 11 in the case of the structure shown. Fig. 9 , Fig.10 In the structure shown, Fig.11 Similarly, the management device 11 of the second embodiment displays an image based on the imaging information obtained from the monitoring camera 10a as the wide-area image 90 indicating the monitoring target area E1.

[0220] In this case, the management device 11 also stores the above-mentioned correspondence information in which the coordinates of the above-mentioned wide area image 90 are associated with the longitude and latitude. The management device 11 uses this correspondence information to execute Fig.15 Processing shown.

[0221] <Modification 2 of the imaging system 1 according to the second embodiment>

[0222] The imaging system 1 of the second embodiment may be, for example, Fig.12 , Fig.13 In this case, the image displayed by the management device 11 is, for example, Fig.17 The wide-area image 90 and the detailed image 91 shown are the same. However, the management device 11 displays a digital zoom image of the area designated by the area designation cursor 90a in the wide-area image 90 cut out and enlarged as the detailed image 91 representing the local area e1. In this case, the management device 11 stores the above-mentioned correspondence information that associates the coordinates and the longitude and latitude of the wide-area image 90. The management device 11 uses the correspondence information to execute Fig.15 Processing shown.

[0223] <Modification example of moving body>

[0224] In embodiments 1 and 2, humans (workers W1 to W3) are used as examples of mobile objects having terminal devices 100, but mobile objects having terminal devices 100 are not limited to humans. Any mobile object that moves with the terminal device 100, such as a vehicle (for example, vehicle V1) equipped with the terminal device 100, can be used.

[0225] <Storage medium of information processing program>

[0226] In the above-mentioned operation control examples, the information processing program of each embodiment is stored in the storage device 60B of the management device 11, and the CPU 60A of the management device 11 executes the information processing program through the memory 60C. However, the technology of the present invention is not limited to this.

[0227] Fig.18 1 is a diagram showing an example of a method of installing an information processing program in the control device 60 of the management device 11 from a storage medium storing the information processing program of the action control example. Fig.18 As shown, as an example, the information processing program 221 can be stored in a storage medium 220 which is a non-temporary storage medium. Fig.18 In the illustrated example, the information processing program 221 stored in the storage medium 220 is installed in the control device 60 , and the CPU 60A executes the above-described processes according to the information processing program 221 .

[0228] Various embodiments have been described above, but the present invention is certainly not limited to these examples. Obviously, those skilled in the art can think of various variations or modifications within the scope described in the technical solution, and it should be understood that these variations or modifications naturally also fall within the technical scope of the present invention. Furthermore, the constituent elements in the above embodiments can be arbitrarily combined within the scope of the purpose of the invention.

[0229] In addition, this application is based on the Japanese patent application (Japanese Patent Application No. 2022-088615) filed on May 31, 2022, and the contents are incorporated herein by reference.

[0230] Explanation of symbols

[0231] 1-camera system, 10, 10a-surveillance camera, 11-management device, 12-communication line, 13a, 43B-display, 13b-keyboard, 13c-mouse, 14-secondary storage device, 15-optical system, 15B-lens group, 15B1-anti-vibration lens, 15B2-zoom lens, 16-rotation mechanism, 17, 21-lens actuator, 19-computer, 22, 23, 75, 76-driver, 22-BIS driver, 23-OIS driver, 25-imaging element, 25A-light receiving surface, 27-imaging element actuator, 28-lens driver, 29, 45-correction mechanism, 31-DSP, 32-image memory, 33-correction unit, 34, 34a, 66 to 69, 79, 80-communication I / F, 35, 60C, 102-memory, 36, 60B-storage device, 37, 60A-CPU, 38, 70, 109-bus, 39, 47-position sensor, 40-quantity detection sensor, 43-UI system device, 43A, 62-receiving device, 60-control device, 71-yaw axis rotation mechanism, 72-pitch axis rotation mechanism, 73, 74-motor, 90-wide area image, 90a-area designation cursor, 91-detailed image, 100-terminal device, 101-processor, 103-communication interface, 104-GNSS unit, 105-user interface, 161-movement history image, 220-storage medium, 221-information processing program, E1-monitoring object area, e1-local area, V1-vehicle, W1~W3-staff.

Claims

1. An information processing device comprising a processor, wherein: The processor performs the following processing: Acquire first position information of a position of a photographed object correspondingly associated with a photographed image based on a photographing system and second position information of the terminal device obtained through the terminal device in a photographing area of ​​the photographing system; and The first data is generated based on the first position information and the second position information.

2. The information processing device according to claim 1, wherein: The terminal device is a terminal device possessed by the moving object included in the captured image.

3. The information processing device according to claim 1, wherein: The first data includes instruction information for setting the terminal device as a transmission destination.

4. The information processing device according to claim 3, wherein: The processor performs the following processing: Regarding a plurality of terminal devices acquiring the second location information; and The instruction information is generated to set the terminal device whose second location information is included in the range based on the first location information among the plurality of terminal devices as the transmission destination.

5. The information processing device according to claim 4, wherein: The processor sets the range based on the first position information and zoom information of the imaging system.

6. The information processing device according to claim 4, wherein: The processor sets the range based on the first position information and the number of moving objects having the terminal device detected from the captured image.

7. The information processing device according to claim 3, wherein: The processor performs the following control: When information indicating abnormality of the holder of the terminal device is received from the terminal device, the captured image indicating the area indicated by the second position information in the imaging area is displayed on a display device.

8. The information processing device according to claim 1, wherein: The first data is information related to the movement history of the mobile body having the terminal device.

9. The information processing device according to claim 8, wherein: The processor performs the following processing: Based on the first position information and the second position information, an image obtained by superimposing an image showing the movement history of the moving object on the captured image is generated as information related to the movement history.

10. The information processing device according to claim 1, wherein: The processor performs the following control: A wide area image showing the imaging area and a detailed image showing a narrower range of the imaging area than the wide area image at a higher magnification than the wide area image are displayed on a display device.

11. The information processing device according to any one of claims 1 to 10, wherein: The camera system includes a camera device that obtains the camera image by shooting and a rotating mechanism that rotates the camera device. The first position information is associated with information indicating a rotation state of the camera device based on the rotating mechanism when the camera image is obtained by the camera system.

12. An information processing method, wherein: The processor of the information processing device performs the following processing: Acquire first position information of a position of a photographed object correspondingly associated with a photographed image based on a photographing system and second position information of the terminal device obtained through the terminal device in a photographing area of ​​the photographing system; and The first data is generated based on the first position information and the second position information.

13. An information processing program for causing a processor of an information processing device to perform the following processing: Acquire first position information of a position of a photographed object correspondingly associated with a photographed image based on a photographing system and second position information of the terminal device obtained through the terminal device in a photographing area of ​​the photographing system; and The first data is generated based on the first position information and the second position information.

Images