Image processing system, image processing device, image processing method, and program

Abstract

To enable the acquisition of images used for image processing in places where it is difficult to install lighting.SOLUTION: An image processing device includes an audio acquisition unit that acquires audio in the monitored area, a launch detection unit that detects the launch of a light-emitting object that illuminates the monitored area from the acquired audio, a control unit that operates an image acquisition device that acquires images of the monitored area in response to the detection of the launch, and an image processing unit that performs image processing on the images acquired by the image acquisition device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system, an image processing device, an image processing method, and a program. [Background technology]

[0002] As a related technique, Patent Document 1 discloses an electronic device that detects gaze input. The electronic device described in Patent Document 1 has an infrared camera and a visible light camera. When gaze input is enabled, the electronic device detects the ambient brightness using an illuminance sensor. If the ambient brightness is greater than a predetermined threshold, the electronic device turns on the visible light camera, which then captures an image of the user's face. The electronic device then performs face recognition processing on the captured facial image. If the user's face is recognized, the electronic device turns on the infrared camera and performs a gaze detection process for the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-142851 Summary of the Invention [Problem to be solved by the invention]

[0004] Consider performing person detection, face detection, etc. in a dark environment, such as at night. In that case, for example, when performing person detection in a relatively large area, a large number of lighting fixtures must be installed in that area to obtain images within the area that can be used for person detection. However, installing a large number of lighting fixtures in a large area is not realistic.

[0005] The electronic device described in Patent Document 1 can turn on the visible light camera in conjunction with turning on the lamp, and can perform face recognition processing using images from the visible light camera. However, installing a lamp in an electronic device increases the power consumption of the electronic device. The electronic device described in Patent Document 1, which does not have a lamp, cannot acquire images that can be used for face recognition processing, for example, at night, and therefore cannot perform face recognition processing.

[0006] An object of the present disclosure is to provide an image processing system, an image processing device, an image processing method, and a program that are capable of acquiring images used for image processing in places where it is difficult to install lighting. [Means for solving the problem]

[0007] A program according to a first aspect of the present disclosure causes a computer to execute a process that includes acquiring audio in a monitored area, detecting from the acquired audio the launch of an illuminating object that illuminates the monitored area, acquiring an image of the monitored area in response to the detection of the launch, and performing image processing on the acquired image.

[0008] An image processing method according to a second aspect of the present disclosure includes acquiring audio in a monitored area, detecting from the acquired audio the launch of an illuminating object that illuminates the monitored area, acquiring an image of the monitored area in response to the detection of the launch, and performing image processing on the acquired image.

[0009] An image processing device according to a third aspect of the present disclosure includes an audio acquisition unit that acquires audio in a monitored area, a launch detection unit that detects the launch of an illuminating object that illuminates the monitored area from the acquired audio, a control unit that operates an image acquisition device that acquires images of the monitored area in response to the detection of the launch, and an image processing unit that performs image processing on the images acquired by the image acquisition device.

[0010] An image processing system according to a fourth aspect of the present disclosure includes the image processing device described above and an image acquisition device that acquires an image of the area to be monitored. [Effects of the Invention]

[0011] The image processing system, image processing device, image processing method, and program according to the present disclosure can acquire images to be used for image processing in places where it is difficult to install lighting. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an image processing system according to the present disclosure. [Figure 2] FIG. 1 is a diagram schematically illustrating a monitoring target area and an image acquisition device. [Figure 3] 10 is a flowchart showing an operation procedure of the image processing device. [Figure 4] FIG. 1 is a block diagram illustrating an example of the configuration of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. In addition, in each drawing, the same or similar elements are designated by the same reference numerals, and duplicate explanations are omitted as necessary.

[0014] Fig. 1 is a block diagram showing an example of the configuration of an image processing system according to the present disclosure. One embodiment of the present disclosure will be described using Fig. 1. The image processing system 100 shown in Fig. 1 includes an image processing device 110 and an image acquisition device 130. Note that, although Fig. 1 shows one image acquisition device 130, the number of image acquisition devices 130 is not limited to one. The image processing system 100 may include multiple image acquisition devices 130.

[0015] The image acquisition device 130 is a device that acquires images of the monitored area. The image acquisition device 130 is configured as an imaging device such as a visible light camera. In addition to images, the image acquisition device 130 may also acquire audio, i.e., audio data, in the monitored area. The image acquisition device 130 may be attached to a mobile object that can move within the monitored area. For example, the image acquisition device 130 may be mounted on an airborne object that flies over the monitored area. The airborne object that flies over the monitored area includes, for example, a drone. The image acquisition device 130 may be mounted on a mobile object that moves on land. The mobile object that moves on land includes, for example, a car. The image acquisition device 130 may also be mounted on a mobile object that moves on water. The mobile object that moves on water includes, for example, a ship. The image acquisition device 130 may also be attached to a person.

[0016] FIG. 2 is a diagram schematically illustrating a monitored area and an image acquisition device 130. In this example, the image acquisition device 130 is mounted on a drone 170 that flies above the monitored area. The drone 170 flies, for example, autonomously over the monitored area. The drone 170 may be capable of manually controlling its flight path as desired. The image processing device 110 is configured to be able to communicate with the drone 170 via wireless communication.

[0017] It should be noted that the image processing device 110 does not necessarily have to be placed near the area to be monitored. The image processing device 110 may be placed away from the area to be monitored. In this case, the image processing device 110 may communicate with the drone 170 via a communication device installed near the area to be monitored.

[0018] In this embodiment, a light-emitting object is used to illuminate the area to be monitored in a dark environment such as at night. The light-emitting object may be, for example, an object that is launched from the ground, a vehicle, or a ship and emits light for a certain period of time as it burns. The light-emitting object does not necessarily have to be launched from the ground; the light-emitting object may also be launched from an aircraft. FIG. 2 shows an example in which fireworks 150 are used as the burning light-emitting object. In this example, the fireworks 150 illuminate the area to be monitored after being launched. The burning light-emitting object is not limited to fireworks 150. The burning light-emitting object may also be a light-emitting object such as a flare.

[0019] 1, the image processing device 110 includes an audio acquisition unit 111, a launch detection unit 112, a control unit 113, and an image processing unit 114. The image processing device 110 may be physically configured as a computer device having one or more memories and one or more processors. In the image processing device 110, the one or more processors execute processing in accordance with instructions read from the one or more memories, thereby realizing at least a portion of the functions of each unit within the image processing device 110.

[0020] The voice acquisition unit 111 acquires voice data in the area to be monitored. For example, the voice acquisition unit 111 acquires the voice data from a microphone placed in the area to be monitored. The voice acquisition unit 111 may also acquire the voice data from the image acquisition device 130.

[0021] The launch detection unit 112 detects the launch of a burning light-emitting object, for example, fireworks 150, from the audio data acquired by the audio acquisition unit 111. For example, the launch detection unit 112 analyzes the audio data and determines whether the acquired audio data includes a sound related to the launch of a light-emitting object. When the launch detection unit 112 detects a sound related to the launch of a light-emitting object such as fireworks 150, the launch detection unit 112 detects the launch of the light-emitting object.

[0022] The control unit 113 controls image acquisition by the image acquisition device 130. In response to the launch detection unit 112 detecting the launch of a light-emitting object, the control unit 113 causes the image acquisition device 130 to acquire an image of the area to be monitored. The control unit 113 may also control the operation of a flying object such as a drone 170 on which the image acquisition device 130 is mounted.

[0023] The control unit 113 causes the image acquisition device 130 to start image acquisition, for example, from a time that has a predetermined time relationship with the time when the launch of the light-emitting object is detected. The control unit 113 causes the image acquisition device 130 to start image acquisition, for example, by turning on the power of the image acquisition device 130. Alternatively, the control unit 113 may cause the image acquisition device 130 to start image acquisition by changing the operating state of the image acquisition device 130 from standby to active.

[0024] The control unit 113 causes the image acquisition device 130 to stop acquiring images after a time period corresponding to the light emission duration of the light-emitting object has elapsed. The control unit 113 causes the image acquisition device 130 to stop acquiring images, for example, by turning off the power of the image acquisition device 130. Alternatively, the control unit 113 causes the image acquisition device 130 to stop acquiring images by changing the operating state of the image acquisition device 130 from active to standby.

[0025] The time difference between the time the light-emitting object is launched and the time the launch sound is acquired varies depending on the distance between the location where the sound is acquired and the location where the light-emitting object is launched. The control unit 113 may control the timing at which image acquisition starts depending on the distance between the location where the sound is acquired and the location where the light-emitting object is launched.

[0026] Here, it is considered that the amount of combustion material filled inside a luminous object such as firework 150 varies depending on its size, and the burning time, i.e., the duration of light, changes depending on the size. It is also considered that the audio data at the time of launch and the audio data after launch of a luminous object change depending on the size. For example, control unit 113 may estimate the size of the luminous object based on the audio data, and estimate the duration of light emission of the luminous object based on the estimated size. Control unit 113 may change the image acquisition time of image acquisition device 130, i.e., the time from the image acquisition start time to the image acquisition stop time, depending on the estimated duration of light emission.

[0027] The image processing unit 114 acquires an image of the area to be monitored from the image acquisition device 130. In the acquired image of the area to be monitored, objects such as people are considered to be illuminated by light emitted from the burning light-emitting object. The image processing unit 114 performs predetermined image processing on the image in which the burning light-emitting object is used as illumination. For example, the image processing unit 114 performs at least one of face recognition processing, person detection processing, and crowd analysis processing on the image of the area to be monitored.

[0028] Next, the operation procedure will be explained. Fig. 3 is a flowchart showing the operation procedure of the image processing device 110. The operation procedure of the image processing device 110 corresponds to an image processing method. The audio acquisition unit 111 acquires audio data in the area to be monitored (step S1). The launch detection unit 112 analyzes the acquired audio data and determines whether or not a light-emitting object has been launched (step S2). If it is determined in step S2 that a light-emitting object has not been launched, the process returns to step S1, and the audio acquisition unit 111 acquires audio data. Steps S1 and S2 are repeatedly executed until it is determined in step S2 that a light-emitting object has been launched.

[0029] If it is determined in step S2 that a light-emitting object has been launched, control unit 113 causes image capture device 130 to capture an image of the area to be monitored (step S3). In step S3, control unit 113 operates image capture device 130 in synchronization with the timing of fireworks 150 emitting light, for example, and causes image capture device 130 to capture an image during the period when the light of fireworks 150 illuminates the area to be monitored. Image processing unit 114 performs predetermined image processing on the image captured in step S3 (step S4). Image processing unit 114 outputs the result of the image processing (step S5).

[0030] In this embodiment, the launch detection unit 112 detects the launch of a burning light-emitting object, such as a firework or a flare, based on the audio data acquired by the audio acquisition unit 111. The control unit 113 causes the image acquisition device 130 to start acquiring images in response to the launch of the burning light-emitting object. The image processing unit 114 performs image processing such as face recognition or human detection on the images acquired by the image acquisition device 130. In this way, the image acquisition device 130 can acquire images that utilize the light of the burning light-emitting object in a dark environment. The image processing unit 114 can also perform image processing such as face recognition or human detection on the acquired images.

[0031] Infrared cameras are sometimes used for human and face detection in areas where it is difficult to install floodlights. However, images from infrared cameras have low resolution and are monochrome images without color information. Furthermore, images from infrared cameras are prone to distortion. While such images can be used to capture the general characteristics of an object, they are not suitable for capturing detailed facial features. In this embodiment, illumination from a burning luminous body is effectively utilized to capture visible light images in areas where it is difficult to install floodlights. Therefore, more detailed image features can be extracted during image processing than when an infrared camera is used.

[0032] The image processing device 110 according to this embodiment can be applied to search activities using face recognition or person detection, or crowd analysis, at fireworks venues. In this case, the launch detection unit 112 detects the launch of fireworks, for example, by detecting a bang. The launched fireworks will flash brightly, for example, two to four seconds after launch. Furthermore, if the launched fireworks are of the type that make a whoosh as they rise, they will flash one second after the whoosh stops. The control unit 113 controls the image capture device 130 to capture images of the monitored area for five seconds from the estimated time when the fireworks will flash, assuming that the average burning time of launched fireworks is five seconds.

[0033] The image processing unit 114 performs face recognition or person detection on images in which fireworks are used as illumination. The image processing unit 114 outputs the results of the face recognition or person detection to the user of the image processing device 110. The user can use the results of the face recognition or person detection to search for people at the fireworks venue. For example, the user can identify the target's movement path by tracking the position of the image acquisition device 130 that detected the target over time. The image processing unit 114 may perform crowd analysis instead of or in addition to face recognition or person detection. In this case, the user can detect dangerous areas at the fireworks venue due to crowding or abnormalities that have occurred at the fireworks venue.

[0034] The image processing device 110 can be applied to search activities using facial recognition or human detection in conflict areas. The image processing device 110 can also be applied to search activities using facial recognition or human detection in border areas. In this case, the launch detection unit 112 detects the launch of a flare, for example, from audio data. The launched flare shines brightly, for example, two to four seconds after launch. The control unit 113 controls the image acquisition device 130 to acquire images of the area to be monitored for 60 seconds from the estimated time when the flare is to be fired, assuming that the average light emission time of the flare is 60 seconds.

[0035] The image processing unit 114 performs face recognition or person detection on the image illuminated by the flare. The image processing unit 114 outputs the results of the face recognition or person detection to the user of the image processing device 110. The user can use the results of the face recognition or person detection to conduct search activities for people in conflict zones or border areas.

[0036] For example, in border areas, it is difficult to install lighting due to power supply issues. Also, in conflict areas and fireworks venues, it is difficult to install lighting due to aesthetic and other issues. Furthermore, because border areas and fireworks venues are large, it is not realistic to install lighting to cover the entire area. In this embodiment, the image acquisition device 130 utilizes the light from a burning light-emitting object to acquire images during the time when the light-emitting object is illuminating the monitored area. In this way, the image acquisition device 130 can acquire images that can be used for image processing without installing lighting in the monitored area.

[0037] Furthermore, because border areas and fireworks venues are large, a large number of cameras would be required to cover the entire area. In this embodiment, when the image capture device 130 is mounted on the drone 170, images can be captured over a large area to be monitored without installing a large number of fixed cameras.

[0038] Furthermore, in this embodiment, because a burning light-emitting object is used as lighting, there is no need to equip the drone 170 with a light to illuminate people. For example, in a conflict zone, if the drone 170 were equipped with a light, the drone 170 could become a target of attack. In this embodiment, there is no need to equip the drone 170 with a light, so the possibility of the drone 170 being attacked can be reduced. Furthermore, if a light is not equipped, the power consumed by the light in the drone 170 can be reduced, allowing for effective use of power.

[0039] In comparison with Patent Document 1, the electronic device described in Patent Document 1 can turn on the visible light camera in response to the surroundings becoming brighter due to the light-emitting combustion body, thereby capturing visible light camera images. However, the electronic device described in Patent Document 1 cannot capture visible light camera images until the light-emitting body has burned and the surroundings have become sufficiently bright. In contrast, in this embodiment, the image processing device 110 detects the launch of the light-emitting combustion body, predicts the timing of the light-emitting body combustion, and starts image capture by the image capture device 130. In this embodiment, the image capture device 130 can more effectively utilize the limited light emission time and capture visible light camera images for a longer period of time, compared to the electronic device described in Patent Document 1.

[0040] Next, a description will be given of the hardware configuration of the image processing device 110. Fig. 4 is a block diagram showing an example configuration of a computer device that can be used as the image processing device 110. The computer device 500 has a processor 510 such as a CPU (Central Processing Unit), a storage unit 520, a ROM (Read Only Memory) 530, a RAM (Random Access Memory) 540, a communication interface (IF: Interface) 550, and a user interface 560.

[0041] The communication interface 550 is an interface for connecting the computer device 500 to a communication network via wired communication means or wireless communication means, etc. The user interface 560 includes a display unit such as a display, and an input unit such as a keyboard, a mouse, and a touch panel.

[0042] The storage unit 520 is an auxiliary storage device that can store various types of data. The storage unit 520 can be used as the product information DB 110. The storage unit 520 does not necessarily have to be a part of the computer device 500, but may be an external storage device or a cloud storage connected to the computer device 500 via a network.

[0043] The ROM 530 is a non-volatile storage device. For example, a semiconductor storage device with a relatively small capacity, such as a flash memory, is used for the ROM 530. The programs executed by the CPU 510 can be stored in the storage unit 520 or the ROM 530. The storage unit 520 or the ROM 530 stores programs for realizing the functions of each unit of the image processing device 110.

[0044] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include RAM, ROM, flash memory, solid-state drive (SSD) or other memory technology, compact disc (CD), digital versatile disc (DVD), Blu-ray disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0045] The RAM 540 is a volatile storage device. Various semiconductor memory devices such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) are used for the RAM 540. The RAM 540 can be used as an internal buffer for temporarily storing data, etc. The CPU 510 loads a program stored in the storage unit 520 or the ROM 530 into the RAM 540 and executes it. The CPU 510 executes the program, thereby realizing the functions of each unit of the image processing device 110. The CPU 510 may have an internal buffer for temporarily storing data, etc.

[0046] In the present disclosure, the image processing device 110 does not necessarily have to be configured as a single physical device. The image processing device 110 may be configured using multiple physically separated devices. For example, the image processing device 110 may be configured using a device including the control unit 113 and a device including the image processing unit 114.

[0047] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be combined with other embodiments as appropriate.

[0048] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0049] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0050] [Appendix 1] Acquires audio in the monitored area, Detecting the launch of a light-emitting object that illuminates the monitored area from the acquired sound; capturing an image of the monitored area in response to a launch being detected; A program that causes a computer to execute a process that includes performing image processing on the acquired image.

[0051] [Appendix 2] The program described in Appendix 1 causes the computer to execute a process of starting acquisition of the image from a time that has a predetermined time relationship with the time when the launch of the luminous object is detected.

[0052] [Appendix 3] The program described in Appendix 2, which causes the computer to execute a process of starting acquisition of the image by turning on the power of an image acquisition device used to acquire the image or changing the operating state of the image acquisition device from standby to active.

[0053] [Appendix 4] 4. The program described in Appendix 2 or 3, which causes the computer to execute a process of stopping the acquisition of the image after a time corresponding to the duration of light emission of the light-emitting object has elapsed.

[0054] [Appendix 5] The program described in Appendix 4, which causes the computer to execute a process of stopping the acquisition of the image by turning off the power of an image acquisition device used to acquire the image or by changing the operating state of the image acquisition device from active to standby.

[0055] [Appendix 6] The program of any one of appendices 1 to 5, which causes the computer to execute a process of detecting the launch of a light-emitting object by analyzing the audio and detecting sounds related to the launch of the light-emitting object.

[0056] [Appendix 7] A program described in any one of appendices 1 to 6, which causes the computer to execute a process of estimating the light-emitting duration of the light-emitting object based on the audio, and controlling the time for acquiring the image in accordance with the estimated light-emitting duration.

[0057] [Appendix 8] The program according to any one of appendices 1 to 7, wherein the luminous object is an object that emits light upon combustion.

[0058] [Appendix 9] The program according to any one of appendices 1 to 8, wherein the luminous object is a firework or a flare launched into the air.

[0059] [Appendix 10] 10. The program of any one of appendices 1 to 9, wherein the image is acquired using an image acquisition device mounted on a flying object flying over the monitored area.

[0060] [Appendix 11] The program according to any one of appendices 1 to 10, wherein the image processing includes at least one of face recognition processing, person detection processing, and crowd analysis processing.

[0061] [Appendix 12] Acquires audio in the monitored area, Detecting the launch of a light-emitting object that illuminates the monitored area from the acquired sound; capturing an image of the monitored area in response to a launch being detected; An image processing method comprising performing image processing on the acquired image.

[0062] [Appendix 13] a voice acquisition unit that acquires voice in a monitoring target area; a launch detection unit that detects the launch of a light-emitting object that illuminates the monitoring target area from the acquired sound; a control unit that, in response to the detection of a launch, activates an image capture device that captures an image of the monitored area; an image processing unit that performs image processing on the image acquired by the image acquisition device.

[0063] [Appendix 14] an image processing device according to Supplementary Note 13; and an image capture device for capturing an image of the area to be monitored;

[0064] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 11 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 12, 13, and 14 in the same dependency relationship as Supplementary Notes 2 to 11. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0065] 100: Image processing system 110: Image processing device 111: Voice acquisition unit 112: Launch detection unit 113: Control unit 114: Image processing unit 130: Image acquisition device 150: Fireworks 170: Drone

Claims

1. Acquires audio in the monitored area, Detecting the launch of a light-emitting object that illuminates the monitored area from the acquired sound; capturing an image of the monitored area in response to a launch being detected; A program that causes a computer to execute a process that includes performing image processing on the acquired image.

2. The program according to claim 1 , causing the computer to execute a process of starting acquisition of the image from a time that has a predetermined time relationship with a time when the launch of the light-emitting object is detected.

3. The program according to claim 2, which causes the computer to execute a process of starting acquisition of the image by turning on the power of an image acquisition device used to acquire the image or by changing the operating state of the image acquisition device from standby to active.

4. The program according to claim 2 or 3, which causes the computer to execute a process of stopping the acquisition of the image after a time corresponding to a light emission duration of the light-emitting object has elapsed.

5. The program according to claim 1 or 3, which causes the computer to execute a process of estimating the duration of light emission of the luminous object based on the audio and controlling the time for acquiring the image in accordance with the estimated duration of light emission.

6. The program according to claim 1 , wherein the light-emitting object is a firework or a flare launched into the air.

7. The program according to claim 1 , wherein the image is acquired using an image acquisition device mounted on a flying object flying over the area to be monitored.

8. Acquires audio in the monitored area, Detecting the launch of a light-emitting object that illuminates the monitored area from the acquired sound; capturing an image of the monitored area in response to a launch being detected; An image processing method comprising performing image processing on the acquired image.

9. a voice acquisition unit that acquires voice in a monitoring target area; a launch detection unit that detects the launch of a light-emitting object that illuminates the monitoring target area from the acquired sound; a control unit that, in response to the detection of a launch, activates an image capture device that captures an image of the monitored area; an image processing unit that performs image processing on the image acquired by the image acquisition device.

10. The image processing device according to claim 9 ; an image capture device for capturing an image of the area to be monitored;

Images