Image processing system, image processing method and program

Abstract

To acquire an ROI image excellent in image quality.SOLUTION: An image processing system comprises: a setting part that sets an attention region including a sound source position or a sound source direction, in an acquired image acquired by an image acquisition part, on the basis of the sound source position or the sound source direction, identified by using a sound signal acquired from a sound acquisition part; and a change part that changes characteristics of an image of the attention region, by image processing based on image information within the attention region.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, imaging devices such as smartphones, wearable cameras, vlog cameras, and surveillance cameras have become widespread. Many of these imaging devices have built-in microphones, allowing them to capture and record images simultaneously.

[0003] Attempts are being made to improve the accuracy of image capture by using sound information. Patent Document 1 discloses a technology that achieves both high image quality and image compression efficiency by controlling image compression encoding inside and outside a region of interest (ROI) preset by a user, using the detection of a predetermined sound as a trigger. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-134323 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology of Patent Document 1, if the position or direction of the sound source of an event related to a sound that may occur arbitrarily is not included within the pre-set ROI, it may be impossible to obtain a high-quality image of the ROI.

[0006] The present invention aims to provide a technique for acquiring an image of an ROI with excellent image quality. [Means for solving the problem]

[0007] The image processing device according to one aspect of the present embodiment includes: a setting unit that sets an attention area including the position or direction of a sound source in an acquired image acquired by an image acquisition unit, based on the position or direction of the sound source identified using a sound signal acquired from a sound acquisition unit; and a modification unit that modifies characteristics of the image in the region of interest through image processing based on image information within the region of interest.

[0008] An image processing method according to another aspect of this embodiment includes a setting step of setting a region of interest including the position or direction of a sound source in an acquired image acquired by an image acquisition unit, based on the position or direction of the sound source identified using a sound signal acquired from a sound acquisition unit; and a modifying step of modifying characteristics of the image in the region of interest based on image information within the region of interest. [Effects of the Invention]

[0009] According to the present invention, an image of an ROI with excellent image quality can be acquired. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to a first embodiment. [Figure 3] 1A is a diagram showing the position of a sound source in an image captured when a sound is generated, and FIG. 1B is a diagram showing an example of ROI when the sound source position is identified in the captured image. [Figure 4] 10A and 10B are diagrams illustrating the effects of the technology of the first embodiment; [Figure 5] FIG. 3 is a diagram showing an example of an image displayed on a display unit according to the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining the flow of processing in the image processing apparatus according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing an example of the arrangement of an image processing apparatus according to a second embodiment. [Figure 8]FIG. 10 is a block diagram showing the configuration of an image processing device of a second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an effect of the technique of the second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to the second embodiment. [Figure 11] FIG. 10 is a flowchart for explaining a processing flow in an image processing apparatus according to a second embodiment. [Figure 12] 10A and 10B are diagrams illustrating an example of the relationship between images stored in a storage unit and time. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] First Embodiment In the first embodiment, a configuration will be described in which a region of interest (ROI) including the position or direction of a sound source is set in an image (acquired image) acquired by an image acquisition unit based on the region of interest set at the sound source position or direction identified using a sound signal, and image characteristics are changed based on image information of the ROI. In this embodiment, an example of changing brightness information (brightness value) will be described as an example of image information. Note that the image information is not limited to this example, and image characteristics of the region of interest can be changed based on various image information. In this embodiment, a process will be described in which the image characteristics of the ROI are changed based on the image information of the ROI, making it easier to see the inside of the ROI.

[0013] <Overall configuration of image processing device> FIG. 1 is a diagram showing the configuration of an image processing device 10 according to the first embodiment. The image processing device 10 has a control unit 130 that processes image information and sound information. In FIG. 1, an image acquisition unit 110 acquires an image and outputs the acquired image signal (image information) to the image processing device 10. Furthermore, a sound acquisition unit 120 acquires sound and outputs a sound signal (sound information) to the image processing device 10. The control unit 130 acquires the image signal (image information) from the image acquisition unit 110 and acquires the sound signal (sound information) from the sound acquisition unit 120 via a bus 150. The control unit 130 performs display control to display the image (image information) processed by each unit of the control unit 130 on a display unit 140. The image processing device 10 of this embodiment can function as a device that monitors a monitoring target 100.

[0014] In FIG. 1, the monitored object 100 is shown as a room as an example, but is not limited to this example and may be any object such as a road, infrastructure, or commercial facility. 101 is a sound source, and FIG. 1 illustrates an example in which glass in a room is broken with a hammer. Other examples of the sound source 101 include sounds resulting from various sound-related events that can occur arbitrarily, such as the destruction of an object, a car crash, a horn, a gun shot, an explosion, talking, screams, and shouts. The monitored object 100 and the sound source 101 do not constitute the image processing device 10 of this embodiment, but are described here to exemplify the processing of the image processing device 10.

[0015] 2 is a block diagram showing the configuration of the image processing device 10 of the first embodiment. The image acquisition unit 110 acquires an image signal (image information) by capturing an image of the monitoring target 100. The image acquisition unit 110 acquires an image signal (image information) of the monitoring target 100 by capturing a moving image and / or a still image. The image acquisition unit 110 has an imaging unit 111 and an imaging unit 112, and together with electrical circuits and the like (not shown), the image acquisition unit 110 (monitoring camera) is configured as a whole.

[0016] The imaging unit 111 can use a lens that collects external light, and forms an image of the incident light on the imaging unit 112. Based on the settings of the imaging control unit 133, the imaging unit 111 can change the aperture diameter, focus, and focal length as imaging conditions.

[0017] The imaging unit 112 converts the image of the monitored object 100 formed by the imaging unit 111 into an electrical image signal (image information) based on the set imaging conditions, and outputs the converted image. The imaging unit 112 can use an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The imaging unit 112 can change imaging conditions such as exposure time and gain based on the settings of the imaging control unit 133.

[0018] Furthermore, the image acquisition unit 110 acquires settings from the imaging control unit 133 via the bus 150, and can change imaging conditions such as brightness, contrast, sharpness, gamma, hue, saturation, white balance, and frame rate based on the acquired settings. The image acquisition unit 110 can capture images of the monitored object 100 using visible light or infrared light. For example, when capturing images using infrared light, the image processing device 10 may be provided with an infrared light illuminator. When capturing images using visible light, the image processing device 10 may be provided with a visible light illuminator.

[0019] The sound acquisition unit 120 receives (collects) sound generated by the sound source 101 and converts it into an electrical sound signal (sound information). The sound acquisition unit 120 can be configured as various types of microphones (microphones), such as a coil-type microphone or a condenser-type microphone. The sound acquisition unit 120 may also be a microphone array consisting of multiple microphones. The sound acquisition unit 120 may be omnidirectional, capable of receiving sound from all directions, or directional, having sensitivity in a specific direction. The sound acquisition unit 120 may be built into the image acquisition unit 110.

[0020] The control unit 130 processes the image signal (image information) acquired by the image acquisition unit 110 and the sound signal (sound information) acquired by the sound acquisition unit 120. The control unit 130 has, as its functional components, an attention area setting unit 131, an image characteristic changing unit 132 (an imaging control unit 133, an image processing unit 134), a storage unit 135, and a communication interface (I / F) 136. The control unit 130 can be configured with a workstation, a personal computer, a tablet PC, a smartphone, a server, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a microcomputer, or the like. All or part of the functional components of the control unit 130 may be provided in the image acquisition unit 110.

[0021] As shown in FIG. 2, the image acquisition unit 110, the sound acquisition unit 120, the control unit 130, and the display unit 140 are connected to each other via a bus 150 so that they can communicate with each other. Here, the bus 150 that connects each unit so that they can communicate with each other may be a network bus. Communication via the bus 150 may be either wired or wireless. Furthermore, the control unit 130 can control the display of the display unit 140 via the bus 150. A communication interface (I / F) 136 of the control unit 130 functions as an interface for receiving various types of information output from the image acquisition unit 110 or the sound acquisition unit 120, or for transmitting various types of information from the control unit 130 to an external device. The configuration of the control unit 130 will be specifically described using FIG. 2.

[0022] (Attention area setting unit 131) The region of interest setting unit 131 uses the sound signal (sound information) acquired from the sound acquisition unit 120 to identify the position or direction of a sound source relative to the position of the sound acquisition unit 120. Then, the region of interest setting unit 131 sets a region of interest (ROI) in an image (acquired image) acquired by the image acquisition unit 110 based on the identified position or direction of the sound source. Here, when the region of interest setting unit 131 identifies the position or direction of the sound source, it is sufficient that, for example, the position and direction in a coordinate system relative to the position of the sound acquisition unit 120 and the position and direction in the coordinate system of the image acquisition unit 110 are calibrated in advance and match.

[0023] When the sound acquisition unit 120 includes multiple microphones (microphone arrays) positioned in advance within the monitored object 100, the sound source location and direction can be estimated based on the difference in arrival time of sound at each microphone and the amplitude (volume) of the sound signal acquired by each microphone. Applications and developments of these methods, such as the beamformer method and delay-and-sum method used in radar technology and ultrasonic echo technology, may also be used. Furthermore, adaptive processing methods such as the Capon method may be used to improve the accuracy of identification. Alternatively, the sound source location and direction may be identified using neural networks, such as deep learning. Analytical methods such as the beamformer method, delay-and-sum method, and Capon method require only the relative positions of multiple microphones to be set in advance. Learning methods such as deep learning allow for the estimation of the sound source location or direction even when the relative positions of multiple microphones are not set in advance by learning the differences in sound signals from multiple microphones relative to different sound source positions. By using a learning model in which the relative positions of multiple microphones are preset and learning the differences in the sound signals from multiple microphones, the accuracy of estimating the position or direction of a sound source can be improved.

[0024] When the sound acquisition unit 120 is configured using a microphone array with narrow directivity, multiple microphones may be arranged so that their directivities do not overlap, and the attention area setting unit 131 may identify the direction of the microphone with the largest amplitude of the sound signal among the sound signals acquired from the multiple microphones as the sound source direction.

[0025] When the sound acquisition unit 120 is configured with a single microphone, the attention area setting unit 131 may acquire (scan) sound signals while mechanically changing the orientation of the sound acquisition unit 120 within the monitored object 100, and identify the sound source direction based on the scanning direction or scanning position at which the amplitude of the acquired sound signal is greatest. In addition to the above-mentioned methods, any method may be used to identify the position or direction of the sound source as long as it can identify the position or direction of the sound source.

[0026] The attention area setting unit 131 sets an ROI on the image captured by the image acquisition unit 110 using the identified (estimated) sound source position or sound source direction. Here, it is assumed that the coordinate system of the position or direction used as a reference in the attention area setting unit 131 and the coordinate system of the image used as a reference in the image acquisition unit 110 are pre-calibrated and matched. The three-dimensional position coordinates of an object on the image can be acquired by a depth information measurement device such as LiDAR (Light-Detection-and-Ranging: not shown), depth information acquisition processing using a neural network typified by deep learning, stereo imaging using multiple image acquisition units 110, or the like.

[0027] 3A and 3B are diagrams illustrating images of a monitored object 100 (room) captured by the image acquisition unit 110. Fig. 3A shows the position of a sound source 101 in an image captured when a sound is generated. The position of the sound source 101 is not actually displayed in the image, but is shown for illustrative purposes.

[0028] 3(b) is a diagram illustrating an example of ROI when a sound source position is identified in a captured image. The rectangular area indicated by the dashed line is the ROI, and the center position (central part) of the rectangular area (ROI) is the sound source position identified by the region of interest setting unit 131. The size of the ROI may be determined based on the position resolution of the algorithm used to identify the sound source position or the directivity of the sound acquisition unit 120, or may be any size set in advance.

[0029] 3(c) shows an example of ROI when the sound source direction is identified. The positional relationship between the image acquisition unit 110 and the sound acquisition unit 120 is assumed to be a configuration in which the sound acquisition unit 120 is provided below the image acquisition unit 110, as shown in FIG. 1. The arrow indicating the sound source direction is indicated by an arrow extending from the bottom of the image captured by the image acquisition unit 110 toward the identified sound source direction, and the dashed line area surrounding the arrow indicating the sound source direction is the ROI. As shown in FIGS. 3(b) and 3(c), the region of interest setting unit 131 sets a partial area including the position or direction of the sound source in the image acquired by the image acquisition unit 110 (acquired image) as a region of interest (ROI). When displaying on the display unit 140, the control unit 130 may perform display control to superimpose a frame display (dashed line in FIGS. 3(b) and 3(c)) indicating the outline of the region of interest (ROI) on the image (acquired image) acquired by the image acquisition unit 110 and display it on the display unit 140. The size of the ROI may be determined based on the directional resolution of the algorithm used to identify the sound source direction or the directivity of the sound acquisition unit 120, or may be any size set in advance. When the sound source direction is identified, specifying an ROI as shown in FIG. 3(c) makes it possible to estimate that the sound source position is included within the range of the ROI, thereby achieving the effects of the disclosed technology. That is, if an ROI is set based on the identification of the sound source direction, even if the specific sound source position is not identified, it is estimated that the sound source position is included within this range. Therefore, it becomes possible to acquire an ROI (ROI image) including the sound source position and sound source direction identified from the sound information. The acquired ROI image can be acquired with high accuracy by processing the acquired ROI image described below.

[0030] (Image characteristic change unit 132) The image characteristic changing unit 132 has an imaging control unit 133 and / or an image processing unit 134. The imaging control unit 133 changes the imaging conditions of the imaging unit 112 and the imaging unit 111 based on the image information of the ROI set by the region of interest setting unit 131 so that the user can easily check (view) the situation within the ROI. When the image information of the ROI is higher than the upper limit of a preset appropriate range of image information or when the image information of the ROI is lower than the lower limit of the appropriate range, the imaging control unit 133 may change the characteristics of the image of the ROI by image processing that changes the imaging conditions of the image acquisition unit 110 so as to adjust the image information of the ROI within the appropriate range. For example, when the image information indicates that the brightness values ​​of the pixels in the ROI are saturated and exceed the upper limit of the appropriate range, making the image too bright, or that the brightness values ​​are beyond the lower limit of the appropriate range and are close to zero (0), making the image too dark, the imaging control unit 133 may adjust the brightness values ​​to within the appropriate range by changing imaging conditions such as the aperture diameter, focus, and focal length of the imaging unit 111, the exposure time and gain of the imaging unit 112, and the brightness, contrast, gamma, saturation, and hue of the image acquisition unit 110.

[0031] The imaging control unit 133 may change the characteristics of the image of the ROI by image processing that changes, as an imaging condition, at least one of the aperture diameter, focus, and focal length of the imaging unit 111 of the image acquisition unit 110. Alternatively, the imaging control unit 133 may change the characteristics of the image of the ROI by image processing that changes, as an imaging condition, the exposure time or gain of the imaging unit 112 of the image acquisition unit 110. Alternatively, the imaging control unit 133 may change the characteristics of the image of the ROI by image processing that changes, as an imaging condition of the image acquisition unit 110, at least one parameter of brightness, contrast, gamma, saturation, and hue.

[0032] Furthermore, the imaging control unit 133 may adjust the brightness value by adjusting the aperture of the imaging unit 111. The imaging control unit 133 may control the imaging unit 111 to focus on the ROI. At this time, the accuracy of the focus may be improved by adjusting the brightness value in the ROI to an appropriate range before adjusting the focus. The imaging control unit 133 may also adjust the focal length of the imaging unit 111 to zoom in on the ROI to make it easier to see.

[0033] The image captured by the image acquisition unit 110 may be a still image or a video. When the image acquisition unit 110 captures an image of a subject using a video, the imaging control unit 133 may increase the frame rate of the image of the ROI compared to the frame rate of an image in which an ROI is not set, to make it easier to see the movement of the subject in the image of the ROI in the monitored object 100. When the image acquisition unit 110 captures an image using lighting, the imaging control unit 133 may adjust the brightness value of the image to an appropriate range by adjusting the lighting intensity, which is an imaging condition.

[0034] The image processing unit 134 performs image processing based on the image information of the ROI set by the region-of-interest setting unit 131 so that the user can easily check the situation within the ROI. For example, the image processing unit 134 may perform image processing by changing various image processing parameters (e.g., brightness, contrast, gamma, saturation, hue, etc.) to change the characteristics of the image. Here, the image processing unit 134 changes the characteristics of the image of the region of interest through image processing based on the image information within the region of interest. Furthermore, the image processing unit 134 changes the characteristics of the image acquired by the image acquisition unit 110 (acquired image) through image processing based on the image information within the region of interest.

[0035] Furthermore, the image processing unit 134 may perform image processing such as emphasizing the subject in the image of the ROI whose characteristics have been changed, based on the image information in the ROI. For example, the image processing unit 134 may perform image processing such as edge enhancement, noise removal, super-resolution, contour extraction of a person or the like, based on the image information in the ROI, frame display of the ROI based on the position information of the ROI, clipping of the ROI, and changing the compression rate. When capturing a video using the image acquisition unit 110, the image processing unit 134 may reduce the frame rate of images in which an ROI is not set (images other than the ROI) compared to the frame rate of images in which an ROI is set, thereby improving the visibility of the movement of the subject in the image of the ROI, and improving the video compression rate based on the similarity between frames.

[0036] The various image processes performed by the image processing unit 134 may be performed only on the internal region of the ROI based on the image information of the ROI set by the region-of-interest setting unit 131, or may be performed on the entire image including the internal region of the ROI and the external region of the ROI. For example, when image processing is performed only on the internal region of the ROI, the visibility of the internal region can be improved while maintaining the visibility of the external region of the ROI. Furthermore, when image processing is performed on the entire image including the internal region of the ROI and the external region of the ROI, natural continuity can be imparted to the image in the vicinity of the ROI.

[0037] The image characteristic change unit 132 may be configured to include both the imaging control unit 133 and the image processing unit 134, or may be configured to include either one of them, and in either case the effects of this embodiment can be obtained.

[0038] The image acquisition unit 110 may be equipped with a pan-tilt-zoom (PTZ) function so that the orientation (imaging direction) of the image acquisition unit 110 can be directed toward the sound source position or sound source direction. The effects of this embodiment can also be achieved in this case. For example, when the orientation of the image acquisition unit 110 is changed toward the sound source position at the center of the image using the pan or tilt function, a high-brightness object such as a room lamp or a street lamp may be reflected at the edge of the image due to the change in orientation. In such a case, the brightness adjustment of the entire image may be strongly affected by the high-brightness object, causing the image near the center where the sound source is located to become dark, making the entire image difficult to see. In this embodiment, at least one of the imaging control unit 133 and the image processing unit 134 changes image characteristics to make the ROI set based on the sound source position more visible, thereby making the image of the ROI near the center where the sound source is located more visible.

[0039] FIG. 4 illustrates an example of the effect of the technology of the first embodiment. FIG. 4(a) shows an image captured when a sound is generated. The position of the sound source 101 is shown for illustrative purposes, although it is not actually displayed in the image. FIG. 4(b) shows a state in which an ROI is set at the sound source position by the region-of-interest setting unit 131. The rectangular region indicated by the dashed line is the ROI, and the center position (central part) of the ROI is the sound source position identified by the region-of-interest setting unit 131. At this point, the vicinity of the sound source position is dark, making it difficult to visually identify the source of the sound or what event has occurred. FIG. 4(c) shows an image in which the overall image characteristics of the image acquired by the image acquisition unit 110 (acquired image) have been changed based on image information within the ROI so that the imaging control unit 133 and the image processing unit 134 can easily confirm the situation within the ROI. As shown in FIG. 4(c), the darkness around the sound source position has disappeared, and the image has been processed so that the situation of glass being broken by a hammer can be confirmed as an event occurring at the sound source. In the example shown in Fig. 4(c), the image has the overall image characteristics changed based on the image information within the ROI. In addition to this example, as shown in Fig. 5(b), it is also possible to change the image characteristics of only the internal region of the ROI image based on the image information within the ROI, making it easier to visually recognize events occurring at the sound source.

[0040] (Archive section 135) The storage unit 135 stores the image in which the ROI has been made more visible through the processing of the image characteristic changing unit 132. The image stored in the storage unit 135 may be a moving image or a still image. The storage unit 135 may be configured by, for example, a memory card including a flash memory, or a non-volatile recording device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0041] FIG. 12 is a diagram illustrating an example of the relationship between images stored by the storage unit 135 and time. The horizontal axis represents time, and time elapses in the direction of the arrow (to the right) from time T0 to time T2. IM illustrates images (video data) stored in the storage unit 135 over time. Time T1 indicates the time when a sound generation event is detected. Time T0 indicates a time that precedes time T1 by a predetermined period TB1 (a predetermined number of frames). Time T2 indicates a time that follows time T1 by a predetermined period TB2 (a predetermined number of frames). The predetermined period TB3 indicates a period before and after the occurrence of a sound event. Here, the predetermined periods TB1 and TB2 may be set arbitrarily. Alternatively, the predetermined period TB2 may be set so that storage begins when a sound generation event is detected (e.g., T1 in FIG. 12), continues while the sound generation continues, and ends when sound is no longer detected (e.g., T2 in FIG. 12). Alternatively, when a moving subject is detected in the image of the ROI, the predetermined period TB2 may be set so that saving ends at the timing when the subject stops moving (for example, T2 in FIG. 12).

[0042] When saving a moving image, the saving unit 135 may save the moving image from the moment when sound is detected by the sound acquisition unit 120, or from a time (for example, T0 in FIG. 12) that is a predetermined period (predetermined frame: for example, TB1 in FIG. 12) before the time when sound is detected (for example, T1 in FIG. 12). When the control unit 130 causes the display unit 140 to display the moving image (for example, IM in FIG. 12) saved in the saving unit 135, the saving unit 135 may save the moving image from time T0 that is a predetermined period (TB1) before the time T1. ) on the display unit 140. By displaying the video from a time going back a predetermined period, it becomes possible to visually confirm the situation immediately before the sound generation event. The video (video data) stored in the storage unit 135 may be sequentially deleted over time, with the video (video data) captured before the predetermined period (TB1). This allows the storage area for image information in the storage unit 135 to be used efficiently.

[0043] Furthermore, when saving a still image, the storage unit 135 may save the still image captured at the moment when sound is detected (for example, at time T1) or within a predetermined period before or after that (for example, within predetermined periods TB1, TB2 before or after time T1).

[0044] The timing at which the storage unit 135 ends storing images (moving images or still images) may be the timing at which a predetermined time or a predetermined period has elapsed since the start of storage. For example, as shown in Fig. 12, storage may start at time T0 and end at the timing (time T2) at which a predetermined period TB3 has elapsed since time T0. By storing images for a predetermined period (TB3) before and after the occurrence of an event, the status of the ROI before and after the occurrence of the event can be visually confirmed.

[0045] The timing to end image saving may be determined based on the movement of a subject detected within the image of the region of interest (ROI). The image processing unit 134 of the image characteristic modification unit 132 performs image processing to detect the presence or absence of movement of the subject by comparing the position of the subject within the image of the region of interest whose characteristics have been modified by image processing between frames of the acquired image. Then, when the image processing unit 134 of the image characteristic modification unit 132 detects a moving subject based on the comparison between frames of the acquired image, it modifies the characteristics of the region of interest that has been moved to match the movement of the subject based on image information within the moved region of interest.

[0046] The image processing unit 134 may detect a moving object (e.g., a human) as a target of interest from the set ROI, and determine that the sound generation event has ended when the detected object stops moving. The storage unit 135 may stop saving the image at the timing (e.g., T2 in FIG. 12) when the image processing unit 134 determines that the event has ended. At this time, the image processing unit 134 may move the position of the ROI set based on the sound source position in accordance with the movement of the object (e.g., a human). For example, the position of the ROI may be moved so that the moving object is located at the center of the internal region of the ROI, thereby making the target of interest (moving object) always easy to see. Furthermore, when sound is generated from a moving object, the attention region setting unit 131 may set the ROI based on the sound source position and move the position of the ROI to track the sound emitted from the object.

[0047] When displaying on the display unit 140, the control unit 130 performs display control to superimpose a region of interest (ROI) that has been moved in accordance with the movement of the subject on the image acquired by the image acquisition unit 110 (acquired image) and display it on the display unit 140.

[0048] The storage unit 135 may store an image (video or still image) of the ROI that has moved in accordance with the movement of the subject through processing by the image processing unit 134 or the region-of-interest setting unit 131. The storage unit 135 may determine that an event has occurred when sound is acquired by the sound acquisition unit 120 (e.g., T1 in FIG. 12 ), store the image acquired by the image acquisition unit 110, determine that an event is occurring while the sound continues, and end the storage when the sound is no longer detected (e.g., T2 in FIG. 12 ). By using the detection of sound associated with the occurrence of an event as a trigger for saving images, the amount of data stored in the storage unit 135 can be reduced. Furthermore, compared to saving images without using sound detection as a trigger, using sound detection as a trigger for saving images can reduce the load of searching for the situation in which the event occurred, and enable quick confirmation of images showing the situation in which the event occurred. A timestamp may be recorded together with the image to determine the time when the event occurred.

[0049] The video captured by the image acquisition unit 110 may be constantly stored in the storage unit 135. In this case, identification information (tag) that can identify the time when the sound was acquired by the sound acquisition unit 120 may be stored in the storage unit 135. When an event occurs, an image search based on the identification information may be performed to make it easier to find the image frame at the time of the event occurrence from among the acquired videos.

[0050] When detecting sound by the sound acquisition unit 120, the control unit 130 may detect only a specific sound signal (sound information) from the sound signal (sound information) acquired from the sound acquisition unit 120. For example, a specific sound may be discriminated by the amplitude or frequency components of the sound signal, or a specific sound may be learned and discriminated using a neural network, such as deep learning. Information about an event may be increased by saving the sound at the same time as saving the image. These processes may be executed as part of the control unit 130.

[0051] For reference and comparison, the storage unit 135 may store the original image output from the image acquisition unit 110 together with the image processed by the control unit 130. For example, since the original image is not specialized for the image quality of the sound source position, it may be easier to see the entire image, which is useful when an event is occurring at a position other than the sound source position.

[0052] The storage unit 135 may store various adjustment parameters used by the image processing unit 134, and when the image processing unit 134 executes image processing, the various adjustment parameters stored in the storage unit 135 may be referenced. In addition, the storage unit 135 may also function as a work area used when the control unit 130 executes various processes.

[0053] (Display section 140) The display unit 140 displays an image in which the ROI has been made easier to see by the image characteristic modification unit 132. Fig. 5 is a diagram showing an example of an image displayed on the display unit 140 of the first embodiment. Fig. 5(a) shows an example of an image displayed in which the characteristics of the entire image have been modified based on image information within the ROI. In the display example of Fig. 5(a), the darkness near the sound source position has been eliminated, and the entire image has been processed so that the state in which the glass has been broken with the hammer can be confirmed. Fig. 5(b) shows an example of an image displayed in which the image characteristics of only the internal region of the ROI image have been modified based on image information within the ROI. In the display example of Fig. 5(b), the region near the sound source position is displayed in white so that the set ROI can be clearly seen, and image processing has been performed to make the ROI easier to see.

[0054] The image processing unit 134 of the image characteristic changing unit 132 can perform image processing to cut out an image of the ROI from the entire image (acquired image) acquired by the image acquiring unit 110 and enlarge it. Specifically, the image processing unit 134 performs image processing to enlarge the image of the ROI cut out from the acquired image and change the characteristics of the enlarged image of the ROI based on the image information in the ROI. Fig. 5(c) shows a display example of an enlarged image 501 in which the ROI is cut out from the state in which the ROI is set at the sound source position (Fig. 4(b)), and the image characteristics in the cut-out ROI are changed to clarify the internal region of the ROI. Note that the enlarged display of the ROI image may be other than this display example, for example, as shown in Fig. 5(b), in which an image of the ROI with changed characteristics is cut out from the acquired image and enlarged.

[0055] In FIGS. 5(a) and 5(b), the image of the ROI set in the monitored object 100 can be made more visible by the processing of the image processing unit 134, and the image of the ROI can be acquired with high accuracy.

[0056] In the display example of FIG. 5(c), there is an effect that the image of the clarified ROI is enlarged to make it easier to specifically check the situation near the sound source. The display unit 140 may display only one of these displays, or may display multiple displays. The size of the enlarged ROI may be set in advance, or the user can set or change the size using an input device (not shown). When multiple sound sources are detected within the monitored object 100, it is possible to display images of multiple ROIs set based on the positions of the multiple sound sources on the display unit 140.

[0057] Furthermore, the display unit 140 may display the original image (captured image) that has not been subjected to image processing and that has been output from the image acquisition unit 110, together with the image that has been subjected to image processing by the image processing unit 134. By displaying a plurality of images in this manner, it is possible to increase the amount of information provided to the user. Furthermore, the storage unit 135 may store images not only in one display method but also for each of these plurality of display methods.

[0058] (Processing flow) 6 is a diagram illustrating the flow of processing in the image processing device 10 of the first embodiment. The processing flow of this embodiment will be described using an example in which moving image capturing is performed by the image acquisition unit 110.

[0059] In step S110, the control unit 130 determines whether a sound signal has been detected by the sound acquisition unit 120. If the sound associated with a specific event has been detected by the sound acquisition unit 120 (S110-YES), the control unit 130 advances the process to S120.

[0060] In step S120, the attention area setting unit 131 identifies the sound source position or sound source direction using the sound signal (sound information) acquired from the sound acquisition unit 120, and sets an ROI in the image captured by the image acquisition unit 110 based on the identified sound source position or sound source direction.

[0061] In step S130, the image characteristic change unit 132 changes the characteristics of the image captured by the image acquisition unit 110 based on the image information of the ROI set in step S120.

[0062] In step S140, storage unit 135 stores the image whose characteristics have been changed by the processing of step S130. In addition to storing the image whose characteristics have been changed, control unit 130 may cause display unit 140 to display the image whose characteristics have been changed by the processing of step S130 under display control of control unit 130. Here, a display example of the image displayed by display unit 140 is as shown in FIG. 5.

[0063] In step S150, the image acquisition unit 110 captures the next frame image. Since the image frame captured by the image acquisition unit 110 and the sound frame captured by the sound acquisition unit 120 do not necessarily coincide, the image processing and the sound processing may be executed in parallel, and information may be exchanged between them asynchronously.

[0064] In step S160, control unit 130 determines whether the sound-related event detected in step S110 is continuing. If the sound-related event is continuing (S160-YES), control unit 130 returns the process to step S140 and repeats the same process. On the other hand, if it is determined in step S160 that the sound-related event is not continuing (S160-NO), control unit 130 returns the process to step S110.

[0065] If it is determined in step S110 that sound associated with a specific event has not been detected by sound acquisition section 120 (S110-NO), control section 130 advances the process to step S170.

[0066] In step S170, the storage unit 135 stores the image captured by the image acquisition unit 110. Furthermore, the display unit 140 displays the image captured by the image acquisition unit 110. Note that, since no sound-related event to be monitored has occurred in this step, this step may be omitted. In an application in which a user monitors the monitoring target 100, this step may involve only displaying the image captured by the image acquisition unit 110.

[0067] In step S180, similar to step S150 described above, the image acquisition unit 110 captures an image of the next frame, and the process returns to step S110. After step S110, the same process is repeatedly executed. Note that although not shown in FIG. 6, the image capture may be terminated at any timing in response to a user instruction or the like.

[0068] According to this embodiment, an ROI is set based on the position and direction of a sound source identified from the sound, and the image characteristics are changed based on the image information within the set ROI, thereby making it possible to obtain an image of the ROI with excellent image quality.

[0069] Second Embodiment In the second embodiment, a configuration is described in which, when a camera with a wide-angle lens is used, image distortion is corrected as information indicating the characteristics of the image based on image information of an ROI, which is a region of interest set at a specified sound source position.

[0070] <Overall configuration of image processing device> 7 is a diagram showing an example of the configuration of an image processing device 20 according to the second embodiment. The image processing device 20 of this embodiment can function as a device for monitoring a monitoring target. The same components as those of the image processing device 10 in the first embodiment are denoted by the same reference numerals, and duplicated descriptions will be omitted in part or in whole.

[0071] 8 is a block diagram showing the configuration of an image processing device 20 of the device of the second embodiment. In the image acquisition unit 210, the imaging unit 111 of the first embodiment is replaced with an imaging unit 211 that is a wide-angle lens with a wide angle of view. The image acquisition unit 210 is attached to the ceiling of the monitored object 100 (room) and is configured to be able to capture an image of the entire monitored object 100.

[0072] The control unit 230 controls each component of the image processing device 20 and processes the image signal (image information) acquired by the image acquisition unit 210 and the sound signal (sound information) acquired by the sound acquisition unit 120. The control unit 230 has, as functional components, a region of interest setting unit 231, an image characteristic changing unit 232 (an imaging control unit 133, an image processing unit 234), and a storage unit 135. The control unit 230 can be configured with a workstation, a personal computer, a tablet PC, a smartphone, a server, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a microcomputer, or the like. All or part of the functional components of the control unit 230 may be provided in the image acquisition unit 210.

[0073] 8, the image acquisition unit 210, the sound acquisition unit 120, the control unit 230, and the display unit 240 are connected to each other so that they can communicate with each other via the bus 150. The configuration of the control unit 230 will be specifically described with reference to FIG.

[0074] Similar to the attention area setting unit 131 in the first embodiment, the attention area setting unit 231 uses the sound signal (sound information) acquired from the sound acquisition unit 120 to identify the sound source position or sound source direction in the monitored object 100, and sets an ROI in the image (acquired image) acquired by the image acquisition unit 210 based on the identified sound source position or sound source direction.

[0075] FIG. 9 is a diagram illustrating an effect of the technology of the second embodiment. FIG. 9(a) shows the position of a sound source 101 in an image captured when sound is generated. The position of the sound source 101 is not actually displayed in the image, but is shown for the purpose of explanation. FIG. 9(b) shows an ROI set around the identified position of the sound source 101 in the captured image. FIG. 9(c) shows the image after distortion correction. As shown in FIG. 9(b), the attention area setting unit 231 sets an ROI with a shape such that the shape of the image after distortion correction by the image processing unit 234 becomes rectangular. The shape of the ROI indicated by the dashed line is converted based on position information on the image, so an appropriate ROI shape can be set by using information on the position of the sound source 101. By making the shape of the image after distortion correction rectangular, for example, as shown in FIG. 9(c), it is possible to provide an image that is easy to observe and less uncomfortable for the user.

[0076] (Image characteristic change unit 232) The image characteristic change unit 232 has an imaging control unit 133 and / or an image processing unit 234. The imaging control unit 133 changes the imaging conditions of the imaging unit 112 and the imaging unit 211 based on the image information of the ROI set by the attention area setting unit 231 so that the user can easily check (make it easier to see) the situation within the ROI.

[0077] The image processing unit 234 corrects image distortion included in the image of the region of interest (ROI) based on position information of the region of interest (ROI) in the acquired image acquired from the image acquisition unit 210 and image information within the region of interest (ROI). Specifically, the image processing unit 234 corrects image distortion within the ROI that may occur when capturing an image using the imaging unit 211 (wide-angle lens) with a wide angle of view, based on the position information of the ROI set by the region of interest setting unit 231 and image information within the ROI. The image processing unit 234 may correct image distortion included in the image of the ROI using a coefficient representing distortion of the image captured by the image acquisition unit 210. Alternatively, the image processing unit 234 may correct image distortion within the ROI so that an object that appears curved on the image but should actually be straight is displayed as a straight line. In the latter case, a neural network, typified by deep learning, may be used.

[0078] In FIG. 9(a), since the image was captured using a wide-angle lens, the boundary between the floor and the wall, which is a straight line, is displayed as a curved line centered on the position of the image capture unit 210. As shown in FIG. 9(a), the image captured by the image capture unit 210 is acquired as a significantly distorted image (omnidirectional image). In particular, since the hammer that broke the glass is distorted, it may be difficult to determine from the distorted image what caused the glass to break. Furthermore, if a person (not shown) is captured as a subject in the image of the ROI, the person will also be displayed as significantly distorted, making it difficult to determine the person's face or stature from the distorted image.

[0079] The image processing unit 134 corrects image distortion contained in the ROI image and generates a distortion-corrected image cut out from the image (omnidirectional image) captured by the image acquisition unit 210. In the distortion-corrected image shown in FIG. 9(c), the distortion of the image near the sound source has been corrected, making it easier to see. In particular, in the distortion-corrected image, the hammer that broke the glass is displayed with less distortion, making it easier to visually identify the cause of the sound event. If a person (not shown) is captured as a subject in the ROI image, the distortion of the person's image is reduced in the distortion-corrected image, making it easier to distinguish the person's face and stature.

[0080] The control unit 230 performs display control to display on the display unit 240 an image in which the ROI has been made easier to see through the processing of the image characteristic changing unit 232. Fig. 10 is a diagram showing an example of an image displayed on the display unit 240 of the second embodiment. An image 1020 in Fig. 10(a) is a diagram showing a display example in which an image within the ROI in which distortion has been corrected is cut out and displayed, and is a rectangular image cut out from the omnidirectional image 1010. Fig. 10(b) is a diagram showing a display example of a panoramic image 1030 converted from the omnidirectional image 1010, and is a display example in which a frame display of the ROI is superimposed on the position of the sound source.

[0081] In this embodiment, the image processing unit 234 may further improve the visibility of the image by changing the image characteristics described in the first embodiment. For example, the image processing unit 234 may perform image processing on the ROI image after distortion correction by changing various adjustment parameters (e.g., brightness, contrast, gamma, saturation, hue, etc.). Alternatively, the image processing unit 234 may perform image processing such as emphasizing the subject in the ROI image, or may perform image processing such as edge enhancement, noise removal, super-resolution, contour extraction of a person, etc., based on image information in the ROI image after distortion correction, or frame display of the ROI based on position information of the ROI. The various image processing by the image processing unit 234 may be performed only on the internal region of the ROI based on the image information of the ROI set by the region-of-interest setting unit 231, or may be performed on the entire image including the internal region of the ROI and the external region of the ROI.

[0082] 10(b), the image processing unit 234 may divide the image (omnidirectional image) captured by the image acquisition unit 210 into N divisions (N is an integer equal to or greater than 2), and generate corrected divided images (1030-1 to 1030-4) by correcting the image distortion for each division. For example, when the imaging area converted from the omnidirectional image 1010 is large, performing distortion correction on the entire omnidirectional image using the same correction parameters may result in large errors. In such a case, by dividing the omnidirectional image into regions where the distortion correction error is small using the same correction parameters, a high-precision panoramic image 1030 with reduced distortion correction errors can be generated. The number of divisions into which the captured image (omnidirectional image) is divided may be set so that the distortion error is equal to or less than a predetermined value. The display example shown in FIG. 10(b) illustrates an example in which the omnidirectional image 1010 is divided into four divided images.

[0083] In generating the panoramic image 1030, the image processing unit 234 may determine the order in which the corrected segmented images are to be joined together, and may align the corrected segmented images and join the images according to the determined order to convert the omnidirectional image 1010 into the panoramic image 1030. The image processing unit 234 may also perform the image processing described above on the panoramic image 1030. The various image processing operations performed by the image processing unit 234 may be performed only on the internal region of the ROI based on image information of the ROI, or may be performed on the entire image including the internal region of the ROI and the external region of the ROI. For example, the image processing unit 234 may perform image processing by changing various adjustment parameters. Alternatively, the image processing unit 234 may perform image processing such as emphasizing the subject in the ROI image, or image processing such as edge enhancement, noise removal, super-resolution, contour extraction of a person, etc., based on image information in the ROI image after distortion correction, or frame display of the ROI based on position information of the ROI. The image processing unit 234 may align the corrected divided images based on the magnitude of correlation of the feature information after changing various adjustment parameters (for example, brightness, contrast, gamma, saturation, hue, etc.).

[0084] 10(a), by displaying the image within the ROI as a rectangular image, the user can instantly check the situation near the sound source. When displaying the image 1020 on the display unit 240, the control unit 230 may perform display control so as to also display an original image in which the ROI is superimposed on the omnidirectional image 1010. By displaying the original image in which the ROI is superimposed (omnidirectional image 1010) and the cropped rectangular image (image 1020) on the display unit 240, the user can easily check where in the monitored object 100 (room) the event that generated the sound occurred.

[0085] In the display example of the panoramic image 1030 shown in FIG. 10(b), images 1030-1, 1030-2, 1030-3, and 1030-4 are divided images obtained by dividing the omnidirectional image 1010. In the panoramic image 1030, which has a larger imaging area converted from the omnidirectional image 1010, it can be time-consuming to find the location of an event. Even in the panoramic image 1030 with a larger imaging area, the user can immediately confirm the location of the event by superimposing a frame display of the ROI on the panoramic image 1030. The internal area of ​​the ROI may be clarified and enlarged by changing the image characteristics within the extracted ROI, making it easier to confirm the status of the sound event. The panoramic conversion process to the panoramic image 1030 can be executed as a function of the control unit 230. By displaying images as shown in FIGS. 10(a) and 10(b), it is possible to provide more information to the user. Furthermore, the storage unit 135 may store images for not only one display method but also multiple display methods.

[0086] (Processing flow) 11 is a diagram illustrating the flow of processing in the image processing device 20 according to the second embodiment. The processing flow according to this embodiment will be described using an example in which moving image capturing is performed by the image acquisition unit 210.

[0087] In step S210, control unit 230 determines whether a sound signal has been detected by sound acquisition unit 120. If control unit 130 detects a sound associated with a specific event from the sound signal of sound acquisition unit 120 (S210-YES), control unit 130 proceeds to step S220.

[0088] In step S220, the attention area setting unit 231 identifies the sound source position or sound source direction using the sound signal (sound information) acquired from the sound acquisition unit 120, and sets an ROI in the image captured by the image acquisition unit 210 based on the identified sound source position or sound source direction.

[0089] In step S230, based on the image information of the ROI set in step S220, the image characteristic change unit 232 corrects distortion of the image captured by the image acquisition unit 210. In this step, the image processing unit 234 in the image characteristic change unit 232 corrects distortion of the image within the ROI and generates a distortion-corrected image cut out from the image (omnidirectional image) captured by the image acquisition unit 210.

[0090] In step S240, the storage unit 135 stores the distortion-corrected image corrected by the processing in step S230. In addition to storing the distortion-corrected image, the control unit 230 may cause the display unit 240 to display the distortion-corrected image. The display unit 240 displays the distortion-corrected image by the processing in step S230 under display control of the control unit 230. Here, display examples of the image displayed by the display unit 240 are as shown in FIGS. 10(a) and 10(b).

[0091] In step S250, the image acquisition unit 210 captures the next frame image. Since the image frame captured by the image acquisition unit 210 and the sound frame captured by the sound acquisition unit 120 do not necessarily coincide, the image processing and the sound processing may be executed in parallel, and information may be exchanged between them asynchronously.

[0092] In step S260, control unit 230 determines whether the sound-related event detected in step S210 is continuing. If the sound-related event is continuing (S260-YES), control unit 230 returns the process to step S240 and repeats the same process. On the other hand, if it is determined in step S260 that the sound-related event is not continuing (S260-NO), control unit 230 returns the process to step S210.

[0093] If it is determined in step S210 that no sound is detected (S210-NO), control unit 230 advances the process to step S270.

[0094] In step S270, the storage unit 135 stores the image captured by the image acquisition unit 210. Furthermore, the display unit 240 displays the image captured by the image acquisition unit 210. Note that, since no sound-related event to be monitored has occurred in this step, this step may be omitted. In an application in which a user monitors the monitoring target 100, only the image captured by the image acquisition unit 210 may be displayed in this step.

[0095] In step S280, similar to step S250 described above, the image acquisition unit 210 captures an image of the next frame, and the process returns to step S210. After step S210, the same process is repeatedly executed. Note that although not shown in FIG. 11, the image capturing may be terminated at any timing in response to a user instruction or the like.

[0096] According to this embodiment, when a camera with a wide-angle lens is used, an ROI is set based on the position of the sound source identified from the sound, and image distortion, which is a characteristic of the ROI image, is corrected based on the image information within the set ROI, thereby making it possible to obtain an image of the ROI with excellent image quality.

[0097] In the first and second embodiments, examples of devices (e.g., surveillance cameras) that monitor a monitoring target have been described above. However, the disclosed technology can be applied to any applications other than surveillance cameras. For example, when capturing images and sounds using a smartphone, wearable camera, vlog camera, or the like to capture an image of a subject such as a person or an animal, the disclosed technology may be applied using a speaker's speech or cries as the sound source. An ROI may be set based on the position or direction of a sound source identified from the sound, such as a speaker's speech or cries, and the image characteristics may be changed based on image information within the set ROI to obtain an ROI image with high image quality.

[0098] (Summary of the embodiment) [Item 1] a setting unit that sets an attention area including the position or direction of the sound source in an acquired image acquired by an image acquisition unit based on the position or direction of the sound source identified using a sound signal acquired from a sound acquisition unit; a modification unit that modifies characteristics of the image in the region of interest by image processing based on image information in the region of interest; An image processing device comprising: [Item 2] 2. The image processing device according to item 1, wherein the change unit changes the characteristics of the acquired image by image processing based on image information within the region of interest. [Item 3] The image processing device described in item 2 is characterized in that the modification unit modifies the characteristics of the image of the region of interest or the characteristics of the acquired image by modifying at least one parameter of brightness, contrast, gamma, saturation, and hue in the image processing. [Item 4] 2. The image processing device according to item 1, wherein the change unit performs image processing to emphasize the subject in the image of the region of interest whose characteristics have been changed, based on image information within the region of interest. [Item 5] 5. The image processing device according to any one of items 1 to 4, wherein the modification unit corrects image distortion contained in the image of the region of interest based on position information of the region of interest in the acquired image and image information within the region of interest. [Item 6] the change unit enlarges the image of the region of interest cut out from the acquired image, 6. The image processing device according to any one of items 1 to 5, wherein image processing is performed to change characteristics of the image of the enlarged region of interest based on image information within the region of interest. [Item 7] 6. The image processing device according to any one of items 1 to 5, wherein the change unit performs image processing to cut out an image of the region of interest whose characteristics have been changed from the acquired image and enlarge it. [Item 8] a storage unit that stores the acquired image and an image of the region of interest; 8. The image processing device according to any one of items 1 to 7, wherein the storage unit stores the acquired image acquired from the image acquisition unit and an image of the region of interest set in the acquired image for a predetermined period before and after the time when the sound signal is detected. [Item 9] Further, a control unit that controls the display of the display unit is provided, Item 9. The image processing device according to item 8, wherein the control unit performs display control to display the acquired image and the image of the region of interest stored in the storage unit on a display unit. [Item 10] the setting unit sets a partial region in the acquired image that includes the position of the sound source or the direction of the sound source as the region of interest; Item 10. The image processing device according to item 9, wherein the control unit performs display control to display a frame indicating the outline of the region of interest on the display unit in a manner superimposed on the acquired image. [Item 11] Item 11. The image processing device according to item 10, characterized in that the change unit performs image processing to detect whether or not the subject is moving by comparing the position of the subject in the image of the region of interest whose characteristics have been changed between frames of the acquired image. [Item 12] Item 12. The image processing device according to item 11, wherein the modification unit, when detecting the subject having movement based on the comparison, modifies the characteristics of the attention area that has been moved in accordance with the movement of the subject based on image information within the moved attention area. [Item 13] Item 13. The image processing device according to item 12, wherein the control unit performs display control to superimpose the attention area, which has been moved in accordance with the movement of the subject, on the acquired image and display it on the display unit. [Item 14] 14. The image processing device according to any one of items 1 to 13, characterized in that the modification unit modifies the characteristics of the image of the region of interest by the image processing that modifies the imaging conditions of the image acquisition unit so as to adjust the image information to within the appropriate range when the image information in the region of interest becomes higher than the upper limit of an appropriate range of image information or when the image information in the region of interest becomes lower than the lower limit of the appropriate range. [Item 15] Item 15. The image processing device according to item 14, wherein the change unit changes the characteristics of the image of the region of interest by the image processing in which at least one of the aperture diameter, focus, and focal length of the imaging unit of the image acquisition unit is changed as the imaging condition. [Item 16] Item 15. The image processing device according to item 14, wherein the change unit changes the characteristics of the image of the region of interest by the image processing in which the exposure time or gain of the imaging unit of the image acquisition unit is changed as the imaging condition. [Item 17] The image processing device described in item 14 is characterized in that the change unit changes the characteristics of the image of the region of interest by image processing in which at least one parameter of brightness, contrast, gamma, saturation, and hue is changed as an imaging condition of the image acquisition unit. [Item 18] a setting step of setting a region of interest including the position or direction of the sound source in an acquired image acquired by an image acquisition unit, based on the position or direction of the sound source identified using a sound signal acquired from a sound acquisition unit; a modifying step of modifying characteristics of the image of the region of interest based on image information within the region of interest; An image processing method comprising: [Item 19] Item 19. A program for causing a computer to execute the image processing method according to Item 18.

[0099] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0100] 10, 20: Image processing device, 110, 210: Image acquisition unit, 120: Sound acquisition unit, 130, 230: Control unit, 131, 231: Attention area setting unit, 132, 232: Image characteristic change unit, 133: Imaging control unit, 134, 234: Image processing unit, 135: Storage unit, 140, 240: Display unit

Claims

1. a setting unit that sets an attention area including the position or direction of the sound source in an acquired image acquired by an image acquisition unit based on the position or direction of the sound source identified using a sound signal acquired from a sound acquisition unit; a modification unit that modifies characteristics of the image in the region of interest by image processing based on image information in the region of interest; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the change unit changes the characteristics of the acquired image by image processing based on image information within the region of interest.

3. 3. The image processing device according to claim 2, wherein the modification unit modifies the image characteristics of the region of interest or the characteristics of the acquired image by modifying at least one parameter of brightness, contrast, gamma, saturation, and hue during the image processing.

4. The image processing device according to claim 1 , wherein the modification unit performs image processing for emphasizing the subject in the image of the region of interest whose characteristics have been modified, based on image information within the region of interest.

5. The image processing device according to claim 1 , wherein the modification unit corrects image distortion contained in the image of the region of interest based on position information of the region of interest in the acquired image and image information within the region of interest.

6. the change unit enlarges the image of the region of interest cut out from the acquired image, 2. The image processing apparatus according to claim 1, wherein image processing is performed to change characteristics of the image in the enlarged region of interest based on image information within the region of interest.

7. The image processing device according to claim 1 , wherein the change unit performs image processing to cut out the image of the region of interest whose characteristics have been changed from the acquired image and enlarge it.

8. a storage unit that stores the acquired image and an image of the region of interest; The image processing device according to claim 1, characterized in that the storage unit stores the acquired image acquired from the image acquisition unit and an image of the region of interest set in the acquired image for a predetermined period before and after the time when the sound signal is detected.

9. Further, a control unit that controls the display of the display unit is provided, The image processing apparatus according to claim 8 , wherein the control unit performs display control to display the acquired image and the image of the region of interest stored in the storage unit on a display unit.

10. the setting unit sets an image of a partial region in the acquired image that includes the position of the sound source or the direction of the sound source as the image of the attention region; The image processing device according to claim 9 , wherein the control unit performs display control to display a frame indicating the outline of the image of the region of interest on the display unit in a manner superimposed on the acquired image.

11. The image processing device according to claim 10, characterized in that the modification unit performs image processing to detect whether or not the subject is moving by comparing the position of the subject in the image of the region of interest whose characteristics have been modified between frames of the acquired image.

12. The image processing device according to claim 11, characterized in that, when the subject having movement is detected based on the comparison, the modification unit modifies the characteristics of the attention area that has been moved in accordance with the movement of the subject based on image information within the moved attention area.

13. The image processing device according to claim 12 , wherein the control unit performs display control to superimpose the attention area, which has been moved in accordance with the movement of the subject, on the acquired image and display it on the display unit.

14. The image processing device described in claim 1, characterized in that the modification unit changes the characteristics of the image of the area of ​​interest by image processing that changes the imaging conditions of the image acquisition unit so as to adjust the image information within the appropriate range when the image information within the area of ​​interest becomes higher than the upper limit of the appropriate range of image information or lower than the lower limit of the appropriate range.

15. The image processing device according to claim 14, characterized in that the change unit changes the characteristics of the image of the region of interest by the image processing in which at least one of the aperture diameter, focus, and focal length of the imaging unit of the image acquisition unit is changed as the imaging condition.

16. The image processing device according to claim 14 , wherein the change unit changes characteristics of the image of the region of interest by the image processing in which an exposure time or a gain of the image capturing unit of the image acquisition unit is changed as the image capturing condition.

17. The image processing device according to claim 14, characterized in that the change unit changes the characteristics of the image of the region of interest by the image processing in which at least one parameter of brightness, contrast, gamma, saturation, and hue is changed as an imaging condition of the image acquisition unit.

18. a setting step of setting a region of interest including the position or direction of the sound source in an acquired image acquired by an image acquisition unit, based on the position or direction of the sound source identified using a sound signal acquired from a sound acquisition unit; a modifying step of modifying characteristics of the image of the region of interest based on image information within the region of interest; An image processing method comprising:

19. A program for causing a computer to execute the image processing method according to claim 18.

Images