Imaging system, display method, and program

Abstract

The imaging system disclosed herein includes an image acquisition unit, a recognition unit, a speech acquisition unit, a speech analysis unit, and a display control unit. The image acquisition unit acquires image data including objects that are speaking subjects. The recognition unit identifies objects included in the image data. The speech acquisition unit acquires speech data related to the objects that are speaking subjects. The speech analysis unit segments the speech data into multiple speech data segments. Based on the recognition results of the recognition unit and the segmentation results of the speech analysis unit, the display control unit displays information about the speech data emitted by the objects that are speaking subjects, correlated with the image data.

Description

Technical Field

[0001] This disclosure relates to imaging systems, display methods, and procedures. Background Technology

[0002] Capturing moving images with a camera includes simultaneously collecting ambient sound via a microphone and recording the collected sound using the captured moving images. During moving image capture, as a method to confirm whether target sound is being recorded appropriately, a method is known where the photographer wears headphones to confirm the camera's monitoring sound during recording by listening to it. Furthermore, a method is known to confirm sound by viewing an audio level meter that indicates the volume during recording. An audio level meter is a visually measuring display that indicates the instantaneous level of an audio signal and typically indicates the peak volume of the sound data in dB (decibels).

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: JP 2009-118318 A Summary of the Invention

[0006] Technical issues

[0007] However, monitoring sound by ear requires advanced skills, and this becomes even more difficult when the cameraman performs sound monitoring independently, alongside video monitoring. Furthermore, the volume that can be confirmed by an audio level meter is the volume of the entire sound and cannot pinpoint the volume of each individual sound source. For example, when recording in noisy environments such as construction sites, it is impossible to determine whether fluctuations in the meter's level are caused by ambient noise or by the target sound being recorded (e.g., a voice), which may not reach the appropriate level.

[0008] In view of this, the present disclosure proposes an imaging system, display method, and procedure capable of measuring the volume of the voice of a target object contained in collected sounds.

[0009] Solution to the problem

[0010] To address the aforementioned problems, the imaging system according to this disclosure includes an image acquisition unit, a recognition unit, a sound acquisition unit, a sound analysis unit, and a display control unit. The image acquisition unit acquires image data including the target object speaking. The recognition unit identifies the object included in the image data. The sound acquisition unit acquires sound data related to the object. The sound analysis unit performs sound source separation on the sound data to obtain multiple sound data streams. Based on the recognition result obtained by the recognition unit and the sound source separation result obtained by the sound analysis unit, the display control unit displays information related to the sound data generated based on the target object speaking, in association with the image data, on a display unit. Attached Figure Description

[0011] Figure 1 This is a functional block diagram illustrating an example of the functional configuration of an imaging apparatus according to an embodiment.

[0012] Figure 2 This is a functional block diagram illustrating a functional configuration example of sound source separation processing via a sound analysis unit according to this embodiment.

[0013] Figure 3 This is a block diagram used to describe an example of sound source separation by a sound analysis unit according to an embodiment.

[0014] Figure 4 This is a diagram showing an example of a display unit according to an embodiment.

[0015] Figure 5 This is a flowchart illustrating an example of the processing procedure of an imaging apparatus according to an embodiment.

[0016] Figure 6 This is a diagram used to describe the gain adjustment method based on the comparison example.

[0017] Figure 7 This is a diagram used to describe the gain adjustment method according to the implementation method.

[0018] Figure 8 This is a flowchart illustrating an example of the processing procedure of an imaging apparatus according to an embodiment.

[0019] Figure 9 This is a flowchart illustrating an example of the processing procedure of the imaging apparatus according to the first modified example.

[0020] Figure 10 This is a flowchart illustrating an example of the processing procedure of the imaging apparatus according to the second variation.

[0021] Figure 11A This is a diagram showing a display example of the display unit according to the third modified example.

[0022] Figure 11B This is a diagram showing a display example of the display unit according to the third modified example.

[0023] Figure 12 This is a diagram showing a configuration example of an imaging system according to the fifth modification.

[0024] Figure 13 This is a functional block diagram illustrating an example of the functional configuration of the monitoring device according to the fifth modification.

[0025] Figure 14 This is a hardware configuration diagram illustrating an example of a computer that implements the functions of an imaging device. Detailed Implementation

[0026] In the following description, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same components are indicated by the same reference numerals to avoid redundant description. Furthermore, in the following description, examples are given of the application of the technology of the present disclosure to an imaging device 10 such as a camera.

[0027] Furthermore, this disclosure is described in the order of the items shown below.

[0028] 1. Configuration of the imaging device

[0029] 2. Processing procedure

[0030] 3. Specific examples of imaging

[0031] 3-1. Gain Adjustment Method in the Comparative Example

[0032] 3-2. Gain Adjustment Method in the Implementation Scheme

[0033] 4. Processing procedure

[0034] 5. Variations

[0035] 5-1. First Variation Example

[0036] 5-1-1. Processing Procedure

[0037] 5-2. Second variation

[0038] 5-2-1. Processing Procedure

[0039] 5-3. Third variation example

[0040] 5-4. Fourth Variation Example

[0041] 5-5. Fifth Variation

[0042] 5-5-1. Configuration of the Imaging System

[0043] 5-5-2. Configuration of Monitoring Devices

[0044] 6. Hardware Configuration

[0045] 7. Conclusion

[0046] 1. Configuration of the imaging device

[0047] Figure 1 This is a functional block diagram illustrating an example of the functional configuration of the imaging apparatus 10 according to an embodiment. It should be noted that... Figure 1 This document only shows the components necessary for describing the features of the embodiments, and omits descriptions of common components. In other words, Figure 1 The components shown are illustrated as functional concepts and do not necessarily have to be physically configured as shown in the figures. For example, specific aspects of the distribution and integration of blocks are not limited to those shown in the figures. All or some blocks may be configured to be functionally or physically distributed or integrated in any unit according to various types of loads, usage conditions, etc.

[0048] Imaging device 10 is, for example, a device capable of capturing images, such as a mirrorless camera, a digital single-lens reflex camera, or a video camera. It should be noted that imaging device 10 can also be a portable information processing device with camera functionality, such as a smartphone.

[0049] like Figure 1 As shown, the imaging device 10 includes a display unit 20, a motion image recording unit 21, an optical unit 22, an imaging unit 23, an image acquisition unit 24, a subject recognition unit 25, a sound acquisition unit 26, a sound analysis unit 27, a display control unit 28, and an operation control unit 29.

[0050] The display unit 20 is, for example, a display provided to the imaging device 10.

[0051] An imaging device 10 has a card slot, in which a memory card is disposed. A motion picture recording unit 21 reads data stored in the memory card and writes data back to the memory card. The memory card is a storage medium, for example, containing a non-volatile semiconductor memory such as flash memory, and capable of rewriting data. The memory card retains the written data. Examples of memory cards include secure digital memory cards (SD cards).

[0052] The optical unit 22 is, for example, an imaging lens disposed in the imaging device 10. The optical unit 22 includes a focusing lens, a lens mechanism for moving the focusing lens, a shutter mechanism for driving the optical shutter, and an aperture mechanism for driving the aperture for adjusting the amount of light passing through the lens. The optical unit 22 adjusts the focus by moving the focusing lens through the lens mechanism and converges the incident light.

[0053] Imaging unit 23 is equipped with an image sensor. The image sensor is arranged such that its optical axis is aligned with the optical axis of optical unit 22. The image converged by optical unit 22 is formed on the image sensor. Imaging unit 23 captures the image through the image sensor and generates image data of the captured image. For example, imaging unit 23 converts the light focused on the image sensor into an electrical signal, outputs the electrical signal, converts and processes the electrical signal into a digital image signal, and generates digital image data.

[0054] The image acquisition unit 24 controls the imaging unit 23 and acquires image data through the imaging unit 23. The image acquisition unit 24 acquires image data including the target object being spoken through the imaging unit 23. When displaying a live view image on the display unit 20 or when capturing moving images, the image acquisition unit 24 controls the imaging unit 23 to capture images sequentially at a predetermined frame rate and sequentially acquires the image data of the captured images.

[0055] The subject recognition unit 25 performs image recognition on the image data acquired by the image acquisition unit 24, and identifies objects contained in the image data. For example, the subject recognition unit 25 performs image recognition on the image data and identifies the subject, subject parts, and scene as objects. Any method can be used as the image recognition method. For example, the subject recognition unit 25 identifies the subject, subject parts, and scene by using a recognition model trained on the subject, subject parts, and scene using artificial intelligence (AI) technology.

[0056] Imaging device 10 includes a microphone. Sound acquisition unit 26 controls the microphone to acquire and process ambient sound, and generates digital sound data. Examples of sound data include linear PCM data.

[0057] The sound analysis unit 27 performs sound source separation processing on the sound data acquired by the sound acquisition unit 26 to generate sound data from multiple sound sources. For example, the sound analysis unit 27 uses a separation model trained for the sound to be separated to generate sound data from the acquired sound data. For example, the sound analysis unit 27 performs sound source separation processing on the sound data and generates sound data from multiple sound sources by using a processing configuration based on a neural network (e.g., DNN) as machine learning.

[0058] Figure 2 This is a functional block diagram illustrating a functional configuration example of the sound source separation processing of the sound analysis unit 27 according to this embodiment. Figure 2 The example shown is an instance of performing sound source separation using a neural network as a separation model.

[0059] The sound analysis unit 27 includes an encoder unit 30, a sequence modeling unit 31, and a decoding unit 32 as processing units.

[0060] The input signal is a time waveform of sound data, or a spectrogram obtained by performing a short-time Fourier transform on the time waveform. The encoding unit 30 is responsible for projecting the input signal into a space that can be easily processed by the sequence modeling unit 31.

[0061] The sequence modeling unit 31 is responsible for modeling the time variation of the output of the encoding unit 30. The decoding unit 32 is responsible for projecting the output of the sequence modeling unit 31 into the same space as the input signal space. Note that the encoder unit 30 and the decoding unit 32 can be omitted as needed.

[0062] The input signals and outputs of each processing unit in the sound analysis unit 27 can be used not only by the processing unit immediately following it, but also, as needed, for skip connection processing in subsequent processing units. The encoding unit 30, sequence modeling unit 31, and decoding unit 32 are implemented using a combination of fully connected layers, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. Here, low-latency processing is achieved by preventing the use of inputs after the output in time when calculating the output of each processing unit. That is, the sound analysis unit 27 is configured such that, as each processing unit calculates its output sequentially, all the data required for that calculation is input to each processing unit. Therefore, the sound analysis unit 27 achieves low-latency processing without waiting for data. For example, this is achieved by employing causal convolutions in the CNN and a unidirectional RNN in the RNN.

[0063] Here, an example of a learning method for the separation model is described. When learning the separation model, first sound data consisting only of the sounds from the sound sources to be separated and second sound data consisting of the sounds from the first sound data and the sounds from another sound source are prepared. Then, in the learning of the separation model, sound source separation processing is performed on the second sound data using a separation model that uses the first sound data as the correct answer data, and learning is performed such that the correct answer data is separated, thereby generating the separation model. A separation model is prepared in advance for each sound source to be separated.

[0064] In the learning of the separation model, various parameters of the separation model are determined by fully collecting sound data of the sound source to be separated and sound data other than the sound source to be separated, and using the collected sound data as learning data to perform machine learning. For example, by performing machine learning using sound data that only includes wind sound and sound data other than wind sound, it is possible to create a separation model configured to separate only wind sound.

[0065] The sound analysis unit 27 performs operations in series or parallel. Figure 2 The sound source separation process shown in the figure separates the sound data into sound data from multiple sound sources. Figure 3This is a block diagram used to describe an example of sound source separation performed by the sound analysis unit 27 according to this embodiment. Figure 3 In this example, speech separation is first performed on the input sound to obtain speech and other sounds besides speech. Then, wind separation and music separation are performed on the sounds besides speech to obtain wind sounds and music sounds. The sound analysis unit 27 outputs the sound data for each separated object. When the input sound does not include the sound of the separated object, the sound data becomes silence data.

[0066] Speech separation is performed by using a separation model created through pre-learning of a large amount of sound data, including human speech and other data, to separate the sound source, thus outputting speech and other sound data. Source separation of the target sound becomes possible by pre-preparing multiple separation models for the sound to be separated. For example, in performing... Figure 3 In the case of sound source separation shown, the sound analysis unit 27 includes a first separation model that separates sound data into sound data related to sound and sound data other than sound based on the sound data. Furthermore, the sound analysis unit 27 includes a second separation model that separates wind-related sound data based on sound data other than speech. Additionally, the sound analysis unit 27 further includes a third separation model that separates music-related sound data based on sound data other than speech. The sound analysis unit 27 uses the first separation model to separate sound and sounds other than sound. Then, the sound analysis unit 27 uses the second separation model to separate wind sound from sounds other than speech. Furthermore, the sound analysis unit 27 uses the third separation model to separate music sound from sounds other than speech.

[0067] Table 1

[0068] Table 1 shows examples of separated sound sources.

[0069] As the number of sound sources to be separated increases, it becomes possible to present the user with a more accurate picture of what type of sound the recorded source is. However, as the number of sound sources to be separated increases, processing time may increase, or separation accuracy may decrease.

[0070] The sound analysis unit 27 measures the volume of each sound data obtained through sound source separation. Examples of methods for measuring volume include methods for calculating PEAK values, methods for calculating root mean square (RMS) values, and methods for calculating loudness values.

[0071] The sound analysis unit 27 adjusts the volume gain of each separated sound data. For example, the sound analysis unit 27 performs an adjustment called automatic gain control (AGC) on each separated sound data to generate sound data with adjusted volume. AGC is a process that maintains the input signal at a constant level by increasing sensitivity when the input signal is weak and decreasing sensitivity when the input signal is strong.

[0072] The motion picture recording unit 21 records the image data sequentially acquired by the image acquisition unit 24 and the sound acquired by the sound acquisition unit 26 as motion picture data (e.g., MPEG-4) in a non-volatile memory (e.g., an SD card). For example, the sound analysis unit 27 adds up the sound data after volume adjustment via AGC to generate volume-balanced sound data. The motion picture recording unit 21 records motion picture data based on the image data sequentially acquired by the image acquisition unit 24 and the volume-balanced sound data.

[0073] The display control unit 28 controls the display unit 20 to display images acquired by the image acquisition unit 24. The display control unit 28 controls the display unit 20 to display images acquired sequentially by the image acquisition unit 24 at a predetermined frame rate when displaying live view images or when capturing moving images. Furthermore, the display control unit 28 controls the display unit 20 to associate information related to sound data generated based on the target object speaking, based on the recognition results obtained by the subject recognition unit 25 and the sound source separation results obtained by the sound analysis unit 27, with the image data. For example, the display control unit 28 causes the display unit 20 to display information representing the volume measured by the sound analysis unit 27. Examples of methods for displaying information include methods for presenting the amplitude of sound included at the current moment, methods for presenting the transition of sound amplitude over the past few seconds, methods for notifying that the amplitude of sound exceeds a specific threshold, and methods for presenting portions where the amplitude of sound exceeds a specific threshold. Additionally, the display control unit 28 generates an OSD image for notifying the user of camera information such as imaging conditions, and causes the display unit 20 to display the generated OSD image.

[0074] The operation control unit 29 receives operations from the user and controls the entire device according to the status.

[0075] Figure 4 This is a diagram showing an example of the display unit 20 according to an embodiment. Figure 4 This illustrates the situation where the display unit 20 displays a moving image when displaying a live view image or when capturing a moving image.

[0076] People, birds, the sun, and clouds are displayed by display unit 20. Additionally, display unit 20 displays display areas 40a to 40c.

[0077] In display area 40a, a time series graph representing the volume changes is displayed for each sound data point used for sound source separation. For example, in display area 40a, a solid line 41a represents the time series graph of the level of the target sound (e.g., human voice), and a dashed line 41b represents the time series graph of the level of sounds other than the target sound (e.g., ambient sounds or noise). The time series graph shows the volume changes over the past few seconds from the current time point.

[0078] Display area 40b displays the volume of a specific sound source at the current moment. For example, display area 40b represents the amplitude of a specific noise included in the current sound, in this case, the amplitude of wind sound.

[0079] In display area 40c, the volume of the sound data at the current moment is displayed separately for each sound source. For example, in display area 40c, the level of sound other than the target sound (i.e., ambient sound (surrounding sound) or noise) is indicated for channel 1 (CH1), and the level of the target sound (e.g., human voice) is indicated for channel 2 (CH2).

[0080] The photographer can monitor the volume of the target's voice contained within the collected sound by viewing the display unit 20. For example, the photographer can monitor the volume of the target voice (e.g., human speech) contained within the collected sound by viewing display areas 40a and 40c. Furthermore, by viewing display areas 40a to 40c, the photographer can confirm how much specific noise is included and take appropriate measures based on the noise, such as changing microphone settings, attaching a windshield, or changing the shooting position.

[0081] 2. Processing procedure

[0082] Figure 5 A flowchart illustrating an example of the processing procedure of the imaging apparatus 10 according to this embodiment. Figure 5 The processing steps for displaying live view images are shown.

[0083] The sound acquisition unit 26 controls the microphone to acquire and process the surrounding sound, and generates sound data such as linear PCM data format (step S10).

[0084] The sound analysis unit 27 performs sound source separation processing on the sound data acquired by the sound acquisition unit 26 to generate sound data for each sound source (step S11). When the input is linear PCM data, the output of the sound analysis unit 27 is also linear PCM data.

[0085] The sound analysis unit 27 measures the volume of each sound data obtained by sound source separation (step S12).

[0086] The display control unit 28 controls the display unit 20 to display the images acquired by the image acquisition unit 24 (step S13). For example, the display control unit 28 controls the display unit 20 to display the images acquired sequentially by the image acquisition unit 24 in a predetermined frame rate order. Furthermore, the display control unit 28 executes control to cause the display unit 20 to associate information related to the sound data generated from the speaking target object with the image data based on the recognition results obtained by the subject recognition unit 25 and the sound source separation results obtained by the sound analysis unit 27. For example, the display control unit 28 causes the display unit 20 to display information representing the volume measured by the sound analysis unit 27. For example, the display control unit 28 measures the level of the sound data of each separated sound source, converts the level into a format easily viewed by the photographer (e.g., PEAK value, RMS value, or loudness value), and displays the converted level.

[0087] The display control unit 28 determines whether to terminate the process (step S14). For example, when the live view image is terminated, the display control unit 28 determines that the process will be terminated. If the process is not completed (step S14: No), the process proceeds to step S10. Conversely, if the process is completed (step S14: Yes), the process ends.

[0088] 3. Specific examples of imaging

[0089] Next, refer to Figure 6 and Figure 7 This describes the gain adjustment method of this disclosure and the gain adjustment method in the comparative examples.

[0090] 3-1. Gain Adjustment Method in the Comparative Example

[0091] Here, as a comparative example, an instance of AGC processing in the prior art is described. Figure 6 This is a diagram illustrating the gain adjustment method based on a comparative example. In the prior art, the imaging device performs AGC on the input sound (input signal) from the microphone to generate sound data. By performing AGC on the input sound, the volume is adjusted to high when the overall volume is low and to low when the overall volume is high. Therefore, the recorded sound is maintained at a constant level and becomes easily audible to the human ear. Furthermore, recording is performed while suppressing peaks, thus preventing the recorded sound from exceeding its maximum value and suppressing sound distortion.

[0092] exist Figure 6In the input sound, AGC is performed to ensure that the volume does not exceed a predetermined upper limit level 50a and does not fall below a predetermined lower limit level 50b. In the output sound of AGC, the gain is adjusted to be lower so that the volume does not exceed the upper limit level 50a and does not fall below the lower limit level 50b, thereby reducing the amplitude of the waveform.

[0093] However, for example, when AGC is performed at a high-level time that includes noise such as wind, the following problem exists: the gain adjustment is performed under the influence of significant noise, so the human speech that is expected to be recorded as the target sound is only recorded at a low level at that time. Furthermore, even when source separation is performed on such sound, there is a problem that the level of the separated human speech varies depending on the timing, for example, the level becomes higher or lower.

[0094] Figure 6 This illustrates a scenario where source separation is performed on the output sound, separating it into speech and other sounds (e.g., wind noise). Figure 6 During periods of high-level noise, including sounds other than speech, the volume of speech in the output sound temporarily decreases because the amplitude of the AGC-adjusted waveform is smaller. As a result, it becomes difficult to discern human speech in the output sound during periods of high-level noise.

[0095] 3-2. Gain Adjustment Method in the Implementation Scheme

[0096] Next, the gain adjustment method of this embodiment will be explained. Figure 7 This is a diagram used to describe the gain adjustment method according to the implementation method.

[0097] The sound analysis unit 27 adjusts the volume gain of each separated sound data. For example, the sound analysis unit 27 performs source separation processing on the input sound data, separating it into multiple sound data sources for each sound source. The sound analysis unit 27 performs AGC on each separated sound data to generate sound data. Then, the sound analysis unit 27 executes AGC to add the sound data together to generate the output sound data.

[0098] exist Figure 7 In this process, the sound data of the input sound (input signal) is divided into sound data from multiple sound sources. Figure 7 This illustrates the case where source separation is performed on the input sound, separating it into speech and other sounds (e.g., wind sound). Then, in... Figure 7 In the process, AGC is performed on the separated sounds to ensure that the volume does not exceed the upper limit level 50a and does not fall below the lower limit level 50b.

[0099] exist Figure 7During the high-level timing of noise, including sounds other than speech, the amplitude of the waveform of the sounds other than speech is adjusted to be smaller by AGC. However, by separating the speech and performing AGC, the volume of the speech included in the output sound can be suppressed from decreasing due to the influence of noise. Therefore, the human speech included in the output sound can be kept in a state that is easy to grasp.

[0100] 4. Processing procedure

[0101] Figure 8 This is a flowchart illustrating an example of the processing procedure of the imaging apparatus 10 according to an embodiment. Figure 8 This illustrates the process of capturing moving images while displaying a live view image. Figure 8 The processing shown is Figure 5 The processing sections shown are identical. Therefore, descriptions of identical sections indicated by the same reference numerals are omitted, and the different sections are described primarily.

[0102] The sound analysis unit 27 adjusts the volume gain of each separated sound data (step S20). For example, the sound analysis unit 27 performs AGC on each separated sound data to generate volume-adjusted sound data. The sound analysis unit 27 adds up the volume-adjusted sound data after AGC to generate volume-balanced sound data.

[0103] The motion picture recording unit 21 records the image data sequentially acquired by the image acquisition unit 24 and the sound acquired by the sound acquisition unit 26 as motion picture data in a non-volatile memory (step S21). For example, the motion picture recording unit 21 records motion picture data based on the image data sequentially acquired by the image acquisition unit 24 and the sound data that has undergone volume balance adjustment.

[0104] exist Figure 8 In the illustrated process, steps S20 and S21 are executed in parallel with steps S12 and S13. Optionally, steps S12 and S13, as well as steps S20 and S21, can be executed serially.

[0105] 5. Variations

[0106] Incidentally, the embodiments described above may include several variations.

[0107] 5-1. First Variation Example

[0108] The sound analysis unit 27 can determine the sound source to be separated based on either the recognition result obtained by the subject recognition unit 25 or the microphone type of the microphone used to collect the sound, and can separate the sound data that will be considered the determined separation target from the acquired sound data. For example, the sound analysis unit 27 can determine the sound source that will be considered the target sound, ambient sound, or noise based on the object recognition result obtained by the subject recognition unit 25. Ambient sound is environmental sound generated by the surrounding environment, etc. For example, when the subject recognition unit 25 recognizes a person, the sound analysis unit 27 can separate "human speech" into the target sound. The sound analysis unit 27 can define a correspondence between the recognized object and the sound source considered as the target sound, ambient sound, or noise, and can separate the sound data corresponding to the sound source of the recognized object identified by the subject recognition unit 25 based on the correspondence. This correspondence can be defined in a one-to-one or one-to-many manner for the sound source to be separated and the recognized object.

[0109] Table 2

[0110] Table 2 shows an example of the correspondence between the identified objects and the sound sources that are considered as separate objects.

[0111] By using the correspondences in Table 2, the sound source to be considered as the separation target can be determined based on either the recognition result obtained by the subject recognition unit 25 or the microphone type of the microphone used to collect the sound. For example, with the correspondences defined as shown in Table 2, the sound analysis unit 27 separates "human voice" as the target sound when a person is recognized, and further separates "sea sound," "bird sound," and "surrounding sound" as environmental sounds when a beach is recognized. Furthermore, when collecting human speech by attaching a lavalier microphone to a person, since lavalier microphones are often used when placed inside clothing, rusting sounds caused by friction with clothing due to the person's movement or the attachment method are mixed in as noise. Accordingly, when a wireless microphone is connected to the imaging device 10, the sound analysis unit 27 separates "clothing friction sound" as noise, based on the assumption that a lavalier microphone (small microphone) is used at the wireless slave microphone end. It should be noted that a correspondence can be defined between the location information associated with the imaging device 10 and the sound source. For example, when the imaging device 10 is located in a city, the sound source to be separated can be defined as "car sound."

[0112] In this way, the imaging apparatus 10 according to the first modification can determine the sound source to be separated based on either the recognition result obtained by the subject recognition unit 25 or the microphone type of the microphone collecting the sound, thus separating only the appropriate sound source. As the number of separations increases, the sound source separation process requires a longer processing time. The imaging apparatus 10 according to the first modification can suppress unnecessary sound source separation, ensuring real-time performance and improving efficiency.

[0113] 5-1-1. Processing Procedure

[0114] Figure 9 This is a flowchart illustrating an example of the processing procedure of the imaging apparatus 10 according to the first modified example. Figure 9 The process for determining the target sound and ambient sound to be separated is shown.

[0115] The subject recognition unit 25 performs image recognition on the image data and determines whether a main subject exists in the image data (step S40). The main subject is the target on which the lens focus is adjusted among the objects included in the image and identified by image recognition. For example, when a person and an animal are present in the image as the target object, and the focus is adjusted on the person, the main subject is the person. The main subject can be determined automatically from a recognition state such as autofocus, or it can be determined by the user selecting a position on the screen where the focus is to be adjusted.

[0116] When a main subject exists (step S40: Yes), the sound analysis unit 27 determines the sound source corresponding to the identified main subject as the target sound by using the correspondence in Table 2 (step S41). On the other hand, if no main subject exists (step S40: No), the sound analysis unit 27 determines that no target sound exists.

[0117] In the correspondence in Table 2, the subject identification unit 25 identifies sound sources other than noise as ambient sounds from among the sound sources related to the identification state other than the main subject and microphone type (step S42). For example, as shown in Table 3, sound sources that will be considered noise are predetermined. The subject identification unit 25 identifies sound sources other than those registered in Table 3 as ambient sounds from among the sound sources related to the identification state and microphone type other than the main subject.

[0118] Table 3

[0119] Table 3 shows examples of sound sources that are considered noise.

[0120] The source of noise is a sound that makes people feel uncomfortable or find annoying, such as the sound of wind, the sound of clothes rubbing, and the noise of household appliances (the sound of air conditioners operating, the sound of refrigerators operating, etc.), and can therefore be uniquely identified regardless of the situation.

[0121] 5-2. Second variation

[0122] Before performing sound source separation, the sound analysis unit 27 can determine the sound source that will be considered as target sound, ambient sound, or noise based on either the recognition result obtained by the subject recognition unit 25 or the microphone type of the microphone that collects the sound. Furthermore, the sound analysis unit 27 can also perform gain adjustment on each sound data segment having a volume adjusted by performing AGC on each sound source (target sound, ambient sound, or noise).

[0123] 5-2-1. Processing Procedure

[0124] Figure 10 This is a flowchart illustrating an example of the processing procedure of the imaging apparatus 10 according to the second variant. Figure 10 This illustrates the process of capturing moving images while displaying a live view image. Figure 10 The processing shown in the figure is the same as Figure 5 and Figure 8 The processing sections shown are identical. Therefore, descriptions of identical sections indicated by the same reference numerals are omitted, and the different sections are described primarily.

[0125] After step S10, the sound analysis unit 27 may determine the sound source that is considered as target sound, ambient sound or noise based on the recognition result obtained by the subject recognition unit 25 and the microphone type of the microphone that collects the sound (step S50), and the processing proceeds to step S11.

[0126] Furthermore, after step S20, the sound analysis unit 27 multiplies each sound data segment whose volume has been adjusted by performing AGC on each sound source (target sound, ambient sound, or noise) by a coefficient, thereby generating multiplied sound data (step S51), and the process proceeds to step S21. The coefficients are set such that the ambient sound is lower than the target sound and the noise is lower than the ambient sound. For example, the sound analysis unit 27 uses the coefficients shown in Table 4 and performs gain adjustment on each sound data segment having a volume adjusted by performing AGC on each sound source (target sound, ambient sound, or noise). This generates sound data. The sound analysis unit 27 adds the gain-adjusted sound data to generate volume-balanced sound data.

[0127] Table 4

[0128] Table 4 shows examples of coefficients used to perform gain adjustment for sound sources including target sound, ambient sound, and noise. Note that coefficients other than those shown in Table 4 are also recorded. The sound analysis unit 27 can perform gain adjustment by multiplying the sound data of sound sources other than target sound, ambient sound, and noise by coefficients recorded for "Other".

[0129] The sound analysis unit 27 attenuates each sound source using coefficients from Table 4, which correspond to the sound source identified as a target sound, ambient sound, or noise. Sound sources such as noise can be attenuated significantly, and sound sources of the atmosphere or surrounding environment (such as ambient sounds) are preferably retained, and coefficients for slight attenuation can be assigned. Furthermore, these coefficients can be set by the user. Additionally, a gain coefficient can be set for each sound source. Coefficients can be used to increase sensitivity by increasing gain in order to maintain the target sound source at or above a specific level. Furthermore, the gain coefficient can be differentiated based on whether the person is facing forward or backward, combined with details related to the recognition state (e.g., when the person is facing backward, the speech entering the microphone is weaker, therefore the gain is increased).

[0130] As described above, the imaging device 10 according to the second modification performs gain adjustment on the sound source that is considered as noise other than ambient sound or target sound, and records the sound source with reduced sensitivity, thereby reducing the level of other sounds, making it easier to hear only the target sound in the recorded sound, and improving functionality.

[0131] 5-3. Third variation example

[0132] The display unit 20 can receive a designation of a sound source that is considered a target sound, ambient sound, or noise. The sound analysis unit 27 can perform sound source separation on the sound data designated as a target sound, ambient sound, or noise source.

[0133] Figure 11A and Figure 11B This is a diagram showing a display example of the display unit 20 according to the third modified example. Figure 11A An example of the category-specified area 42 displayed by the display unit 20 is shown. Figure 11B An example of the display area 40a displayed by the display unit 20 is shown.

[0134] Figure 11AThe category designation area 42 shown is a window displaying sound sources currently considered as separate targets, including target sounds, ambient sounds, and noise. Category designation area 42 is divided into zones 42a to 42c for target sounds (target), ambient sounds (ambience), and noise (noise). Icons 43 representing sound sources considered as target sounds, ambient sounds, and noise are displayed in zones 42a to 42c. By viewing category designation area 42, users can easily identify which sound sources have been detected / separated. The current gain and corresponding volume are also displayed in icon 43. Furthermore, users can move sound source categories by dragging and dropping, and can manually add new sound sources.

[0135] Figure 11B The display area 40a shown is a window that displays the levels of corresponding categories of sounds, including the target sound, ambient sound, and noise, in the time series graph. In display area 40a, the levels of the target sound, ambient sound, and noise are displayed using solid lines, dashed lines, etc., or by using different colors. Figure 11B In the display area 40a, the time-series curve of the target sound is displayed via line 44a, the time-series curve of the ambient sound is displayed via line 44b, and the time-series curve of the noise is displayed via line 44c. When a specific sound source is selected from the window of the category-specified area 42, the display control unit 28 can display only the time-series curve of the selected sound source in the display area 40a. The upper limit level 50a and lower limit level 50b of the target sound are displayed in the display area 40a. The sound analysis unit 27 adjusts the gain so that the volume of the target sound does not exceed the upper limit level 50a and is not lower than the lower limit level 50b. The upper limit level 50a and lower limit level 50b can be freely operated and adjusted by the user. In addition, bars for the upper limit level and lower limit level, not only for the target sound but also for the ambient sound and noise, can be displayed, so that the upper limit level and lower limit level of the ambient sound and noise can be adjusted. Alternatively, bars for the upper limit level and lower limit level can be displayed for each sound source, so that the upper limit level and lower limit level can be adjusted for each sound source.

[0136] As described above, the imaging apparatus 10 according to the third modification can easily select sound sources during imaging to be considered as target sounds, ambient sounds, and noise, and can provide a user interface suitable for easy collection of audible sounds, thereby improving usability.

[0137] 5-4. Fourth variation example

[0138] The operation control unit 29 can control imaging based on the sound sources separated by the sound analysis unit 27. For example, the operation control unit 29 can control imaging based on the separated sound sources according to predetermined rules. Adjustments are performed to generate sound data. For example, the operation control unit 29 can control imaging according to the rule table in Table 5.

[0139] Table 5

[0140] Table 5 shows an example of a rule table used to perform imaging control.

[0141] As shown in Table 5, when a specific condition is detected from the rule base, the operation control unit 29 performs an operation corresponding to the condition. For example, when the isolated sound source "animal sound (or footstep sound)" exceeds a certain level, the operation control unit 29 begins recording. In imaging of wild animals, animals do not approach when humans are present. Therefore, in the imaging of wild animals in related technologies, methods such as installing cameras and setting infrared sensors are commonly used so that imaging automatically begins when an animal approaches.

[0142] Conversely, the imaging device 10 according to the fourth modification begins imaging while using specific separated sounds, and therefore can be implemented independently by the imaging device 10 without the need for another device such as an infrared sensor. Furthermore, the display control unit 28 can also cause the display unit 20 to display a warning prompting the user to install a windproof shield when strong winds are detected. Therefore, the user and the imaging device 10 can cooperate to achieve better sound collection.

[0143] As described above, the imaging apparatus 10 according to the fourth modification can perform imaging suitable for the current situation while using separate sound sources, thereby improving usability.

[0144] 5-5. Fifth Variation

[0145] In the above embodiments, the application of the technology of this disclosure to the imaging apparatus 10 has been described as an example. However, this embodiment is not limited thereto. For example, the technology of this disclosure can be implemented in a distributed manner by the imaging apparatus 10 and another device. In other words, the technology of this disclosure can be implemented by a system configuration including multiple devices.

[0146] 5-5-1. Configuration of the Imaging System

[0147] Figure 12 This is a diagram illustrating a configuration example of the imaging system 1 according to a fifth modification. The imaging system 1 includes an imaging device 10 and a monitoring device 11.

[0148] The monitoring device 11 is, for example, an information processing device such as a smartphone or tablet computer. The monitoring device 11 receives video and sound from the imaging device 10 and sends control signals. The imaging device 10 and the monitoring device 11 are communicatively connected via a wireless or wired network. Examples of communication methods between the imaging device 10 and the monitoring device 11 include wired connections via Universal Serial Bus (USB) cables, Local Area Network (LAN) cables, etc., and wireless connections using Wi-Fi, Bluetooth (registered trademark), etc.

[0149] 5-5-2. Configuration of Monitoring Devices

[0150] Figure 13 This is a functional block diagram showing an example of the functional configuration of the monitoring device 11 according to the fifth modification.

[0151] like Figure 13 As shown, the monitoring device 11 includes a display unit 50, a receiving unit 51, a transmission unit 52, a sound analysis unit 53, and a display control unit 54.

[0152] The display unit 50 is a display provided on the monitoring device 11. The receiving unit 51 receives video, sound, and related metadata from the imaging device 10. The receiving unit 51 receives the recognition result obtained by the subject recognition unit 25 and the microphone type of the microphone collecting the sound via the metadata. The transmission unit 52 transmits control signals to the imaging device 10 to externally operate and control the imaging device 10. The operation control unit 55 receives operations from the user and controls the entire device according to the status.

[0153] The sound analysis unit 53 performs sound source separation processing on the sound data received by the receiving unit 51, generating sound data from multiple sound sources. For example, the sound analysis unit 53 uses a separation model trained for the sound to be separated to generate sound data of the sound to be separated from the acquired sound data. The sound analysis unit 53 measures the volume of each sound data obtained through sound source separation. In addition, the sound analysis unit 53 can also, similarly to the sound analysis unit 27, perform gain adjustment on each separated sound data, generate sound data after adding the gain-adjusted sound data, and send it to the recording device 10 for recording.

[0154] The display control unit 54 performs control to cause the display unit 20 to display the video received via metadata. Furthermore, the display control unit 54 performs control to cause the display unit 20 to associate information related to the sound data generated from the speaking target object with the image, based on the recognition results obtained by the subject recognition unit 25 and the sound source separation results obtained by the sound analysis unit 53. For example, the display control unit 54 causes the display unit 20 to display information representing the volume measured by the sound analysis unit 27. Additionally, the display control unit 54 generates an OSD image for informing the user of camera information (e.g., imaging conditions) and causes the display unit 20 to display the generated OSD image.

[0155] As described above, in the imaging system 1 according to the fifth modification, motion images can be confirmed on the monitoring device 11 connected to the imaging device 10, and therefore the volume of the target object's voice, including the voice in the collected sound, can be monitored on a large screen even at a location far from the imaging device 10. Furthermore, the imaging system 1 according to the fifth modification uses the resources of the monitoring device 11 to perform analysis, thus enabling more advanced processing than that of the imaging device 10, thereby improving functionality.

[0156] In the processes described in the above embodiments of this disclosure, all or some of the processes described as being automatically executed can be performed manually, or all or some of the processes described as being manually executed can be performed automatically by known methods. Furthermore, unless otherwise specified, the processing procedures, specific names, and information including various types of data and parameters described in the above documents and figures are freely changeable. For example, the various types of information shown in the figures are not limited to the information shown.

[0157] The components of the device shown in the accompanying drawings are conceptual in function and do not necessarily have to be physically constructed as shown in the drawings. That is, the specific forms of distribution and integration of the device are not limited to those shown in the accompanying drawings, and all or some of the device may be configured to be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc.

[0158] The embodiments described above can be appropriately combined, as long as the processing content does not contradict each other. Furthermore, the order of steps shown in the sequence diagrams or flowcharts of this embodiment can be appropriately changed.

[0159] 6. Hardware Configuration

[0160] Furthermore, the imaging device 10 and monitoring device 11 according to embodiments of this disclosure are, for example, as... Figure 14 The computer 1000 with the configuration shown is implemented. The imaging device 10 is described as an example. Figure 14This is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of the imaging device 10. The computer 1000 includes a CPU 1100, RAM 1200, ROM 1300, auxiliary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The units of the computer 1000 are connected together via a bus 1050.

[0161] The CPU 1100 operates based on programs stored in ROM 1300 or auxiliary storage device 1400 and controls each unit. For example, the CPU 1100 loads programs stored in ROM 1300 or auxiliary storage device 1400 into RAM 1200 and executes processing corresponding to various types of programs.

[0162] The ROM 1300 stores boot programs (such as the Basic Input / Output System (BIOS) executed by the CPU 1100 when the computer 1000 is activated), programs that depend on the hardware of the computer 1000, etc.

[0163] The auxiliary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the CPU 1100, data used by the programs, etc. Specifically, the auxiliary storage device 1400 is a recording medium that records the program for each process of the imaging apparatus 10 according to an embodiment of the present disclosure, the program being an instance of program data 1450.

[0164] Communication interface 1500 is an interface used to connect computer 1000 to external network 1550. For example, CPU 1100 receives data from other devices or sends data generated by CPU 1100 to other devices via communication interface 1500.

[0165] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as microphones and touch panels via input / output interface 1600. CPU 1100 also transmits data to output devices such as displays or speakers via input / output interface 1600. Furthermore, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium (medium). For example, the medium includes optical recording media such as digital versatile discs (DVDs) or phase-change rewritable discs (PDs), magneto-optical recording media such as magneto-optical discs (MOs), magnetic tape media, magnetic recording media, semiconductor memory, etc.

[0166] For example, when computer 1000 is used as imaging apparatus 10 according to an embodiment of the present disclosure, CPU 1100 of computer 1000 executes a program loaded onto RAM 1200 to implement the functions of image acquisition unit 24, subject recognition unit 25, sound acquisition unit 26, sound analysis unit 27, display control unit 28, and operation control unit 29. Furthermore, auxiliary storage device 1400 stores the program according to the present disclosure and data related to the separation model. Note that CPU 1100 reads and executes program data 1450 from auxiliary storage device 1400. However, in another instance, CPU may obtain the program from another device via external network 1550.

[0167] 7. Conclusion

[0168] As described above, according to embodiments of this disclosure, the imaging apparatus 10 (corresponding to an example of an "imaging system") includes an image acquisition unit 24, a subject recognition unit 25 (recognition unit), a sound acquisition unit 26, a sound analysis unit 27, and a display control unit 28. The image acquisition unit 24 acquires image data including the target object speaking. The subject recognition unit 25 recognizes objects contained in the image data. The sound acquisition unit 26 acquires sound data related to the speaking object. The sound analysis unit 27 performs sound source separation on the sound data to obtain multiple sound data sets. Based on the recognition result obtained by the subject recognition unit 25 and the sound source separation result obtained by the sound analysis unit 27, the display control unit 28 causes the display unit 20 to display information related to the sound data generated from the speaking target object in association with the image data. Thus, the imaging apparatus 10 can grasp the volume of the speaking target object's voice included in the collected sound.

[0169] Furthermore, according to embodiments of this disclosure, the imaging system 1 (an example corresponding to "imaging system") includes an image acquisition unit 24, a subject recognition unit 25 (recognition unit), a sound acquisition unit 26, a sound analysis unit 53, and a display control unit 54. The image acquisition unit 24 acquires image data including the target object speaking. The subject recognition unit 25 recognizes objects contained in the image data. The sound acquisition unit 26 acquires sound data related to the speaking object. The sound analysis unit 53 performs sound source separation on the sound data to obtain multiple sound data sets. Based on the recognition result obtained by the subject recognition unit 25 and the sound source separation result obtained by the sound analysis unit 53, the display control unit 54 causes the display unit 50 to display information related to the sound data generated from the speaking target object in association with the image data. Thus, the imaging system 1 can determine the volume of the speaking target object's voice included in the collected sound data.

[0170] While embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the above embodiments, and various modifications can be made without departing from the spirit of this disclosure. Furthermore, components from different embodiments and modifications can be appropriately combined.

[0171] The effects of the embodiments described in this specification are merely examples and are not limited, and other effects may be generated.

[0172] It should be noted that this technology can be configured as described below. (1)

[0174] An imaging system, comprising: The image acquisition unit is configured to acquire image data including the target object being spoken; The recognition unit is configured to recognize objects included in an image of image data; The sound acquisition unit is configured to acquire sound data related to the target object speaking. The sound analysis unit is configured to perform sound source separation on the sound data to obtain multiple sound data streams; and The display control unit is configured to display information related to the sound data generated based on the target object speaking, in association with the image data, based on the recognition result obtained by the recognition unit and the sound source separation result obtained by the sound analysis unit. (2)

[0176] According to the imaging system of item (1), wherein, The sound analysis unit includes: a first separation model configured to separate speech-related sound data from other sound data based on the sound data. (3)

[0178] According to the imaging system in item (2), where, The sound analysis unit also includes a second separation model, configured to separate wind-related sound data based on sound data other than speech. (4)

[0180] According to the imaging system of item (3), where, The sound analysis unit also includes a third separation model, configured to separate music-related sound data based on sound data other than speech. (5)

[0182] According to the imaging system of item (1), wherein, The image acquisition unit acquires image data sequentially at a predetermined frame rate. The sound acquisition unit acquires sound data, while the image acquisition unit sequentially acquires image data. The display control unit outputs information related to multiple sound data obtained through sound source separation, and displays the information synchronously with the sequentially acquired image data. (6)

[0184] According to the imaging system in item (5), where, The display control unit enables the display unit to show the transition of sound volume for each of the multiple sound data obtained through sound source separation. (7)

[0186] According to the imaging system in item (5), where, The sound analysis unit performs volume gain adjustment on each separated sound data segment, and The imaging system also includes a recording unit configured to record motion image data based on image data acquired sequentially by the image acquisition unit and sound data with adjusted volume balance. (8)

[0188] According to the imaging system of item (1), wherein, The sound analysis unit determines the sound source to be separated based on either the recognition result obtained by the recognition unit or the type of microphone used to collect the sound, and performs sound source separation to obtain the sound data of the determined sound source from the sound data obtained by the sound acquisition unit. (9)

[0190] According to the imaging system of item (8), wherein, The sound analysis unit determines the sound source that will be considered as target sound, ambient sound, or noise based on the object recognition results obtained by the recognition unit. (10)

[0192] According to the imaging system in item (6), where, The display unit shows the volume of each of the target sound, ambient sound, and noise, and accepts the designation of the sound source to be considered the target sound, ambient sound, or noise. The sound analysis unit performs sound source separation on the sound data displayed on the display unit that is designated as the target sound, ambient sound, or noise source. (11)

[0194] The imaging system according to item (1) also includes: The operation control unit is configured to control imaging based on sound sources separated by the sound analysis unit. (12)

[0196] A display method, comprising: Acquire image data including the target object speaking; Identify objects included in an image of image data; Acquire audio data related to the target of the speech; Perform source separation on the audio data to obtain multiple audio data sets; and The execution control uses object recognition results and sound source separation results to display information related to the sound data generated based on the target object speaking, in association with the image data. (13)

[0198] A program that causes a computer to perform the following processes: Acquire image data including the target object speaking; Identify objects included in an image of image data; Acquire audio data related to the target of the speech; The sound data is processed by source separation to obtain multiple sound data sets; and The execution control uses object recognition results and sound source separation results to display information related to the sound data generated based on the target object speaking, in association with the image data.

[0199] Reference number list

[0200] 1 Imaging System

[0201] 10 Imaging Devices

[0202] 11 Monitoring devices

[0203] 20, 50 display units

[0204] 21 Motion Picture Recording Unit

[0205] 22 optical units

[0206] 23 imaging units

[0207] 24 Image Acquisition Units

[0208] 25 Subject Identification Units

[0209] 26 Sound Acquisition Units

[0210] 27, 53 Sound Analysis Units

[0211] 28, 54 Display control unit

[0212] 29, 55 Operation control unit

[0213] 30 Sequence coding units

[0214] 31 Modeling Units

[0215] 32 Decoding Units

[0216] Display areas 40a, 40b, and 40c

[0217] 42 Category-Specific Areas

[0218] Areas 42a, 42b, and 42c

[0219] 51 Receiving Unit

[0220] 52. Transmission Unit.

Claims

1. An imaging system, comprising: The image acquisition unit is configured to acquire image data including the target object being spoken; The recognition unit is configured to recognize objects included in the image data. The sound acquisition unit is configured to acquire sound data related to the target object speaking; The sound analysis unit is configured to perform sound source separation on the sound data to obtain multiple sound data; as well as The display control unit is configured to display information related to the sound data generated based on the sound data of the target object speaking, in association with the image data, based on the recognition result obtained by the recognition unit and the sound source separation result obtained by the sound analysis unit.

2. The imaging system according to claim 1, wherein, The sound analysis unit includes a first separation model configured to separate speech-related sound data from other sound data based on the sound data.

3. The imaging system according to claim 2, wherein, The sound analysis unit also includes a second separation model configured to separate wind-related sound data based on sound data other than the speech.

4. The imaging system according to claim 3, wherein, The sound analysis unit also includes a third separation model, configured to separate music-related sound data based on sound data other than the speech.

5. The imaging system according to claim 1, wherein, The image acquisition unit acquires image data sequentially at a predetermined frame rate. The sound acquisition unit acquires sound data, and simultaneously the image acquisition unit sequentially acquires the image data. The display control unit outputs information related to multiple audio data obtained through sound source separation, and displays the information synchronously with the sequentially acquired image data.

6. The imaging system according to claim 5, wherein, The display control unit causes the display unit to display the transition of sound volume for each of the multiple sound data obtained through sound source separation.

7. The imaging system according to claim 5, wherein, The sound analysis unit performs volume gain adjustment on each separated sound data, and The imaging system further includes a recording unit configured to record motion image data based on the image data sequentially acquired by the image acquisition unit and the sound data with adjusted volume balance.

8. The imaging system according to claim 1, wherein, The sound analysis unit determines the sound source to be separated based on either the recognition result obtained by the recognition unit or the type of microphone used to collect the sound, and performs sound source separation to obtain sound data of the determined sound source from the sound data obtained by the sound acquisition unit.

9. The imaging system according to claim 8, wherein, The sound analysis unit determines the sound source that will be considered as target sound, ambient sound, or noise based on the object recognition result obtained by the recognition unit.

10. The imaging system according to claim 6, wherein, The display unit displays the volume of each of the target sound, ambient sound, and noise, and accepts the designation of a sound source that will be considered as the target sound, the ambient sound, or the noise. The sound analysis unit performs sound source separation on the sound data displayed on the display unit that is designated as the target sound, the ambient sound, or the noise.

11. The imaging system according to claim 1, further comprising: The operation control unit is configured to control imaging based on the sound source separated by the sound analysis unit.

12. A display method executed by a computer, the display method comprising: Acquire image data including the target object speaking; Identify objects included in the image data; Acquire audio data related to the target object speaking; Perform source separation on the sound data to obtain multiple sound data streams; and The execution control is based on the object recognition result and the sound source separation result to cause the display unit to display information related to the sound data generated according to the target object speaking, in association with the image data.

13. A program that causes a computer to perform the following processes: Acquire image data including the target object speaking; Identify objects included in the image data; Acquire audio data related to the target object speaking; Perform source separation on the sound data to obtain multiple sound data streams; and The execution control is based on the object recognition result and the sound source separation result to cause the display unit to display information related to the sound data generated according to the target object speaking, in association with the image data.

Images