Signal processing device, method, and program

Abstract

The present technology pertains to a signal processing device, method, and program with which it is possible to obtain high-quality intended sounds. This signal processing device is provided with a interval detection unit for detecting the time interval, in a recorded signal, of a sound emitted by a moving object, on the basis of a recorded signal obtained by collecting sounds from around the moving object in a state in which other moving objects are present around the moving object, and a sensor signal outputted by a sensor mounted on the moving object. The present technology can be applied to recording systems.

Description

[Technical field]

[0001] The present technology relates to a signal processing device, a method, and a program, and more particularly to a signal processing device, a method, and a program that are capable of obtaining a high-quality target sound. [Background technology]

[0002] When reproducing a sound field from any viewpoint, such as a bird's-eye view or walk-through, it is important to record the sound of the desired sound source with a high signal-to-noise ratio (SNR), and at the same time, it is necessary to obtain information indicating the position and direction of each sound source.

[0003] Specific examples of the target sound source include human voices, general human movement sounds such as walking and running sounds, and movement sounds specific to content such as sports and theater, such as the kicking of a ball.

[0004] Furthermore, for example, as a technology relating to user behavior recognition, a technology has been proposed that makes it possible to obtain behavior recognition results for one or more users by analyzing distance measurement sensor data detected by multiple distance measurement sensors (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-205213 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when recording sports, theater, or other free viewpoint content, there are multiple sound sources in the recording space, and these sound sources may move in complex ways, making it difficult to obtain the sound of the desired sound source with a high signal-to-noise ratio. In other words, it is difficult to obtain high-quality desired sound.

[0007] The present technology has been made in consideration of such circumstances, and makes it possible to obtain a high-quality target sound. [Means for solving the problem]

[0008] A signal processing device according to one aspect of the present technology detects a time period of a sound emitted by a moving object in a recorded signal obtained by collecting a sound around the moving object in a state where another moving object is present around the moving object, based on a sensor signal output by a sensor attached to the moving object. a section detection section; and a data generation section that generates an audio signal of the voice or operation sound of the moving object from the recorded signal based on a result of the detection of the time section, and outputs object sound source data including the audio signal and position information indicating the position of the moving object. Equipped with.

[0009] A signal processing method or program according to one aspect of the present technology includes: determining a time period of a sound emitted by a moving object in a recorded signal obtained by collecting a sound around the moving object in a state where another moving object is present around the moving object, based on a sensor signal output by a sensor attached to the moving object; and generating an audio signal of the voice or operation sound of the moving object from the recorded signal based on the result of the detection of the time period, and outputting object sound source data including the audio signal and position information indicating the position of the moving object. Includes steps.

[0010] In one aspect of the present technology, a time period of a sound emitted by a moving object is detected in a recorded signal obtained by collecting a sound around the moving object in a state where another moving object is present around the moving object, and a sensor signal output by a sensor attached to the moving object. Based on the detection result of the time period, an audio signal of the voice or operation sound of the moving object is generated from the recorded signal, and object sound source data including the audio signal and position information indicating the position of the moving object is output. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of the configuration of a recording system. [Diagram 2] FIG. 2 is a diagram illustrating an object and an object sound source. [Diagram 3] FIG. 11 is a diagram showing an example of sound source type section information. [Figure 4] 11 is a diagram illustrating generation of sound source type section information. FIG. [Diagram 5] FIG. 13 is a diagram illustrating selection of an object to be removed. [Figure 6] 13 is a flowchart illustrating a recording process. [Figure 7] 11 is a flowchart illustrating a data generation process. [Figure 8] FIG. 1 illustrates an example of the configuration of a recording system. [Figure 9] 11 is a flowchart illustrating a data generation process. [Figure 10] FIG. 1 illustrates an example of the configuration of a recording system. [Figure 11] 13 is a flowchart illustrating a recording process. [Figure 12] 11 is a flowchart illustrating a data generation process. [Figure 13] FIG. 1 illustrates an example of the configuration of a recording system. [Figure 14] 11 is a flowchart illustrating a data generation process. [Figure 15] FIG. 1 illustrates an example of the configuration of a recording system. [Figure 16] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0013] First embodiment <Example of recording system configuration> This technology enables the acquisition of high-quality target sounds by equipping multiple moving objects within a target space with microphones, ranging devices, cameras, etc., and distinguishing and extracting the sounds of oneself and others based on recorded sound signals, the position information of the moving objects, movement information of the moving objects, and surrounding images.

[0014] Specifically, examples of content suitable for application of this technology include the following:

[0015] -Content that recreates fields where team sports are played -Content that plays performances by orchestras, marching bands, etc. Content that recreates a space with multiple performers, such as in musicals, operas, plays, etc. Content that recreates any space, such as in athletic meets, live concerts, various events, theme park parades, etc.

[0016] Hereinafter, the space to be recorded will be referred to as the target space.

[0017] In particular, it is assumed here that a plurality of moving objects exist within the same target space, and each of these moving objects is equipped with or has a recording device built therein for recording content.

[0018] A moving object to which a recording device is attached or built is treated as an object, and the sound generated by the object is recorded (collected) as the sound of an object sound source.

[0019] For example, an object (moving body) in the target space may be a person such as an athlete, or may be a robot, vehicle, drone, or other flying object equipped with or equipped with a recording device.

[0020] For example, if the object is a person, it is desirable for the recording device worn by the person to be as small as possible so as not to affect the performance of the person and so as not to be visible to those around.

[0021] The recording device also includes, for example, a microphone for picking up the sound of the object sound source, sensors such as a 9-axis sensor for measuring the movement and direction (azimuth) of the object, a ranging device for measuring the position, and a camera for capturing images of the surrounding area.

[0022] Here, the distance measuring device is, for example, a GPS (Global Positioning System) device for measuring the position of an object or a beacon receiver for indoor distance measuring, and the distance measuring device can provide position information indicating the position of an object.

[0023] In addition, the output of a sensor installed in the recording device can provide motion information indicating the object's movement, such as speed and acceleration, as well as the object's orientation (direction).

[0024] The recording device uses a built-in microphone, sensor, and distance measuring device to obtain a recording signal, which is an audio signal obtained by picking up sounds around the object, position information of the object, and motion information of the object. In addition, if the recording device is equipped with a camera, a video signal of an image around the object can also be obtained.

[0025] The recorded signal, position information, movement information, and video signal thus obtained for each object are used to obtain an object sound source signal, which is an audio signal of the sound of the object sound source, which is the target sound.

[0026] Here, the sound of the object sound source that is the target sound is, for example, the voice spoken by a person who is an object, the walking sound or running sound of an object, the clapping sound, or other motion sounds.

[0027] The recorded signal obtained for each object includes not only the sound emitted by the object itself, but also the sounds emitted by other objects in the vicinity. Furthermore, the recorded signal includes the sound of the object itself, such as the object's own speech and movement sounds, and also includes a number of different object sound sources, i.e., different types of sounds.

[0028] With this technology, by using the position information, movement information, and video signals obtained for each object as needed, it is possible to discriminate (distinguish) the sounds of each object sound source contained in the recorded signal and extract the object sound source signal of each object sound source from the recorded signal.

[0029] Specifically, for example, by identifying the motion state of an object based on the motion information, it is possible to detect a time period in the recorded signal in which the sound of each object sound source is included.

[0030] Therefore, by extracting the signal of the object sound source sound section from the recorded signal, and then performing appropriate signal processing such as sound quality correction, sound source separation, and noise removal on the extracted signal, it is possible to obtain a high-quality object sound source signal with a high signal-to-noise ratio.

[0031] Furthermore, by integrating information such as position information, movement information, and video signals obtained for multiple objects, the detection results for the time interval of the sound of the object sound source can be made more accurate, making it possible to obtain an object sound source signal of even higher quality.

[0032] Now, the present technology will be described in more detail below.

[0033] FIG. 1 is a diagram showing an example of the configuration of an embodiment of a recording system to which the present technology is applied.

[0034] In the example shown in FIG. 1, the recording system includes a recording device 11 attached to an object, which is a moving body, and a server 12 that receives transmission data from the recording device 11 and generates an object sound source signal.

[0035] Although the recording device 11 may be built into a mobile object, the following description will be given assuming that the recording device 11 is attached to the mobile object.

[0036] The recording device 11 is attached to an object, which is a mobile body that can move freely within the target space to be recorded, and generates transmission data including a recording signal, position information, and motion information and transmits it to the server 12.

[0037] Although only one recording device 11 is illustrated here, in reality there are multiple recording devices 11, and these multiple recording devices 11 are attached to multiple different objects.

[0038] The server 12 outputs object sound source data consisting of object sound source signals and metadata of each object sound source as content data based on the transmission data received from the multiple recording devices 11. Note that the server 12 does not need to be placed in the target space.

[0039] The recording device 11 also has a microphone 21 , a movement measuring unit 22 , a position measuring unit 23 , a recording unit 24 , and a transmission unit 25 .

[0040] The microphone 21 picks up sounds around the recording device 11 and supplies the resulting recording signal to the recording unit 24. Note that the recording signal may be a monaural signal, but the following description will be given assuming that the recording signal is a multi-channel signal.

[0041] In the recording device 11, sound is picked up by the microphone 21 in a state in which not only the object to which the recording device 11 is attached but also other objects are present in the surrounding area, so the sound based on the recorded signal includes sounds from multiple sound sources.

[0042] The motion measurement unit 22 is composed of sensors for measuring the motion and orientation of an object, such as a 9-axis sensor, a geomagnetic sensor, an acceleration sensor, or a gyro sensor, and outputs a sensor signal indicating the measurement result (sensing value) to the recording unit 24 as motion information.

[0043] In particular, the movement measurement unit 22 measures the movement and direction of the object while the microphone 21 is collecting sound, and outputs motion information indicating the results.

[0044] Although an example will be described here in which the sensor signal is directly used as motion information, motion information may be generated from the sensor signal by appropriately subjecting the sensor signal to signal processing by the recording unit 24.

[0045] In addition, the movement measurement unit 22 may be provided outside the recording device 11 and attached to a position on the object different from the position where the recording device 11 is attached.

[0046] The position measurement unit 23 is composed of a distance measurement device such as a GPS device or an indoor distance measurement beacon receiver, measures the position of the object to which the recording device 11 is attached, and outputs position information indicating the measurement result to the recording unit 24.

[0047] It should be noted that the recorded signals, the motion information, and the position information are acquired simultaneously during the same period of time.

[0048] The recording unit 24 performs AD (Analog to Digital) conversion, etc., on the recorded signal supplied from the microphone 21, the movement information supplied from the movement measurement unit 22, and the position information supplied from the position measurement unit 23 as appropriate, and supplies them to the transmission unit 25.

[0049] The transmission unit 25 performs compression processing on the recorded signal, movement information, and position information supplied from the recording unit 24 to generate transmission data including the recorded signal, movement information, and position information, and transmits the obtained transmission data to the server 12 via a wireless network, etc.

[0050] The server 12 also includes a receiving unit 31 , a section detecting unit 32 , and an object sound source data generating unit 33 .

[0051] The receiving unit 31 receives the transmission data sent from each of the multiple recording devices 11, and extracts the recording signal, position information, and movement information from the transmission data.

[0052] The receiving unit 31 supplies the recorded signal to the section detection unit 32 and the object sound source data generation unit 33. The receiving unit 31 also supplies the motion information to the section detection unit 32, and supplies the motion information and position information to the object sound source data generation unit 33.

[0053] Based on the recorded signal and movement information supplied from the receiving unit 31, the section detection unit 32 detects, for each recorded signal, the type (kind) of sound of the object sound source contained in the recorded signal, i.e., the type of object sound source and the time section in which the sound of the object sound source is contained.

[0054] The section detection unit 32 supplies the object sound source data generation unit 33 with sound source type section information indicating the type and time section of the sound of the object sound source detected from the recorded signal.

[0055] Furthermore, the section detection unit 32 supplies sound source type information indicating an object corresponding to the recorded signal and a type of sound of the object sound source detected from the recorded signal to the object sound source data generation unit 33. In other words, the sound source type information is information indicating the type of the object sound source that is the sound source of the sound based on the object sound source signal and the object that is the source of the sound.

[0056] The object sound source data generation unit 33 generates object sound source data based on the recorded signal, movement information, and position information supplied from the receiving unit 31, and the sound source type section information and sound source type information supplied from the section detection unit 32, and outputs the object sound source data to a downstream playback device, etc.

[0057] The object sound source data generating unit 33 includes a signal processing unit 41 and a metadata generating unit .

[0058] The signal processing unit 41 applies a predetermined signal processing to the recorded signal supplied from the receiving unit 31 based on the sound source type section information supplied from the section detection unit 32 and the motion information and position information supplied from the receiving unit 31, thereby generating an object sound source signal.

[0059] Here, for example, based on the sound source type section information, one or more signal processing steps are performed to generate the object sound source signal, such as extracting a time section of the object sound source sound from the recorded signal and muting a time section of the recorded signal where there is no sound from the object sound source.

[0060] In addition, the metadata generation unit 42 generates metadata for each object sound source, i.e., each object sound source signal, including the sound source type information supplied from the section detection unit 32 and the motion information and position information supplied from the receiving unit 31.

[0061] The object sound source data generating unit 33 outputs object sound source data, which is made up of the object sound source signal thus obtained and metadata, to a subsequent stage.

[0062] <About each part of the server> Next, each unit constituting the server 12 will be described in more detail.

[0063] First, the section detection unit 32 will be described.

[0064] In the following description, a predetermined object of interest will also be referred to as a target object, and objects other than the target object will also be referred to as other objects, where appropriate.

[0065] The section detection unit 32 distinguishes whether the sound contained in the recorded signal is a sound emitted from the target object or from another object, identifies the type of sound emitted from the target object, and detects the time section of the sound emitted from the target object.

[0066] As described above, the section detection unit 32 receives the recorded signal and the motion information as input, and outputs sound source type section information and sound source type information in response to the input.

[0067] In this case, for example, as shown in Fig. 2, a moving object equipped with a recording device 11 is an object, each part of the object is an object sound source, and the sound of the object sound source is emitted from the object. More specifically, a musical instrument held by the object can also be an object sound source.

[0068] Moreover, in the recording device 11 and the server 12, what types of object sound sources are available are determined in advance.

[0069] For example, the types of object sound sources, that is, the types of sounds of the object sound sources, may be common to all contents and may differ for each content.

[0070] Specifically, as shown on the right side of Figure 2, examples of types of object sound sources that are common to all content include the voice emitted by a person who is an object, the sound of that person walking, running, clapping, etc.

[0071] Also, for example, examples of types of object sound sources specific to sports content include passing sounds, shooting sounds, whistle sounds, etc., and examples of types of object sound sources specific to music content include instrument sounds, etc. Furthermore, examples of types of object sound sources specific to theater, dance, etc. include sounds related to the actions of performers, such as the rustling of clothes and stepping sounds.

[0072] The section detection unit 32 generates sound source type section information indicating which type of object sound source sound is included in which time section of the recorded signal.

[0073] This sound source type section information may be in any format, such as binary information indicating 0 or 1, or probability information expressed by continuous values, as shown in FIG. 3, and may be section information for a time signal or section information for each frequency bin.

[0074] For example, in the example shown in the upper left of FIG. 3, the sound source type section information is binary information for each object sound source, which indicates whether or not the sound of the object sound source is included at each time of the recorded signal, which is a time signal.

[0075] In this example, each broken line indicates whether or not the object sound sources "background noise," "walking / running sounds," "shooting sounds," and "voice" are included at each time.

[0076] In particular, the horizontal direction of each polygonal line indicates time, and an interval where the polygonal line is convex upward indicates that the interval includes the sound of an object sound source.

[0077] In the example shown in the upper right of FIG. 3, the sound source type section information is continuous value information for each object sound source, indicating the probability value that the sound of the object sound source is included at each time of the recorded signal, which is a time signal.

[0078] In this example, each curve indicates the probability value that the object sound sources "background noise," "walking sound / running sound," "shooting sound," and "voice" are included at each time.

[0079] The continuous probability value in which the sound of each object sound source is included is, for example, the output value of a DNN (Deep Neural Network) obtained by machine learning when detecting object sound sources as a multi-class classification problem.

[0080] Furthermore, in the example shown in the lower left of FIG. 3, the sound source type section information is binary information in a time-frequency mask format generated for each type of object sound source.

[0081] In this binary information in the time-frequency mask format, for each component of each time-frequency bin of the recorded signal, whether or not the sound of the object sound source is included in each time interval (time) of the recorded signal is expressed by a binary value. In particular, in this example, the vertical axis indicates the time-frequency bin, and the horizontal axis indicates the time.

[0082] 3, the sound source type section information is generated for each type of object sound source as continuous value information in the time-frequency mask format. In this example, the vertical axis also indicates the time-frequency bin, and the horizontal axis indicates the time.

[0083] In this continuous value information in the time-frequency mask format, the probability that the sound of an object sound source is included in each time interval (time) of the recorded signal is expressed by a continuous value for each component of each time-frequency bin of the recorded signal.

[0084] Note that the sound source type section information is not limited to the example shown in FIG. 3 and may be in any format. The format of the sound source type section information to be used may be appropriately determined depending on the type of signal processing to be performed in the downstream signal processing unit 41.

[0085] In generating the sound source type section information, the section detection unit 32 detects the sounds of the various object sound sources from each time section of the recorded signal. In other words, the time sections of the sounds of the various object sound sources are detected.

[0086] The motion information obtained by the recording device 11 is information that indicates the motion and orientation of the object when sound is picked up by the microphone 21 to obtain a recording signal.

[0087] Therefore, by detecting the time intervals of the sound of the object sound source using the motion information, it is possible to distinguish whether the sound contained in each time interval in the recorded signal is a sound emitted from the object or a sound emitted from a surrounding object.

[0088] For example, object sound sources include various motion sounds such as walking sounds, running sounds, clapping sounds, soccer shot sounds, and dance step sounds.

[0089] As one method for detecting the time period of the operation sound, for example, a method for detecting the time period of the operation sound by a simple algorithm such as threshold processing using a threshold can be adopted.

[0090] In such a case, for example, a time period during which the sensing value of a sensor as motion information falls within a specific range determined for the operation sound to be detected is regarded as the time period of the operation sound.

[0091] In addition, a classifier such as a DNN may be generated by multimodal learning, and the classifier may be used to detect the time period of the operation sound.

[0092] In this case, for example, a classifier such as a DNN is generated by learning, which takes the sensor values ​​of sensors such as an acceleration sensor, geomagnetic sensor, and gyro sensor, which are obtained as movement information, and the recorded signal as input, and outputs the presence or absence of operation sound in each time interval of the recorded signal.

[0093] As the classifier, for example, one that detects a plurality of operation sounds common to all contents, or one that detects operation sounds specific to a content, can be used.

[0094] Here, a specific example of detecting a time period of an operation sound will be described.

[0095] For example, when detecting a time period of an object's walking sound or running sound as an action sound, a sensor value indicating the upward and downward acceleration of the object measured by an acceleration sensor may be used as the motion information.

[0096] In this case, the walking or running of the object can be detected by the fluctuation of the sensor value, and for example, a time period in which the frequency of the time waveform of the sensor value, i.e., the vibration frequency, is about 2 Hz or less is determined as a period in which the object is walking, i.e., a time period in which the walking sound is generated. Similarly, for example, a time period in which the vibration frequency of the sensor value is about 3 Hz to 4 Hz is determined as a period in which the object is running, i.e., a time period in which the running sound is generated.

[0097] Furthermore, when detecting a time interval of a ball kick sound as an action sound or a time interval of a shot-related sound in a ball game, information such as a rotation angle indicating the rotation of an object, measured mainly by a gyro sensor, can be used as the motion information, because a person as an object rotates his or her body when performing a ball kicking action or a shot action.

[0098] Furthermore, when detecting the time interval of the sound of a finger snap as an action sound or the time interval of the sound of an object hitting its own body, for example, fluctuations in sensor values ​​such as an acceleration sensor, gyro sensor, and geomagnetic sensor may be used.

[0099] In this case, for example, an acceleration sensor, gyro sensor, or geomagnetic sensor serving as the motion measuring unit 22 is attached to the torso, wrist, arm, etc. of a person as an object, and the movement of the torso, hand, etc. of the object is detected based on the amount of variation in the sensor value according to the part where it is attached.

[0100] Furthermore, for example, when detecting a time period of breathing sounds of a person, which is an object, as a time period of an action sound, a sensor value indicating minute vertical displacement of the object measured by an acceleration sensor can be used as motion information.

[0101] In this case, the breathing movement of the object can be detected from fluctuations in the sensor value. For example, a time period in which the frequency of the sensor value is approximately 0.5 Hz to 1 Hz is determined to be a time period in which breathing movement is occurring and audible breathing sounds are recorded, i.e., a time period in which the breathing sounds of the object are recorded.

[0102] Furthermore, by utilizing the characteristics of each movement when an object such as those described above makes a sound, the time period of the sound of each object sound source may be detected using a classifier such as a DNN that receives the recorded signal and movement information as input and outputs the presence or absence of sound from the object sound source.

[0103] For example, as shown in FIG. 4, by using a recorded signal and a sensor value (sensor signal) of an acceleration sensor as motion information, a classifier can detect the time period of walking sounds as sounds of an object sound source with high accuracy.

[0104] In Fig. 4, the part indicated by arrow Q11 shows the time waveform of the recorded signal, and the part indicated by arrow Q12 shows the spectrum of the recorded signal. Additionally, the part indicated by arrow Q13 shows the time waveform of the sensor signal of the acceleration sensor, and the part indicated by arrow Q14 shows the spectrum of the sensor signal. In each part indicated by arrows Q11 to Q14, the horizontal direction in the figure indicates time.

[0105] In this example, for example, in a portion indicated by an arrow A11 in the recorded signal, the walking sound of the target object and the walking sounds of other objects around the target object are mixed together.

[0106] In such cases, it is difficult to determine whether the walking sound components contained in the recorded signal are from the target object or from another object based only on the time waveform and spectrum of the recorded signal.

[0107] Therefore, in this example, not only the recorded signal but also the sensor signal (motion information) is used to determine (discriminate) whether the signal is from the target object or another object.

[0108] The time waveform of the sensor signal indicated by arrow Q13 fluctuates periodically in the vertical direction, and this time waveform, that is, the value of the vertical component, indicates the vertical component of the floor reaction force of the target object.

[0109] In particular, here, for example, the part indicated by arrow A12, a single upwardly convex mountain part in the figure corresponds to one step of the body movement of the target object, and it can be seen that the sensor signal contains information indicating the body movement of the target object with a high signal-to-noise ratio.

[0110] It can also be seen that the shading pattern in the spectrum of the sensor signal indicated by arrow Q14 clearly corresponds to the time waveform of the sensor signal indicated by arrow Q13.

[0111] In this way, the sensor signal contains information indicating the body movement of the target object with a high S / N ratio, but does not contain any information indicating the body movements of other objects.

[0112] Therefore, by using the recorded signal and the motion information, it is possible to detect the time period of the sound of the object sound source of the target object with high accuracy.

[0113] Specifically, for example, if the sound of another object is included in the recorded signal at the same sound pressure as the sound of the target object, the time period of the sound of the target object cannot be detected with high accuracy using only the recorded signal. However, by using not only the recorded signal but also the motion information, the time period of the sound of the target object can be detected with high accuracy.

[0114] Generally, many methods have been proposed in the field of behavior recognition to estimate the behavior of an object from sensor values ​​such as acceleration sensors, gyro sensors, and geomagnetic sensors.

[0115] In contrast, the section detection unit 32 uses the recorded signal and movement information as described above to distinguish between sounds emitted from the target object and sounds emitted from other objects according to the type of sound of the object sound source.

[0116] In addition, in fields such as behavior recognition and physical therapy, walking and running are considered to be continuous movements, and are often described as continuous state transitions such as a stance phase and a swing phase.

[0117] In contrast, the interval detection unit 32 detects, for example, the time interval during which actual walking sounds or running sounds occur, i.e., the time interval from when the human foot as an object, or more specifically, the heel or toe, touches the ground to when it takes off, as the time interval of the walking sounds or running sounds.

[0118] In addition, the time interval of the object's speech as the sound of the object sound source can also be detected with high accuracy from the motion information.

[0119] For example, when the movement measurement unit 22 is attached around the neck or head of a person as an object, when the target object speaks, information indicating body movements caused by the speech is observed with a high signal-to-noise ratio in the sensor signal as movement information.

[0120] Therefore, just as in the case of action sounds, by using the recorded signal and movement information for the time period of speech, it is possible to distinguish with high accuracy between sounds emitted from the target object and sounds emitted from other objects.

[0121] It may also be impossible to obtain motion information that contains information indicating the body movement of the target object when speaking with a high signal-to-noise ratio.

[0122] However, in such a case, for example, when the speech of a target object wearing recording device 11 is picked up by each of the multiple microphones that make up microphone 21, the property that the direction from which the speech arrives at each microphone is approximately constant can be utilized.

[0123] Specifically, for example, in the section detection unit 32, DS (Delay and Sum Beamforming) is performed on the recorded signals obtained by each of the multiple microphones, thereby emphasizing the components of the recorded signals in the direction from which the sound of the target object arrives.

[0124] By using the recorded signal thus obtained and the motion information, it is possible to discriminate with high accuracy between the speech of the target object and the speech of other objects.

[0125] Also, for example, the segment detection unit 32 may use a null beamformer (NBF) to suppress the speech component of the target object contained in the recorded signal.

[0126] In this case, a time period of the speech of the target object detected from the recorded signal before suppression is compared with a time period of the speech of the target object detected from the recorded signal after suppression. Then, among the time periods detected from the recorded signal before suppression, a time period that is not detected from the recorded signal after suppression is determined as the final time period of the speech of the target object.

[0127] Next, the processing performed in the signal processing unit 41 will be described in more detail.

[0128] The signal processing unit 41 performs signal processing based on the sound source type section information obtained by the section detection unit 32, the movement information, the position information, and the recorded signal, to generate an object sound source signal, which is an audio signal for each type of object sound source.

[0129] For example, the signal processing unit 41 performs signal processing on the recorded signal, such as sound quality correction, sound source separation, noise removal, distance correction, sound source replacement, and a combination of two or more of these processes.

[0130] More specifically, for example, sound quality correction processing is performed to improve the quality (sound quality) of the sound of the object sound source by suppressing unintended sounds such as noise that occur at the contact point between the recording device 11 and the object when the object moves.

[0131] In other words, sound quality correction processing includes processes for suppressing noise, such as filter processing and gain correction to suppress frequency bands where noise is dominant, and muting sections that contain a lot of noise, unnecessary sections, or sections that contain sounds that are inappropriate when watching content.

[0132] It is possible to detect a time section including an inappropriate sound, for example, from sound source type section information or by performing a voice recognition process on a recorded signal.

[0133] Also, the sound quality correction process may be a process for improving the quality of the sound of the object sound source, such as a filter process for increasing high-frequency components in a time section of the recorded signal that includes the sound of the object sound source, which is prone to attenuation of high frequencies. In such a case, for example, a process determined for each type of object sound source may be performed as the sound quality correction process for each time section of the recorded signal based on the sound source type section information.

[0134] Furthermore, by referring to the sound source type section information, for example, it is possible to identify a time section in which sounds from a plurality of object sound sources are included in the recorded signal.

[0135] Therefore, based on the identification results, a sound source separation process based on independent component analysis that separates the sounds of each object sound source according to differences in amplitude values ​​and probability density distribution for each type of object sound source may be performed on the recorded signal.

[0136] Also, beamforming or the like may be performed as a sound source separation process according to the difference in the direction of the object sound source as seen from the object, and the sound signal of each object sound source may be separated from the recorded signal.

[0137] Furthermore, if it is determined from the sound source type section information that the time section of the recorded signal contains only the sound of one object sound source, the signal of that time section is extracted as an object sound source signal as a sound source separation process.

[0138] Through these processes, a signal containing only the sound of one object sound source can be obtained, and this signal can be used as an object sound source signal.

[0139] Also, similar to the sound quality correction process, when unwanted sounds in the recorded signal, such as background noise, steady noises such as cheers, and noises from wind, are included in the time interval of the sound of the object sound source, a noise removal process may be performed to suppress the noise for that time interval.

[0140] In addition, for example, from the position information and movement information of each object, it is possible to determine whether there are other objects around the target object, the relative orientation of the other objects with respect to the target object, and the distance from the target object to the other objects.

[0141] Therefore, based on these identification results and the sound source type section information, it is possible to identify whether the time section containing the sound of the object sound source of the target object contains the sound of other objects, and it is possible to extract (separate) only the sound of the object sound source for the target object by sound source separation using DNN.

[0142] In addition, in such sound source separation, for example, as shown in Fig. 5, other objects located within a region R11 of a circle having a predetermined radius centered on a target object OB11 are set as objects to be removed. In Fig. 5, each point represents an object.

[0143] Also, taking into consideration the distance to the object to be removed and the relative orientation of the object to be removed, processing such as sound source separation is performed for a time interval in which the sound of the object sound source in the recorded signal of the target object OB11 is included, to remove the sound of the object to be removed that is included in that time interval. In other words, the sound signal of the object sound source for the target object OB11 is extracted.

[0144] At this time, the distance from the target object OB11 to the object to be removed can be obtained from the position information of those objects, and the relative orientation of the object to be removed from the target object OB11 can be obtained from the orientation and position information indicated by the motion information of those objects.

[0145] Furthermore, objects outside the region R11, that is, objects that are a predetermined distance or more away from the target object OB11, are not considered to be objects to be removed.

[0146] This is because sounds from distant objects that mix with the recorded signal of the target object OB11 are attenuated by distance, and therefore there is no need to take into account the voices and operation sounds of such objects.

[0147] Furthermore, when removing (separating) the sound of the object to be removed, the gain or intensity at the time of separation of the sound of the object to be removed may be changed according to the distance from the target object OB11 to the object to be removed. In other words, the amount of mixing (contribution rate) may be treated as changing continuously according to the distance.

[0148] In addition, for example, the distance correction process performed as signal processing is a process that corrects the influence of the distance attenuation and transfer characteristics from the object sound source to the position of the microphone 21 during recording, which are convoluted with the absolute sound pressure of the sound emitted by the object sound source.

[0149] Specifically, for example, as the distance correction process, a process may be performed in which the inverse characteristic of the transfer characteristic from the object sound source to the microphone 21 is added to the recorded signal.

[0150] This makes it possible to correct deterioration in sound quality of the object sound source due to distance attenuation, transmission characteristics, etc., and also to restore the relative relationship of the absolute sound pressure of the sounds of each object sound source according to the positional relationship of each object sound source during content playback.

[0151] Furthermore, for example, the sound source replacement process, which is performed as signal processing, is a process in which the sound of a specific object sound source type indicated by the sound source type section information is replaced with a sound different from the recorded sound, such as a sound prepared in advance, to generate an object sound source signal.

[0152] In other words, in the sound source replacement process, a portion of a recorded signal, or a portion of an object sound source signal obtained from a recorded signal, is replaced with another audio signal that has been prepared in advance or that has been dynamically generated, based on the sound source type section information.

[0153] Here, for example, a signal with a high S / N ratio prepared in advance according to the type of object sound source can be used as the object sound source signal. Such a sound source replacement process is particularly effective when the amplitude of the sensor value as motion information is large, that is, the movement of the object is large, and the sound quality of the recorded object sound source is not good. Therefore, for example, whether or not to perform the sound source replacement process may be determined based on the result of threshold processing for the motion information.

[0154] Also, for example, in the sound source replacement process, a sound signal that is parametrically generated by substituting acceleration or the like as motion information into a function may be used as the object sound source signal.

[0155] Furthermore, for example, in a sound source replacement process, if there is a time period in which the sound is not appropriate as an object sound source when viewing the content, a predetermined sound signal prepared in advance may be used as the object sound source signal for that time period.

[0156] The object sound source signal obtained by the signal processing unit 41 may be a signal covering only the time interval in which the sound of the object sound source is present, or a signal covering the entire time interval but with a silent signal during the time interval in which the sound of the object sound source is not present.

[0157] In addition, the sound quality correction process, sound source separation process, noise removal process, distance correction process, and sound source replacement process described above can be realized in any case, such as when processing is performed online for each frame of the recorded signal, when processing is performed using a look-ahead frame, when processing is performed offline, etc. In this case, it is sufficient to hold the recorded signal, sound source type section information, motion information, position information, etc. for frames older than the frame to be processed of the recorded signal as necessary.

[0158] <Explanation of recording process> Next, the operations of the recording device 11 and the server 12 will be described.

[0159] First, there will be described the operation of the recording device 11. The recording device 11 is attached to an object and performs a recording process for a predetermined period of time, such as while the object is performing a performance or playing a game.

[0160] The recording process by the recording device 11 will be described below with reference to the flowchart of FIG.

[0161] In step S11, the recording unit 24 records surrounding sounds.

[0162] That is, when the microphone 21 picks up surrounding sounds and outputs the resulting recorded signal, the recording unit 24 obtains the recorded signal output from the microphone 21, thereby obtaining a recorded signal of the recorded sound.

[0163] In step S12, the recording unit 24 acquires the motion information and the position information from the motion measuring unit 22 and the position measuring unit 23.

[0164] The recording unit 24 performs AD conversion and the like as necessary on the recorded signal, the movement information, and the position information obtained in this manner, and supplies them to the transmission unit 25.

[0165] The transmitting unit 25 also generates transmission data consisting of the recorded signal, the motion information, and the position information supplied from the recording unit 24. At this time, the transmitting unit 25 performs compression processing on the recorded signal, the motion information, and the position information as necessary.

[0166] In step S 13 , the transmission unit 25 transmits the transmission data to the server 12 .

[0167] Note that, here, an example will be described in which the transmission data obtained by recording is sequentially sent to the server 12 in real time (online) during recording, but it is also possible to accumulate the transmission data during recording and then send all the transmission data to the server 12 in one go offline after recording.

[0168] In step S14, the recording unit 24 determines whether or not to end the process. For example, when a button (not shown) provided on the recording device 11 is operated to instruct the end of recording, it is determined that the process is to be ended.

[0169] If it is determined in step S14 that the process is not yet to be ended, the process returns to step S11, and the above-mentioned process is repeated.

[0170] On the other hand, if it is determined in step S14 that the process is to be ended, each unit of the recording device 11 stops the operation that it is performing, and the recording process ends.

[0171] In this manner, the recording device 11 collects sound and measures the movement and position of the object, and transmits transmission data including the recording signal, the movement information, and the position information to the server 12. In this manner, the server 12 can obtain a high-quality target sound.

[0172] <Description of Data Generation Process> Furthermore, the server 12 performs data generation processing and outputs object sound source data when transmission data is transmitted from each recording device 11. Hereinafter, the data generation processing performed by the server 12 will be described with reference to the flowchart of FIG.

[0173] In step S 41 , the receiving unit 31 receives the transmission data sent from the recording device 11 .

[0174] Furthermore, the receiving unit 31 performs decompression processing on the received transmission data as necessary, and extracts the recorded signal, the motion information, and the position information from the transmission data.

[0175] The receiving unit 31 then supplies the recorded signal and motion information to the section detection unit 32, supplies the recorded signal, motion information, and position information to the signal processing unit 41, and supplies the motion information and position information to the metadata generation unit .

[0176] In step S42, the section detection unit 32 generates sound source type section information for each object (recording device 11) based on the recording signal and motion information of the object supplied from the receiving unit 31, and supplies the generated information to the signal processing unit 41.

[0177] For example, the section detection unit 32 performs threshold processing on the recorded signal as described above, performs calculations by substituting the recorded signal and motion information into a classifier such as a DNN, or performs DS or NBF on the recorded signal to identify the type of object sound source contained in each time section, and generates sound source type section information.

[0178] In addition, the section detection unit 32 generates sound source type information indicating the type of the object sound source and the object of the sound contained in the recorded signal based on the result of identifying the type of the object sound source contained in each time section, and supplies the sound source type information to the metadata generation unit 42.

[0179] In step S 43 , the signal processing unit 41 generates an object sound source signal based on the recorded signal, the motion information, and the position information supplied from the receiving unit 31 and the sound source type section information supplied from the section detection unit 32 .

[0180] Specifically, the signal processor 41 generates an object sound source signal for each object by appropriately performing the above-mentioned sound quality correction process, sound source separation process, noise removal process, distance correction process, and sound source replacement process on the recorded signal. At this time, not only the motion information, position information, and sound source type section information of the target object, but also the motion information, position information, and sound source type section information of other objects are used to generate the object sound source signal of the target object.

[0181] In step S44, the metadata generating unit 42 generates metadata including the sound source type information supplied from the section detecting unit 32 and the motion information and position information supplied from the receiving unit 31 for each object sound source of each object.

[0182] When the object sound source signal and metadata are obtained for each object sound source in this manner, the object sound source data generating unit 33 outputs the object sound source data including the object sound source signal and the metadata for each object sound source to a subsequent stage.

[0183] In step S45, the server 12 determines whether or not to end the process. For example, in step S45, if all the transmission data received from the recording device 11 has been processed, it is determined that the process is to end.

[0184] If it is determined in step S45 that the process is not yet to be ended, then the process returns to step S41, and the above-mentioned process is repeated.

[0185] On the other hand, if it is determined in step S45 that the process is to be ended, each unit of the server 12 stops the process being performed, and the data generation process ends.

[0186] In the above description, the recording device 11 sequentially transmits transmission data in real time, and the server 12 also sequentially generates object sound source data from the transmission data.

[0187] However, the transmission data from the recording device 11 may be stored and the stored transmission data may be processed in a batch to generate the object sound source data. Also, when the transmission data is sent in a batch from the recording device 11, the received transmission data may be processed in a batch to generate the object sound source data.

[0188] In this manner, the server 12 receives transmission data from a plurality of recording devices 11, generates object sound source data from the transmission data, and outputs the object sound source data.

[0189] At this time, by generating sound source type section information using not only the recorded signal but also the motion information, and then using that sound source type section information to generate object sound source data, a high-quality target sound, i.e., an object sound source signal, can be obtained.

[0190] Second embodiment <Example of recording system configuration> In the above, an example has been described in which the sound source type section information of each object is generated without using information obtained about other objects. However, for example, the sound source type section information may be improved in accuracy by integrating information obtained about each object.

[0191] In such a case, the recording system is configured, for example, as shown in Fig. 8. In Fig. 8, the parts corresponding to those in Fig. 1 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0192] The recording system shown in FIG. 8 includes a recording device 11 and a server 12, and the configuration of the recording device 11 is the same as that in FIG.

[0193] 8, the server 12 includes a receiving unit 31, a section detecting unit 32, an integrating unit 71, and an object sound source data generating unit 33. The object sound source data generating unit 33 includes a signal processing unit 41 and a metadata generating unit 42.

[0194] The configuration of this server 12 differs from that of the server 12 shown in FIG. 1 in that an integration unit 71 is newly provided, but in other respects it has the same configuration as the server 12 shown in FIG.

[0195] In this example, the sound source type section information generated by the section detection unit 32 is supplied to the integration unit 71. In addition to the sound source type section information from the section detection unit 32, the integration unit 71 is also supplied with the recorded signal, the motion information, and the position information from the receiving unit 31.

[0196] Based on these recorded signals, sound source type section information, movement information, and position information, the integration unit 71 generates final sound source type section information and supplies it to the signal processing unit 41, and also generates sound source type information and supplies it to the metadata generation unit 42.

[0197] In particular, the integrating unit 71 integrates each piece of information such as the motion information and the position information obtained by each recording device 11, thereby generating more accurate sound source type section information.

[0198] In the following, an example will be described in which the integrating unit 71 is provided separately from the section detection unit 32, but the integrating unit 71 may be provided in the section detection unit 32. In such a case, the section detection unit 32 simultaneously performs the processing described above and the processing of the integrating unit 71 described below to generate sound source type section information and sound source type information.

[0199] The integrating unit 71 will now be described in more detail.

[0200] For example, the section detection unit 32 detects a time section that is estimated to be the operation sound or voice of the object for each object, that is, for each recording device 11, and generates sound source type section information.

[0201] However, there are still cases where a time period of the operation sound or sound of another object is mistakenly detected as a time period of the operation sound or sound of the target object, or a time period of the operation sound or sound of the target object that should be detected is not detected.

[0202] Therefore, the integration unit 71 integrates the information obtained from each recording device 11, thereby making it possible to obtain more accurate sound source type section information.

[0203] Specifically, the integration unit 71 obtains final sound source type section information by performing position information comparison processing, time section integration processing, and section smoothing processing based on the sound source type section information, recorded signal, motion information, and position information, for example, for each frame of a predetermined time length.

[0204] In other words, the integration unit 71 generates sound source type section information of the target object based on at least one of the recorded signal, movement information, and position information of the target object and the recorded signals, movement information, and position information of other objects.

[0205] Examples of the position information comparison process, the time interval integration process, and the interval smoothing process will be described in further detail below.

[0206] First, all objects are selected in sequence as target objects, and a position information comparison process, a time interval integration process, and an interval smoothing process are performed for each target object.

[0207] In the position information comparison process, the distance from the target object to other objects is calculated based on the position information of each object.

[0208] Then, based on the determined distance, another object that may have an effect on the sound of the object sound source of the target object, that is, another object near the target object, is selected as a reference object.

[0209] Specifically, for example, an object whose distance to the target object is equal to or less than a predetermined threshold is selected as the reference object. In this example, since each object is equipped with a recording device 11, the distance between the recording devices 11 is approximately equal to the distance between the objects, and therefore the distance calculated from the position information is used to select the reference object.

[0210] In this example, a reference object is selected based on the distance, and information about the reference object is used to perform the time interval integration process.

[0211] However, all objects may be treated as reference objects, and time interval integration processing may be performed using information on the reference objects with a weight according to the distance from the target object.

[0212] In the time interval integration process, first, it is determined whether or not there is an object selected as a reference object in the position information comparison process.

[0213] Then, when no object is selected as the reference object, the sound source type section information of the target object obtained by the section detection unit 32 is output as it is as the final sound source type section information to the signal processing unit 41. This is because, when there is no other object near the target object, the sound of the other object is not mixed into the recorded signal.

[0214] On the other hand, if there are objects selected as reference objects, the sound source type section information of the target object is updated using the position information and motion information of the reference objects, i.e., the final sound source type section information is generated.

[0215] Specifically, among the reference objects, a reference object having a section overlapping with a time section indicated by the sound source type section information of the target object as a time section of the sound of the object sound source is selected as the final reference object.

[0216] In other words, even if an object is determined to be a reference object in the position information comparison process, if the time period indicated by the sound source type section information does not overlap with the time period indicated by the sound source type section information of the target object, the object is excluded from the reference objects.

[0217] Next, based on the position information and movement information of the reference object and the position information and movement information of the target object, the relative orientation (direction) of the reference object as seen from the target object in the three-dimensional space is estimated, and relative orientation information indicating the estimation result is generated. More specifically, for example, the direction (orientation) of the mouth of the reference object as seen from the front direction of the target object is estimated. Note that the generation of the relative orientation information may be made using only the position information, without using the movement information.

[0218] Furthermore, an NBF filter is formed based on the position information of the target object, the orientation of the target object indicated by the motion information, and the relative orientation information of each reference object.

[0219] This NBF filter is a filter that realizes beamforming that maintains the gain of sounds coming from the direction of the target object's mouth, which is indicated by the orientation of the target object, while suppressing sounds coming from the direction indicated by the relative orientation information.

[0220] The integration unit 71 performs a convolution process between the NBF filter obtained in this manner and a time section indicated by the sound source type section information of the target object in the recorded signal of the target object.

[0221] Furthermore, the integration unit 71 performs processing similar to that performed by the section detection unit 32, i.e., threshold processing and arithmetic processing using a classifier such as DNN, based on the signal obtained by the convolution processing and the motion information of the target object, to generate sound source type section information. In this way, the sound emitted from the reference object is suppressed, and more accurate sound source type section information can be obtained.

[0222] In addition, in the computational processing in a classifier such as a DNN, motion information and position information of a reference object, recorded signals, etc. may also be used as input.

[0223] Finally, the integration unit 71 performs a section smoothing process on the sound source type section information obtained by the time section integration process, thereby obtaining final sound source type section information.

[0224] For example, for each type of object sound source, the average minimum duration of a sound when the sound of the object sound source of that type is generated is obtained in advance as an average minimum duration.

[0225] In the section smoothing process, smoothing is performed using a smoothing filter that connects the time sections of the sound of the object sound source that have been fragmented (divided) so that the length of the time section in which the sound of the object sound source is detected is equal to or longer than the average minimum duration.

[0226] In other words, in the section smoothing process, multiple consecutive time sections in which the same type of object sound source sound is detected in the recorded signal are connected to form a final time section, where the multiple time sections to be connected include at least one time section whose duration is less than the average minimum duration.

[0227] For example, the integration unit 71 holds in advance a smoothing filter that is generated based on the average minimum duration of each object sound source type.

[0228] The integration unit 71 performs filtering (filtering) on ​​the sound source type section information obtained in the time section integration process as section smoothing processing based on the smoothing filter, obtains final sound source type section information, and supplies it to the signal processing unit 41. In the section smoothing processing, filtering may be performed on sound source type section information of multiple consecutive frames depending on the type of object sound source, i.e., the average minimum duration.

[0229] Furthermore, the integration unit 71 generates sound source type information from the obtained sound source type section information, and supplies the generated information to the metadata generation unit 42.

[0230] In this manner, the integration unit 71 can remove information about the sounds of other objects that was not completely removed (excluded) from the sound source type section information obtained by the section detection unit 32, thereby obtaining more accurate sound source type section information.

[0231] For example, the section detection unit 32 may perform DS or NBF as necessary on the recorded signal as described above.

[0232] However, for example, in DS, the components of the direction from which the sound of the target object is coming may not be sufficiently emphasized, and in such a case, if the volume of the sound of other objects is loud, it may not be possible to obtain correct sound source type section information.

[0233] Also, for example, even in the case of NBF, if there is another object in the vicinity of the target object in a direction close to the direction from which the sound of the target object arrives, and the target object and the other object speak at approximately the same time, correct sound source type section information may not be obtained.

[0234] On the other hand, the integration unit 71 can obtain more accurate sound source type section information by using not only information on the target object but also motion information and position information of other objects and sound source type section information.

[0235] <Description of Data Generation Process> When the recording system has the configuration shown in FIG. 8, each recording device 11 performs the recording process described with reference to FIG.

[0236] Then, the server 12 performs the data generation process shown in Fig. 9. Hereinafter, the data generation process by the server 12 shown in Fig. 8 will be described with reference to the flowchart in Fig. 9.

[0237] The processes in steps S71 and S72 are similar to those in steps S41 and S42 in FIG. 7, and therefore will not be described.

[0238] However, in step S 71 , the recorded signal, the motion information, and the position information extracted from the transmission data by the receiving unit 31 are also supplied to the integrating unit 71 .

[0239] In addition, in step S72, the generated sound source type section information is supplied from the section detection unit 32 to the integration unit 71.

[0240] In step S 73 , the integration unit 71 integrates the information supplied from the section detection unit 32 and the information supplied from the receiving unit 31 .

[0241] That is, the integration unit 71 performs position information comparison processing, time interval integration processing, and interval smoothing processing based on the recorded signal, movement information, and position information supplied from the receiving unit 31 and the sound source type interval information supplied from the interval detection unit 32, to obtain final sound source type interval information.

[0242] The integration unit 71 supplies the obtained final sound source type section information to the signal processing unit 41, and also generates sound source type information from the final sound source type section information and supplies it to the metadata generation unit .

[0243] Once the sound source type section information is obtained in this manner, the processes of steps S74 to S76 are performed and the data generation process is terminated. However, since these processes are similar to the processes of steps S43 to S45 in FIG. 7, their description will be omitted.

[0244] In this manner, the server 12 receives transmission data from a plurality of recording devices 11, generates object sound source data from the transmission data, and outputs the object sound source data.

[0245] At this time, by generating the final sound source type section information of the target object using information of other objects as well, it is possible to obtain a target sound of even higher quality.

[0246] Third embodiment <Example of recording system configuration> In the above description, the recorded signal and position information are used to generate the sound source type section information, but video information may also be used.

[0247] In such a case, the recording system is configured, for example, as shown in Fig. 10. In Fig. 10, the parts corresponding to those in Fig. 8 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0248] The recording system shown in FIG.

[0249] In this example, the recording device 11 has a microphone 21 , a movement measuring unit 22 , a position measuring unit 23 , an image capturing unit 101 , a recording unit 24 , and a transmission unit 25 .

[0250] The configuration of the recording device 11 shown in FIG. 10 differs from that of the recording device 11 shown in FIG. 8 in that a new image capturing unit 101 is provided, but in other respects it has the same configuration as the recording device 11 in FIG.

[0251] The photographing unit 101 is made up of a small camera, and photographs an image including a part of an object as a subject, for example, with the position of the object as a viewpoint, and supplies the obtained image information (image signal) to the transmission unit 25. Note that there may be cases where the object is not included as a subject in the image based on the image information.

[0252] The transmission unit 25 generates transmission data including the recorded signal, the motion information, and the position information supplied from the recording unit 24 and the video information supplied from the imaging unit 101 , and transmits the transmission data to the server 12 .

[0253] The server 12 also has a receiving unit 31, a section detecting unit 32, an integration unit 71, and an object sound source data generating unit 33, and the object sound source data generating unit 33 is provided with a signal processing unit 41 and a metadata generating unit .

[0254] That is, the configuration of the server 12 shown in FIG. 10 is the same as the configuration of the server 12 shown in FIG. 8, but in the server 12 shown in FIG. 10, the video information extracted from the transmission data by the receiving unit 31 is supplied to the section detection unit 32 and the integration unit 71.

[0255] Therefore, in the section detection section 32, sound source type section information is generated based on the recorded signal, the motion information, and the video information supplied from the receiving section 31.

[0256] For example, when a part of a target object is included as a subject in an image based on video information, the video information is used to detect the motion of the target object.

[0257] Specifically, for example, the sound source type section information is corrected based on the movement of the target object at each time point detected based on the video information.

[0258] Alternatively, for example, the video information, movement information, and recorded signal may be substituted into a classifier such as a DNN for calculation, and the presence or absence of motion sound at each time of the recorded signal may be obtained.

[0259] Similarly, the integrating unit 71 also performs position information comparison processing, time interval integration processing, and interval smoothing processing based on the recorded signal, the motion information, the position information, the video information, and the sound source type interval information.

[0260] At this time, the video information may be used for detecting the movement of the target object and for time interval integration processing, as in the case of the interval detection unit 32, or may be used for detecting whether there are other objects around the target object and for detecting the movement of other objects, etc.

[0261] <Explanation of recording process> Next, the operation of the recording device 11 and the server 12 shown in FIG. 10 will be described.

[0262] First, the recording process by the recording device 11 will be described with reference to the flowchart of FIG.

[0263] The processes in steps S101 and S102 are similar to those in steps S11 and S12 in FIG. 6, and therefore will not be described.

[0264] In step S103 , the photographing unit 101 photographs an object, that is, the surroundings of the recording device 11 as a subject, and supplies the resulting video information to the transmission unit 25 .

[0265] The transmission unit 25 generates transmission data including the video information supplied from the imaging unit 101 and the recording signal, the motion information, and the position information supplied from the recording unit 24 .

[0266] After the transmission data is generated, the processes of steps S104 and S105 are performed and the recording process ends. However, since these processes are similar to the processes of steps S13 and S14 in FIG. 6, their description will be omitted.

[0267] In this manner, the recording device 11 captures the surrounding subject, generates transmission data including the obtained video information, and transmits the data to the server 12. In this manner, the server 12 can obtain a higher quality target sound by using not only the motion information and the position information but also the video information.

[0268] <Description of Data Generation Process> Next, the data generation process performed by the server 12 shown in FIG. 10 will be described with reference to the flowchart in FIG.

[0269] 9, the processing in step S131 is the same as the processing in step S71 in Fig. 9, so a description thereof will be omitted. However, in step S131, the receiving unit 31 also extracts video information from the transmission data and supplies it to the section detection unit 32 and the integrating unit 71.

[0270] In step S132 , the section detection unit 32 generates sound source type section information based on the recorded signal, the motion information, and the video information supplied from the receiving unit 31 , and supplies the information to the integration unit 71 .

[0271] In step S132, the same process as in step S72 in FIG. 9 is performed, but here, the video information is used to detect the motion of the target object, and sound source type section information is generated.

[0272] In step S133, the integration unit 71 integrates the information supplied from the section detection unit 32 and the receiving unit 31 to generate final sound source type section information.

[0273] In step S133, the same process as in step S73 in Fig. 9 is performed, but here, not only the sound source type section information, the recorded signal, the motion information, and the position information, but also the video information is used to perform the position information comparison process, the time section integration process, and the section smoothing process. That is, for example, the video information is used for selecting the reference object and generating the relative orientation information.

[0274] In this way, when the final sound source type section information is obtained, the processes of steps S134 to S136 are then performed and the data generation process is terminated. However, since these processes are similar to the processes of steps S74 to S76 in FIG. 9, their description will be omitted.

[0275] In this manner, the server 12 receives transmission data from a plurality of recording devices 11, generates object sound source data from the transmission data, and outputs the object sound source data.

[0276] At this time, by generating sound source type section information of the target object using video information as well, it is possible to obtain a target sound of higher quality.

[0277] Fourth embodiment <Example of recording system configuration> In addition, in the recording system shown in Figure 10, an example has been described in which video information with the position of each object as the viewpoint is used, but video information with the object equipped with each recording device 11, i.e., the entire target space in which all objects are located, as the subject may also be used.

[0278] In such a case, the recording system is configured, for example, as shown in Fig. 13. In Fig. 13, the parts corresponding to those in Fig. 8 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0279] The recording system shown in FIG. 13 includes a recording device 11, a photographing apparatus 131, and a server 12, and the configurations of the recording device 11 and the server 12 are the same as those in FIG.

[0280] The photographing device 131 is, for example, a camera, photographs the entire target space in which the object wearing the recording device 11 is located as a subject, and transmits the resulting video information to the server 12. Note that the photographing by the photographing device 131 continues while the recording device 11 is recording, that is, while the microphone 21 is collecting sound.

[0281] In addition, the receiving unit 31 of the server 12 receives not only the transmission data transmitted by the recording device 11 but also the video information transmitted by the photographing device 131 .

[0282] The receiving unit 31 supplies the received video information to the integrating unit 71. The integrating unit 71 generates final sound source type section information based on the recorded signal, the motion information, the position information, and the video information supplied from the receiving unit 31 and the sound source type section information supplied from the section detecting unit 32.

[0283] In this example, the video information is used in the integration unit 71 to detect the motion of each object.

[0284] For example, since the integration unit 71 is also supplied with position information of each object, by using the position information, it is possible to identify which object each object on the image obtained by image recognition or the like of the image information is. Also, it is possible to identify what action each object is performing by image recognition or the like of the image information. In other words, it is possible to identify what kind of object sound source is being emitted from the object at each time.

[0285] The integration unit 71 uses the motion of each object thus identified in the time interval integration process, etc., to generate final sound source type section information. In addition, for example, video information may be used as an input for arithmetic processing in a classifier such as a DNN in the time interval integration process.

[0286] Note that the section detection unit 32 may also use video information to detect the motion of each object.

[0287] <Description of Data Generation Process> When the recording system has the configuration shown in Fig. 13, each recording device 11 performs the recording process described with reference to Fig. 6, and transmits transmission data to the server 12. In addition, video information is transmitted from the imaging device 131 to the server 12.

[0288] Then, the server 12 performs the data generation process shown in Fig. 14. Hereinafter, the data generation process by the server 12 shown in Fig. 13 will be described with reference to the flowchart in Fig. 14.

[0289] In step S 161 , the receiving unit 31 receives the video information transmitted from the image capturing device 131 and supplies it to the integrating unit 71 .

[0290] Furthermore, since the server 12 receives transmission data from the recording device 11, the processes of steps S162 and S163 are carried out to generate sound source type section information.

[0291] The processes in steps S162 and S163 are similar to those in steps S71 and S72 in FIG. 9, and therefore will not be described.

[0292] In step S164, the integration unit 71 integrates the information.

[0293] That is, the integration unit 71 performs position information comparison processing, time interval integration processing, and interval smoothing processing based on the video information, recorded signal, motion information, and position information supplied from the receiving unit 31 and the sound source type interval information supplied from the interval detection unit 32, to obtain final sound source type interval information. At this time, the video information is used, for example, for selecting a reference object, etc.

[0294] The integration unit 71 supplies the obtained final sound source type section information to the signal processing unit 41, and also generates sound source type information from the final sound source type section information and supplies it to the metadata generation unit .

[0295] Once the sound source type section information is obtained in this manner, the processes of steps S165 to S167 are performed and the data generation process is terminated. However, since these processes are similar to the processes of steps S74 to S76 in FIG. 9, their description will be omitted.

[0296] In this manner, the server 12 receives transmission data from the multiple recording devices 11 and also receives video information from the photographing device 131, and generates and outputs object sound source data from the transmission data and video information. By using the video information in this manner, it is possible to obtain a target sound of even higher quality.

[0297] Fifth embodiment <Example of recording system configuration> Although an example in which the sound source type section information is generated on the server 12 side has been described above, the sound source type section information may be generated on the recording device 11 side.

[0298] In such a case, for example, as shown in Fig. 15, the recording device 11 is provided with the above-mentioned section detection unit 32. Note that in Fig. 15, parts corresponding to those in Fig. 8 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0299] The recording system shown in FIG. 15 includes a recording device 11 and a server 12 .

[0300] The recording device 11 also has a microphone 21 , a movement measurement unit 22 , a position measurement unit 23 , a recording unit 24 , a section detection unit 32 , and a transmission unit 25 .

[0301] The configuration of the recording device 11 shown in FIG. 15 differs from the configuration of the recording device 11 shown in FIG. 8 in that a section detection unit 32 is newly provided, but in other respects it has the same configuration as the recording device 11 in FIG.

[0302] The section detection unit 32 generates sound source type section information based on the recorded signal and movement information supplied from the recording unit 24, and supplies the obtained sound source type section information, along with the recorded signal, movement information, and position information supplied from the recording unit 24 to the transmission unit 25.

[0303] The transmission unit 25 generates transmission data including the recorded signal, the motion information, the position information, and the sound source type section information supplied from the section detection unit 32 , and transmits the data to the server 12 .

[0304] The server 12 also has a receiving section 31, an integration section 71, and an object sound source data generating section 33, and the object sound source data generating section 33 is provided with a signal processing section 41 and a metadata generating section .

[0305] The configuration of this server 12 differs from that of the server 12 shown in FIG. 8 in that the section detection unit 32 is not provided, but in other respects it has the same configuration as the server 12 shown in FIG.

[0306] In the example shown in FIG. 15, the receiving unit 31 in the server 12 extracts the recorded signal, the motion information, the position information, and the sound source type section information from the received transmission data.

[0307] Then, the receiving unit 31 supplies the recorded signal, movement information, position information, and sound source type section information to the integration unit 71, supplies the recorded signal, movement information, and position information to the signal processing unit 41, and supplies the movement information and position information to the metadata generation unit 42.

[0308] In addition, the integration unit 71 generates final sound source type section information based on the recorded signal, movement information, position information, and sound source type section information supplied from the receiving unit 31 and supplies it to the signal processing unit 41, and also generates sound source type information and supplies it to the metadata generation unit 42.

[0309] In this way, by generating the sound source type section information on the recording device 11 side, it is possible to reduce the processing load of the server 12 and obtain a high-quality target sound. Note that in the recording systems shown in Fig. 10 and Fig. 13, the sound source type section information may be generated on the recording device 11 side.

[0310] As described above, according to the present technology, in an environment where multiple moving bodies (objects) are present and these moving bodies emit sounds, by utilizing movement information, position information, and video information, it is possible to distinguish between the sound of a target object contained in a recorded signal and the sounds of other objects.

[0311] This makes it possible to detect time intervals containing sounds of each type of object sound source, realize signal processing for each type of object sound source, and recognize the behavior of the object's operating state.

[0312] For example, time intervals of sounds for each type of object sound source can be detected with high accuracy, such as the sound of walking, running, kicking a soccer ball, hitting a baseball or catching a ball, clapping, rustling clothes, and stepping sounds.

[0313] Generally, operation sounds cannot be obtained from sensor signals alone, and in order to distinguish between the operation sounds of the target object and similar objects in the microphone recorded signal, information on the direction and the speaker (individuality) of the voice is required.

[0314] In this regard, the present technology is capable of detecting the time period of the sound of the object sound source with higher accuracy than when only the sensor signal or only the recorded signal is used, and thus it is possible to obtain an object sound source signal of higher quality.

[0315] Specifically, when detecting a time period of an operation sound only from a recorded signal, it is assumed that a target object and another object are close to each other.

[0316] In such cases, in order to distinguish between the operation sounds of the target object and the operation sounds of other objects, the direction of the sound source will be estimated using multiple microphones, just as in the case of audio, and that direction will be used.

[0317] However, when the time interval of an action sound such as a walking sound is short, or when the direction of the sound source changes over time, it is often difficult to distinguish which object the action sound comes from.

[0318] On the other hand, the motion information includes only body motion information based on the motion of the target object, and does not include information resulting from the motion of other objects.

[0319] Therefore, by detecting the time period of the movement sound by combining the recorded signal and the motion information as in the present technology, the time period of the movement sound of the target object can be detected with high accuracy.

[0320] For example, when detecting the time period of walking sounds as a type of motion sound, the condition of the ground and shoes will have a significant effect on the detection accuracy when using only the recorded signal; however, by combining it with motion information, it becomes possible to detect the time period of walking sounds with high accuracy.

[0321] In addition, the present technology can detect time periods in recorded content such as sports and theater where there is a significant object sound source sound in audio playback, and can avoid transmitting object sound source signals in unnecessary time periods. This can reduce the amount of information on the content to be transmitted or recorded, particularly the amount of information on the object sound source signals, and the amount of processing in the subsequent stages.

[0322] Furthermore, in the present technology, an object sound source signal is generated for each object or for each object sound source of an object. Therefore, sound image localization can be set for each object sound source in a later stage, and sound images can be localized with higher accuracy.

[0323] In addition, since the present technology generates an object sound source signal for each object sound source, it is possible to selectively play only the sounds of some object sound sources, for example, in a sports broadcast, without playing the voice, and playing only the action sounds, etc. This improves the functionality during playback.

[0324] In addition, according to the present technology, when the server 12 processes content such as a sports match in real time during recording, it is possible to provide information on the motion status of each player, which is useful additional information, for example, in a situation where video review is currently being performed.

[0325] Specifically, for example, information indicating a time period of a predetermined action sound or a time period of a voice can be provided from the sound source type section information as information regarding the motion state of a player.

[0326] Furthermore, this technology is not limited to recording content, but can also be used in a variety of situations, such as when there are multiple vehicles on the road, multiple drones or other flying objects are flying, or multiple robots are present.

[0327] For example, by installing a recording device 11 in a vehicle and using the recorded signals and movement information obtained by the recording device 11, together with information obtained by a drive recorder installed in the vehicle, it is possible to perform contact determination with other vehicles.

[0328] Example of computer configuration The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0329] FIG. 16 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0330] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0331] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0332] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0333] In a computer configured as described above, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the series of processes described above.

[0334] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0335] In the computer, the program can be installed in the recording unit 508 via the input / output interface 505 by mounting the removable recording medium 511 in the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0336] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0337] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0338] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0339] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0340] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0341] Furthermore, the present technology can also be configured as follows.

[0342] (1) The apparatus further includes a section detection unit that detects a time section of a sound emitted by a moving object in a recorded signal obtained by collecting a sound around the moving object when another moving object is present around the moving object, based on the recorded signal and a sensor signal output by a sensor attached to the moving object. Signal processing device. (2) The apparatus further includes a data generating unit that generates an audio signal of the voice or operation sound of the moving object from the recorded signal based on the detection result of the time period. A signal processing device as described in (1). (3) The data generating unit outputs object sound source data including the audio signal and position information indicating a position of the moving object. A signal processing device as described in (2). (4) The data generating unit outputs object sound source data including the audio signal and information indicating a direction of the moving object. A signal processing device according to (2) or (3). (5) The data generating unit outputs object sound source data including the audio signal and sound source type information indicating a type of sound based on the audio signal. A signal processing device according to any one of (2) to (4). (6) The section detection unit detects the time section of the sound emitted by the moving object based on the recorded signal and the sensor signal of the moving object and the recorded signal or the sensor signal of the other moving object. A signal processing device according to any one of (1) to (5). (7) The section detection unit detects the time section of the sound emitted by the moving body based on a distance from the moving body to the other moving body. A signal processing device according to (6). (8) The section detection unit detects the time section of the sound emitted by the moving object based on the orientation of the moving object and the position of the other moving object. A signal processing device according to (6) or (7). (9) The section detection unit obtains a final detection result of the time section by connecting a plurality of time sections in which the same type of sound is consecutively arranged in the recorded signal, the time sections including the time sections shorter than a predetermined time width, based on the detection result of the time section. A signal processing device according to any one of (6) to (8). (10) The section detection unit connects the plurality of time sections by performing a smoothing process on a result of the detection of the time sections. A signal processing device according to (9). (11) The data generating unit generates the audio signal by performing sound source separation on the recorded signal based on the detection result of the time period. A signal processing device according to any one of (2) to (5). (12) The data generating section generates a final audio signal by replacing a part of the recorded signal or a part of the audio signal with another signal based on the result of the detection of the time section. A signal processing device according to any one of (2) to (5). (13) A signal processing device, A time period of a sound emitted by a moving object is detected based on a recorded signal obtained by collecting a sound around the moving object when other moving objects are present around the moving object and a sensor signal output by a sensor attached to the moving object. Signal processing methods. (14) A time period of a sound emitted by a moving object is detected based on a recorded signal obtained by collecting a sound around the moving object when other moving objects are present around the moving object and a sensor signal output by a sensor attached to the moving object. A program that causes a computer to execute a process including steps. [Explanation of symbols]

[0343] 11 recording device, 12 server, 21 microphone, 24 recording section, 25 transmission section, 31 receiving section, 32 section detection section, 33 object sound source data generation section, 41 signal processing section, 42 metadata generation section, 71 integration section

Claims

1. a section detection unit that detects a time section of a sound emitted by a moving object in a recorded signal obtained by collecting a sound around the moving object in a state where other moving objects are present around the moving object, based on the recorded signal and a sensor signal output by a sensor attached to the moving object; a data generating unit that generates an audio signal of the voice or the operation sound of the moving object from the recorded signal based on the detection result of the time section, and outputs object sound source data including the audio signal and position information indicating the position of the moving object; A signal processing device comprising:

2. The data generating unit outputs the object sound source data including information indicating a direction of the moving object. The signal processing device according to claim 1 .

3. The data generating unit outputs the object sound source data including sound source type information indicating a type of sound based on the audio signal. The signal processing device according to claim 1 .

4. The section detection unit detects the time section of the sound emitted by the moving object based on the recorded signal and the sensor signal of the moving object and the recorded signal or the sensor signal of the other moving object. The signal processing device according to claim 1 .

5. The section detection unit detects the time section of the sound emitted by the moving body based on a distance from the moving body to the other moving body. The signal processing device according to claim 4.

6. The section detection unit detects the time section of the sound emitted by the moving object based on the orientation of the moving object and the position of the other moving object. The signal processing device according to claim 4.

7. The section detection unit obtains a final detection result of the time section by connecting a plurality of time sections in which the same type of sound is consecutively arranged in the recorded signal, the time sections including the time sections shorter than a predetermined time width, based on the detection result of the time section. The signal processing device according to claim 4.

8. The section detection unit connects the plurality of time sections by performing a smoothing process on a result of the detection of the time sections. The signal processing device according to claim 7.

9. The data generating unit generates the audio signal by performing sound source separation on the recorded signal based on the detection result of the time period. The signal processing device according to claim 1 .

10. The data generating section generates a final audio signal by replacing a part of the recorded signal or a part of the audio signal with another signal based on the result of the detection of the time section. The signal processing device according to claim 1 .

11. A signal processing device, a time period of a sound emitted by a moving object is detected based on a recorded signal obtained by collecting a sound around the moving object in a state where another moving object is present around the moving object and a sensor signal output by a sensor attached to the moving object; Based on the result of the detection of the time period, an audio signal of the voice or the operation sound of the moving object is generated from the recorded signal, and object sound source data including the audio signal and position information indicating the position of the moving object is output. Signal processing methods.

12. a time period of a sound emitted by a moving object is detected based on a recorded signal obtained by collecting a sound around the moving object in a state where another moving object is present around the moving object and a sensor signal output by a sensor attached to the moving object; Based on the result of the detection of the time period, an audio signal of the voice or the operation sound of the moving object is generated from the recorded signal, and object sound source data including the audio signal and position information indicating the position of the moving object is output. A program that causes a computer to execute a process including steps.

Images