Information processing system, information processing method, and program

The information processing system using event cameras efficiently restores sound by counting pixel events, deriving local amplitudes and phases, and applying spatially blurred phase signals, addressing the limitations of visual microphones and event cameras in cost and safety, enabling high-frequency sound recording.

JP2024165819APending Publication Date: 2024-11-28UNIV OF TSUKUBA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023082342
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Visual microphones using high-speed cameras require expensive equipment, have high computational costs, and safety concerns with laser light, while event cameras handle less data but still have room for reducing calculation costs.

Method used

An information processing system that utilizes an event camera to receive event data, counts pixel events, extracts specific pixels, derives local amplitudes and phases, and restores sound using spatially blurred phase signals through principal component analysis and blind source separation.

Benefits of technology

Enables efficient sound restoration with reduced computational costs and safe, inexpensive equipment, achieving sharp directivity and high-frequency sound recording from a distance without microphones.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165819000001_ABST
    Figure 2024165819000001_ABST
Patent Text Reader

Abstract

To provide an information processing system, an information processing method, and a program that can restore a sound.SOLUTION: An information processing system comprises: a receiving unit that receives a plurality of pieces of event data obtained by photographing a vibrating target; an extraction unit that counts the number of events occurring in each of a plurality of pixels on the basis of the plurality of pieces of event data received by the receiving unit, and extracts a first number of pixels on the basis of a result of counting the number of events occurring in each of the plurality of pixels; a creation unit that, on the basis of the first number of pixels extracted by the extraction unit, creates a local reaction for each of the first number of pixels; a derivation unit that derives a local amplitude and a local phase for each of the first number of pixels on the basis of the local reaction created by the creation unit, and derives a spatially blurred phase signal from the local reaction, local amplitude, local phase, and the Gaussian kernel; and a restoration unit that restores a sound on the basis of the spatially blurred phase signal derived by the derivation unit.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] A visual microphone records the vibrations of an object, i.e., the sounds emitted by the object and the sounds around the object, without a microphone, by capturing the minute vibrations of the object with a camera and processing the images (see, for example, Non-Patent Document 1). Because it captures sound from the vibrations of an object, it has a very sharp directionality and can record sounds from far away compared to a normal microphone. Regarding technology for acquiring sound from the vibration of an object, one that uses an event camera is known (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Abe Davis; Michael Rubinstein; Neal Wadhwa; Gautham J. Mysore; Fr´edo Durand; and William T. Freeman,“The Visual Microphone: Passive Recovery of Sound from Video”, July 2014, Article No.: 79, pp 1-10 [Non-Patent Document 2] Charles Dorn; Sudeep Dasari; Yongchao Yang; AMASCE; Charles Farrar; Garrett Kenyon; Paul Welch; and David Mascarenas, “Efficient Full-Field Vibration Measurements and Operational Modal Analysis Using Neuromorphic Event-Based Imaging”, J. Eng. Mech., 2018, 144(7) Summary of the Invention [Problem to be solved by the invention]

[0004] Visual microphones, such as those using high-speed cameras, can only record a narrow frequency band of sound, and require expensive equipment and high computational costs. Laser-based microphones raise safety concerns. Event cameras capture vibrations by recording changes in the brightness of an object. Compared to conventional cameras, event cameras handle a smaller amount of data, are capable of high-speed shooting and high-speed data transmission, and are inexpensive to use. However, there is room for further reduction in computational costs when using event cameras.

[0005] An object of the present invention is to provide an information processing system, an information processing method, and a program capable of restoring sound. [Means for solving the problem]

[0006] One embodiment of the present invention is an information processing system comprising: a reception unit that receives a plurality of event data of an image of a vibrating target; an extraction unit that counts the number of events occurring at each of a plurality of pixels based on the plurality of event data received by the reception unit, and extracts a first number of pixels based on the counting result of the number of events occurring at each of the plurality of pixels; a creation unit that creates a local response for each of the first number of pixels based on the first number of pixels extracted by the extraction unit; a derivation unit that derives local amplitudes and local phases based on the local responses created by the creation unit for each of the first number of pixels, and derives a spatially blurred phase signal from the local responses, the local amplitudes, the local phases, and a Gaussian kernel; and a restoration unit that restores sound based on the spatially blurred phase signal derived by the derivation unit.

[0007] In one embodiment of the present invention, in the aforementioned information processing system, the first number is less than the number of pixels in which an event is considered to have occurred in the plurality of event data accepted by the accepting unit.

[0008] In one embodiment of the present invention, in the aforementioned information processing system, the extraction unit extracts a first few pixels having a larger number of occurring events based on the results of counting the number of events occurring in each of a plurality of pixels.

[0009] One embodiment of the present invention, in the above-mentioned information processing system, further includes a detection unit that detects the direction in which the target is vibrating, and the creation unit creates a local reaction of the target detected by the detection unit.

[0010] One embodiment of the present invention, in the above-mentioned information processing system, further includes an input unit for inputting the direction in which the target is vibrating, and the creation unit creates the local reaction of the target input by the input unit.

[0011] In one embodiment of the present invention, in the aforementioned information processing system, the restoration unit reduces a dimension of a matrix configured from the first several spatially blurred phase signals derived by the derivation unit.

[0012] In one embodiment of the present invention, in the aforementioned information processing system, the restoration unit divides the result of reducing the dimension of a matrix composed of the first several spatially blurred phase signals into independent signals.

[0013] In one embodiment of the present invention, in the information processing system described above, the restoration unit reconstructs the results of the division into independent signals into a time vector.

[0014] One embodiment of the present invention is an information processing method executed by an information processing system, comprising the steps of: receiving a plurality of event data photographing a vibrating target; counting the number of events occurring at each of a plurality of pixels based on the plurality of event data received in the receiving step; extracting a first number of pixels based on the counted number of events occurring at each of the plurality of pixels; creating a local response for each of the first number of pixels based on the first number of pixels extracted in the extracting step; deriving a local amplitude and a local phase based on the local response created in the creating local response for each of the first number of pixels, deriving a spatially blurred phase signal from the local response, the local amplitude, the local phase, and a Gaussian kernel; and restoring sound based on the spatially blurred phase signal derived in the derivation step.

[0015] One embodiment of the present invention is a program that causes a computer to execute the steps of: accepting a plurality of event data photographing a vibrating target; counting the number of events occurring in each of a plurality of pixels based on the plurality of event data accepted in the accepting step, and extracting a first number of pixels based on the counted number of events occurring in each of the plurality of pixels; creating a local response for each of the first number of pixels based on the first number of pixels extracted in the extracting step; deriving a local amplitude and a local phase based on the local response created in the step of creating the local response for each of the first number of pixels, and deriving a spatially blurred phase signal from the local response, the local amplitude, the local phase, and a Gaussian kernel; and restoring sound based on the spatially blurred phase signal derived in the derivation step. Effect of the Invention

[0016] According to the embodiments of the present invention, it is possible to provide an information processing system, an information processing method, and a program capable of restoring sound. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system 10 according to an embodiment of the present invention. [Diagram 2] FIG. 11 illustrates an example of acquiring event data. [Diagram 3] FIG. 11 is a diagram illustrating an example of event data. [Figure 4] FIG. 2 is a diagram for explaining an example of processing of the information processing system 10 according to the present embodiment. [Figure 5A] FIG. 2 is a diagram illustrating an example of processing of the information processing system 10 according to the present embodiment. [Figure 5B] FIG. 2 is a diagram illustrating an example of processing of the information processing system 10 according to the present embodiment. [Figure 5C] FIG. 2 is a diagram illustrating an example of processing of the information processing system 10 according to the present embodiment. [Figure 6] 4 is a flowchart illustrating an example of an operation of the information processing system according to the present embodiment. [Figure 7] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system 10a according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Next, an information processing system, an information processing method, and a program according to the present embodiment will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. In addition, in all the drawings for explaining the embodiments, the same reference numerals are used for the parts having the same functions, and the repeated explanation is omitted. In addition, in this application, "based on XX" means "based on at least XX" and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0019] (Embodiment) (Information Processing System) 1 is a diagram showing an example of a configuration of an information processing system 10 according to an embodiment of the present invention. A plurality of pieces of event data D obtained by capturing an image of a vibrating target are input to the information processing system 10. The information processing system 10 accepts the input plurality of pieces of event data D. For example, a plurality of pieces of event data D obtained by capturing an image with an event camera EC are input to the information processing system 10. An example of the plurality of pieces of event data D is time-series data of the event data D.

[0020] When a luminance change of a certain level or more occurs in a pixel, the event camera EC outputs event data D including the pixel coordinates, time, and polarity. The pixel coordinates specify the two-dimensional vertical and horizontal position of the pixel in the event camera EC where the luminance has changed by a threshold or more using coordinate values. The polarity value specifies the direction of increase or decrease of the luminance change by a threshold or more using a binary value. Thus, in the event data D, the direction of increase or decrease of the luminance change at the pixel coordinates where the event occurred is represented by a pair of polarity values.

[0021] The target is, for example, an object that vibrates and produces sound like a guitar GU string, or an object that vibrates minutely due to environmental sounds, etc. In this embodiment, as an example, the description will continue for the case where the target is an object that vibrates and produces sound like a guitar GU string.

[0022] The information processing system 10 receives information indicating the direction in which the target is vibrating. In the present embodiment, as an example, the case where the user inputs the direction in which the target is vibrating into the input unit will be described. For example, the vibration direction may be input according to the direction in which the strings of the guitar GU are vibrating. The direction in which the target is vibrating may be represented by an angle. For example, the angle in which the target is vibrating may be represented with respect to a certain plane such as a horizontal plane. Examples of angles are 0 degrees, 45 degrees, 90 degrees, etc.

[0023] The information processing system 10 counts the number of events occurring in each of the multiple pixels based on the multiple received event data D. For example, the information processing system 10 counts the number of events occurring in each of the multiple pixels based on the multiple event data D received during a predetermined period (time). The information processing system 10 extracts a first number of pixels (hereinafter referred to as "active pixels") from the multiple pixels in which a larger number of events have occurred. Here, the first number is less than the number of pixels in which an event has been determined to have occurred based on the multiple received event data D. The first number may be set in advance, or may be configured to be input.

[0024] The information processing system 10 creates a local response R for each of the first few active pixels. The information processing system 10 derives a local amplitude A and a local phase Φ based on the created local response R for each of the first few active pixels. The information processing system 10 derives a spatially blurred phase signal from the derived local response R, local amplitude A, local phase Φ, and a Gaussian kernel.

[0025] This allows the information processing system 10 to derive a spatially blurred phase signal for each direction in a local region around the active pixel. For example, the information processing system 10 may derive a spatially blurred phase signal for a region including the active pixel and eight pixels around the active pixel, or may derive a spatially blurred phase signal for a region including the active pixel and 24 pixels around the active pixel, or may derive a spatially blurred phase signal for a region including the active pixel and more than 24 pixels around the active pixel. The number of pixels around the active pixel may be preset or may be configured to be input.

[0026] The information processing system 10 restores the sound based on the derived spatially blurred phase signals. For example, the information processing system 10 constructs a matrix δ from the derived spatially blurred phase signals, and restores the sound based on the constructed matrix δ. The information processing system 10 performs a principal component analysis (PCA) on the matrix δ composed of the spatially blurred phase signals derived for each of the first several active pixels, and extracts only one of the top 1st component to the top 4th component. In this embodiment, as an example, the information processing system 10 will be described below with reference to a case where only the second most significant components are extracted. The information processing system 10 performs blind source separation (BSS) on the extracted second most significant components to separate them into several signals, and linearly combines them to restore the sound. The information processing system 10 will be described below in detail.

[0027] The information processing system 10 is realized by a device such as a personal computer, a server, a smartphone, a tablet computer, or an industrial computer. The information processing system 10 includes, for example, an input unit 11, a reception unit 12, an extraction unit 13, a creation unit 14, a derivation unit 15, a restoration unit 16, an output unit 17, and a storage unit 18.

[0028] The input unit 11 inputs information. As an example, the input unit 11 may have an operation unit such as a keyboard and a mouse. In this case, the input unit 11 inputs information according to an operation performed by a user on the operation unit. As another example, the input unit 11 may input information from an external device. The external device may be, for example, a portable storage medium.

[0029] A plurality of pieces of event data D obtained by capturing an image of a vibrating target are input to the input unit 11. An example of the plurality of pieces of event data D is time-series data of the event data D. An example of the target is an object that vibrates and produces a sound, such as a string of a guitar GU. 2 is a diagram showing an example of event data acquisition. Sound is generated by vibrating the strings of a guitar GU. An event camera EC captures the guitar GU (strings) and outputs event data D when a certain level of change in luminance occurs in a pixel.

[0030] Fig. 3 is a diagram showing an example of event data D. Fig. 3 shows a schematic example of the event data D output by the event camera EC when the strings of the guitar GU are vibrated. Fig. 3 also shows the sound hole of the guitar GU. According to Fig. 3, the parts where the luminance has changed due to the vibration are shown with white hatching or dotted hatching. Specifically, the six strings of the guitar GU are indicated by white hatching or dotted hatching. The event camera EC outputs event data D for pixels in the portion indicated by white hatching or dotted hatching. Returning to FIG. 1, the explanation will be continued. Further, information indicating the direction in which the target is vibrating is input to the input unit 11. One example of the information indicating the direction in which the target is vibrating is the angle at which the target is vibrating with respect to a certain reference. Here, the certain reference may be set in advance or may be configured to be input.

[0031] The receiving unit 12 receives the plurality of event data D input to the input unit 11 and information indicating the direction in which the target is vibrating. The receiving unit 12 stores the plurality of received event data D in the storage unit 18.

[0032] The extraction unit 13 acquires the multiple event data D accepted by the acceptance unit 12. The extraction unit 13 counts the number of events occurring in each of the multiple pixels based on the acquired multiple event data D. The extraction unit 13 extracts a first number of active pixels from the multiple pixels in which a greater number of events have occurred. An example of the first number is 1000 to 2000. By increasing the first number, the accuracy of sound restoration can be improved. Also, an example of the first number may be less than 1000 or more than 2000.

[0033] For example, when the number of event data accumulated in each pixel over a small backward time window is represented as S(x, y, t), the following formula (1) is satisfied. The extraction unit 13 derives S(x, y, t) based on formula (1).

number

[0034] The creation unit 14 obtains the first few active pixels from the extraction unit 13. The creation unit 14 obtains information indicating the direction in which the target is vibrating from the reception unit 12. The creation unit 14 creates a local response R based on the obtained first few active pixels. The creation unit 14 effectively uses a single spatial scale that corresponds to the direction of motion. A single-scale Gabor filter Gωo is applied locally in the spatial domain to a spatial neighborhood S(x,y,t) surrounding each event.

number

[0035] For each event, equation (2) is the function defined in equation (1) evaluated in an n×n (n is an integer > 0) spatial neighborhood centered on where the event occurred. The local response R, expressed in equation (4) to a Gabor filter Gωo, is used to calculate the local amplitude A and local phase φ caused by the event. The occurrence of an event only affects the phase in an n×n spatial neighborhood centered on that event.

number

[0036] The derivation unit 15 acquires the local response R from the creation unit 14. Based on the acquired local response R, the derivation unit 15 derives a local amplitude A from A=|R|, and derives a local phase φ from φ=angle(R). The derivation unit 15 reconstructs the phase signal at each pixel throughout the video by convolving the spatial neighborhood surrounding each event. However, the measurements of the event camera EC contain a non-negligible amount of sensor noise. To reduce the noise, the derivation unit 15 may apply an amplitude-weighted Gaussian blur to the filter response. The spatially blurred phase signal (weighted local phase) is given by Equation (5):

[0037] ^φ=(Aφ*Kρ) / (A*Kρ) (5) In this embodiment, a Gaussian kernel with a standard deviation ρ=8 pixels and expressed by equation (6) is used.

number

[0038] Fig. 4 is a diagram for explaining an example of processing of the information processing system 10 according to the present embodiment. Fig. 4 illustrates processing for deriving a spatially blurred phase signal for an area including an active pixel and pixels surrounding the active pixel. As an example, in Fig. 4, a case is explained in which a phase signal that is spatially blurred in the direction in which the target is vibrating is derived for an area including an active image and 80 pixels around the active pixel. As the ratio of white to black increases, the value becomes smaller.

[0039] 4, it can be seen that in a region including an active pixel and its surrounding pixels, the region is spatially blurred from the active pixel to its surrounding pixels, with the values ​​being high in the active pixel region and gradually decreasing in the region of pixels surrounding the active pixel.

[0040] The restoration unit 16 acquires time series data of the spatially blurred phase signal created for each of the first several active pixels from the derivation unit 15. The restoration unit 16 derives the asynchronous time phase signal φ at each pixel (x=i, y=j) from the time series data of the spatially blurred phase signal extracted for each of the acquired first several active pixels. ij The restoration unit 16 obtains the plurality of asynchronous phase signals φ ij The matrix δ is created by arranging (t). The restoration unit 16 performs principal component analysis on the matrix δ composed of the spatially blurred phase signals, and extracts and adds up only the second most significant components to derive the matrix δ expressed by equation (7).

number

[0041] In equation (7), U = [u1, u2, u3, , uN] is an N × N matrix. The diagonal mixing matrix (N > T) shown in equation (8) has singular values ​​σi that decrease by T along the diagonal. The matrices of left and right singular vectors are expressed by equations (9) and (10).

number

number

number

[0042] The rank of the phase signal δ is the number r of non-zero singular values, and the active vibration modes are captured by r ≈ n principal components. Therefore, the restoration unit 16 can significantly reduce the spatial dimension by linearly projecting the matrix δ, which is made up of the spatially blurred phase signals by performing PCA, onto a small number of principal components. The top r components of U are extracted and denoted as U. * r Then, the principal components are expressed by the formula (11). The restoration unit 16 may use each row component of the matrix δ at this time as the restored sound.

[0043]

Number

Number

Number

[0044] Here, Equation (12) is the first column r(<<N) of U, and the i-th row of Equation (13) is the i-th principal component of the matrix δ consisting of spatially blurred phase signals. In a general structure, the principal components are a mixture of modal coordinates, and further processing is required to solve the coupling. The restoration unit 16 uses BSS to separate the mixed modal responses obtained by PCA. Some BSS use a type of machine learning and split the mixed signals into independent signals. The principal components can be expressed as a linear combination of modal coordinates represented by Equation (14). Since PCA and BSS enforce the independence of phase signals at a certain time, the restoration unit 16 reconstructs the non-uniform (asynchronous) phase signals onto a uniform time vector.

Number

[0045] The output unit 17 acquires the reconstructed signal obtained by reconstructing the non-uniform phase signal from the restoration unit 16 onto a time vector. The output unit 17 outputs the acquired reconstructed signal.

[0046] FIG. 5A is a diagram showing an example of the processing of the information processing system according to the present embodiment. FIG. 5A shows an example of the spatially blurred phase signal acquired by the restoration unit 16. According to FIG. 5A, it can be seen that the waveform is distorted.

[0047] Fig. 5B is a diagram showing an example of processing of the information processing system according to the present embodiment. Fig. 5B shows a result of reducing the dimension by performing principal component analysis on a matrix δ consisting of spatially blurred phase signals by the restoration unit 16, and dividing the mixed signal into independent signals by BSS on the dimension-reduced phase signals. Fig. 5B shows that the waveform distortion is reduced compared to Fig. 5A.

[0048] Fig. 5C is a diagram showing an example of processing of the information processing system according to the present embodiment. Fig. 5C shows an example of a reconstructed signal. According to Fig. 5C, a non-uniform phase signal is reconstructed onto a uniform time vector. The storage unit 18 is realized by a hard disk drive (HDD), a flash memory, a random access memory (RAM), a read only memory (ROM), etc., and stores information. The storage unit 18 stores a plurality of event data D output by the reception unit 12.

[0049] The input unit 11, the reception unit 12, the extraction unit 13, the creation unit 14, the derivation unit 15, the restoration unit 16, and the output unit 17 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a computer program (software) stored in the memory unit 18.

[0050] Furthermore, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware.

[0051] The computer program may be stored in advance in a storage device such as a hard disk drive (HDD) or flash memory, or may be stored in a removable storage medium such as a digital versatile disc (DVD) or CD-ROM and installed by inserting the storage medium into a drive device.

[0052] (Operation of Information Processing System 10) Fig. 6 is a flowchart showing an example of the operation of the information processing system of this embodiment. With reference to Fig. 6, the operation after a plurality of event data D and information indicating the direction in which the target is vibrating are input to the information processing system 10 will be described. (Step S1) An input unit 11 receives a plurality of event data D obtained by capturing an image of a vibrating target.

[0053] (Step S2) To the input unit 11, information indicating the direction in which the target is vibrating is input. (Step S3) The receiving unit 12 receives the plurality of event data D input to the input unit 11 and information indicating the direction in which the target is vibrating. The receiving unit 12 stores the plurality of received event data D in the storage unit 18.

[0054] (Step S4) The extraction unit 13 acquires the multiple pieces of event data D accepted by the acceptance unit 12. The extraction unit 13 counts the number of events occurring in each of the multiple pixels based on the acquired multiple pieces of event data D. The extraction unit 13 extracts a first few active pixels having a larger number of events occurring from the multiple pixels.

[0055] (Step S5) The creation unit 14 acquires the first several active pixels from the extraction unit 13. The creation unit 14 acquires information indicating the direction in which the target is vibrating from the reception unit 12. The creation unit 14 creates a local response R using the acquired first several active pixels.

[0056] (Step S6) The derivation unit 15 acquires the local response R from the creation unit 14. The derivation unit 15 derives the local amplitude A and the local phase φ based on the acquired local response R. The derivation unit 15 derives a spatially blurred phase signal from the local response R, the local amplitude A, and the local phase φ Gaussian kernel. (Step S7) The restoration unit 16 obtains time series data of the spatially blurred phase signals created for each of the first several active pixels from the derivation unit 15. The restoration unit 16 performs principal component analysis on a matrix δ composed of the spatially blurred phase signals, and extracts and adds up only the second most numerous components to derive a phase signal.

[0057] (Step S8) The restoration unit 16 splits the mixed signal into independent signals by blind source separation using the derived phase signal. (Step S9) The restoration unit 16 reconstructs the non-uniform phase signals onto a uniform time vector.

[0058] (Step S10) The output unit 17 acquires a reconstructed signal obtained by reconstructing the non-uniform phase signal onto a time vector from the restoration unit 16. The output unit 17 outputs the acquired reconstructed signal.

[0059] In the above embodiment, the case where information indicating the direction in which the target is vibrating is input to the input unit 11 has been described, but the present invention is not limited to this example. For example, the information processing system may detect the direction in which the target is vibrating.

[0060] 7 is a diagram showing a configuration example of an information processing system 10a according to the present embodiment. The information processing system 10a is different from the information processing system 10 in that the information processing system 10a detects the direction in which the target is vibrating. The information processing system 10a is realized by a device such as a personal computer, a server, a smartphone, a tablet computer, or an industrial computer. The information processing system 10a includes, for example, an input unit 11, a receiving unit 12, an extracting unit 13, a creating unit 14, a derivation unit 15, a restoring unit 16, an output unit 17, a storage unit 18, and a detecting unit 19a.

[0061] The detection unit 19a detects the direction in which the target is vibrating by analyzing an image obtained by photographing the target. The detection unit 19a outputs information indicating the direction in which the target is vibrating to the reception unit 12.

[0062] The receiving unit 12 receives a plurality of event data D input to the input unit 11 and information indicating the direction in which the target detected by the detection unit 19a is vibrating.

[0063] The detection unit 19a is realized, for example, by a hardware processor such as a CPU executing a computer program (software) stored in the storage unit 18. In addition, some or all of these functional units may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0064] In the above embodiment, the restoration unit 16 performs a principal component analysis on the matrix δ composed of spatially blurred phase signals extracted for each of the first few active pixels derived by the derivation unit 15, extracts and adds only the second most significant components to derive a phase signal, and divides the derived phase signal into independent signals. However, the present invention is not limited to this example. For example, the process of the restoration unit 16 dividing the phase signal into independent signals may be omitted. In this case, the restoration unit 16 performs a principal component analysis on a matrix δ composed of spatially blurred phase signals created for each of the first several active pixels, and reconstructs the matrix based on a phase signal derived by extracting and adding only the second-highest order components. This configuration can reduce the processing load.

[0065] The information processing system 10 according to this embodiment includes a reception unit 12 that receives a plurality of event data of an image of a vibrating target, an extraction unit 13 that counts the number of events occurring in each of a plurality of pixels based on the plurality of event data received by the reception unit 12 and extracts a first few pixels based on the counting result of the number of events occurring in each of the plurality of pixels, a creation unit 14 that creates a local response for each of the first few pixels based on the first few pixels extracted by the extraction unit 13, a derivation unit 15 that derives local amplitudes and local phases based on the local responses created by the creation unit 14 for each of the first few pixels and derives a spatially blurred phase signal from the local responses, the local amplitudes, the local phases, and a Gaussian kernel, and a restoration unit 16 that restores sound based on the spatially blurred phase signal derived by the derivation unit 15.

[0066] By configuring in this manner, the information processing system 10 can count the number of events occurring at each of the multiple pixels based on the multiple event data, and extract a first several pixels based on the counting results of the number of events occurring at each of the multiple pixels, thereby enabling the sound to be restored and recorded with a smaller amount of calculations compared to processing all of the pixels in which an event is determined to have occurred based on the multiple event data D received.

[0067] For example, if the target is a vibrating object such as a guitar string, the sound emitted by the object can be recorded. Also, if the target is an object that vibrates due to a sound source around the object, the surrounding sounds can be recorded. Even in an environment where various musical instruments are being produced, such as an orchestra, the vibration of a single instrument can be photographed to safely and quickly record only the sound produced by that single instrument, and high-pitched sounds that were previously difficult to record can also be recorded.

[0068] Furthermore, in the case of recording music production, the information processing system 10 can be used to separate and record the sounds of musical instruments, allowing editing of specific sound sources. Furthermore, by incorporating the information processing system 10 into a normal camera, it becomes possible to record distant sounds in a zoomed-in manner that was previously difficult.

[0069] In addition, by using the event data acquired by the event camera EC, it is possible to achieve extremely sharp directivity and recording of sounds from a long distance without using a microphone, compared to the case of using a conventional microphone. Also, by simply photographing the vibration of the target with a safe and inexpensive device, it is possible to record high-frequency sounds like a normal microphone.

[0070] Conventional microphones can only be used in limited situations due to the limited sound frequencies they can record, the high cost of the equipment required, and issues with safety and high computational costs. Event cameras can capture the vibrations of objects, so they handle a small amount of data, can capture images at very high speeds, transmit data at high speeds, and require inexpensive equipment. The information processing system 10 can perform calculations more efficiently than conventional systems, and therefore can achieve smaller memory capacity and reduced calculation costs. The information processing system 10 can record high frequency sounds at low cost, with less memory and at lower calculation costs than conventional systems.

[0071] In the information processing system 10, the first number is less than the number of pixels in which an event is considered to have occurred in the plurality of event data received by the receiving unit.

[0072] With this configuration, the information processing system 10 can count the number of events occurring in each of the multiple pixels based on the multiple event data, and extract a first number of pixels that is less than the number of pixels in which an event is said to have occurred in the multiple event data accepted by the accepting unit 12 based on the counting result of the number of events occurring in each of the multiple pixels, so that it is possible to restore and record sound with a smaller amount of calculation compared to the case where processing is performed on all of the pixels in which an event is said to have occurred based on the accepted multiple event data D. Furthermore, because the amount of calculation can be reduced, the time it takes for the user to obtain the sound that he or she wants to record can be shortened.

[0073] In the information processing system 10, the extraction unit 13 extracts a first few pixels in descending order of the number of events occurring therein, based on the counting results of the number of events occurring in each of a plurality of pixels.

[0074] With this configuration, the information processing system 10 counts the number of events occurring in each of the multiple pixels based on multiple event data, and extracts the first several pixels with the largest number of events occurring based on the counting results of the number of events occurring in each of the multiple pixels, so that it is possible to restore and record sound with a smaller amount of calculation compared to the case where processing is performed on all pixels in which an event is determined to have occurred based on the received multiple event data D. Also, it is possible to record the desired sound with very sharp directionality even in a noisy environment where recording is difficult with a normal microphone.

[0075] The information processing system 10 further includes a detection unit 19a that detects the direction in which the target is vibrating. The creation unit 14 creates a local response in the direction in which the target is vibrating, detected by the detection unit 19a.

[0076] By configuring in this manner, the information processing system 10a can detect the direction in which the target is vibrating, and therefore, based on a first number of pixels extracted by the extraction unit 13, can create a local response for each of the first number of pixels based on the direction in which the target is vibrating detected by the detection unit 19a.

[0077] The information processing system 10 further includes an input unit 11 for inputting the direction in which the target is vibrating, and the creation unit 14 creates a local response in the direction in which the target is vibrating input by the input unit 11.

[0078] By configuring in this manner, the direction in which the target is vibrating can be input to the information processing system 10a, so that, based on the first few pixels extracted by the extraction unit 13, a local response can be created for each of the first few pixels based on the direction in which the target is vibrating input to the input unit 11.

[0079] In the information processing system 10, the restoration unit 16 reduces the dimension of a matrix formed from the first several spatially blurred phase signals derived by the derivation unit 15. With this configuration, the information processing system 10 can reduce the dimension of the matrix formed from the first several spatially blurred phase signals derived by the derivation unit 15, and therefore can reduce the spatial dimension.

[0080] In the information processing system 10, the restoration unit 16 divides the dimension-reduced result of the matrix δ composed of the first several spatially blurred phase signals into independent signals. Configured in this manner, the information processing system 10 is able to separate the mixed-mode responses obtained by reducing the dimensionality of the phase signal.

[0081] In the information processing system 10, the restoration unit 16 reconstructs the results of the division into independent signals into a time vector. By configuring in this way, the information processing system 10 can restore clear sound.

[0082] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents described in the claims.

[0083] The information processing system 10 and the information processing system 10a described above each have a computer therein. The process of each of the above-mentioned devices is stored in a computer-readable recording medium in the form of a program, and the above-mentioned process is performed by the computer reading and executing the program.

[0084] Here, computer-readable recording media refers to magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, semiconductor memories, etc. In addition, this computer program may be distributed to a computer via a communication line, and the computer that receives this distribution may execute the program. The program may be for realizing part of the functions described above. Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program). [Explanation of symbols]

[0085] REFERENCE SIGNS LIST 10, 10a...information processing system, 11...input unit, 12...reception unit, 13...extraction unit, 14...creation unit, 15...derivation unit, 16...restoration unit, 17...output unit, 18...storage unit, 19a...detection unit

Claims

1. a reception unit that receives a plurality of event data obtained by photographing a vibrating target; an extracting unit that counts the number of events occurring in each of a plurality of pixels based on the plurality of event data received by the receiving unit, and extracts a first number of pixels based on a result of counting the number of events occurring in each of the plurality of pixels; a creating unit that creates a local response for each of the first number of pixels based on the first number of pixels extracted by the extracting unit; a derivation unit that derives a local amplitude and a local phase based on the local response created by the creation unit for each of the first several pixels, and derives a spatially blurred phase signal from the local response, the local amplitude, the local phase, and a Gaussian kernel; a restoration unit that restores sound based on the spatially blurred phase signal derived by the derivation unit; An information processing system comprising:

2. The information processing system according to claim 1 , wherein the first number is less than the number of pixels in which an event has occurred based on the plurality of event data received by the receiving unit.

3. The information processing system according to claim 1 , wherein the extraction section extracts a first few pixels in which a greater number of events have occurred based on a count of the number of events occurring in each of a plurality of pixels.

4. A detection unit for detecting a direction in which the target is vibrating Further equipped with The information processing system according to claim 1 , wherein the creation unit creates a local response in the direction in which the target detected by the detection unit is vibrating.

5. An input unit for inputting the direction in which the target is vibrating Further equipped with The information processing system according to claim 1 , wherein the creation unit creates a local response in the direction in which the target input by the input unit is vibrating.

6. The information processing system according to claim 1 , wherein the restoration unit reduces a dimension of a matrix composed of the first several spatially blurred phase signals.

7. The information processing system according to claim 5 , wherein the restoration unit divides a result of reducing a dimension of a matrix formed from the first several spatially blurred phase signals into independent signals.

8. The information processing system according to claim 7 , wherein the restoration unit reconstructs the results of the division into independent signals into a time vector.

9. receiving a plurality of event data images of a vibrating target; counting the number of events occurring in each of a plurality of pixels based on the plurality of event data received in the receiving step, and extracting a first number of pixels based on a result of counting the number of events occurring in each of the plurality of pixels; generating a local response for each of the first number of pixels based on the first number of pixels extracted in the extracting step; deriving a local amplitude and a local phase based on the local response generated in the step of generating the local response for each of a first number of said pixels, and deriving a spatially blurred phase signal from the local response, the local amplitude, the local phase and a Gaussian kernel; restoring the sound based on the spatially blurred phase signal derived in the deriving step; An information processing method executed by an information processing system comprising:

10. On the computer, receiving a plurality of event data images of a vibrating target; counting the number of events occurring in each of a plurality of pixels based on the plurality of event data received in the receiving step, and extracting a first number of pixels based on a result of counting the number of events occurring in each of the plurality of pixels; generating a local response for each of the first number of pixels based on the first number of pixels extracted in the extracting step; deriving a local amplitude and a local phase based on the local response generated in the step of generating the local response for each of a first number of said pixels, and deriving a spatially blurred phase signal from the local response, the local amplitude, the local phase and a Gaussian kernel; restoring the sound based on the spatially blurred phase signal derived in the deriving step; A program to execute.