Audio with embedded timing for synchronization

Abstract

Techniques for audio processing are described herein. For example, techniques may include receiving an audio signal from one or more microphones; receiving a periodic timing signal based on the periodic timing signal; combining the audio signal with the periodic timing signal into an audio stream; generating a timestamp based on the received periodic timing signal; and adding the generated timestamp to the audio stream based on a periodic timing signal in the audio stream.

Description

Technical Field

[0001] The present disclosure generally relates to audio processing (e.g., generating a digital audio stream or file from an audio input and / or decoding a digital audio stream or file into audio data). For example, aspects of the present disclosure relate to systems and techniques for generating audio with embedded timing information for synchronization (such as across devices). Background Art

[0002] Audio synchronization generally refers to a technique whereby audio recordings or samples obtained from multiple sources are aligned in time. For example, a device with multiple microphones may generate an audio recording for each microphone. Sound waves may arrive at the microphones of the multiple microphones of the device at slightly different times, and it may be necessary to synchronize the audio recordings of the multiple microphones, for example, to generate a single audio stream with better quality than a single microphone may have. As another example, audio recordings made on multiple devices across multiple microphones on the same device are synchronized in time to help accurately determine when each microphone receives a particular sound wave. This time synchronization can help determine the angle of arrival of the sound wave, which can be used to locate where the sound is coming from, or to combine sounds across devices to form a large microphone array. Time synchronization of microphones across multiple devices can pose challenges. Summary of the invention

[0003] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Therefore, the following summary of the invention should neither be considered as an exhaustive overview related to all conceived aspects, nor should it be considered to identify key or decisive elements related to all conceived aspects or to delineate the scope associated with any particular aspect. Accordingly, the following overview presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form before the detailed description presented below.

[0004] Systems and techniques for audio processing are described. In one illustrative example, an apparatus for audio processing includes: a receiver configured to output a periodic timing signal; one or more microphones; a microphone interface coupled to the receiver and to the one or more microphones, wherein the microphone interface is configured to: receive an audio signal from the one or more microphones; and receive the periodic timing signal from the receiver; and one or more processors coupled to the microphone interface and to the receiver, wherein the one or more processors are configured to: combine the audio signal with the periodic timing signal into an audio stream; generate a timestamp based on the received periodic timing signal; and add the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0005] In another illustrative example, a method for audio processing includes: receiving an audio signal from one or more microphones; receiving a periodic timing signal based on a periodic timing signal; combining the audio signal and the periodic timing signal into an audio stream; generating a timestamp based on the received periodic timing signal; and adding the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0006] In another illustrative example, a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive an audio signal from one or more microphones; receive a periodic timing signal based on the periodic timing signal; combine the audio signal and the periodic timing signal into an audio stream; generate a timestamp based on the received periodic timing signal; and add the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0007] In another illustrative example, an apparatus for audio processing includes: a component for receiving an audio signal from one or more microphones; a component for receiving a periodic timing signal based on a periodic timing signal; a component for combining the audio signal and the periodic timing signal into an audio stream; a component for generating a timestamp based on the received periodic timing signal; and a component for adding the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0008] In some aspects, the apparatus includes a mobile device (e.g., a mobile phone or so-called "smart phone", a tablet computer, or other type of mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television), a vehicle (or a computing device or system of a vehicle), or other device. In some aspects, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more images and / or one or more videos including video frames. In some aspects, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus includes a transmitter configured to send one or more video frames and / or grammatical data to at least one device on a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing device or component.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 illustrates example implementations of a system on a chip (SoC) according to some examples;

[0013] Figure 2 is a block diagram illustrating the reception of an audio signal using a separate microphone according to various aspects of the present disclosure;

[0014] Figure 3 is a block diagram of an example audio device for generating audio with embedded timing information according to aspects of the present disclosure;

[0015] Figure 4 is a flow chart illustrating a technique for generating audio with embedded timing information for synchronization according to aspects of the present disclosure;

[0016] Figure 5 An example computing device architecture is illustrated that illustrates an example computing device that may implement the various techniques described herein. DETAILED DESCRIPTION

[0017] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0018] The following description provides only example embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example embodiments will provide an enabling description for implementing the example embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0019] Various aspects of the disclosure will be described with respect to the figures. Figure 1 An example implementation of a system on chip (SOC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0020] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one specific implementation, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0021] The SOC 100 and / or its components may be configured to perform audio capture using embedded timing. For example, the sensor processor 114 may receive and / or process audio input from a sensor, such as one or more microphones (not shown) of the device. In some cases, the sensor processor 114 may also receive outputs of one or more processing blocks of the connectivity block 110 as audio input. Additional processing of the audio input may be performed by other components of the SOC 100, such as the CPU 102, the DSP 106, and / or the NPU 108.

[0022] Figure 2An example 200 for estimating the direction of an audio event is illustrated. In order to estimate the direction from which an audio event originates from an audio source 210, it is desirable to obtain widely separated audio recordings. For example, when two (or more) closely spaced microphones 204 are used, sound waves 202 may arrive at very close times. When closely spaced microphones are used, very accurate timing information may be required to determine the direction of the sound waves. However, for more widely spaced (e.g., baseline-increased) microphones, there will be more time between sound waves arriving at different microphones. This increased time may allow the use of less accurate timing measurements and / or significantly increased accuracy.

[0023] Generally speaking, audio synchronization across multiple audio inputs on a single device, such as multiple microphones, is straightforward because the device's single clock source can be used to obtain timing information across the multiple microphones. However, due to device size constraints, especially for a single device, there are practical limits on how far apart the microphones on a single device can be. More accurate calculations can be made with multiple devices separated by relatively large distances.

[0024] In some cases, synchronizing audio information recorded on multiple devices helps increase the baseline between microphones. However, combining data across devices may depend on using a common timing reference to align audio samples. In addition, even when a timing reference signal is available on multiple devices, there may be unknown delays within individual devices that may cause errors in determining the exact time when the audio source was sampled. Therefore, it may be difficult to synchronize audio information across multiple devices because these multiple devices may not share a common clock source.

[0025] According to aspects of the present disclosure, systems and techniques for providing a common timing reference embedded or included with audio data are described. The common timing reference can be any periodic signal. Illustrative examples of periodic signals include, but are not limited to, certain global positioning system (GPS) signals, Wi-Fi signals (e.g., Wi-Fi beacons), Bluetooth signals, cellular signals, etc.

[0026] Figure 3is a block diagram of an example audio device 300 for generating audio with embedded timing information according to aspects of the present disclosure. The audio device 300 may include an audio subsystem 320, a global navigation satellite system (GNSS) receiver 302, one or more microphones 306, and an application processor 312. The audio subsystem 320 may include a digital microphone interface (DMIC) 304 for receiving audio signals from the one or more microphones 306 and an audio processor 308 for processing the received audio signals. The application processor 312 may be any general purpose processor, such as a CPU, a core of a multi-core CPU, etc. The application processor 312 may include an input interface, such as one or more general purpose input / output (GPIO) pins 310. In some cases, the DMIC 304 and the audio processor 308 may be included as Figure 1 The GPS1PPS signal can also be input to the application processor 312 (e.g., Figure 1 The audio subsystem 320 and the application processor 312 may be integrated on a single chip, such as a SoC.

[0027] In some cases, the GNSS receiver 302 may include one or more GNSS receivers or transceivers for determining the location of the audio device 300 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, GPS in the United States, GLONASS in Russia, BeiDou Navigation Satellite System (BDS) in China, and Galileo GNSS in Europe. In this example, the GNSS receiver 302 may receive GPS signals and generate periodic timing signals, such as a pulse per second (1PPS) signal 314. The GPS 1PPS may have a pulse width of 100 ms. Although the GNSS / GPS signals may be used as Figure 3 The timing signals used in the present invention are illustrative examples of commonly found reference signals, but any other commonly found reference signals may be used, such as Wi-Fi signals (e.g., Wi-Fi beacons, advertising signals, etc.), Bluetooth signals, cellular signals, etc.

[0028] According to aspects of the invention, the GPS1PPS signal may be fed into a microphone input, such as a digital microphone (DMIC) input 304, as an audio input. Feeding the GPS1PPS signal as an audio input embeds the GPS1PPS signal into an audio sample stream as a sound signal indicating timing information (e.g., one pulse per second). In some cases, the embedded GPS1PPS sound signal in the audio stream may be characterized as a waveform of a certain set frequency and amplitude (e.g., sound), which upon playback of the audio sample stream may sound like a tone, pulse, beep, click, or other periodic sound that occurs once per second and lasts for 100 ms in the audio sample stream.

[0029] When the age of the samples is determined relative to GPS time (e.g., by the application processor 312), the exact audio samples that coincide with the pulses per second (from the 1PPS signal) can be determined by processing the audio stream to locate the embedded GPS 1PPS sound. When an audio stream receives audio samples at a particular rate, the 1PPS signal can be used to determine the "true" sample rate (e.g., by counting audio samples between instances of the 1PPS signal) and the 1PPS signal provides a high-resolution timing indicator (e.g., a clock reference) across all received audio streams.

[0030] As noted above, in some cases, a GPS1PPS signal (clock) reference 314 may be input to an input port of a DMIC input 304 (e.g., or other digital or analog audio front end). The DMIC 304 may also be coupled to one or more microphones 306 and may receive audio signals from one or more microphones 306. The DMIC 304 may be coupled to an audio processor 308, such as an audio DSP. In some cases, audio samples from multiple microphone inputs of a device (e.g., for all microphones 306 and GPS1PPS signals input to the DMIC 304) may be synchronized by the audio subsystem 320 (e.g., by the DMIC 304 and / or the audio processor 308) to produce a single audio stream that may be output to the application processor 312. For example, an audio device (such as the audio device 300) may include multiple microphones 306, and the audio signals received from each microphone may have a different amount of delay between when the audio signal is received by the microphone and when the audio signal arrives at the DMIC 304, and the audio subsystem 320 may be configured to correct for the delay difference (e.g., delay correction).

[0031] Assuming sample synchronization within the audio device 300 (e.g., delay correction), the embedded timing information derived from one microphone input can be applied to audio samples from all microphones 306 on the same audio device 306. Unlike conventional approaches, such as comparing explicit timing data from a GPIO pin (e.g., GPIO pin 310) with audio samples that may be across a bus, such as from a different processor (e.g., as may occur when an application processor 312 is trying to apply timing data received from GNSS 302 directly to audio samples / streams from an audio subsystem), this approach does not introduce unknown delays or jitter. Here, the audio processor can output an audio stream with timing information embedded in the audio stream.

[0032] In some cases, an audio stream with embedded timing information from the audio processor 308 may be input to the application processor 312. As indicated above, the application processor 312 may also receive the GPS IPPS signal 314. The application processor 312 may also receive additional GPS information, such as location and time of week (TOW) information. In an illustrative example, the GPS TOW information may be a 10-digit number indicating the week number based on the GPS system's definition of week 0 and the number of seconds elapsed in the week. The application processor 312 may extract the embedded timing information from the audio stream and use the timing information to synchronize the audio stream with the TOW information. Timestamps may be generated based on the TOW information, and these timestamps may be attached to the audio stream, for example as metadata tags corresponding to the synchronization timing. In some cases, the location information may be additionally or alternatively added to the audio stream, for example as metadata.

[0033] When augmented with TOW that may be independently obtained from GPS receiver 302, an appropriate GPS timestamp may be created, for example, by application processor 312. After the timestamp is added to the audio stream, peer devices may exchange location information and timing information related to the audio event that may be properly aligned in time. For example, for a particular audio event, multiple peer devices that detect the audio event may exchange timing information indicating when they detected the audio event. When multiple peer devices are synchronized based on a common timing signal (e.g., a GPS1PSS timing signal), the exchanged timing information may already be aligned (e.g., synchronized), and any differences in when the audio event was heard by the peer devices may be based on the peer devices' relative position to the audio source of the audio event (e.g., Figure 2 The location of the audio source 210).

[0034] Each device may then perform certain operations based on the synchronization. In one illustrative example, one or more of the peer devices, such as audio device 300, may perform a time difference of arrival (TDOA) calculation to estimate the audio source (e.g., Figure 2The location of the audio source 210 shown in FIG. 1 is known because the microphone locations, device relative locations, and audio timing are known through the exchanged timing information and location information (for the peer devices). It should be noted that the larger separation distance allows for the calculation of smaller arrays than on a single device (e.g., Figure 3 In another illustrative example, the periodic timing signal can be used to combine audio streams for many other purposes, such as to improve the fidelity of an audio recording of a musical performance when captured from many different locations by multiple recording devices.

[0035] In some aspects, in addition to or as an alternative to using GPS1PSS signals as a reference or timing signal for synchronization, Wi-Fi signals (e.g., Wi-Fi timing beacons, announcement beacons, or other periodic beacons) can be used to embed timing information into the audio stream. For example, a Wi-Fi system can broadcast announcement and / or beacon signals periodically (e.g., at regular intervals), and the beacon signals can be detected by multiple devices in the vicinity of the Wi-Fi system. The beacon signals can be used as a means to synchronize multiple audio devices such as Figure 3 The reference signal of the audio device 300. In some cases, pre-processing (e.g., to reduce the frequency of the signal, the signal width, etc.) may be applied to allow the Wi-Fi signal to fit into the low bandwidth audio signal.

[0036] On the other hand, periodic cellular signals can be used to embed timing information into the audio stream. Cellular signals may include those used for broadband wireless communication systems, including but not limited to first generation analog wireless telephone services (1G), second generation (2G) digital wireless telephone services (including transitional 2.5G networks), third generation (3G) high-speed data wireless devices with Internet capabilities, and fourth generation (4G) services (e.g., Long Term Evolution (LTE), WiMax). Examples of broadband wireless communication systems include code division multiple access (CDMA) systems, time division multiple access (TDMA) systems, frequency division multiple access (FDMA) systems, orthogonal frequency division multiple access (OFDMA) systems, global system for mobile communications (GSM) systems, etc. As an example of embedding cellular signals in an audio stream, the vehicle-to-everything (V2X) standard (which may be based on 4G LTE and / or NR standards) includes periodic beacons sent at a rate of 10 Hz. When properly preprocessed, this can be a sufficiently low pulse rate to be fed into the audio input as a periodic timing signal.

[0037] Figure 44 is a flow chart illustrating a process 400 for generating audio with embedded timing information for synchronization according to aspects of the present disclosure. At operation 402, process 400 may include receiving an audio signal from one or more microphones. At operation 404, process 400 may include receiving a periodic timing signal. In some cases, the periodic timing signal is received from a global positioning system (GPS) or other global navigation satellite system receiver. In some cases, the periodic timing signal includes a pulse signal received per second by a GPS receiver. In some cases, the periodic timing signal is received from a Wi-Fi receiver. In some cases, the periodic timing signal is received from a cellular receiver.

[0038] At operation 406, process 400 may include combining the audio signal and the periodic timing signal into an audio stream. At operation 408, process 400 may include generating a timestamp based on the received periodic timing signal. In some cases, process 400 may also include receiving a weekly time signal from a GPS receiver and generating a timestamp based on the weekly time signal and the periodic timing signal. In some cases, the generated timestamp is added to the audio stream as metadata.

[0039] At operation 408, process 400 may include adding the generated timestamp to the audio stream based on a periodic timing signal in the audio stream. In some cases, process 400 may also include obtaining first position information associated with one or more microphones and outputting the first position information and the audio stream for transmission to another device. In some cases, process 400 may also include: obtaining first position information associated with one or more microphones; receiving an additional audio stream with a timestamp and a second position information; and identifying the location of the audio event based on the first position information, the second position information, the audio stream with the generated timestamp, and the additional audio stream with the timestamp.

[0040] Figure 5 An example computing device architecture 500 is illustrated of an example computing device that may implement the various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. For example, the computing device architecture 500 may include Figure 1 SOC 100 and / or Figure 3The components of the computing device architecture 500 are shown as being in electrical communication with each other using a connection 505, such as a bus. The example computing device architecture 500 includes a processing unit (CPU or processor) 510 and a computing device connection 505 that couples various computing device components including a computing device memory 515, such as a read-only memory (ROM) 520, and a random access memory (RAM) 525 to the processor 510.

[0041] The computing device architecture 500 may include a cache of high-speed memory directly connected to the processor 510, very close to the processor, or integrated as part of the processor. The computing device architecture 500 may copy data from the memory 515 and / or the storage device 530 to the cache 512 for fast access by the processor 510. In this way, the cache can provide a performance improvement that avoids the processor 510 from delaying when waiting for data. These modules and other modules can control or be configured to control the processor 510 to perform various actions. Other computing device memories 515 can also be used. The memory 515 may include a variety of different types of memory with different performance characteristics. The processor 510 may include any general-purpose processor and hardware or software services configured to control the processor 510, such as service 1 532, service 2 534, and service 3 536 stored in the storage device 530, and a special-purpose processor in which software instructions are incorporated into the processor design. The processor 510 may be a self-contained system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0042] To enable user interaction with the computing device architecture 500, the input device 545 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 535 may also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-mode computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 500. The communication interface 540 may generally govern and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0043] The storage device 530 is a non-volatile memory and may be a hard disk or other type of computer readable medium that can store computer accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 525, a read-only memory (ROM) 520, and hybrids thereof. The storage device 530 may include services 532, 534, 536 for controlling the processor 510. Other hardware or software modules are contemplated. The storage device 530 may be connected to the computing device connector 505. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer readable medium connected to the necessary hardware components (such as the processor 510, the connector 505, the output device 535, etc.) to perform the function.

[0044] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0045] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific embodiments. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0046] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0047] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0048] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that configure a general purpose computer, a special purpose computer, or a processing device to perform a function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0049] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as flash memory), memory or memory devices, disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, etc. Computer-readable media may store thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0050] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0051] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0052] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0053] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0054] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0055] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0056] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0057] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0058] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. The technician can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0059] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. A computer-readable data storage medium may form a part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0060] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0061] Exemplary aspects of the present disclosure include:

[0062] Aspect 1: A device for audio processing, the device comprising: a receiver configured to output a periodic timing signal; one or more microphones; a microphone interface coupled to the receiver and to the one or more microphones, wherein the microphone interface is configured to: receive an audio signal from the one or more microphones; and receive a periodic timing signal from the receiver based on the periodic timing signal; and one or more processors coupled to the microphone interface and to the receiver, wherein the one or more processors are configured to: combine the audio signal with the periodic timing signal into an audio stream; generate a timestamp based on the received periodic timing signal; and add the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0063] Aspect 2. The apparatus of claim 1, wherein the receiver comprises a Global Positioning System (GPS) receiver.

[0064] Aspect 3. An apparatus according to claim 2, wherein the periodic timing signal comprises a pulse signal received by the GPS receiver every second.

[0065] Aspect 4. An apparatus according to any one of claims 2 or 3, wherein the one or more processors are further configured to: receive a weekly time signal from the GPS receiver; and generate the timestamp based on the weekly time signal and the periodic timing signal.

[0066] Aspect 5. The device according to any one of claims 1 to 4, wherein the one or more processors are further configured to: obtain first position information associated with the one or more microphones; and output the first position information and the audio stream for transmission to another device.

[0067] Aspect 6. An apparatus according to any one of claims 1 to 5, wherein the one or more processors are further configured to: obtain first position information associated with the one or more microphones; receive an additional audio stream with a timestamp and second position information from a device; and identify the position of an audio event based on the first position information, the second position information, the audio stream with the generated timestamp, and the additional audio stream with the timestamp.

[0068] Aspect 7. The apparatus of any one of claims 1 to 6, wherein the receiver comprises a Wi-Fi receiver.

[0069] Aspect 8. The apparatus of any one of claims 1 to 6, wherein the receiver comprises a cellular receiver.

[0070] Aspect 9. The apparatus according to any one of claims 1 to 8, wherein the generated timestamp is added to the audio stream as metadata.

[0071] Aspect 10. A method for audio processing, the method comprising: receiving an audio signal from one or more microphones; receiving a periodic timing signal based on the periodic timing signal; combining the audio signal and the periodic timing signal into an audio stream; generating a timestamp based on the received periodic timing signal; and adding the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0072] Aspect 11. The method of claim 10, wherein the periodic timing signal is received from a Global Positioning System (GPS) receiver.

[0073] Aspect 12. The method of claim 11, wherein the periodic timing signal comprises a pulse signal received by the GPS receiver every second.

[0074] Aspect 13. The method according to any one of claims 11 or 12, further comprising: receiving a weekly time signal from the GPS receiver; and generating the timestamp based on the weekly time signal and the periodic timing signal.

[0075] Aspect 14. The method according to any one of claims 10 to 13, further comprising: obtaining first position information associated with the one or more microphones; and outputting the first position information and the audio stream for transmission to another device.

[0076] Aspect 15. The method according to any one of claims 10 to 14 further includes: obtaining first position information associated with the one or more microphones; receiving an additional audio stream with a timestamp and second position information; and identifying the position of an audio event based on the first position information, the second position information, the audio stream with the generated timestamp, and the additional audio stream with the timestamp.

[0077] Aspect 16. The method of any one of claims 10 to 15, wherein the periodic timing signal is received from a Wi-Fi receiver.

[0078] Aspect 17. The method according to any one of claims 10 to 15, wherein the periodic timing signal is received from a cellular receiver.

[0079] Aspect 18. The method according to any one of claims 10 to 17, wherein the generated timestamp is added as metadata to the audio stream.

[0080] Aspect 19. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to: receive an audio signal from one or more microphones; receive a periodic timing signal based on the periodic timing signal; combine the audio signal and the periodic timing signal into an audio stream; generate a timestamp based on the received periodic timing signal; and add the generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

[0081] Aspect 20. The non-transitory computer-readable medium of claim 19, wherein the periodic timing signal is received from a Global Positioning System (GPS) receiver.

[0082] Aspect 21. The non-transitory computer-readable medium of claim 20, wherein the periodic timing signal comprises a one-per-second pulse signal received by the GPS receiver.

[0083] Aspect 22. A non-transitory computer-readable medium according to any one of claims 20 or 21, wherein the instructions further cause the one or more processors to: receive a weekly time signal from the GPS receiver; and generate the timestamp based on the weekly time signal and the periodic timing signal.

[0084] Aspect 23. The non-transitory computer-readable medium of any one of claims 19 to 55, wherein the instructions further cause the one or more processors to: obtain first position information associated with the one or more microphones; and output the first position information and an audio stream for transmission to another device.

[0085] Aspect 24. A non-transitory computer-readable medium according to any one of claims 19 to 23, wherein the instructions further cause the one or more processors to: obtain first position information associated with the one or more microphones; receive an additional audio stream with a timestamp and second position information from a device; and identify the location of an audio event based on the first position information, the second position information, the audio stream with the generated timestamp, and the additional audio stream with the timestamp.

[0086] Aspect 25. The non-transitory computer-readable medium of any one of claims 19 to 24, wherein the periodic timing signal is received from a Wi-Fi receiver.

[0087] Aspect 26. The non-transitory computer-readable medium of any one of claims 19 to 24, wherein the periodic timing signal is received from a cellular receiver.

[0088] Aspect 27. The non-transitory computer-readable medium of any one of Claims 19 to 26, wherein the generated timestamp is added to the audio stream as metadata.

[0089] Aspect 28. An apparatus comprising means for performing the operations according to any one of aspects 1 to 27.

Claims

1. A device for audio processing, the device comprising: a receiver configured to output a periodic timing signal; one or more microphones; a microphone interface coupled to the receiver and to the one or more microphones, wherein the microphone interface is configured to: receiving an audio signal from the one or more microphones; as well as receiving the periodic timing signal from the receiver; and one or more processors coupled to the microphone interface and to the receiver, wherein the one or more processors are configured to: combining the audio signal and the periodic timing signal into an audio stream; generating a timestamp based on the received periodic timing signal; as well as The generated timestamp is added to the audio stream based on the periodic timing signal in the audio stream.

2. The apparatus of claim 1, wherein the receiver comprises a Global Positioning System (GPS) receiver.

3. The apparatus of claim 2, wherein the periodic timing signal comprises a pulse signal output by the GPS receiver every second.

4. The apparatus of claim 2, wherein the one or more processors are further configured to: receiving a time of week signal from the GPS receiver; and The time stamp is generated based on the weekly time signal and the periodic timing signal.

5. The apparatus of claim 1 , wherein the one or more processors are further configured to: obtaining first position information associated with the one or more microphones; and The first position information and the audio stream are output for transmission to another device.

6. The apparatus of claim 1, wherein the one or more processors are further configured to: obtaining first position information associated with the one or more microphones; receiving an additional audio stream having a timestamp and second position information from a device; as well as A position of an audio event is identified based on the first position information, the second position information, the audio stream with the generated time stamp, and the additional audio stream with the time stamp.

7. The device of claim 1, wherein the receiver comprises a Wi-Fi receiver.

8. The apparatus of claim 1, wherein the receiver comprises a cellular receiver.

9. The apparatus of claim 1, wherein the generated timestamp is added to the audio stream as metadata.

10. A method for processing audio data, the method comprising: receiving audio signals from one or more microphones; receiving a periodic timing signal; combining the audio signal and the periodic timing signal into an audio stream; generating a timestamp based on the received periodic timing signal; as well as The generated timestamp is added to the audio stream based on the periodic timing signal in the audio stream.

11. The method of claim 10, wherein the periodic timing signal is received from a Global Positioning System (GPS) receiver.

12. The method of claim 11, wherein the periodic timing signal comprises a one-per-second pulse signal output by the GPS receiver.

13. The method according to claim 11, further comprising: receiving a time of week signal from the GPS receiver; as well as The time stamp is generated based on the weekly time signal and the periodic timing signal.

14. The method according to claim 10, further comprising: obtaining first position information associated with the one or more microphones; as well as The first position information and the audio stream are output for transmission to another device.

15. The method according to claim 10, further comprising: obtaining first position information associated with the one or more microphones; receiving an additional audio stream with a timestamp and second position information; as well as A position of an audio event is identified based on the first position information, the second position information, the audio stream with the generated time stamp, and the additional audio stream with the time stamp.

16. The method of claim 10, wherein the periodic timing signal is received from a Wi-Fi receiver.

17. The method of claim 10, wherein the periodic timing signal is received from a cellular receiver.

18. The method of claim 10, wherein the generated timestamp is added to the audio stream as metadata.

19. A non-transitory computer readable medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to: receiving an audio signal from one or more microphones; receiving a periodic timing signal; combining the audio signal and the periodic timing signal into an audio stream; generating a timestamp based on the received periodic timing signal; as well as The generated timestamp is added to the audio stream based on the periodic timing signal in the audio stream.

20. The non-transitory computer readable medium of claim 19, wherein the periodic timing signal is received from a Global Positioning System (GPS) receiver.

21. The non-transitory computer readable medium of claim 20, wherein the periodic timing signal comprises a one pulse signal per second received by the GPS receiver.

22. The non-transitory computer readable medium of claim 20, wherein the instructions further cause the one or more processors to: receiving a time of week signal from the GPS receiver; and The time stamp is generated based on the weekly time signal and the periodic timing signal.

23. The non-transitory computer readable medium of claim 19, wherein the instructions further cause the one or more processors to: obtaining first position information associated with the one or more microphones; and The first position information and the audio stream are output for transmission to another device.

24. The non-transitory computer readable medium of claim 19, wherein the instructions further cause the one or more processors to: obtaining first position information associated with the one or more microphones; receiving an additional audio stream having a timestamp and second position information from a device; as well as A position of an audio event is identified based on the first position information, the second position information, the audio stream with the generated time stamp, and the additional audio stream with the time stamp.

25. The non-transitory computer readable medium of claim 19, wherein the periodic timing signal is received from a Wi-Fi receiver.

26. The non-transitory computer readable medium of claim 19, wherein the periodic timing signal is received from a cellular receiver.

27. The non-transitory computer readable medium of claim 19, wherein the generated timestamp is added to the audio stream as metadata.

28. An apparatus for processing audio data, the apparatus comprising: means for receiving audio signals from one or more microphones; A component for receiving a periodic timing signal; means for combining said audio signal with said periodic timing signal into an audio stream; means for generating a time stamp based on the received periodic timing signal; as well as Means for adding a generated timestamp to the audio stream based on the periodic timing signal in the audio stream.

29. The apparatus of claim 28, wherein the periodic timing signal comprises a pulse signal per second.

30. The apparatus of claim 28, further comprising: A component for receiving a weekly time signal; as well as Means for generating the time stamp based on the weekly time signal and the periodic timing signal.

Images