Sparse auditory pulse coding method and device fusing masking effect and dynamic threshold

Abstract

The invention is suitable for the technical field of audio signal processing, and provides a sparse auditory pulse coding method and device fusing a masking effect and a dynamic threshold. According to the auditory rarefaction method provided by the embodiment of the invention, a human auditory perception mechanism can be combined, and the threshold group characteristics are utilized to realize time sequence pulse coding, so that the audio signal can directly adapt to the pulse neural network model, the coding efficiency and fidelity are improved, and meanwhile, the data redundancy is reduced. While the perception quality is ensured, the number of pulse events can be remarkably reduced, the energy efficiency and accuracy of a recognition task are improved, and the method is naturally adaptive to brain-like hardware and a pulse neural network. According to the embodiment of the invention, the masking effect of psychoacoustics is fused into the pulse coding process for the first time, and is highly consistent with the feeling and processing mode of organisms on sound signals. The biological reasonability is relatively high.

Description

Technical Field

[0001] This application belongs to the field of audio signal processing technology, and in particular relates to a sparse auditory pulse coding method and apparatus that integrates masking effect and dynamic threshold. Background Technology

[0002] Currently, Artificial Neural Networks (ANNs) are widely used in auditory signal processing tasks such as speech recognition and sound classification. However, ANNs suffer from large parameter counts, long training times, and high energy consumption. These problems become increasingly prominent when faced with ever-growing volumes of data. With the rapid development of neuromorphic computing and Spiking Neural Networks (SNNs), lower-energy-consumption and faster-training neural networks have become possible. Spiking Neural Networks represent continuous information in the physical world as discrete temporal events in the form of pulses. Through pulse coding, continuous information is converted into discrete pulse events, and computation is performed only when a pulse occurs, thus significantly reducing training energy consumption. Therefore, efficiently and accurately converting continuous environmental sound signals into time-series pulse signals has become a key step in realizing low-power intelligent auditory perception.

[0003] Existing auditory front-end feature extraction methods (such as short-time Fourier transform, Mel-Cepstral coefficients, discrete cosine transform, etc.) mostly use amplitude or energy spectrum as input features, which are difficult to directly adapt to time-driven spiking neural network structures, resulting in data redundancy, high energy consumption, and poor real-time performance.

[0004] Existing pulse coding methods mainly include rate coding and time coding. Rate coding (such as counting rate coding, density rate coding, and population rate coding) encapsulates information in the number of pulses emitted per unit time, without considering the precise timing of the pulses, resulting in problems such as latency and high energy consumption. Time coding (such as pulse interval coding, correlation and synchronization coding) characterizes sound energy features through the delay time of pulse emission, with information encapsulated in the precise timing of pulse emission and the interval between pulses. Both rate coding and time coding differ significantly from the actual biological auditory mechanisms, lacking sufficient biological interpretability. Summary of the Invention

[0005] This application provides a sparse auditory pulse coding method and apparatus that integrates masking effects and dynamic thresholds, which can solve the technical problem that existing audio feature extraction methods are difficult to directly adapt to time-driven spiking neural networks.

[0006] In a first aspect, embodiments of this application provide a sparse auditory pulse coding method that integrates masking effects and dynamic thresholds, including: The original sound signal is preprocessed to obtain the preprocessed sound signal; The preprocessed sound signal is input into a filter bank, and the energy time-frequency matrix is ​​determined based on the time-frequency matrix output by the filter bank. The energy time-frequency matrix is ​​encoded based on dynamic threshold group coding to generate a time-frequency pulse matrix; The time-frequency pulse matrix and the binary masking matrix are subjected to a Hamada product operation to obtain a sparse pulse event set that conforms to psychoacoustics.

[0007] In one possible implementation of the first aspect, the preprocessing includes at least one of the following: pre-emphasis processing, frame splitting processing, and windowing processing.

[0008] In one possible implementation of the first aspect, the step of inputting the preprocessed audio signal into a filter bank and determining the energy time-frequency matrix based on the time-frequency matrix output by the filter bank includes: The preprocessed audio signal is input into a filter bank to obtain the time-frequency matrix output by the filter bank; The time-frequency matrix is ​​divided into frames using frame windows, and the logarithmic energy of each frame window is calculated to obtain the energy time-frequency matrix.

[0009] In one possible implementation of the first aspect, encoding the energy time-frequency matrix based on dynamic threshold group coding to generate a time-frequency pulse matrix includes: For each frequency band k, an over-threshold detection is performed. The energy sequence of frequency band k is traversed. When the energy of frequency band k exceeds a set threshold at a certain moment and continues to exceed the set threshold for a certain number of frames in the following period, a pulse event is triggered. The coding index of the neuron is combined with the firing time of the spike event to form an event pair; All frequency band event pairs are merged to form a pulse event set, and then arranged in ascending order of time to form a time-frequency pulse matrix.

[0010] In one possible implementation of the first aspect, the binary masking matrix is ​​constructed based on a comprehensive masking threshold, which is the maximum value among an absolute auditory threshold, a simultaneous masking threshold, and a temporal masking threshold.

[0011] Secondly, embodiments of this application provide a sparse auditory pulse coding device that integrates masking effects and dynamic thresholds, comprising: The first acquisition module is used to preprocess the original sound signal to obtain the preprocessed sound signal; The second determining module is used to input the preprocessed sound signal into the filter bank and determine the energy time-frequency matrix based on the time-frequency matrix output by the filter bank. The third generation module is used to encode the energy time-frequency matrix based on dynamic threshold group coding to generate a time-frequency pulse matrix; The fourth acquisition module is used to perform a Hamada product operation on the time-frequency pulse matrix and the binary masking matrix to obtain a sparse pulse event set that conforms to psychoacoustics.

[0012] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device performs the method as described in any one of the first aspects above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of the first aspects above.

[0014] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the method as described in any one of the first aspects above to be performed.

[0015] The beneficial effects of the first aspect of this application compared with the prior art are: This application provides an auditory sparsity method that combines human auditory perception mechanisms with threshold group characteristics to achieve temporal pulse coding. This allows audio signals to be directly adapted to spiking neural network models, improving coding efficiency and fidelity while reducing data redundancy. While ensuring perceptual quality, it significantly reduces the number of pulse events, improving the energy efficiency and accuracy of recognition tasks, and is naturally compatible with neuromorphic hardware and spiking neural networks. This application is the first to integrate the masking effect of psychoacoustics into the pulse coding process, which is highly consistent with biological perception and processing patterns of sound signals. It has strong biological rationality. This application uses a dynamically increasing threshold group to finely classify each frequency band of the filter bank. Each threshold within a different frequency band corresponds to a neuron. When the energy first exceeds the threshold, a pulse is emitted, thus preserving the precise timing, amplitude, and frequency band information of the pulse emission. Compared to the rate coding and time coding methods in the prior art, the multi-level biomimetic processing chain designed in this application forms a complete low-power auditory front end and a complete coding process from acoustic input to pulse event output, with higher information density, lower latency, and stronger interpretability.

[0016] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a sparse auditory pulse coding method that integrates masking effects and dynamic thresholds according to an embodiment of this application. Figure 2 This is a schematic diagram of the process of inputting a preprocessed audio signal into a filter bank according to an embodiment of this application; Figure 3 This is a schematic diagram of the encoding process based on dynamic threshold group coding provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a sparse auditory pulse coding device that integrates masking effect and dynamic threshold provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] Figure 1 This is a flowchart illustrating a sparse auditory pulse coding method that integrates masking effects and dynamic thresholds, provided in an embodiment of this application.

[0026] S11, preprocess the original sound signal to obtain the preprocessed sound signal.

[0027] The original audio signal can be acquired using an auditory sensor. Preprocessing of the original audio signal includes, but is not limited to, pre-emphasis processing, framing, and windowing. Pre-emphasis processing enhances the high-frequency components of the signal, compensates for their loss, and strengthens the original audio signal to a certain extent. The pre-emphasized signal can then undergo framing to approximate a stationary signal for a short period. The duration of each frame can be selected based on the specific audio signal to be encoded; for example, any value between 10ms and 30ms can be used for speech, while longer values, such as 64ms to 256ms, can be used for musical tones. Preprocessing the original audio signal yields a preprocessed audio signal. Those skilled in the art should understand that the pre-emphasis, framing, and windowing methods described above are merely examples, and other methods of preprocessing the original audio signal also fall within the scope of this application.

[0028] S12, input the preprocessed sound signal into the filter bank, and determine the energy time-frequency matrix according to the time-frequency matrix output by the filter bank.

[0029] Filter banks include, but are not limited to, gammatone filter banks and filter banks based on the constant Q transform. For the sake of simplicity, this application mainly uses gammatone filter banks as examples.

[0030] The preprocessed audio signal to be encoded is input into a gamma-ton filter bank, where each filter corresponds to a different center frequency. The time-frequency matrix output by the filter bank after filtering can be obtained by calculating the energy of each filter channel and taking the logarithm.

[0031] The center frequencies of the gammatone filter bank can be determined based on the equivalent rectangular bandwidth principle. This center frequency bank contains multiple distinct center frequencies, each ranging from 10Hz to 2000Hz. Using these center frequencies, a gammatone filter bank with multiple different center frequencies can be designed, yielding its time-domain and frequency-domain expressions. The preprocessed audio signal to be encoded is then input into the designed gammatone filter bank, where each filter corresponds to a different center frequency. Each filter can be represented as a neuron in the human auditory nerve, and the weight of each neuron represents the time-domain impulse response of the corresponding filter. The computation process for each filter's corresponding neuron and the preprocessed audio signal input is essentially a time-domain convolution process.

[0032] By inputting the preprocessed audio signal into a filter bank, the audio signal can be frequency-divided. Different filters are used to process different frequency bands, simulating the perception process of the biological auditory system. This better reflects the biological response to sound and is biologically interpretable.

[0033] In some embodiments, S12 involves inputting the preprocessed audio signal into a filter bank and determining the energy time-frequency matrix based on the time-frequency matrix output by the filter bank, including the following S121 and S122.

[0034] S121, the preprocessed sound signal is input into a filter bank to obtain the time-frequency matrix output by the filter bank.

[0035] Each filter in the gammatone filter bank can be convolved in the time domain with the preprocessed audio signal to be encoded, where each filter in the filter bank corresponds to a neuron. The output of each neuron in the filter bank is a vector with a length equal to the number of filters in the filter bank. The signal to be encoded after decomposition by the gammatone filter bank is a time-frequency matrix.

[0036] S122, the time-frequency matrix is ​​divided into frames using frame windows, and the logarithmic energy of each frame window is calculated to obtain the energy time-frequency matrix.

[0037] For the output signal of each filter in each filter bank, a frame window of the same length can be used, sliding along the time-frequency matrix to divide the signal into frames. The sliding step size can be half the length of the frame window. The logarithmic frame energy in each frame window is calculated to simulate the logarithmic perception of sound intensity by the human ear, thus obtaining the energy time-frequency matrix.

[0038] See also Figure 2 , Figure 2 The upper area (Area A) represents the pre-processed encoded audio signal. Figure 2 The middle region (region B) represents a set of pre-designed gammatone filters, corresponding to neurons in the auditory nerve. Each filter in the filter bank processes the pre-processed audio signal to be encoded within a certain time window, thus obtaining the response of each filter to the audio signal at a certain moment. The final output of the filter bank neuron is a vector of length K, where K represents the number of filters in each time window. In the time domain, shifting the convolution time window along the output signal to be encoded yields an energy time-frequency matrix with the same width as the original audio signal, such as... Figure 2 The area below (area C) is shown.

[0039] Specifically: Assume the preprocessed audio signal of the original input is Where L is the number of samples of the input audio signal. A well-designed gammatone filter... , Let be the k-th filter in the filter bank, and let its impulse response be a vector of length L. Then the output of the k-th filter is calculated as shown in the following formula (1).

[0040]

[0041] in, . Indicates the first A subset of the input audio samples within each window. Its length is the same as the length of the filter impulse response.

[0042] window The filter bank will move along with the input audio samples. At each time step, the filter bank produces a vector of length K. After L such samples, the filter bank will finally output a... Time-frequency matrix .

[0043] The time-frequency matrix obtained above Perform frame segmentation. For the waveform output from each filter channel, a frame of length W and step size can be used. The frame window is divided into frames, and the logarithmic energy of each frame window is calculated to simulate the human brain's processing of auditory signals. The specific calculation method is shown in the following calculation formula (2).

[0044]

[0045] in, . This represents the logarithmic energy of the nth frame in the kth filter channel. The final energy time-frequency matrix can be obtained. .

[0046] S13, the energy time-frequency matrix is ​​encoded based on dynamic threshold group coding to generate a time-frequency pulse matrix.

[0047] Raw pulse events can be generated through dynamic threshold quorum coding. For the energy trajectory of each frequency band in the energy-time-frequency matrix, several threshold neuron groups can be set. When the energy first exceeds the threshold, the firing time is recorded, generating a raw pulse event, thus forming the time-frequency pulse matrix. A schematic diagram of the raw pulse events obtained from the energy-time-frequency matrix through dynamic threshold quorum coding is shown below. Figure 3 As shown.

[0048] In some embodiments, S13, encoding the energy time-frequency matrix based on dynamic threshold group coding to generate a time-frequency pulse matrix, includes the following S131 to S133.

[0049] S131, perform rising threshold detection for each frequency band k, traverse the energy sequence of frequency band k, and when the energy of frequency band k exceeds a set threshold at a certain moment and continues to exceed the set threshold for a certain number of frames in the following period, trigger a pulse event to avoid interference jitter.

[0050] For the time-frequency pulse matrix formed in the previous step, set dynamic thresholds, with different thresholds set for each different frequency band. All frequency bands share a common set of thresholds.

[0051] Each frequency band is allowed to trigger pulse events multiple times within a certain period of time. However, a refractory period is applied to the threshold of each frequency band. When a pulse event is triggered in a frequency band at a certain moment, the function of triggering pulse events in that frequency band will be blocked for a certain period of time in the following period of time, thereby avoiding frequent triggering.

[0052] S132, the coding index of the neuron is combined with the firing time of the pulse event to form an event pair.

[0053] The timing of pulse events is precisely calculated. If the energy of the current frame exceeds the threshold and a pulse event is triggered, the timing of the previous frame and the current frame is linearly interpolated to obtain the precise pulse timing.

[0054] Neurons are encoded according to different frequency bands, mapping the frequency bands of the filter to the encoding of the neuron. The encoding index of the neuron is combined with the firing time of the spike event to form an event pair.

[0055] S133, merge all frequency band event pairs into a pulse event set, and arrange them in ascending order of time to form a time-frequency pulse matrix.

[0056] The time-frequency pulse matrix is ​​a sparse temporal event stream that can be directly input into a spiking neural network.

[0057] Specifically, for each frequency channel k of the filter bank, a set of L incremental pulse activation thresholds is preset. These preset thresholds can be globally uniform, or they can be adaptively adjusted according to each filter frequency channel.

[0058] Next, the pulse-coded neurons are mapped. Since the filter bank has K frequency channels, and each frequency channel has L pulse activation thresholds, the global index of the neuron corresponding to the nth threshold of the kth frequency channel is: .in, Represents the global index of a neuron.

[0059] For time-frequency matrix Perform threshold detection, and in the time dimension, for each filter channel in the filter bank. and each pulse activation threshold level Find the first time index that satisfies the condition that the energy exceeds the pulse activation threshold. . .in, This represents the time when the energy on the k-th frequency channel in the filter bank first exceeds the activation threshold of the n-th pulse. If such a time is not found throughout the entire signal time, the neuron will not fire a pulse. Performing the same threshold detection on each frequency channel of the filter bank yields the final set of pulse events: .in, This represents the final set of pulse events, where each pulse event contains the time of pulse delivery. and neurons that emit pulses . This means that within each frequency channel, the time indices that exceed the pulse activation threshold are combined with the corresponding global neuron indices, and the sets of pulse events under all k and n combinations are merged into a large set.

[0060] An auditory time-frequency pulse matrix is ​​formed based on the set of pulse events constructed in the previous step. .in Through pulse event set The generation is shown in the following calculation formula (3).

[0061] (3) The auditory pulse coding algorithm constructed in this application includes a dynamic threshold group coding mechanism, which can encode the filtered audio signal (i.e., the real-time frequency map) into a pulse signal, retaining the precise time information of the pulse occurrence and the corresponding different frequency bands, thereby increasing the amount of information contained in the pulse coding and improving the reconfigurability of the signal.

[0062] S14, perform a Hamada product operation on the time-frequency pulse matrix and the binary masking matrix to obtain a sparse pulse event set that conforms to psychoacoustics.

[0063] Based on the psychoacoustic model, the masking threshold can be calculated and a binary masking matrix (also known as a binary masking map) can be constructed. By performing the Hamad product operation on the time-frequency pulse matrix obtained in the previous step and the binary masking matrix, a sparse pulse event set that conforms to psychoacoustics can be obtained.

[0064] In some embodiments, the binary masking matrix is ​​constructed based on a comprehensive masking threshold, which is the maximum value among an absolute auditory threshold, a simultaneous masking threshold, and a temporal masking threshold.

[0065] The absolute hearing threshold can be calculated using a psychoacoustic model, reflecting the minimum audible intensity at different frequencies. The simultaneous masking threshold can be calculated, describing the masking effect of adjacent frequency band signals. The temporal masking threshold can be calculated by setting an exponential decay model based on the forward masking effect. The maximum value among the absolute hearing threshold, simultaneous masking threshold, and temporal masking threshold is taken as the comprehensive masking threshold. A binary masking matrix is ​​constructed by comparing the comprehensive masking threshold with the energy intensity of the time-frequency matrix formed in the preceding steps. In the binary masking matrix, if the comprehensive masking threshold is greater than or equal to the energy intensity of the time-frequency matrix, the value of the binary masking matrix is ​​1; if the comprehensive masking threshold is less than the energy intensity of the time-frequency matrix, the value of the binary masking matrix is ​​0. Performing a Hamad product operation between the binary masking matrix (i.e., the binary masking map) and the original time-frequency impulse matrix yields a sparsified set of impulse events.

[0066] Specifically, the absolute hearing threshold is first calculated based on a psychoacoustic model. At the same time, the masking threshold and time masking threshold Absolute hearing threshold This is reflected in frequency channel k as the minimum audible intensity of the sound. Simultaneously, the masking threshold... This reflects the masking effect of signals from adjacent frequency bands. Time masking threshold. This is reflected in the mutual masking effect between sounds occurring in close proximity, based on the forward masking effect. It can be represented by an exponential decay model.

[0067] Set the absolute hearing threshold At the same time, the masking threshold and time masking threshold Then, the overall masking threshold can be selected: .

[0068] A binary masking matrix can be constructed based on the selected comprehensive masking threshold. .in, As shown in the following calculation formula (4).

[0069] (4) This means that in the k-th frequency channel and within the n-th time frame, if the energy of the frame window exceeds the selected comprehensive masking threshold, the sound signal is not masked; otherwise, the sound signal is masked. This simulates the human auditory mechanism.

[0070] The masked auditory pulse code is obtained by multiplying the binary masking matrix element by element with the auditory pulse code matrix. Specifically, it is shown in the following calculation formula (5).

[0071] (5) in, express and Hamada product operation. This represents the masked auditory impulse code, also known as the sparse impulse event set.

[0072] Compared with existing auditory signal encoding methods, the embodiments of this application incorporate auditory masking effects, integrating the masking effects in psychoacoustics into the encoding of sound signals. Furthermore, it takes into account absolute auditory thresholds, frequency domain masking (simultaneous masking thresholds), and time domain masking (temporal masking thresholds), which can automatically filter out imperceptible signal components and achieve high sparsity encoding.

[0073] This application provides an auditory sparsity method that combines human auditory perception mechanisms with threshold group characteristics to achieve temporal pulse coding. This allows audio signals to be directly adapted to spiking neural network models, improving coding efficiency and fidelity while reducing data redundancy. While ensuring perceptual quality, it significantly reduces the number of pulse events, improving the energy efficiency and accuracy of recognition tasks, and is naturally compatible with neuromorphic hardware and spiking neural networks. This application is the first to integrate the masking effect of psychoacoustics into the pulse coding process, which is highly consistent with biological perception and processing patterns of sound signals. It has strong biological rationality. This application uses a dynamically increasing threshold group to finely classify each frequency band of the filter bank. Each threshold within a different frequency band corresponds to a neuron. When the energy first exceeds the threshold, a pulse is emitted, thus preserving the precise timing, amplitude, and frequency band information of the pulse emission. Compared to the rate coding and time coding methods in the prior art, the multi-level biomimetic processing chain designed in this application forms a complete low-power auditory front end and a complete coding process from acoustic input to pulse event output, with higher information density, lower latency, and stronger interpretability.

[0074] Figure 4 This is a schematic diagram of the structure of a sparse auditory pulse coding device that integrates masking effect and dynamic threshold provided in an embodiment of this application.

[0075] like Figure 4 As shown, the sparse auditory pulse coding device that integrates masking effects and dynamic thresholding includes: The first acquisition module 41 is used to preprocess the original sound signal to obtain the preprocessed sound signal; The second determining module 42 is used to input the preprocessed sound signal into the filter bank and determine the energy time-frequency matrix according to the time-frequency matrix output by the filter bank. The third generation module 43 is used to encode the energy time-frequency matrix based on dynamic threshold group coding to generate a time-frequency pulse matrix; The fourth obtaining module 44 is used to perform a Hamada product operation on the time-frequency pulse matrix and the binary masking matrix to obtain a sparse pulse event set that conforms to psychoacoustics.

[0076] Another embodiment of the present invention discloses a sparse auditory pulse coding device that integrates masking effects and dynamic thresholding. This embodiment is based on the above... Figure 4 Based on the corresponding embodiments, the preprocessing includes at least one of the following: pre-emphasis processing, frame splitting processing, and windowing processing.

[0077] Another embodiment of the present invention discloses a sparse auditory pulse coding device that integrates masking effects and dynamic thresholding. This embodiment is based on the above... Figure 4 Based on the corresponding embodiment, the second determining module 42 is used for: The preprocessed audio signal is input into a filter bank to obtain the time-frequency matrix output by the filter bank; The time-frequency matrix is ​​divided into frames using frame windows, and the logarithmic energy of each frame window is calculated to obtain the energy time-frequency matrix.

[0078] Another embodiment of the present invention discloses a sparse auditory pulse coding device that integrates masking effects and dynamic thresholding. This embodiment is based on the above... Figure 4 Based on the corresponding embodiment, the third generation module 43 is used for: For each frequency band k, an over-threshold detection is performed. The energy sequence of frequency band k is traversed. When the energy of frequency band k exceeds a set threshold at a certain moment and continues to exceed the set threshold for a certain number of frames in the following period, a pulse event is triggered. The coding index of the neuron is combined with the firing time of the spike event to form an event pair; All frequency band event pairs are merged to form a pulse event set, and then arranged in ascending order of time to form a time-frequency pulse matrix.

[0079] Another embodiment of the present invention discloses a sparse auditory pulse coding device that integrates masking effects and dynamic thresholding. This embodiment is based on the above... Figure 4 Based on the corresponding embodiment, the binary masking matrix is ​​constructed based on a comprehensive masking threshold, which is the maximum value among the absolute auditory threshold, the simultaneous masking threshold, and the temporal masking threshold.

[0080] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0082] This application also provides an electronic device, such as... Figure 5 As shown, the electronic device 5 includes: at least one processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 executes the computer program 52 to implement the steps in any of the above-described method embodiments.

[0083] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0084] This application provides a computer program product, including a computer program, which, when run, causes the steps in the above-described method embodiments to be executed.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0088] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A sparse auditory pulse coding method that integrates masking effect and dynamic threshold, characterized in that, include: The original sound signal is preprocessed to obtain the preprocessed sound signal; The preprocessed sound signal is input into a filter bank, and the energy time-frequency matrix is ​​determined based on the time-frequency matrix output by the filter bank. The energy time-frequency matrix is ​​encoded based on dynamic threshold group coding to generate a time-frequency pulse matrix; The time-frequency pulse matrix and the binary masking matrix are subjected to a Hamada product operation to obtain a sparse pulse event set that conforms to psychoacoustics.

2. The method as described in claim 1, characterized in that, The preprocessing includes at least one of the following: pre-emphasis processing, frame splitting processing, and windowing processing.

3. The method as described in claim 1, characterized in that, The step of inputting the preprocessed audio signal into a filter bank and determining the energy time-frequency matrix based on the time-frequency matrix output by the filter bank includes: The preprocessed audio signal is input into a filter bank to obtain the time-frequency matrix output by the filter bank; The time-frequency matrix is ​​divided into frames using frame windows, and the logarithmic energy of each frame window is calculated to obtain the energy time-frequency matrix.

4. The method as described in claim 1, characterized in that, The step of encoding the energy time-frequency matrix based on dynamic threshold group coding to generate a time-frequency pulse matrix includes: For each frequency band k, an over-threshold detection is performed. The energy sequence of frequency band k is traversed. When the energy of frequency band k exceeds a set threshold at a certain moment and continues to exceed the set threshold for a certain number of frames in the following period, a pulse event is triggered. The coding index of the neuron is combined with the firing time of the spike event to form an event pair; All frequency band event pairs are merged to form a pulse event set, and then arranged in ascending order of time to form a time-frequency pulse matrix.

5. The method as described in claim 1, characterized in that, The binary masking matrix is ​​constructed based on a comprehensive masking threshold, which is the maximum value among the absolute auditory threshold, the simultaneous masking threshold, and the temporal masking threshold.

6. A sparse auditory pulse coding device that integrates masking effect and dynamic threshold, characterized in that, include: The first acquisition module is used to preprocess the original sound signal to obtain the preprocessed sound signal; The second determining module is used to input the preprocessed sound signal into the filter bank and determine the energy time-frequency matrix based on the time-frequency matrix output by the filter bank. The third generation module is used to encode the energy time-frequency matrix based on dynamic threshold group coding to generate a time-frequency pulse matrix; The fourth acquisition module is used to perform a Hamada product operation on the time-frequency pulse matrix and the binary masking matrix to obtain a sparse pulse event set that conforms to psychoacoustics.

7. The apparatus as claimed in claim 6, characterized in that, The preprocessing includes at least one of the following: pre-emphasis processing, frame splitting processing, and windowing processing.

8. The apparatus as claimed in claim 6, characterized in that, The second determining module is used for: The preprocessed audio signal is input into a filter bank to obtain the time-frequency matrix output by the filter bank; The time-frequency matrix is ​​divided into frames using frame windows, and the logarithmic energy of each frame window is calculated to obtain the energy time-frequency matrix.

9. The apparatus as claimed in claim 6, characterized in that, The third generation module is used for: For each frequency band k, an over-threshold detection is performed. The energy sequence of frequency band k is traversed. When the energy of frequency band k exceeds a set threshold at a certain moment and continues to exceed the set threshold for a certain number of frames in the following period, a pulse event is triggered. The coding index of the neuron is combined with the firing time of the spike event to form an event pair; All frequency band event pairs are merged to form a pulse event set, and then arranged in ascending order of time to form a time-frequency pulse matrix.

10. The apparatus as claimed in claim 6, characterized in that, The binary masking matrix is ​​constructed based on a comprehensive masking threshold, which is the maximum value among the absolute auditory threshold, the simultaneous masking threshold, and the temporal masking threshold.

Images