Temporal convolution network for cochlear processing of audio stimuli

Abstract

The present invention relates to a computer-implemented method for emulating cochlear processing of auditory stimuli, the method comprising the steps of: - providing a multi-layer temporal convolution network (TCN), the network comprising: a. a plurality of stacked memory blocks, each memory block comprising at least one pointwise convolution and at least one depthwise convolution with a dilation factor; b. at least one nonlinear unit for applying a nonlinear transformation to the activation maps generated by at least one convolutional layer of the network; c. a plurality of residual and / or skip connections between the plurality of memory blocks, configured for forwarding the output of lower layer number memory blocks to at least one higher layer number memory block; d. an input layer, configured for receiving an input sequence to the network; and, e. an output layer, configured for generating for each input to the network, N output sequences of cochlear response parameters; - providing at least one input sequence of any length, wherein the input sequence is indicative of a time-sampled auditory stimulus; and, - applying the at least one input sequence to the input layer of the neural network to obtain the N output sequences of cochlear response parameters.

Description

[0001] TEMPORAL CONVOLUTION NETWORK FOR COCHLEAR PROCESSING OF AUDIO STIMULI FIELD OF THE INVENTION

[0002] The present invention relates to the field of audio processing. More specifically, it relates to computer-implemented methods and devices for emulating cochlear processing of auditory stimuli.

[0003] BACKGROUND OF THE INVENTION

[0004] The human cochlea (or, inner ear) is an active, nonlinear system which transduces sound into cochlear travelling waves and neural signs which can be characterized as basilar membrane (BM) displacement or velocity, inner-hair-cell (IHC) transduction, or auditory-nerve fibre (ANF) and auditory neuron firing patterns. Modelling these cochlear responses travelling waves can be useful for better understanding the mechanisms of hearing, compensating for hearing impairments and even to improve audio and machine hearing applications.

[0005] WO2020249532 Al discloses a method for emulating cochlear processing of auditory stimuli, which solves some of the critical dysfunctionalities of a multilayer convolutional encoder-decoder neural network model. This model is herein referred to as the "prior CoNNear" model. During the encoder stage, the feature dimensions are decreased through the strided convolution. Conversely, in the decoder phase, the feature maps undergo upsampling via transpose convolution with a stride.

[0006] Colindres et al. "Temporal Convolutional and Recurrent Neural Networks for Sequence Modeling", 2023, discusses application of convolutional neural networks for sequential data (time series analysis).

[0007] The main disadvantage of the prior CoNNear model relates to generation of tonal artifacts and undesirable distortions that are generated by its original encoder-decoder structure. These distortions are generated by the transposed convolutions in the decoder, and result in additional tones or resonances that were not present in the input signal. Furthermore, these distortions are present in the human frequency range of hearing (20Hz - 20 kHz).

[0008] The prior CoNNear model can simulate cochlear responses that correspond to different processing stages along the auditory pathway (e.g., CoNNear_cochlear, CoNNearJHC, and CoNNear_ANF), and each of these intermediate cochlear response calculations can be placed in cascade (e.g. CoNNear_cochlear serves as the input to the CoNNEarJHC model). Unfortunately, when the different encoder-decoder CoNNear models are placed in cascade, the distortions may propagate throughout the entire model structure and become overamplified, thereby rendering them audible when the models are used for audio applications (e.g. hearing-aid processing, speech enhancement, machine hearing). A second problem with the prior CoNNear implementation is that the original CoNNear activation function that is placed between consecutive layers of the model, is not suitable for embedded system implementations. We previously motivated that a smooth tanh-activation function (and not PReLU-based nonlinearities) is a requirement for the prior CoNNear to capture the nonlinear properties of cochlear processing. However, implementing a smooth function in an embedded context would require many parameters and this is both energy inefficient while putting a high burden on the overall latency that is required to compute CoNNear. Because embedded applications based on CoNNear require a processing latency of less than 10 ms to perceive the processing as real-time by the user, it is necessary to move away from smooth tanh activation functions to improve the network architecture for translation into embedded systems.

[0009] Consequently, there is a need for improved cochlear processing systems that are easy to compute and have short latencies, while capturing the key nonlinear, coupling and frequency-selectivity properties of (human) cochlear processing. Essential for audio and machine hearing applications, is that no undesired artifacts or distortions are generated in the network, while the network preferably operates at a reduced computational complexity to enable real-time applications.

[0010] To circumvent the main problem of audible artifacts that are created by the transposed convolutions in the decoder architecture, it is necessary to replace the transposed convolution in the decoder structure by another method that does not introduce artifacts, while the quality of the original cochlear processing modules is maintained. Several alternative upsampling techniques have been proposed in the literature to replace the encoder-decoder structure of conventional convolutional neural network models, but they fail in circumventing the problem of audible artifacts, as well as not being suitable to be integrated within an embedded system.

[0011] In summary, while there are different methods available to circumvent the problem of distortions created by the upsampling operation in standard encoder-decoder CNN models, such as that used in the prior CoNNear models, none of the existing methods removes the tonal distortions while maintaining the quality of cochlear processing.

[0012] SUMMARY OF THE INVENTION

[0013] The present invention overcomes one or more of these issues. Preferred embodiments of the present invention overcome one or more of these issues.

[0014] It is an object of embodiments of the present invention to provide good methods and systems for emulating cochlear processing of auditory stimuli, good hearing aids using such methods and systems for modelling hearing, as well as methods for audio applications or assisting in hearing, using such modelling methods. While the prior CoNNear model used an encoder-decoder architecture, the present invention, herein referred to as the "new CoNNear", employs a novel architecture based on a combined temporal convolution network (TCN) and deep-feedforward sequential memory network (DFSMN) memory block, such that there is no encoder and decoder required in the present invention to circumvent the issue of distortion. The details of the TCN network are furthermore tailored to capture essential nonlinear cochlear processing characteristics, while keeping the overall network parameter numbers low to facilitate translation into embedded systems.

[0015] The methods according to the present invention ensure that the input dimensions remain unchanged after each convolution, eliminating the need for upsampling. This design choice effectively prevents the generation of artifacts associated with upsampling. It is an advantage of embodiments of the present invention that artifacts are minimized and / or avoided.

[0016] Modelling long-term dependencies in sequential data stands as a pivotal attribute in the field of cochlear processing of auditory stimuli. To effectively capture these extended dependencies, the prior CoNNear (comprising an encoder-decoder structure) utilized strided convolutions within the encoder, while the present invention employs dilated convolution in the memory blocks instead. It is an advantage of embodiments of the present invention that for a given number of model layers and kernel sizes, long-term dependencies in sequential data are more accurately processed than other model architectures with the same number of layers and kernel sizes.

[0017] It is an advantage that the presented method minimizes artifacts that are associated with the gated units adopted in other dilated convolutional networks, by presenting a method that is based on a temporal convolution network (TCN), combined with a deep feedforward sequential memory network (DFSMN)-based memory block, whereby the combined system is modified to perform accurate cochlear processing while keeping the computational complexity low.

[0018] The methods according to the present invention are not just accurate, but also computationally efficient. It is an advantage of embodiments of the present invention that they provide good modelling systems and methods that allow accurate modelling in a fast way. It is an advantage of embodiments of the present invention that they may allow faster emulation when implemented on devices that support (massively) parallelized computation, for instance on multiple processing elements (PEs) such as graphical processing units (GPUs). Embodiments of the present invention have the advantage that they can be implemented with short processing delays to afford live or close-to-live audio processing capabilities, for example on embedded systems or consumer products.

[0019] The methods according to the present invention are not just computationally efficient, but also accurate. It is an advantage of embodiments of the present invention that a transformation of an auditory stimulus into cochlear responses such as basilar membrane vibration, inner-hair-cell transduction, or auditory-nerve firing is accurately emulated. It is an advantage of embodiments of the present invention that a wide range of auditory systems across human and non-human animal beings can be emulated in an accurate manner. It is an advantage of embodiments of the present invention that normal hearing and hearing impairments can be accurately emulated.

[0020] It is an advantage of embodiments of the present invention that the processing can be individualized to hearing impairment characteristics of the individual i.e. frequency-specific patterns of outer-hair-cell, inner-hair-cell, or auditory-nerve fibre damage patterns, and hence can be used in backpropagation methods which aim to compensate for a single aspect, or combinations of aspects, of hearing damage.

[0021] Methods and devices in accordance with the present invention can be advantageously used for improving currently available hearing aids. It is an advantage of embodiments of the present invention that the methods and devices for emulating cochlear processing, can be easily implemented and integrated into hearing devices, hearing aids, automatic speech recognition systems and machine hearing applications.

[0022] It is an advantage of embodiments of the present invention that they can be easily combined with a wide range of available speech recognition software or algorithms, for which they may be used as a speech or audio signal pre-processing module.

[0023] Methods and devices in accordance with some embodiments of the present invention can perform reliably in the presence of noise affected auditory stimuli.

[0024] In a first aspect, the invention relates to a computer-implemented method for emulating cochlear processing of auditory stimuli. The method preferably comprises the steps of: providing a multi-layer temporal convolution network (TCN); providing at least one input sequence of any length, wherein the input sequence is indicative of a time-sampled auditory stimulus; and, applying the at least one input sequence to the input layer of the neural network to obtain the N output sequences of cochlear response parameters.

[0025] The multi-layer temporal convolution network (TCN), preferably comprises: a. a plurality of stacked memory blocks, each memory block comprising at least one pointwise convolution and at least one depthwise convolution with a dilation factor; b. at least one nonlinear unit for applying a nonlinear transformation to the activation maps generated by at least one convolutional layer of the network; c. a plurality of residual and / or skip connections between the plurality of memory blocks, configured for forwarding the output of lower layer number memory blocks to at least one higher layer number memory block; d. an input layer, configured for receiving an input sequence to the network; and, e. an output layer, configured for generating for each input to the network, N output sequences of cochlear response parameters.

[0026] In some preferred embodiments, the nonlinear transformation comprises an activation function having a compression growth and a zero-crossing, wherein said activation function generates positive and negative outputs. In some preferred embodiments, the nonlinear transformation comprises an activation function selected from a hyperbolic tangent type function, preferably selected from the group comprising: hardtanh, tanh, or piecewise-tanh functions.

[0027] In some preferred embodiments, the activation function is a hardtanh function.

[0028] In some preferred embodiments, the method comprises the following step, preferably performed as dilated convolution in the plurality of memory blocks, of: processing one-dimensional waveform inputs through a sequence of stacked memory blocks, wherein the dilation factors increase.

[0029] In some preferred embodiments, the dilation factors exponentially increase; preferably the dilation factors increase as 2k l.

[0030] In some preferred embodiments, the outputs of each memory block are aggregated through a weighted summation process and subsequently directed to a convolutional layer to obtain the cochlear response; thereby providing a skip connection.

[0031] In some preferred embodiments, in each memory block, the input from the preceding block is utilised, preferably by comprising the steps of: processing one-dimensional waveform input through a first pointwise convolution, followed by depthwise convolution; preferably with dilation factors of 2k l; and, summing the pointwise convolution and depthwise convolution and passing through a non-linear transformation.

[0032] In some preferred embodiments, the method further comprises the step of: obtaining an output of the memory block subsequent to a second pointwise convolution.

[0033] In some preferred embodiments, the second pointwise convolution occurs after applying the at least one nonlinear transformation.

[0034] In some preferred embodiments, the TCN comprises at least 8 and at most 16 memory blocks, preferably at least 10 and at most 14 memory blocks, most preferably 12 memory blocks. In a second aspect, the invention relates to a data processing device comprising means for carrying out the method of the first aspect or (preferred) embodiments thereof. The data processing device preferably comprises:

[0035] - input means for receiving at least one input sequence indicative of a time-sampled auditory stimulus;

[0036] - a memory unit for storing at least the network parameters; and,

[0037] - output means for outputting N output sequences of cochlear response parameters.

[0038] In a third aspect, the invention relates to a hearing device comprising the data processing device according to the second aspect, or (preferred) embodiments thereof. The hearing device preferably further comprises: a pressure detection means for detecting a time-varying pressure signal indicative of at least one auditory stimulus; sampling means for sampling the detected auditory stimulus to obtain an input sequence comprising a plurality of input samples; and, at least one transducer for converting output sequences generated by the neural network into audible time-varying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus.

[0039] In a fourth aspect, the invention relates to a computer program comprising instructions which, when the program is executed by a computer, perform the method steps of the method according to the first aspect, or (preferred) embodiments thereof.

[0040] In a fifth aspect, the invention relates to a computer-readable medium comprising instructions which, when executed by a computer, perform the method steps of the method according to the first aspect, or (preferred) embodiments thereof.

[0041] In a sixth aspect, the invention relates to a use of the method according to the first aspect, or (preferred) embodiments thereof, to increase the receptive field to capture long-term dependencies of an auditory signal.

[0042] (Preferred) embodiments of the first aspect of the invention are also (preferred) embodiments of the second, third, fourth, fifth, and sixth aspects of the invention, and vice versa.

[0043] BRIEF DESCRIPTION OF THE FIGURES

[0044] The following description of the figures relate to specific embodiments of the disclosure which are merely exemplary in nature and not intended to limit the present teachings, their application, or use. FIG. 1 illustrates how one-dimensional waveform inputs are processed through a sequence of stacked memory blocks according to a preferred embodiment of the invention, wherein the dilation factors exponentially increase.

[0045] FIG. 2 illustrates the architectural design of the memory block according to preferred embodiments of the invention, wherein the input from the preceding block is utilised. FIG. 2 further illustrates cochlear processing capabilities of the proposed TCN architecture that uses a memory block with a singular (panel 2A) versus a double (panel 2B) depthwise dilated convolution. The target simulations are cochlear processing simulations conducted with a non-neural-based model of cochlear processing [TL, 11] and show the tuning of cochlear filtering (QERB) as a function of frequency for a stimulation level of 40 dB SPL (panel 2C). Both proposed new CoNNear architectures match the reference TL simulations well and can thus capture this essential feature of cochlear processing.

[0046] FIG. 3A-D illustrate cochlear excitation patterns for various models, mor specifically simulated cochlear excitation patterns to a 1-kHz pure tone stimuli of different sound pressure levels between 0 and 90 dB SPL (legend). The TL target simulations (panel 3D) refer to cochlear processing simulations with a non-neural-network-based model of cochlear processing. The target responses are characterized by broader patterns as the stimulus level increases and a non-linear, compressive growth of the peak response as the stimulus level increases. These features are key properties of cochlear processing and should also be captured by neural-network-based models of cochlear processing (panels 3A, 3B, and 3C).

[0047] FIG. 4A-E illustrate cochlear filter tuning (QERB) for various models, more specifically simulated cochlear filtering tuning (QERB) as a function of frequency for three stimulation levels comparing the target responses (panel 4D) to those simulated with different implementations of TCN based models. Panel 4A: without skip connections in the architecture, panel 4B: new CoNNear with skip connections and tanh nonlinearity in the architecture and dilation factors d of [1,2, 4, 8, 1,2, 4, 8, 1,2, 4, 8] across the layers, panel 4C: with skip connections but a PReLU nonlinearity in the architecture. Panel 4E shows an example with skip connections and tanh nonlinearity in the architecture but using dilations factors of [1,1, 1,1, 1,1, 1,1, 1,1, 1,1],

[0048] FIG. 5 illustrates responses to speech input for various models, more specifically simulated speech processing of a part of a sentence (top panel 5A) comparing the target cochlear responses (panel 5B) to those simulated with different implementations of TCN based models. Panel 5C: simulations with new CoNNear. Panel 5D shows the impact of different design choices on the error signal between the target response and the chosen TCN model architecture.

[0049] RECTIFIED SHEET (RULE 91 ) ISA / EP FIG. 6A-D illustrate the comparison of the distortion-product otoacoustic emission (DPOAE) between various models. Simulated cochlear distortion products that are a desired and natural consequence of cochlear processing. Simulated responses are shown to two pure tones of fl of 2.0 and f2 of 2.4 kHz. In humans and the target model simulations (panel 6D), the most pronounced distortion product (DP) occurs at 2fl-f2 (1.6 kHz). Panel 6B shows the new CoNNear model with skip connections and tanh nonlinearity, otherTCN structures without skip connections (panel 6A) or PReLU nonlinearity (panel 6C) capture the reference TL response less good. Replacing the nonlinearity by a PReLU gave the worst degradation.

[0050] FIG. 7A-C illustrate the comparison of the root-mean-square (RMS) error between excitation patterns between various models. FIG. 7A-C illustrate that the model incorporating skip connections (panel 7A) demonstrates a comparable RMS value compared to the model without skip connections (panel 7B). However, at low Ncf, the model with skip connections exhibits a lower RMS value. Using Hardtanh activation shows less RMS than using PReLU (panel 7C).

[0051] FIG. 8A and 8B illustrate responses to a lk Hz pure tone and a step function input, respectively, using methods according to the prior art ("previous CoNNear").

[0052] FIG. 8C and 8D illustrate responses to a lk Hz pure tone and a step function input, respectively, without a skip connection but with a hardtanh activation function.

[0053] FIG. 8E and 8F illustrate responses to a lk Hz pure tone and a step function input, respectively, with a skip connection and with a hardtanh activation function.

[0054] FIG. 8G and 8H illustrate responses to a lk Hz pure tone and a step function input, respectively, with a skip connection and with a PReLU activation function.

[0055] FIG. 9 illustrates simulations of the new CoNNear model with causal setting. When comparing the simulation of the model with interpolation, the model causal setting fails to capture an essential property of cochlear processing, (QERB) target that was well captured by the new CoNNear model with non-causal setting.

[0056] FIG. 10 Shows an example of a closed-loop system wherein normal-hearing and personalised hearing-impaired CoNNear models are used to train the parameters of a CoNNEar-based HA algorithm. The CoNNear models simulate cochlear responses of different processing stages along the pathway: human cochlear, IHC and ANF processing. A loss function is used to train the parameters of the HA model corresponding to the auditory nerve (AN) response iy of an NH auditory system, and one corresponding to the response of an HI system. The result of this network is a trained HA algorithm (x) that can be applied to a speech sample to restore the hearing in the impaired cochlea, i.e., the target of the loss function and the trained x is to bring iy as close as possible to the original i . FIG. 11A and 11B show the outputs of the HA model (x) trained with using either the prior (A) or present (B) CoNNear-based closed-loop framework from FIG. 10 to compensate for a typical sensorineural high-frequency sloping hearing loss (HI type "Slope35-7,0,0"). A spectral analysis of the HA response to a given a 1-kHz tonal sound input is shown together with the stimulus itself.

[0057] FIG. 12 illustrates the effect of HA compensation on simulated HI AN population responses. The closer the HA processed models are to the NH response (solid black), the better the performance. All models showed improvements over the HI response (dashed squares), and performance differences between CoNNear architectures were small. The specific hearing loss tested was cochlear synaptopathy.

[0058] FIG. 13 depicts the RMS energy of the CoNNear_cochlea basilar membrane response assessed at CFs spanning the speech frequency range for a speech segment that. The upper and lower bounds reflect the NH (solid black) and uncompensated HI (dashed squares) responses respectively, and the other lines illustrate the performance of a HI model that is given the new-CoNNear-based HA processing. The closer the RMS energy of a specific HI_CoNNear simulation is to the upper bound, the better the HA was able to compensate for the hearing damage. New CoNNear-HA variations with different hyperparameters were tested: the number of memory blocks (L), the kernel size (R_k) and the dilation factor (R_s).

[0059] FIG. 14 illustrates how a new-CoNNear-trained HA processed speech segment was evaluated in testing how well it improved the HI AN population response. The maximum cross correlation (NCC) between the HA-compensated response and the NH response is calculated. The lower bound (dashed bottom curve) is given by the NCC between the (uncompensated) HI response and the NH response. The higher the NCC performance above this line, the more the specific HA model was able to compensate for the hearing loss. New CoNNear-HA variations with different hyperparameters were tested: the number of memory blocks (layers), the kernel size (R_k) and the dilation factor (R_s).

[0060] FIG. 15 illustrates the maximum cross correlation (NCC) of new-CoNNear-based HA processed speech using a 12-layer model (serving as a reference), alongside other models. The first bar shows the performance of the model with 12 memory blocks (layers) model with itself, yielding a value of 1 and thus optimal performance. The other bars represent the maximum NCC responses between each HA processing model with the reference 12-layers model. The number of memory blocks (layers) was altered across the model versions, the kernel size (R_k) and the dilation factor (R_s), and models that reached an NCC performance close to 1 performed best.

[0061] FIG. 16 presents a specific implementation of the general new CoNNear architecture presented in FIG. 2B which was optimized for efficient hardware implementation. Specifically, the dashed lines represent how a context retraining network can be added to enable a more efficient hardware implementation of the new CoNNear architecture.

[0062] FIG. 17 Performance in processing a segment of speech of the context-retraining implementation version of new CoNNear in comparison to the new CoNNear architecture depicted in FIG.2B. The performance of both networks is identical, demonstrating that the context retraining is merely a more efficient hardware implementation of the original new CoNNear general architecture depicted in FIG.2B.

[0063] DETAILED DESCRIPTION

[0064] In the following detailed description, the technology underlying the present disclosure will be described in the form of various technological aspects. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and make part of this disclosure. This description is meant to aid the reader in understanding the technological concepts more easily, but it is not meant to limit the scope of the present disclosure, which is limited only by the claims.

[0065] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment.

[0066] As used herein, the terms "comprising", "comprises" and "comprised of" as used herein are synonymous with "including", "includes" or "containing", "contains", and are inclusive or open- ended and do not exclude additional, non-recited members, elements, or method steps. The terms "comprising", "comprises" and "comprised of" when referring to recited members, elements or method steps also include embodiments which "consist of" said recited members, elements, or method steps. The singular forms "a", "an", and "the" include both singular and plural referents unless the context clearly dictates otherwise.

[0067] As used herein, relative terms, such as "left," "right," "front," "back," "top," "bottom," "over," "under," and the like, are used for descriptive purposes and not necessarily for describing permanent relative positions. It is to be understood that such terms are interchangeable under appropriate circumstances and that the embodiment as described herein are capable of operation in other orientations than those illustrated or described herein unless the context clearly dictates otherwise. Objects described herein as being "adjacent" to each other reflect a functional relationship between the described objects, that is, the term indicates the described objects must be adjacent in a way to perform a designated function which may be a direct ( / .e. physical) or indirect ( / .e. close to or near) contact, as appropriate for the context in which the phrase is used.

[0068] Objects described herein as being "connected" or "coupled" reflect a functional relationship between the described objects, that is, the terms indicate the described objects must be connected in a way to perform a designated function which may be a direct or indirect connection in an electrical or nonelectrical ( / .e. physical) manner, as appropriate for the context in which the term is used.

[0069] As used herein, the term "substantially" refers to the complete or nearly complete extent or degree of an action, characteristic, property, state, structure, item, or result. For example, an object that is "substantially" enclosed would mean that the object is either completely enclosed or nearly completely enclosed. The exact allowable degree of deviation from absolute completeness may in some cases depend on the specific context. However, generally speaking the nearness of completion will be so as to have the same overall result as if absolute and total completion were obtained. The use of "substantially" is equally applicable when used in a negative connotation to refer to the complete or near complete lack of an action, characteristic, property, state, structure, item, or result.

[0070] As used herein, the term "about" is used to provide flexibility to a numerical value or range endpoint by providing that a given value may be "a little above" or "a little below" said value or endpoint, depending on the specific context. Unless otherwise stated, use of the term "about" in accordance with a specific number or numerical range should also be understood to provide support for such numerical terms or range without the term "about". For example, the recitation of "about 30" should be construed as not only providing support for values a little above and a little below 30, but also for the actual numerical value of 30 as well.

[0071] The recitation of numerical ranges by endpoints includes all numbers and fractions subsumed within the respective ranges, as well as the recited endpoints. Furthermore, the terms first, second, third and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order, unless specified. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the disclosure described herein are capable of operation in other sequences than described or illustrated herein.

[0072] Reference in this specification may be made to devices, structures, systems, or methods that provide "improved" performance (e.g. increased or decreased results, depending on the context). It is to be understood that unless otherwise stated, such "improvement" is a measure of a benefit obtained based on a comparison to devices, structures, systems, or methods in the prior art. Furthermore, it is to be understood that the degree of improved performance may vary between disclosed embodiments and that no equality or consistency in the amount, degree, or realization of improved performance is to be assumed as universally applicable.

[0073] In addition, embodiments of the present disclosure may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic based aspects of the present disclosure may be implemented in software (e.g., instructions stored on non-transitory computer-readable medium) executable by one or more processing units, such as a microprocessor and / or application specific integrated circuits. As such, it should be noted that a plurality of hardware and software-based reaction or output in response to a stimulus devices, as well as a plurality of different structural components may be utilized to implement the technology of the present disclosure. For example, "servers" and "computing devices" described in the specification can include one or more processing units, one or more computer-readable medium modules, one or more input / output interfaces, and various connections connecting the components.

[0074] The term "cochlea" refers to a spiral-shaped, fluid-filled structure in the inner ear that plays a key role in hearing by converting sound waves into electrical signals that the brain can interpret; while the term "response" refers to a reaction or output in response to a stimulus, in this case, the stimulus being sound. The term "cochlear response parameter" is therefore understood by the person skilled in the art and refers to any measurable characteristic or factor that quantifies how any of the cochlear physiological elements (such as middle ear, inner or outer hair cells, and / or auditory nerve) responds to sound stimuli. This term encompasses a range of variables used in auditory research to describe cochlear function or performance. Examples of cochlear response parameters include basilar membrane vibration, inner-hair-cell transduction, or auditory-nerve firing and indirect correlates thereof such as measured through ear-canal pressure or impedance responses, otoacoustic emissions, population responses or potentials (e.g. compound action or summating potentials).

[0075] In a first aspect, the invention relates to a computer-implemented method for emulating cochlear processing of auditory stimuli. The method preferably comprises the steps of: providing a multi-layer temporal convolution network (TCN); providing at least one input sequence of any length, wherein the input sequence is indicative of a time-sampled auditory stimulus; and, applying the at least one input sequence to the input layer of the neural network to obtain the N output sequences of cochlear response parameters.

[0076] The multi-layer temporal convolution network (TCN), preferably comprises: a. a plurality of stacked memory blocks, each memory block comprising at least one pointwise convolution and at least one depthwise convolution with a dilation factor; b. at least one nonlinear unit for applying a nonlinear transformation to the activation maps generated by at least one convolutional layer of the network; c. a plurality of residual and / or skip connections between the plurality of memory blocks, configured for forwarding the output of lower layer number memory blocks to at least one higher layer number memory block; d. an input layer, configured for receiving an input sequence to the network; and, e. an output layer, configured for generating for each input to the network, N output sequences of cochlear response parameters.

[0077] The method emulates cochlear processing of auditory stimuli. In some embodiments, the input sequence comprises a time-sampled auditory stimulus. In some embodiments, the output comprises sound waves. In some embodiments, the output comprises an electrical signal. In some embodiments, the output comprises deep-brain stimulation. The output signals may be used to modify sound characteristics of consumer audio devices, hearables, or hearing-aids. Electrical signal outputs may be used to drive cochlear implants whereas deep-brain stimulation signals may be used for neural stimulation devices.

[0078] In the human cochlear model, the input may comprise sound waves while the output comprises basilar membrane (BM) displacements. In the IHC model, the input may comprise basilar membrane (BM) displacement while the output comprises electrical signals. In the ANF model, the input may comprise electrical signals while the output comprises neural signs.

[0079] The method adequately captures cochlear processing time-dependencies and adaptation. The method also reduces spectral tonal artifacts. Capturing cochlear processing time-dependencies and adaptation properties is essential in generating new-CoNNear-based audio applications for augmented hearing (e.g. human-like noise reduction, ideal hearing-aid algorithm compensation). FIG. 3D, 4D, 5B, and 6D illustrate the target cochlear processing responses as simulated using a transmission-line (TL) based cochlear processing model that can simulate basilar-membrane vibration. The TL model is a realistic model which can capture essential properties of the human cochlear such as adaptation, cochlear compression, and / or cochlear filter tuning. It is a goal of the new CoNNear model to match well with these simulations of the TL model. FIG. 3B, 4B, 5D, 6B, and 7B show the new CoNNear responses, which show good matches with the TL responses. Thus, the new CoNNear model can simulate key cochlear processing aspects such as cochlear nonlinearities, filter characteristics, time-dependencies, and adaptation well.

[0080] Minimizing the generation of tonal artifacts is essential for the audio waveform outputs that can be created with CoNNear. Tonal artifacts in the CoNNear architectures were examined by plotting the frequency response to a 1-kHz pure tone and a step function (DC input). The new CoNNear architecture (FIG. 8C and 8D) can minimize artifacts that were generated by the encoder-decoder structure of the prior CoNNear model (FIG. 8A and 8B).

[0081] The model comprises a multi-layerTemporal Convolution Network (TCN). ATCN is a type of neural network architecture specifically designed for processing sequential data, such as time-series data, audio signals, and natural language sequences. TCNs are inspired by the traditional convolutional neural networks (CNNs) commonly used for image processing, but they are adapted to handle ID data, like sequences, in a more effective manner. TCNs comprise one or more temporal convolutional layers. These layers apply convolutional operations to the input sequence. TCNs employ ID filters that move along the temporal dimension of the sequence. This allows the network to capture local and global patterns in the data. A benefit of the TCN architecture over encoder-decoder based CNNs, is that it uses dilated convolutions instead of transposed convolutions; such that the receptive field size of the network can be increased without increasing the kernel length. Consequently, for the receptive-field size required to adequately capture cochlear processing time-dependencies and adaptation, a TCN with a given kernel size and layers can perform this task with fewer parameters than a traditional CNN.

[0082] In some preferred embodiments, the outputs of each memory block are aggregated through a weighted summation process and subsequently directed to a convolutional layer to obtain the cochlear response; thereby providing a skip connection.

[0083] In some preferred embodiments, in each memory block, the input from the preceding block is utilised, preferably by comprising the steps of: processing one-dimensional waveform input through a first pointwise convolution, followed by depthwise convolution; preferably with dilation factors of 2k l; and, summing the pointwise convolution and depthwise convolution and passing through a non-linear transformation.

[0084] In some preferred embodiments, the method further comprises the step of: obtaining an output of the memory block subsequent to a second pointwise convolution.

[0085] In some preferred embodiments, the second pointwise convolution occurs after applying the at least one nonlinear transformation.

[0086] The TCN preferably comprises a plurality of stacked memory blocks, each memory block comprising at least one pointwise convolution and at least one depthwise convolution with a dilation factor. The inventors have found that high dilation factors enable the network to capture the essential cochlear mechanics properties and do not increase computational complexity.

[0087] High dilation factors allow to capture the cochlear filter tuning and stimulus-level dependency across the different centre frequencies (CF) as can be seen in FIG. 4. The figure evaluates the CoNNears' QERB characteristics across frequency and for different stimulation levels. FIG. 4D illustrates the QERB of target cochlear processing responses as simulated using a transmissionline (TL) model. FIG. 4B shows the QERB of model with high dilation factors (d) [1,2, 4, 8, 1,2, 4, 8, 1,2, 4, 8] across the layers and FIG. 4E shows the model with minimal dilation factors [1,1, 1,1, 1,1, 1,1, 1,1, 1,1] which represent an implementation with normal convolutions. Comparing the simulated QERB of FIG 4. B / E with the target cochlear processing shown in FIG. 4D demonstrates that the network with minimal dilation factors (FIG. 4E) fails to capture the leveldependence of cochlear filter tuning that is present in the target (FIG. 4D) and the new CoNNear model with exponentially increasing dilation (FIG. 4B). Because the new CoNNear is based on dilated convolutions, it can capture these essential cochlear mechanics feature while the overall network parameters are the same for the models with dilation factors [1,2, 4, 8, 1,2, 4, 8, 1,2, 4, 8] and [1,1, 1,1, 1,1, 1,1, 1,1, 1,1].

[0088] The TCN thus preferably comprises a plurality of stacked memory blocks. As used herein, the term "memory block", also referred to as a "residual block" or "dilated convolution block," is a fundamental building block used to process sequential data efficiently. Each memory block comprises a depthwise convolution with dilation factors (d) and this helps to capture long-range dependencies (dilations), while decreasing the required parameter numbers of the model (depthwise) compared to a traditional 1-d convolution block. The memory blocks typically comprise dilated (atrous) convolutions and residual connections. The dilated convolutions may be causal or non-causal. The original DFSMN uses causal convolutions, whereas in the present invention it has been found that the network works best for non-causal convolutions. Preferably, the memory blocks comprise dilated non-causal convolutions.

[0089] Advantageously, residual connections can make the model deeper and help to prevent the vanishing gradient problem. The vanishing gradient problem refers to the issue of diminishing gradients during the training of deep neural networks. It occurs when the gradients propagated backward through the layers become very small, making it difficult for the network to update the weights effectively. Preferably, the TCN comprises at least 4 memory blocks, preferably at least 6, preferably at last 8. Preferably, the TCN comprises at least 8 and at most 16 memory blocks, preferably at least 10 and at most 14 memory blocks, most preferably 12 memory blocks.

[0090] As used herein, the term "convolutional layer" refers to the main building block of a CNN; it comprises a set of filters (or kernels), parameters of which are to be learned throughout the training.

[0091] The memory blocks comprise pointwise and depthwise convolutions. The multi-layer temporal convolution network comprises at least one pointwise convolution, preferably at least two pointwise convolutions.

[0092] As used herein, the term "depthwise convolution" refers to a type of convolution where a single convolutional filter is applied for each input channel. Unlike a traditional convolution, which applies a separate filter to each input channel, depthwise convolutions uses a separate filter for each channel but shares these filters across all spatial positions. The depthwise convolution was designed for mobile and embedded applications, since this operation can reduce the memory compared to applying a standard convolution. The integration of depthwise convolution is employed as a means to reduce the overall number of model parameters, thereby enhancing the efficiency of the model.

[0093] As used herein, the term "pointwise convolution" refers to a type of convolution operation with a kernel size of 1. It operates on the output of the depthwise convolution, typically with the goal of combining and transforming the channels. The term "pointwise" refers to the fact that it performs a convolution at each spatial location (each "point") independently.

[0094] As used herein, the term "dilated convolution" refers to a type of convolution that "inflates" the kernel (filter) by inserting holes between the kernel elements. It can cover a larger area of the input (increase the receptive field). Dilated convolutions, also referred to as atrous convolutions, have gaps between filter elements, which allows the network to capture a broader range of contextual information without significantly increasing the number of parameters. This is particularly useful for handling sequences of varying lengths.

[0095] While a depthwise convolution involves the number of the filters (kernel) which is designed to reduce the model size; a dilated convolution involves inflating the kernel elements to cover a larger area of the sequence signal. The depthwise convolution can be performed with arbitrary dilations. The depthwise convolution shown in the figures thus refers to a depthwise convolution with different dilation factors, or a depthwise dilated convolution, which combines the depthwise separable convolution and dilated convolution techniques.

[0096] In the term "dilation factor (d) of 2k l", the variable "k" refers to the index or step number of the convolutional layer in the network. This variable represents the layer index or the dilation step within a sequence of layers in the network. It typically starts from 1 and increases with each subsequent layer. The dilation factor as such determines how the input signal is sampled during the convolution operation. In dilated convolutions, the dilation factor specifies the spacing between the elements of the convolutional filter as it moves across the input. A higher dilation factor means that the filter covers a larger range of the input with fewer samples, effectively expanding the receptive field without increasing the number of parameters or layers.

[0097] In some preferred embodiments, the method comprises the following step, preferably performed as dilated convolution in the plurality of memory blocks, of: processing one-dimensional waveform inputs through a sequence of stacked memory blocks, wherein the dilation factors increase.

[0098] In some preferred embodiments, the method comprises the following step, preferably performed as dilated convolution in the plurality of memory blocks, of: processing one-dimensional waveform inputs as the at least one input sequence through the plurality of stacked memory blocks, wherein the dilation factors increase.

[0099] In some preferred embodiments, the dilation factors exponentially increase; preferably the dilation factors increase as 2k l.

[0100] The dilation factor of the convolutional layer is a key parameter in the memory block. By stacking multiple memory blocks with increasing dilation rates, a TCN can capture a wide range of temporal dependencies in the input sequence. The blocks with smaller dilation rates capture local patterns, while those with larger dilation rates capture more global patterns. The dilation factor may be chosen individually in each memory block. In some embodiments, the dilation factor is the same in each memory block. It is a goal of the invention to increase the receptive field to capture longterm dependency of the signal (higher receptive field, longer dependency). The receptive field (RF) estimates whether the adaptation time course is sufficiently captured by the architecture, which can be defined as:

[0101] N denotes the number of the memory blocks, fndenotes the filter length and dLdenotes the dilation factor. Preferably, the dilation factors exponentially increase. Preferably, the dilation factors increase as 2k l. The TCN preferably comprises a plurality of residual and / or skip connections between the plurality of memory blocks, configured for forwarding the output of lower layer number memory blocks to at least one higher layer number memory block. The inventors have found that these connections allow to capture the cochlear mechanics features accurately at high CFs (above 7 kHz). Given that humans can hear frequencies up to 8 kHz well, it is important that the cochlear mechanics features CoNNear simulates are accurate at higher CFs. The benefits of adding skip connections are further demonstrated when evaluating the successful capturing of the following cochlear mechanics features: cochlear excitation patterns and cochlear filter tuning (QERB).

[0102] FIG. 3A and 3B show cochlear excitation patterns of networks without and with skip connections, respectively. It demonstrates that adding skip connection between the memory blocks reduces the fluctuation of the simulations at CFs above 4 kHz. The new CoNNear simulations with skip connections (FIG. 3B) match the target cochlear excitation patterns (FIG. 3D) better than the network without skip connections (FIG. 3A). FIG. 4A and 4B show cochlear filter tuning (QERB) of models without and with skip connections, respectively. Compared to the target QERB simulations in FIG. 4D, the model with skip connections (FIG. 4B) matches the QERBs better than the model without skip connections, which exhibits a pronounced decrease beyond 7 kHz (FIG. 4A). Lastly, FIG. 7 investigates the root-mean-square (RMS) error between the simulated cochlear excitation patterns of models without (FIG. 7A) and with (FIG. 7B) skip connections. The new CoNNear with skip connections achieves better performance at low CFs (under 1kHz) and high CFs (above 8kHz) than the model without skip connections, especially for input levels above 70 dB (legend).

[0103] As used herein, the term "skip connection" refers to a connection that computes weighted sums of the outputs from multiple memory blocks, preferably from each memory block, and feeds them into a convolutional layer. This results in the acquisition of the cochlear response across the model architecture. In some embodiments, outputs of each memory block are aggregated through a weighted summation process and subsequently directed to a convolutional layer to obtain the cochlear response; thereby providing a skip connection.

[0104] The minimum number of residual connections preferably equals the number of memory blocks minus one. The number of skip connections preferably equals the number of memory blocks.

[0105] As used herein, the term "residual connection" refers to an interconnection between memory blocks as such. This enables the output of a lower layer memory block to be efficiently directed to a higher layer memory block. Residual connections are typically employed within TCNs to facilitate the training of deep networks and mitigate the vanishing gradient problem.

[0106] In the prior CoNNear architecture, skip connections helped to propagate the information from encoder to the decoder while preserving phase information in the signal. In the new CoNNear, the skip connections serve to aggregate the outputs from all memory blocks, thereby enhancing the model's ability to capture fine temporal details of the input across the entire model architecture. The residual connections enable the direct flow of information from earlier memory block to later block, aiding in preserving the gradient and promoting better convergence.

[0107] The TCN comprises a nonlinear unit for applying a nonlinear transformation. This is also referred to as an activation function. An activation function is preferably applied to introduce non-linearity and enhance the network's capability to model complex patterns. The inventors found that it is preferred to choose a nonlinearity that (i) does not generate undesired distortions such as observed in the gating units networks, and / or that (ii) is capable of capturing key nonlinear aspects of cochlear processing.

[0108] In some preferred embodiments, the nonlinear transformation comprises an activation function having a compression growth and a zero-crossing, wherein said activation function generates positive and negative outputs. In some preferred embodiments, the nonlinear transformation comprises an activation function selected from a hyperbolic tangent type function, preferably selected from the group comprising: hardtanh, tanh, or piecewise-tanh functions. The nonlinear transformation may comprise an activation function selected from a hyperbolic tangent type function selected from the group comprising: hardtanh, tanh, or piecewise-tanh functions; preferably a hardtanh function. In some preferred embodiments, the nonlinear transformation comprises an activation function selected from a hyperbolic tangent type function selected from the group comprising: hardtanh, tanh, or piecewise-tanh functions.

[0109] The term "compression growth" can be understood as the property of an activation function where the output grows at a slower rate than the input, particularly as the input magnitude becomes large. This behaviour assists in controlling the range of outputs and prevents them from becoming excessively large, which is crucial for stabilising the training of deep neural networks.

[0110] In the context of TCNs or any neural network, activation functions that exhibit compression growth ensure that as input values increase, the output values do not grow proportionally but rather are "compressed" or "squeezed" into a smaller range. This can be especially useful in avoiding the issue of exploding gradients during backpropagation in deep networks.

[0111] The term "zero-crossing" refers to the property of an activation function where the function's output changes sign when its input crosses zero. An activation function exhibits zero crossing if it outputs negative values for negative inputs and positive values for positive inputs, and specifically, it outputs zero when the input is exactly zero. This property is particularly useful in neural networks because it can help with the symmetry of the learning process. Activation functions with zero crossing ensure that the model's response to negative inputs is directly opposite to its response to positive inputs, which can be beneficial for balancing gradients during backpropagation and for capturing patterns in the data that involve negative correlations.

[0112] In some preferred embodiments, the activation function is a hardtanh function.

[0113] The benefit of using an activation function with compressive growth and zero-crossing instead of RELU type of non-linearity is that the resulting cochlear processing model can capture the necessary cochlear compression and tuning properties.

[0114] Distortions or artifacts relate to undesired alterations or deformities of the sound input that are not part of the intended cochlear processing, but that are generated through the design architecture of the neural network model. It is possible to isolate the neural-network-generated distortions by comparing simulations performed with a target (reference) deterministic cochlear processing model with simulations performed using a neural-network-based cochlear processing model such as "prior" or "new" CoNNear. FIG. 8A and 8B show an example of tonal distortions that are generated by the prior CoNNear model (indicated by the stars on the magnitude spectrum). Distortions can become audible when their frequency content falls within the human frequency range of hearing (20Hz-10kHz) and the magnitude of the distortions exceeds the threshold of human hearing. For example, for a 1-kHz input sound presented at 70 dB SPL, distortions should be attenuated by more than 60 dB to render them inaudible.

[0115] In some embodiments, the method comprises the step of:

[0116] - receiving at least one input sequence indicative of a time-sampled auditory stimulus, preferably comprising a time-sampled auditory stimulus; preferably from an input means.

[0117] In some embodiments, the method comprises the step of:

[0118] - detecting a time-varying pressure signal indicative of at least one auditory stimulus, preferably comprising a time-sampled auditory stimulus; preferably from a pressure detection means; and,

[0119] - sampling the detected auditory stimulus to obtain an input sequence comprising a plurality of input samples; preferably by a sampling means.

[0120] In some embodiments, the method comprises the step of:

[0121] - outputting N output sequences of cochlear response parameters; preferably through an output means.

[0122] In some embodiments, the method comprises the step of: converting output sequences generated by the neural network into audible timevarying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus; preferably through at least one transducer.

[0123] In a second aspect, the invention relates to a data processing device comprising means for carrying out the method of the first aspect or (preferred) embodiments thereof. The data processing device preferably comprises:

[0124] - input means for receiving at least one input sequence indicative of a time-sampled auditory stimulus;

[0125] - a memory unit for storing at least the network parameters; and,

[0126] - output means for outputting N output sequences of cochlear response parameters.

[0127] In a third aspect, the invention relates to a hearing device comprising the data processing device according to the second aspect, or (preferred) embodiments thereof. The hearing device preferably further comprises: a pressure detection means for detecting a time-varying pressure signal indicative of at least one auditory stimulus; sampling means for sampling the detected auditory stimulus to obtain an input sequence comprising a plurality of input samples; and, at least one transducer for converting output sequences generated by the neural network into audible time-varying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus.

[0128] The hearing device is preferably a cochlear implant or wearable hearing aid.

[0129] In some embodiments, the output comprises sound waves. In some embodiments, the output comprises an electrical signal. In some embodiments, the output comprises deep-brain stimulation. In some embodiments, the input means comprises a microphone.

[0130] In some embodiments, the data processing device comprises a processor, with a dedicated processor for parallel computations (e.g. GPU, VPU, Al-accelerator) being the best choice, since it can compute the output of the NN-based model much faster compared to a CPU.

[0131] The processing device may be a specially designed processing unit such as an ASIC, or may be a dedicated, energy-efficient machine learning hardware module, for instance a convolution accelerator chip, suitable for portable and embedded applications, e.g. battery-powered applications.

[0132] In some embodiments, the output means comprises at least one transducer.

[0133] In some embodiments, the output means is configured to provide audible time-varying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus, for example the transducer may be configured to convert output sequences generated by the neural network into audible time-varying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus.

[0134] In a fourth aspect, the invention relates to a computer program comprising instructions which, when the program is executed by a computer, perform the method steps of the method according to the first aspect, or (preferred) embodiments thereof.

[0135] In a fifth aspect, the invention relates to a computer-readable medium comprising instructions which, when executed by a computer, perform the method steps of the method according to the first aspect, or (preferred) embodiments thereof. The present invention also relates to a computer program, or a computer program product directly loadable into the internal memory of a computer, or a computer program product stored on a computer readable medium, or a combination of such computer programs or computer program products, configured for performing the method as described herein, or embodiments thereof.

[0136] In a sixth aspect, the invention relates to a use of the method according to the first aspect, or (preferred) embodiments thereof, for capturing long-term dependencies of an auditory signal. The use of the method allows to increase the receptive field to capture long-term dependencies of an auditory signal.

[0137] EXAMPLES

[0138] The present invention allows to avoid (audible) artifacts and allows to reduce model size and computational cost, while maintaining optimal cochlear processing performance, as illustrated by below examples.

[0139] FIG. 1 illustrates how one-dimensional waveform inputs are processed through a sequence of stacked memory blocks, wherein the dilation factors exponentially increase. The outputs of each memory block are aggregated through a weighted summation process and subsequently directed to a convolutional layer to obtain the cochlear response. This aggregation is herein referred to as a skip connection. The summed responses are passed through a tanh-based nonlinearity and then sent to a final pointwise convolution to generate waveforms of cochlear processing. The inputoutput transformation can thus be seen as an "end-to-end" operation. Aside from the skip connections in the model, there are also residual connections between the memory blocks.

[0140] FIG. 2A and 2B illustrate the architectural design of the memory block, wherein the input from the preceding block is utilised. The input undergoes processing through pointwise convolution, followed by depthwise convolution with dilation factors of 2k l. In FIG. 2A, a singular depthwise dilated convolution is employed to concurrently capture both historical and future contextual information, which allows the network to capture cochlear processing. FIG. 2B illustrates the deployment of two depthwise dilated convolutions, each independently capturing historical and future information, a preferred choice in this exemplary embodiment.

[0141] The outputs from the pointwise convolution and depthwise convolution are summed up and then passed through the activation function, for example a piecewise linear approximation tangent hyperbolic (pwltanh). The outputs of the memory block are obtained subsequent to the pointwise operation. FIG. 2A or 2B illustrates a residual connection. A residual connection is also observed part of the memory block design, and a skip connection sends the output of each memory block through the entire network.

[0142] Example 1

[0143] In this example, a comparison is made between a memory block that uses a singular depthwise dilated convolution and a memory block that uses two dilated convolutions. The general TCN parameters adopted in this simulation were: 12 memory blocks, each with dilation factors delineated as [1,2, 4, 8, 1,2, 4, 8, 1,2, 4, 8], Each hidden layer had a channel number of 512. Within every memory block, the kernel lengths for the depthwise dilated convolutions were uniformly set to 64. Throughout the model, the hard Hyperbolic Tangent (hardtanh) served as the activation function. The performance of the model was evaluated using a cochlear processing simulation of QERB. Both model architectures capture this essential feature of (human) cochlear processing well.

[0144] Example 2

[0145] In this example, a comparison is made between a network, depicted in FIG. 2B, with or without skip connections in addition to residual connections. A comparison is also made between PReLU and hardtanh activation functions. The simulations below show a benchmarking of new / different / added model features against the prior art model (herein referred to as the prior CoNNear model) and / or cochlear mechanics model (the transmission-line (TL) model or target as described in prior CoNNear). It is an object of the invention to captures all the original TL cochlear mechanics features (click response, yrms excitation patterns, QERB: cochlear filter tuning across centre frequency and stimulus level), while keeping the distortions to a minimal level and reducing the model size. The general parameters of the prior and new CoNNear parameters were as follows: new CoNNear: 12 memory blocks with double depthwise convolution, each memory block with dilation factors delineated as [1,2, 4, 8, 1,2, 4, 8, 1,2, 4, 8], Each hidden layer had a channel number of 512 and the kernel lengths for the two depthwise dilated convolutions was 64. Throughout the model, the hard Hyperbolic Tangent (hardtanh) served as the activation function. Prior CoNNear: four encoder layers and four decoder layers, each with a stride of 2. Every layer consisted of 128 filters with a length of 64, each followed by a Tanh activation function, except for the last layer of the decoder.

[0146] - FIG. 3D, 4D, 5B, 6D illustrate the target cochlear processing responses as simulated using a transmission-line (TL) based cochlear processing model that can simulate basilar-membrane vibration (prior CoNNear). This model does not use a neural-network-based architecture and captures essential nonlinear features of human hearing. The new CoNNear model should capture the target simulations as closely as possible to be an adequate model of cochlear processing.

[0147] - FIG. 3A, 4A, 6A, 7A illustrate a TCN-based model without a skip connection and a hardtanh activation.

[0148] - FIG. 3B, 4B, 5D, 6B, 7B illustrate a TCN-based model with a skip connection and a hardtanh activation.

[0149] - FIG. 3C, 4C, 6C and 7C illustrate a TCN-based model with a skip connection and a PReLU activation.

[0150] Cochlear Excitation Patterns FIG. 3A-D demonstrate that both models can capture the property of level-dependent filter tuning, and that using skip connection can reduce the fluctuation at high CFs. Whereas PReLU activation function was unable to capture the level-dependence of cochlear filter tuning, hardtanh activation can capture the level-dependent filter tuning property.

[0151] Cochlear filter : FIG. 4A-D demonstrate that the model without skip connection exhibits a pronounced decrease beyond 7 kHz, whereas the presence of a skip connection obviates this phenomenon. QERB functions of PReLU remained invariant to stimulus level changes which indicates that PReLU nonlinearity failed to capture level-dependent cochlear filter tuning. FIG. 5 illustrates that there is no notable difference between the transmission line (TL) model (B) and new CoNNear model (C). When using a PReLU activation function instead of the preferred hardtanh function, the error between target TL simulation and

[0152] CoNNear is significant.

[0153] Distortion-product otoacoustic emission : FIG 6A-D illustrate simulations in response to two pure tones of fl of 2.0 and f2 of 2.4 kHz. In humans, the most pronounced distortion product (DP) occurs at 2fl-f2 (1.6 kHz). Both models can capture the level of the DP at 1.6kHz. The level of this DP was better captured using hardtanh activation.

[0154] FIG. 6 Panel B shows the new CoNNear model with skip connections and tanh nonlinearity, other TCN structures without skip connections (panel A) or PReLU nonlinearity (panel C) capture the reference TL response less good. Replacing the nonlinearity by a PReLU gave the worst degradation.

[0155] RECTIFIED SHEET (RULE 91 ) ISA / EP Root-mean-square (RMS) error between the excitation patterns: FIG. 7A-C illustrate that the model incorporating skip connections demonstrates overall comparable RMS value compared to the model without skip connections. At high stimulation levels (90dB), and low stimulation frequencies (Ncf), the new CoNNear with skip connection and hardtanh shows smaller RMS errors (FIG. 7B), than the model without skip connections (FIG. 7A). The RMS error is overall greater than the model with PReLU activation function.

[0156] Example 3

[0157] The generation of undesired distortion caused by design choices of the neural network architecture were evaluated by plotting the frequency response to lk Hz pure tone and a step function (DC input). Artifacts in the prior CoNNear simulation illustrate how the original decoder architecture needed to be avoided. The encoder-decoder structure was replaced by a TCN to avoid upsampling artifacts.

[0158] Furthermore, the architecture of the memory block (activation function choice, skip connections) was important to ensure no distortions were generated by the new CoNNear architecture. The new CoNNear architecture (FIG. 8C and 8D) can overcome artifacts that were generated by the encoder-decoder prior CoNNear model (FIG. 8A and 8B). The new CoNNear simulations correspond well to the target while the distortions are no longer present. FIG. 8E and 8F show that the skip connections are not an essential feature to minimize the distortions, but they were priorly shown to be essential to capture essential DPOAE and speech processing properties of cochlear processing (see FIG. 5 and FIG. 6). Skip connections furthermore imply a on overall model performance improvement, as described in FIG. 4A and 4B which show that skip connections benefit the filter tunning properties at high CFs (above 7kHz). Additionally, FIG. 7A and 7B show that skip connections achieved better performance at low CFs (under 1kHz) and high CFs (above 8kHz), especially at high input levels. FIG. 8G and 8H show that using an activation function in the memory unit that is purely based on PReLU was insufficient to remove the artifacts generated by the neural-network architecture.

[0159] Example 4: model performance benchmarks

[0160] Latency comparison: We conducted a latency comparison between the prior CoNNear model and the new CoNNear model using a stimulus window consisting of 1048 samples, both executed on a NVIDIA GTX1080 GPU. The latency exhibited by the new CoNNear model, exemplified by 7.31 milliseconds, demonstrates its competitive equivalence with the prior CoNNear model (7.27 milliseconds). Furthermore, it is anticipated that ongoing optimization efforts for the new CoNNear model will result in even lower latency when compared to prior CoNNear. Parameters and computational complexity: For the number of parameters, the new CoNNear (6.7 million) is nearly half of the prior CoNNear (11.7 million). For the computation complexity, the new CoNNear is 15.3G MAC (multiply-accumulate operation) while the prior CoNNear is 12.2G MAC.

[0161] Receptive field and parameters: For prior CoNNear, the receptive filed is 47.3ms (20kHz sampling rate) while the number of parameters is 11.7M; for new CoNNear, the receptive filed is 94.6ms while the number of parameters is 6.9M. The new CoNNear has less parameters but a longer receptive field.

[0162] Example 5

[0163] In the example of hearing-aid processing, we evaluate the artifacts of hearing-aid (HA) algorithms trained using a DNN-based closed-loop system that includes CoNNear models of normal and hearing-impaired cochlear processing and a loss function. We compare the performance of the resulting HA processing for the new CoNNear architecture and compare it to prior CoNNear architectures. The closed-loop system for individualized DNN-based HA model training is shown in FIG. 10. It consists of two auditory pathways: One corresponding to the auditory nerve (AN) response r of an NH auditory system, and one corresponding to the response i of a HI system. The CoNNear models simulate cochlear responses of different processing stages along the pathway: human cochlear, IHC, and ANF processing. The HA model is trained by minimizing a predefined loss function between the simulated NH and HI responses. The trained HA algorithm (x) is then applied to a speech sample to restore the hearing in the impaired cochlea, i.e., the target of the loss function and the trained x is to bring ff- as close as possible to the original . FIG. 11A and 11B show the outputs of the HA model (x) trained with two different CoNNear-based frameworks to compensate for a typical sensorineural high-frequency sloping hearing loss (HI type "Slope35-7,0,0"). A spectral analysis of the HA response to a given a 1-kHz tonal sound input is shown together with the stimulus itself.

[0164] In prior CoNNear-based closed-loop systems, prior CoNNear model architectures were adopted for the human auditory processing (z and r ) and DNN-based HA model blocks. However, the prior CoNNear architecture creates undesired artifacts, which propagate through the closed-loop network to produce audible artifacts that compromise the sound quality of the resulting HA model. These artifacts are clearly visible in FIG. 11A as spectral tonal artifacts in the frequency spectrum that are not part of the stimulus frequency (1kHz) or its harmonics. The distortion energy is strong compared to the original stimulus and implicates that prior CoNNear-based closed-loop systems, and by extension those that are based on classical encoder-decoder structures, are difficult to use for HA or audio applications. In the new CoNNear-based framework, we simulated the human cochlear, IHC and ANF processing based on the new CoNNear architecture to circumvent the problem of artifacts created by the encoder-decoder structure. FIG. 11B illustrates the 1-kHz tone response of the HA model trained by new CoNNear-based framework. It is clearly observed that the spectral tonal artifacts are removed in this new CoNNear framework and that only energy remains at the stimulus frequency and its harmonics. This example thus demonstrates that the new CoNNear-based closed-loop framework can overcome artifacts generated by the prior CoNNear model architectures. As a result, the sound quality of the resulting HA algorithm thus markedly improved.

[0165] The details of the adopted CoNNear and HA parameters in the closed-loop network of FIG. 10 and the resulting HA simulations shown in FIG. 11 are listed in Table 1 and described here. The new CoNNear comprises a sequence of stacked memory blocks with dilation factors exponentially increasing. K blocks with dilation factors 2A(k - 1) are repeated R times, yielding a total of N = K * R memory blocks. The training dataset for new CoNNear and HA models consisted of 2310 randomly selected recordings from the TIMIT speech corpus. The model training was performed in two phases. First, we trained the auditory (cochlear, IHC and ANF) model elements, using reference analytical BM, IHC and ANF models, respectively. The new CoNNear models were optimized with the mean absolute error (MAE) between the analytical and predicted new CoNNear outputs. The trained new CoNNears simulate cochlear responses corresponding to 201 centre frequencies between 100 Hz and 12 kHz. When training the HA model, Ncf = 21 equally spaced frequency channels were selected out of the 201 to speed up the training procedure. The HA-model was trained with the mean-squared-error (MSE) between new CoNNear-simulated NH and HI auditory nerve responses, while parameters of the new CoNNear elements were kept frozen. A learning rate of 0.0001 was used with an Adam optimizer for all training procedure and the entire framework was developed in PyTorch.

[0166] Table 1. Hyperparameters of the new CoNNear elements and HA model. fndenotes the kernel length of the 1-D dilated convolutional layer in each memory block; H indicates the channel number of hidden layers; L, Li, and Lrrepresent the input length, left context, and right context, respectively. Example 6

[0167] This example illustrates the new CoNNear hyperparameter choice and impact on hearing-aid restoration quality and execution load.

[0168] To optimize the hyperparameters of the new CoNNear based HA model for future hardware implementations, various HA architectures and their efficacy in HI compensating were investigated. Ideally, the HA architecture becomes as small as possible in the total number of required parameters, while it maintains a good HA compensation that brings the cochlear response of a HI patient as close as possible to the cochlear response of a NH patient. Multiple new CoNNear models (with different hyperparameter choices) are trained using the specifications outlined in Table 1, and their results were compared to the baseline HA model with the most parameters and best HA compensation efficacy to determine the most optimal model implementation in terms of parameter number and processing latency. Different model architectures were compared with different layers (N), dilation factors (d, or R_s) and right kernel (R_k) sizes to identify the best configuration.

[0169] FIG. 12 shows the quality of the CS compensation for the simulated auditory-nerve (AN) population response of a HI model when a new-CoNNear-based HA algorithm was used to process the audio. A sentence was processed with one of the new-CoNNear models and the resulting AN population responses were evaluated in CFs that cover the speech frequency range. The solid black line represents the target NH AN populations response, while the dashed line with square markers represents the HI model in response to unprocessed speech. The other dashed lines represent the responses of each model. The closer a HA processed responses is to the NH AN response (black line), the better the HA performs. All models improved the HI response which indicates that models with different hyperparameter choices can yield a good HA performance, and that - depending on the final application - different hyperparameters combinations can be chosen.

[0170] FIG. 13 shows RMS energy of simulated CoNNear_cochlea basilar-membrane responses (y_bm) across the CFs that span the speech frequency range. The target NH response (solidblack) and unprocessed HI response (dashed with squares) are shown as well as various compensated responses which reflect the HA processing for new CoNNear models with different hyperparameter choices. It becomes apparent that at CFs lower than 1 kHz, the model with memory blocks L of 8, kernel length R_k of 32 and dilation factor R_s of [1,2,..] and the model of L of 6, R_k of 32 and R_s of [1,1..] have responses that are closest to the NH response, and thus show best performance. Between CFs of 1 kHz and 2 kHz, the L 12, R_k 32, and R_s[l,2..] model and the L 6, R_k 32, R_k [1,1..] model perform better than the other versions. Moreover, for CFs higher than 2 kHz, the L 4, R_k 32, and R_s[2,2..] and 2 kHz, the L 4, R_k 16, and R_s[l,l..] models are more effective in aligning the HA-processed y_bm with the NH target. These simulations show that depending on the hardware characteristics and the speech frequency range of interest, new CoNNears with various hyperparameter combinations may perform well to offer HA compensation.

[0171] The cross-correlation serves as a metric for assessing the similarity between two waveforms relative to their time delay. For two jointly stationary random processes, and the cross-correlation sequence is expressed as:

[0172] Here, the asterisk represents complex conjugation, and E denotes the expected value operator. Additionally, the normalized cross-correlation (NCC) is defined as:

[0173] FIG. 14 illustrates the maximum cross correlation (NCC) for the HA-compensated AN population response with the NH AN population response for new CoNNear models with various hyperparameter choices. The NCC values are shown for a processed sentence and evaluated across the CFs that cover the speech frequency range. The bottom dashed curve with circles represents the NCC response between NH and unprocessed HI responses, and acts as lower performance limit. The upper bound of performance is established by the NCC of NH with itself, yielding a value of 1. The other lines depict the maximum NCC of each HA model with different hyperparameters and shows that all models have a benefit with the best performing model for the CFs above 6 kHz that with L 4, R_k 32, and R_S [2,2,..] hyperparameters.

[0174] FIG. 15 shows the maximum NCC output of HA processed speech using a model with 12 memory blocks (layers), alongside models that used different hyperparameters. The first bar shows the maximum NCC of the 12-layers model with itself, yielding a value of 1. The other bars represent the maximum NCC responses between each model and the reference 12-layer model. The model with L=6, R_k 32, and R_S [1,2, ..] (horizontally hashed bar) performs best on this task.

[0175] To effectively select the optimal model for hardware implementation, it is crucial to prioritize the processing capabilities of the hardware. After careful evaluation of factors such as number of parameters, latency, and model performance in the tasks simulated in FIG.12-15, the 6-layer CoNNear HA model with a dilation factor of 1 is as a choice for initial hardware deployment.

[0176] Example 7 This example illustrates new CoNNear architecture optimization for optimal hardware implementation.

[0177] Every sample processed by the CoNNear algorithm requires additional context to preserve the adaptation properties of cochlear processing. Some future and historic data is thus required to calculate the right and left depth-wise convolutions respectively in FIG. 2B.

[0178] The length of the required context data depends on the chosen kernel size (fn) and dilation factor (d). For example, a dilation factor d of 2A4 with a kernel length fn of 32, requires a context length of 2A4*31+32 = 528 samples. At a sampling rate of 16 kHz, this corresponds to 33ms of sound. To enable real-time processing of the algorithm for hardware implementations, the total latency resulting from the processing should be limited to 10 ms to not perceive the processing delay by the human ear. Hence, the batch size should be limited to a fraction of this. E.g. 2ms, corresponding to 32 samples. However, if for every 32 samples an additional context of 528 samples needs to be evaluated and this would yield an unacceptable computational cost.

[0179] To overcome this limitation and to optimize the CoNNear implementation for hardware applications, the context can be retained between batches. The network in FIG. 16 shows a possible implementation of a new CoNNear memory element that can perform batch processing. The notation in brackets shows the dimension of the signals propagating on the wires in (features, batch size) where batch size is the number of audio samples that are processed in a single iteration of the network.

[0180] The example shows one of the inner memory elements with 256 input and output features which performs processing on 32 audio samples. In this example, the left convolution has a dilation of d and a kernel size (fn) of 32, the right convolution has no dilation and a kernel size of 32. Hence, to calculate a single output sample, the right convolution requires 32 samples, or to calculate 32 samples corresponding to the 32 batch size, 31 additional samples are required, resulting in a total of 61 samples.

[0181] The left convolution requires 31*(2Ak-l)+32 samples to calculate a single output, and an additional sample per input sample. Hence for a batch size of 32, 31*2Ak+32 samples are required. The example implements two features that enable real time processing:

[0182] A delay is introduced such that the right convolution does not require future data. In this example a delay of 2 ms is introduced. A 4-layer implementation with a batch size of 32 samples results in a total delay of 10 ms, such that hearing-aid latencies are respected.

[0183] The context is remembered between iterations of the network (dash lines). The right convolution is provided with the most recent features coming from the pointwise convolution (convlD) and extended with 31 most recent context samples. The direct feed is a 32 samples-delayed version. The left convolution requires all context samples. The 32 output samples of the convlD calculation are concatenated with the most recent context samples to obtain the context for the next iteration of the network

[0184] This implementation depicted in FIG. 16 results in no additional calculations due to the required context for calculating the network. At the same time, the performance of the context-aware new CoNNear implementation is no different from that of the new CoNNear implementation depicted in FIG. 2B. FIG. 17 compares the output of one of the features of a single layer network when simulated with a large input batch size (40000 samples) and when the batch size is reduced to 32 samples and the network is iteratively evaluated while the context is retained (i.e., context-aware retraining). The context-aware network is populated with the same trained coefficients as the original new CoNNear network, so no additional training is required. Both networks have identical output as can be seen in FIG. 17, proving the validity of the approach.

Claims

CLAIMS1. A computer-implemented method for emulating cochlear processing of auditory stimuli, the method comprising the steps of: providing a multi-layer temporal convolution network (TCN), the network comprising: a. a plurality of stacked memory blocks, each memory block comprising at least one pointwise convolution and at least one depthwise convolution with a dilation factor; b. at least one nonlinear unit for applying a nonlinear transformation to the activation maps generated by at least one convolutional layer of the network; c. a plurality of residual and / or skip connections between the plurality of memory blocks, configured for forwarding the output of lower layer number memory blocks to at least one higher layer number memory block; d. an input layer, configured for receiving an input sequence to the network; and, e. an output layer, configured for generating for each input to the network, N output sequences of cochlear response parameters; providing at least one input sequence of any length, wherein the input sequence is indicative of a time-sampled auditory stimulus; and, applying the at least one input sequence to the input layer of the neural network to obtain the N output sequences of cochlear response parameters.

2. The method according to claim 1, wherein the nonlinear transformation comprises an activation function having a compression growth and a zero-crossing, wherein said activation function generates positive and negative outputs.

3. The method according to any one of the previous claims, wherein the activation function is selected from a hyperbolic tangent type function selected from the group comprising: hardtanh, tanh, or piecewise-tanh functions, most preferably a hardtanh function.

4. The method according to any one of the previous claims, comprising the following step, performed as dilated convolution in the plurality of memory blocks, of: processing one-dimensional waveform inputs as the at least one input sequence through the plurality of stacked memory blocks, wherein the dilation factors increase.

5. The method according to claim 4, wherein the dilation factors exponentially increase; preferably wherein the dilation factors increase as 2k l.

6. The method according to any one of claims 4 or 5, wherein the outputs of each memory block are aggregated through a weighted summation process and subsequently directed to a convolutional layer to obtain the cochlear response; thereby providing a skip connection.

7. The method according to any one of claims 4 to 6, whereby in each memory block, the input from the preceding block is utilised, by comprising the steps of: processing one-dimensional waveform input through a first pointwise convolution, followed by depthwise convolution; preferably with dilation factors of 2k l; and, summing the pointwise convolution and depthwise convolution and passing through a non-linear transformation.

8. The method according to claim 7, further comprising the step of: obtaining an output of the memory block subsequent to a second pointwise convolution.

9. The method according to claim 8, wherein the second pointwise convolution occurs after applying the at least one nonlinear transformation.

10. The method according to any one of the previous claims, wherein the TCN comprises at least 4 memory blocks, preferably at least 6 memory blocks, preferably at least 8 memory blocks; preferably wherein the TCN comprises at least 8 and at most 16 memory blocks, preferably at least 10 and at most 14 memory blocks, most preferably 12 memory blocks.

11. The method according to any one of the previous claims, comprising the steps of: detecting a time-varying pressure signal comprising a time-sampled auditory stimulus from a pressure detection means; and, sampling the detected auditory stimulus to obtain an input sequence comprising a plurality of input samples by a sampling means; and,converting the output sequences generated by the neural network into audible timevarying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus.

12. A data processing device comprising means for carrying out the method of any of the preceding claims, the data processing device further comprising:- input means for receiving at least one input sequence indicative of a time-sampled auditory stimulus;- a memory unit for storing at least the network parameters; and,- output means for outputting N output sequences of cochlear response parameters.

13. A hearing device comprising the data processing device of the previous claim, further comprising: a pressure detection means for detecting a time-varying pressure signal indicative of at least one auditory stimulus; sampling means for sampling the detected auditory stimulus to obtain an input sequence comprising a plurality of input samples; and, at least one transducer for converting output sequences generated by the neural network into audible time-varying pressure signals, basilar membrane vibrations, or corresponding auditory nerve stimuli associated with the at least one auditory stimulus.

14. A computer-readable medium comprising instructions which, when executed by a computer, perform the method steps of the method of any one of the claims 1 to 10.

15. Use of the method of any one of the claims 1 to 10 for capturing long-term dependencies of an auditory signal.

Images