Sound scene enhancement system and forming method thereof
Through the artificial intelligence model and data acquisition system, the appropriate masking settings are automatically selected, which solves the problem of masking in the prior art selection statically or relying on expert guidance, and achieves the effect of real-time optimization of sound scene perception attributes and acoustic comfort.
Patent Information
- Application Number
- CN202380036570.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-27
- Filing Date
- 2023-04-26
- Publication Date
- 2024-12-06
AI Technical Summary
Existing soundscape enhancement technologies are difficult to automatically select suitable masking settings, and masking selection usually depends on static or expert guidance, and cannot adapt to dynamic acoustic environments in real time, affecting acoustic comfort and overall perception.
采用人工智能模型,结合数据获取系统、感知属性预测器和掩蔽设置排序系统,自动选择基于周围声景数据生成的预定义感知属性指标上的最佳掩蔽设置,实时调整掩蔽器以优化声景感知。
It realizes automated masking setting selection, optimizes the sound and scene perception attributes in real time, improves acoustic comfort and overall sound and scene quality, adapts to dynamic acoustic environment, and reduces the dependence of user intervention and expert guidance.
Smart Images

Figure CN119096291A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of priority to Singapore application number 10202204451S filed on April 27, 2022, the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field
[0003] Embodiments of the present application relate to a soundscape enhancement system. Embodiments of the present application relate to a method for forming a soundscape enhancement system. Background Art
[0004] The World Health Organization (WHO) has ranked noise exposure as the second most pollutant in the environment after air pollution. Noise exposure has also been likened to second-hand smoke. Urgent calls for action to reduce noise exposure stem from growing evidence of the health effects of noise, such as increased risk of ischemic heart disease, irritability and sleep disturbances. The WHO has also highlighted the impact of environmental noise exposure on mental health and well-being, with emerging evidence showing adverse effects on indicators of depression and anxiety.
[0005] A paradigm shift from noise management to soundscape management is occurring. ISO 12913-1 defines “soundscape” as “the acoustic environment as perceived or experienced and / or as understood by one or more persons in context”. In contrast to noise management, the soundscape management framework considers sound as a resource rather than a waste; it focuses on preferred sounds rather than unpleasant sounds; and it manages for desired sounds to mask undesirable sounds, and manages to reduce undesirable sounds instead of just reducing sound levels. Thus, there are soundscape intervention techniques based on the enhancement of “masking” sounds or the introduction of “masking” sounds into an acoustic environment to improve the overall perception of acoustic comfort. Typically, the intervention involves the enhancement of natural sounds (e.g., through loudspeakers) and has been trialed in outdoor recreational settings and indoors, such as nursing homes, to improve acoustic comfort. Soundscape enhancement with desired sounds is similar to sound masking systems, which are commonly used in office environments to reduce distractions. The key difference between sound masking systems and soundscape-based augmentation systems is that masking is based on objective metrics, whereas soundscape augmentation is based primarily on subjective metrics based on context. Context "includes the interrelationships of people, activities, and places in space and time," and is further detailed in ISO12913-1 with examples. It is also worth noting that soundscapes are not limited to real-world environments, but can also include virtual environments and even memories. Summary of the invention
[0006] Embodiments of the present application may provide a soundscape enhancement system. The soundscape enhancement system may include a data acquisition system configured to provide ambient soundscape data. The soundscape enhancement system may also include a database, the database including multiple masking settings. The soundscape enhancement system may also include a perceptual attribute predictor, the perceptual attribute predictor is coupled to the data acquisition system and the database, the perceptual attribute predictor is configured to generate a prediction representing perception on one or more predefined perceptual attribute indicators for each masking setting in the multiple masking settings based on the ambient soundscape data. The soundscape enhancement system may additionally include a masking setting ranking system, the masking setting ranking system is configured to determine one or more optimal masking settings based on the prediction generated by the perceptual attribute predictor. The soundscape enhancement system may also include a playback system, the playback system is configured to play or reproduce one or more optimal masking settings.
[0007] Embodiments of the present application may relate to a method of forming a soundscape enhancement system. The method may include providing a data acquisition system, the data acquisition system being configured to provide ambient soundscape data. The method may also include providing a database, the database including a plurality of masking settings. The method may further include coupling a perceptual attribute predictor to the data acquisition system and the database, the perceptual attribute predictor being configured to generate, based on the ambient soundscape data, for each of the plurality of masking settings, a prediction representing perception on one or more predefined perceptual attribute indicators. The method may additionally include providing a masking setting ranking system, the masking setting ranking system being configured to determine one or more optimal masking settings based on the predictions generated by the perceptual attribute predictor. The method may further include providing a playback system, the playback system being configured to play or reproduce the one or more optimal masking settings. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In the accompanying drawings, like reference characters generally refer to the same parts in different views. The accompanying drawings are not necessarily drawn to scale, but emphasis is placed essentially on illustrating the principles of the various embodiments. In the following description, various embodiments of the present invention will be described with reference to the following drawings.
[0009] Figure 1 A schematic diagram of a soundscape enhancement system according to various embodiments is shown.
[0010] Figure 2 A schematic diagram showing a method of forming a soundscape enhancement system according to various embodiments.
[0011] Figure 3 A schematic diagram of a soundscape enhancement system according to various embodiments is shown.
[0012] Figure 4 The two-dimensional circumferential eight-point model of the emotional quality attribute proposed by ISO 12913-3 is shown, and the prediction of the perceptual attribute predictor according to various embodiments may be based on the two-dimensional circumferential eight-point model.
[0013] Figure 5 is a schematic diagram of a perceptual attribute predictor according to various embodiments.
[0014] Figure 6 is a schematic diagram of a perceptual attribute predictor according to various embodiments.
[0015] Figure 7 is a schematic diagram of a perceptual attribute predictor according to various embodiments.
[0016] Figure 8 The training and inference schemes of an automatic masker selection system (AMSS) according to various embodiments are shown.
[0017] Fig. 9 Shown are (a) a basic convolutional recurrent neural network (CRNN) architecture used according to various embodiments; (b) a possible implementation of the feature mapping block in (a) according to various embodiments; and (c) other possible implementations of the feature mapping block in (a) according to various other embodiments.
[0018] Fig.10 is a table showing mean fold mean squared errors (MSEs) (± standard deviation) of the probabilistic perceptual attribute predictor (PPAP) in 10 runs (top: cross validation set, bottom: test set) tested on each setting according to various embodiments.
[0019] Fig.11 shows mask selection using 50 models (dot product variants) on the test set with naive maximum μ according to various embodiments. k Value selection scheme (left) and random sampling scheme according to various embodiments to incentivize masker exploration (right).
[0020] Fig. 12A A schematic diagram of a probabilistic perceptual attribute predictor (PPAP) according to various embodiments is shown.
[0021] Fig. 12B Three feature enhancement methods according to calculation formulas (8), (9) and (10) in various embodiments are shown.
[0022] Fig.13 An algorithm for reducing the overall runtime according to various embodiments is shown.
[0023] Fig.14 Shown is (top) a graph of mean squared error (MSE) as a function of attention block type, which shows a violin graph of the validation MSE of pleasantness prediction for each attention block type and feature enhancement method according to various embodiments; and (bottom) a graph of mean absolute error (MAE) as a function of attention block type and feature enhancement method, which shows a violin graph of the validation MAE of pleasantness prediction according to various embodiments.
[0024] Fig.15A is a graph of ISO pleasantness as a function of log gain showing gain interpolation for masker class “water” using additive attention (AA) and CONV augmentation (seed 5, validation fold 2), according to various embodiments.
[0025] Fig. 15B is a graph of ISO pleasantness as a function of log gain showing gain interpolation for masker class “traffic” using additive attention (AA) and CONV augmentation (seed 5, validation fold 2), according to various embodiments.
[0026] Fig. 15C is a graph of ISO pleasantness as a function of log gain showing gain interpolation for masker class “bird” using additive attention (AA) and CONV augmentation (seed 5, validation fold 2), according to various embodiments.
[0027] Fig.15D is a graph of ISO pleasantness as a function of log gain, showing gain interpolation for masker class “Construction” using Additive Attention (AA) and CONV augmentation (seed 5, validation fold 2), according to various embodiments.
[0028] Fig.15E is a graph of ISO pleasantness as a function of log gain showing gain interpolation for masker class “silence” using additive attention (AA) and CONV augmentation (seed 5, validation fold 2), according to various embodiments. DETAILED DESCRIPTION
[0029] The following detailed description refers to the accompanying drawings, which show by way of illustration specific details and embodiments in which the present invention may be implemented. These embodiments are described in sufficient detail to enable those skilled in the art to implement the present invention. Other embodiments may be used, and structural, logical, and electrical changes may be made without departing from the scope of the present invention. The various embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.
[0030] Features described in the context of one embodiment may be applicable to the same or similar features in other embodiments accordingly. Features described in the context of one embodiment may be applicable to other embodiments accordingly, even if not explicitly described in these other embodiments. In addition, additions and / or combinations and / or substitutions described for a feature in the context of one embodiment may be applicable to the same or similar features in other embodiments accordingly.
[0031] In the context of various embodiments, the articles “a,” “an,” and “the” used with respect to features or elements include reference to one or more features or elements.
[0032] In the context of the various embodiments, the term "about" or "approximately" as applied to a numerical value encompasses the exact value and reasonable variances, such as within 10% of the specified value.
[0033] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0034] “Including” means including but not limited to whatever follows the word “including.” Thus, the use of the term “including” indicates that the listed elements are required or mandatory, but other elements are optional and may or may not be present.
[0035] “Consisting of” means including and limited to whatever is between the phrase “consisting of.” Thus, the phrase “consisting of” indicates that the listed elements are required or mandatory, and that no other elements are present.
[0036] Embodiments described in the context of one soundscape enhancement system are equally valid for other soundscape enhancement systems. Similarly, embodiments described in the context of a method are equally valid for a soundscape enhancement system and vice versa.
[0037] Current state-of-the-art augmentation or masking interventions involve static playback of sound through loudspeakers, where the sound level and track must be manually selected. Various embodiments may utilize a pioneering artificial intelligence model to automatically select a track that produces the highest (or lowest) value on any subjective metric (e.g., perceived loudness, ISO12913-3's CircumplexOctant Scale - pleasantness, annoyance, tranquility, calmness, vibrancy, eventfulness, etc.) at the most appropriate sound level. The artificial intelligence (AI) model was trained by a Nanyang Technological University (NTU) research team on a large dataset of human subjective responses from a local population in Singapore collected based on the ISO12913-2 standard. The ISO12913 series of standards represents a paradigm shift in sound environment management and details a perception-based approach to comprehensively assess and analyze sound environments. Furthermore, the AI model can also double as a predictive tool for automatically assessing the perceived quality of the sound environment without conducting tedious and labor-intensive surveys or questionnaires as required by ISO12913-2.
[0038] Figure 1 is a schematic diagram of a soundscape enhancement system according to various embodiments. The soundscape enhancement system may include a data acquisition system 102, which is configured to provide ambient soundscape data. The soundscape enhancement system may also include a database 104, which includes a plurality of masking settings. The soundscape enhancement system may further include a perceptual attribute predictor 106, which is coupled to the data acquisition system 102 and the database 104, and the perceptual attribute predictor 106 is configured to generate a prediction representing perception on one or more predefined perceptual attribute indicators for each masking setting in the plurality of masking settings based on the ambient soundscape data. The soundscape enhancement system may additionally include a masking setting ranking system 108, which is configured to determine one or more optimal masking settings based on the prediction generated by the perceptual attribute predictor 106. The soundscape enhancement system may also include a playback system 110, which is configured to play or reproduce the one or more optimal masking settings.
[0039] In other words, the soundscape enhancement system may include a perceptual attribute predictor 106, which may be connected to the data acquisition system 102 and the database 104. The soundscape enhancement system may also include a masking setting ranking system 108 connected to the perceptual attribute predictor 106 and a playback system 110 connected to the masking setting ranking system 108.
[0040] For the avoidance of doubt, Figure 1 It is intended to illustrate some features of the soundscape enhancement system according to various embodiments, and is not intended to limit, for example, the size, shape, orientation, arrangement, etc. of the various components.
[0041] In various embodiments, the data acquisition system 102 may also be configured to provide group demographic or contextual data. The sensory attribute predictor 106 may also be configured to generate predictions based on the group demographic or contextual data. The group demographic or contextual data may be directly received, measured, or sensed by the data acquisition system 102, or received from an external source, such as a portable health monitoring device or a storage device, as described in more detail below.
[0042] The demographic or situational data may include data related to environmental parameters, demographic data of one or more listeners, psychological data of one or more listeners, physiological data of one or more listeners, or any combination thereof.
[0043] In this regard, the one or more audiences may include real audiences and / or imaginary audiences.
[0044] In various embodiments, environmental data related to environmental parameters may be or may include data related to location, data related to the visual environment, the number of people nearby, air temperature, humidity, wind speed, or any combination thereof. Environmental parameters may be non-acoustic environmental parameters. For example, the data acquisition system 102 may include or may be coupled to a global positioning system (GPS) sensor to determine data related to location. The data acquisition system 102 may include or may be coupled to a camera to determine data related to the number of people nearby and / or obtain visual information from the environment (e.g., determining the presence of a particular type of furniture, pets, etc., determining the movement of people, determining the percentage of greenery or water present, etc.). The data acquisition system 102 may include or may be coupled to a thermometer to determine data related to air temperature. The data acquisition system 102 may include or may be coupled to a hygrometer to determine data related to humidity. The data acquisition system 102 may include or may be coupled to an anemometer to determine data related to wind speed. In this context, the term "coupled" may refer to a direct connection or an indirect connection. The connection may be established, for example, by wire or wirelessly.
[0045] In various embodiments, the demographic data of the one or more listeners may be or may include data related to age, gender, occupation, or any combination thereof. In various embodiments, the demographic data of the one or more listeners provided by the data acquisition system 102 may be obtained by the data acquisition system 102 from an external source. In various embodiments, the demographic data of the one or more listeners may be provided in a digitized form attached to the data acquisition system 102, or may be provided as preloaded data on a storage device coupled to (e.g., directly coupled or coupled via a cloud-based server) the data acquisition system 102. In various embodiments, the demographic data of the one or more listeners may be provided to the data acquisition system 102 manually, such as manually entered into the data acquisition system 102, or sent to the data acquisition system 102 from a remote system according to manual instructions provided by a user. In various embodiments, the data acquisition system 102 may be configured to automatically obtain the demographic data of the one or more listeners, such as by wired or wireless means from a remote system.
[0046] In various embodiments, the psychological data of the one or more listeners may be or may include data related to noise sensitivity (using, for example, the Weinstein Noise Sensitivity Scale), perceived stress (using, for example, the Cohen's Perceived Stress Scale), a well-being index score (using, for example, the WHO-5 Well-Being Index), or any combination thereof. In various embodiments, the psychological data of the one or more listeners provided by the data acquisition system 102 may be obtained by the data acquisition system 102 from an external source. In various embodiments, the psychological data of the one or more listeners may be provided in a digitized form attached to the data acquisition system 102, or may be provided as pre-loaded data on a storage device coupled to (e.g., directly coupled or coupled through a cloud-based server) the data acquisition system 102. In various embodiments, the psychological data of the one or more listeners may be provided manually to the data acquisition system 102, such as manually typed into the data acquisition system 102, or sent from a remote system to the data acquisition system 102 according to manual instructions provided by a user. In various embodiments, the data acquisition system 102 may be configured to automatically obtain psychological data of one or more listeners, for example, via a wired or wireless method from a remote system.
[0047] In various embodiments, the physiological data of one or more listeners may be or may include data related to heart rate, blood pressure, body temperature, or any combination thereof. For example, the data acquisition system 102 may include or may be coupled to (e.g., directly coupled or coupled through a cloud-based server) one or more portable health monitoring devices, which are configured to determine or measure the physiological data of one or more listeners. The data acquisition system 102 may be configured to obtain the physiological data of one or more listeners from the one or more portable health monitoring devices via a wired or wireless manner.
[0048] In various embodiments, the perceptual property predictor 106 may be arranged to generate the prediction also based on one or more masking gain inputs.For example, the one or more masking gain inputs may be one or more digital gain levels or one or more masking gain waveforms.
[0049] In various embodiments, the one or more predefined perceptual attribute indicators may include pleasantness, liveliness, calmness, perceived loudness, sound quality, sharpness, roughness, or any combination thereof. In various embodiments, the one or more predefined perceptual attribute indicators may be selected from the emotional quality attributes defined in ISO 12913-3.
[0050] In various embodiments, the generated prediction may or may not be a single numeric value. In various embodiments, the generated prediction may be non-deterministic. In various other embodiments, the generated prediction may be deterministic. In various embodiments, the prediction may be a probability distribution, a random value extracted from a probability distribution, a vector or multivariate representation of one or more predefined perceptual attributes. In various embodiments, the generated prediction may be a predicted attribute distribution.
[0051] In various embodiments, the perceptual attribute predictor 106 may be or may include a probabilistic perceptual attribute predictor. In various embodiments, the perceptual attribute predictor 106 may be configured to generate a prediction by mixing, combining or adding the ambient soundscape data and each masking setting. In various embodiments, the ambient soundscape data and each masking setting may be added, mixed or combined, for example, in an inference engine, to form enhanced soundscape data before inputting the enhanced soundscape data into the perceptual attribute predictor 106. The masking waveform of each masking setting may be weighted (i.e., multiplied) with a masking gain input, such as a masking gain waveform, to generate a weighted masking waveform. The weighted masking waveform may be combined with (i.e., added to) the ambient soundscape waveform of the ambient soundscape data to generate an enhanced soundscape waveform. The prediction block of the perceptual attribute predictor 106 may receive the enhanced soundscape waveform to generate the prediction.
[0052] In various embodiments, the perceptual property predictor 106 may include a soundscape feature extractor configured to extract features of the ambient soundscape data. The perceptual property predictor 106 may include a masking feature extractor configured to extract features of each masking setting. The perceptual property predictor 106 may include a feature level enhancer configured to generate one or more enhanced soundscape features based on the extracted features of the ambient soundscape data, the extracted features of each masking setting, and a masking gain input such as a digital gain level. The perceptual property predictor 106 may include a prediction block configured to generate a prediction based on one or more enhanced soundscape features, the extracted features of the ambient soundscape data, and the extracted features of each masking setting. The perceptual property predictor 106 may be configured to generate a prediction based on the extracted features of the ambient soundscape data and the extracted features of each masking setting.
[0053] In various embodiments, the perceptual attribute predictor 106 may include an objective feature block configured to extract objective features from the ambient soundscape data and each masking setting. The perceptual attribute predictor 106 may be further configured to extract objective features based also on the masking gain input. The perceptual attribute predictor 106 may include a subjective feature block configured to extract subjective features from group statistics or contextual data. The subjective feature block may also be configured to process the extracted subjective features and the extracted objective features to generate predictions. The perceptual attribute predictor 106 may be configured to generate predictions based on the extracted objective features and the extracted subjective features.
[0054] In various embodiments, the perceptual property predictor 106 may include a linear regression model arranged to generate predictions based on acoustic or psychoacoustic parameters calculated based on the ambient soundscape data and each masking setting.
[0055] In various embodiments, the perceptual property predictor 106 may include one or more deep neural networks configured to generate predictions based on the original audio, a spectrogram rendering, or a combination of the original audio and a spectrogram rendering computed based on the ambient soundscape data and each masking setting.
[0056] In various embodiments, the perceptual attribute predictor 106 may include a composite network as described herein or any combination of networks and / or models as described herein.
[0057] In various embodiments, the ambient soundscape data may be directly received, recorded, measured or sensed by the data acquisition system 102, or may be received from an external source. In various embodiments, the data acquisition system 102 may be configured to provide or generate the ambient soundscape data based on input from a recording device, a receiving device and / or a storage device. The data acquisition system 102 may be configured to provide or generate the ambient soundscape data based on input from one or more microphones, one or more antennas, a storage medium or any combination thereof. In various embodiments, the data acquisition system 102 may be configured to provide or generate the ambient soundscape data by directly recording or sensing the ambient soundscape of the environment.
[0058] In various embodiments, the plurality of masking settings may include one or more tracks of recorded or synthesized sounds, one or more tracks of silence, or one or more tracks derived from the one or more tracks of recorded or synthesized sounds and the one or more tracks of silence.
[0059] In various embodiments, the playback system 110 may include one or more speakers, one or more virtual reality or augmented reality headsets, one or more headphones, one or more earbuds, or any combination thereof.
[0060] In various embodiments, one or more computing, processing or electronic devices or systems may be used to implement various components of the soundscape enhancement system. In one example, the data acquisition system 102, the database 104, the perceptual attribute predictor 106, the masking setting ranking system 108, and the playback system 110 may be implemented in a single computing or processing device or system including a sound receiver and a speaker, such as a physical server or a cloud-based server, a personal computer system, or a mobile device. In another example, the data acquisition system 102 may be implemented by a mobile device (with a sound receiver), while the database 104, the perceptual attribute predictor 106, the masking setting ranking system 108, and the playback system 110 may be implemented by a personal computer system having a speaker that directly or indirectly communicates wirelessly (e.g., WiFi or Bluetooth) with the mobile device. In another example, the data acquisition system 102 may be implemented with a laptop computer remotely connected to the sound receiver and the environmental sensor. The database 104, the perceptual attribute predictor 106, and the masking setting ranking system 108 may be implemented using another computing device that communicates with the laptop computer via a server. The playback system 110 may be a headset that communicates wirelessly with the computer device.
[0061] Figure 2 A schematic diagram of a method of forming a soundscape enhancement system according to various embodiments is shown. The method may include providing a data acquisition system at step 202, the data acquisition system being configured to provide ambient soundscape data. The method may also include providing a database including a plurality of masking settings at step 204. The method may further include coupling a perceptual attribute predictor to the data acquisition system and the database at step 206, the perceptual attribute predictor being configured to generate, for each of the plurality of masking settings, a prediction representing perception on one or more predefined perceptual attribute indicators based on the ambient soundscape data. The method may additionally include providing a masking setting ranking system at step 208, the masking setting ranking system being configured to determine one or more optimal masking settings based on the predictions generated by the perceptual attribute predictor. The method may further include providing a playback system at step 210, the playback system being configured to play or reproduce the one or more optimal masking settings.
[0062] In other words, the method may include coupling a perceptual property predictor to a data acquisition system and a database containing a plurality of masking settings. The method may also include coupling a masking setting ranking system to the perceptual property predictor, and coupling a playback system to the masking setting ranking system.
[0063] For the avoidance of doubt, Figure 2 The order of the various steps is not intended to be limited. For example, step 202 may occur before, after, or simultaneously with step 204.
[0064] In various embodiments, the data acquisition system may also be configured to provide group demographic or contextual data. The perceptual attribute predictor may also be configured to generate predictions based on the group demographic or contextual data.
[0065] In various embodiments, demographic or contextual data may include data related to environmental parameters, demographic data of one or more listeners, psychological data of one or more listeners, physiological data of one or more listeners, or any combination thereof.
[0066] In various embodiments, the data related to environmental parameters may include data related to location, data related to the visual environment, the number of people nearby, air temperature, humidity, wind speed, or any combination thereof.
[0067] In various embodiments, demographic data of one or more listeners may include data related to age, gender, occupation, or any combination thereof.
[0068] In various embodiments, the psychological data of the one or more listeners may include data related to noise sensitivity, perceived stress, a happiness index score, or any combination thereof.
[0069] In various embodiments, the physiological data of the one or more listeners may include data related to heart rate, blood pressure, body temperature, or any combination thereof.
[0070] In various embodiments, the perceptual property predictor may be arranged to generate a prediction based on one or more masking gain inputs.
[0071] In various embodiments, the one or more predefined perceptual attribute indicators may include pleasantness, liveliness, eventfulness, calmness, perceived loudness, sound quality, sharpness, roughness, or any combination thereof.
[0072] In various embodiments, the perceptual property predictor may be arranged to generate a prediction by combining the ambient soundscape data with each masking setting.
[0073] In various embodiments, the perceptual property predictor may include a soundscape feature extractor configured to extract features of the ambient soundscape data. The perceptual property predictor may include a masking feature extractor configured to extract features of each masking setting. The perceptual property predictor may be configured to generate a prediction based on the extracted features of the ambient soundscape data and the extracted features of each masking setting.
[0074] In various embodiments, the perceptual property predictor may include an objective feature block configured to extract objective features from the ambient soundscape data and each masking setting. The perceptual property predictor may include a subjective feature block configured to extract subjective features from population statistics or contextual data. The perceptual property predictor may be configured to generate predictions based on the extracted objective features and the extracted subjective features.
[0075] In various embodiments, the perceptual property predictor may comprise a linear regression model arranged to generate predictions based on acoustic or psychoacoustic parameters calculated based on the ambient soundscape data and each masking setting.
[0076] In various embodiments, the perceptual property predictor may include one or more deep neural networks configured to generate predictions based on the original audio, a spectrogram rendering, or a combination of the original audio and the spectrogram rendering, the spectrogram rendering being calculated based on the ambient soundscape data and each masking setting.
[0077] In various embodiments, the data acquisition system may be configured to provide or generate ambient soundscape data based on input from one or more microphones, one or more antennas, a storage medium, or any combination thereof.
[0078] In various embodiments, the plurality of masking settings may include one or more tracks of recorded or synthesized sounds, one or more tracks of silence, or one or more tracks derived from the one or more tracks of recorded or synthesized sounds and the one or more tracks of silence.
[0079] In various embodiments, the playback system may include one or more speakers, one or more virtual reality or augmented reality headsets, one or more headphones, one or more earbuds, or any combination thereof.
[0080] In various embodiments, the generated predictions may be non-deterministic.
[0081] Figure 3Schematic diagram of a soundscape enhancement system according to various embodiments is shown. The soundscape enhancement system may also be referred to as an automatic masker selection system (AMSS). The soundscape enhancement system may include a data acquisition system 302, which is responsible for providing ambient soundscape data 312, i.e., audio of the acoustic environment and / or parameters calculated directly from the audio of the acoustic environment. To obtain the ambient soundscape data 312 from the acoustic environment, the data acquisition system 302 may use, for example, real-time recording of one or more microphones, digital audio signals received from wireless antennas, data stored on an attached storage medium, or any combination thereof.
[0082] In addition, the soundscape enhancement system may include a database including a plurality of masking settings 304 (alternatively referred to as a candidate masking setting group). The term "masking setting" may refer to any audio track that can be added to the acoustic environment in 312 by any form of playback, combined with other parameters or effects required for the intended playback of the audio track. The masking setting may alternatively be referred to as a "masker". The parameters or effects may include, for example, the application of digital gain levels, spatialization, dynamic envelopes, and / or filtering. Examples of masking settings may include, but are not limited to, audio tracks consisting of recorded or synthesized sounds played at different volumes, audio tracks consisting only of silence, and / or audio tracks derived from a function of one or more other audio tracks in 304, the playback of which does not constitute any sound addition to the acoustic environment in 312. The masking setting may alternatively or additionally refer to the process of selecting a masker, such as the process of selecting a masker from the database 304 of audio tracks. The masking setting strategy can be autonomous and can adapt to the dynamics of the current soundscape (e.g. amplitude, frequency content, context) (e.g. by monitoring the microphone or through digitized input).
[0083] The data acquisition system 302 may also optionally be responsible for collecting group statistics or contextual data 314 as auxiliary parameters of the acoustic environment itself, and for providing group statistics or contextual data 314 . The group demographic or contextual data 314 may include, but is not limited to, (1) non-acoustic environmental parameters, such as location (collected by a GPS sensor), number of people nearby (collected by a camera), air temperature (collected by a thermometer), humidity (collected by a hygrometer) and / or wind speed (collected by an anemometer); (2) group demographic data of real or hypothetical listeners in the acoustic environment (e.g., collected in a digital form attached to the system or as preloaded data on a storage device), such as age, gender and / or occupation; (3) psychological data of real or hypothetical listeners in the acoustic environment (collected by, for example, methods similar to group demographic methods), such as noise sensitivity (obtained using, for example, the Weinstein Noise Sensitivity Scale), perceived stress (obtained using, for example, the Cohen Perceived Stress Scale), and / or happiness index scores (obtained using, for example, the WHO-5 Happiness Index); and / or physiological data of real or hypothetical listeners in the acoustic environment (obtained by, for example, a portable health monitoring device), such as heart rate, blood pressure and / or body temperature.
[0084] The data acquisition system 302 may provide the ambient soundscape data 312 and optional group statistics or context data 314 to the perceptual attribute predictor 306. Given the ambient soundscape data 312, the masking settings of the database 304, and the optional group statistics or context data 314 as inputs, the perceptual attribute predictor 306 may output, for each masking setting in the database 304, a prediction on one or more predefined perceptual attribute metrics (predictions for listeners corresponding to the data in 314) as the playback of each masking setting would be achieved in the acoustic environment. For example, the one or more predefined perceptual attribute metrics may be derived from the ISO 12913-3 circular octave metrics for each masking setting in the database 304. For example, the one or more predefined perceptual attribute metrics may be based on pleasantness, eventfulness, sound quality, perceived loudness, sharpness, roughness, etc. for each masking setting in the database 304. Figure 4 A two-dimensional circumferential octant model of the emotional quality attributes presented in ISO 12913-3 is shown, on which the prediction of the perceptual attribute predictor 306 according to various embodiments may be based.
[0085] exist Figure 3, the set of predictions is represented by 316. The prediction 316 may or may not be a single numeric value. Similarly, in various embodiments, the generated prediction 316 may be non-deterministic, while in various other embodiments, the generated prediction 316 may be deterministic. The term "deterministic" may mean that whenever the same input of ambient soundscape data 312, multiple masking settings of the database 304, and optional population statistics or contextual data 314 is provided to the perceptual attribute predictor 306, the same output will be produced. A probability distribution (or a random value drawn from a probability distribution), a vector, or any other multivariate representation of a perceptual attribute may be a possible output as a prediction 316.
[0086] The perceptual attribute predictor 306 may be or may include any model that takes the same inputs from the ambient soundscape data 312, the plurality of masking settings from the database 304, and optionally the group statistics or context data 314 to output a prediction 316. The model may internally transform the ambient soundscape data 312, the plurality of masking settings from the database 304, and the group statistics or context data 314 before delivering the prediction 316 as an output. Examples of such models may include, but are not limited to: (1) a linear regression model that takes acoustic or psychoacoustic parameters computed from the ambient soundscape data 312, the plurality of masking settings from the database 304, and an aggregated average computed from the group statistics or context data 314 to output a prediction 316; (2) a deep neural network that takes a representation computed from the ambient soundscape data 312, the plurality of masking settings from the database 304, in series with a representation of the group statistics or context data 314, to output a prediction 316; (3) a neural network that takes a representation computed from the ambient soundscape data 312, the plurality of masking settings from the database 304, in series with a representation of the group statistics or context data 314, to output a prediction 316; Figures 5 to 7 The composite network shown; and / or (4) any combination as described herein.
[0087] Figure 5 is a schematic diagram of a perceptual attribute predictor 506 according to various embodiments. In various embodiments, the perceptual attribute predictor 506 may be configured to generate a prediction by combining the ambient soundscape data with each masking setting, for example, in an inference engine with pre-calibrated soundscape-to-masker ratios. The masking waveform 504a of each masking setting may be weighted (i.e., multiplied) with a masking gain input 504b, such as a masking gain waveform, to generate a weighted masking waveform. The weighted masking waveform may be combined with (i.e., added to) the ambient soundscape waveform 512 of the ambient soundscape data to generate an enhanced soundscape waveform 518. The prediction block 520 of the perceptual attribute predictor 506 may receive the enhanced soundscape waveform 518 to generate the prediction 510.
[0088] Various other embodiments may not require mixing, combining or adding the ambient soundscape data and each weighted masking waveform to form the enhanced soundscape waveform before inputting the enhanced soundscape waveform into the perceptual attribute predictor. Without the need to mix, combine or add the ambient soundscape waveform and each weighted masking waveform to form the enhanced soundscape data before inputting the enhanced soundscape waveform into the perceptual attribute predictor, various embodiments may allow for more efficient computation and bandwidth systems in practical deployments.
[0089] Figure 6 6 is a schematic diagram of a perceptual property predictor 606 according to various embodiments. In various embodiments, the perceptual property predictor 606 may include a soundscape feature extractor 622, which is configured to extract features 624 of the ambient soundscape data, i.e., extract features 624 of the ambient soundscape waveform 612 of the ambient soundscape data. The perceptual property predictor 606 may include a masking feature extractor 626, which is configured to extract features 628 of each masking setting, i.e., extract features 628 of the masking waveform 604a of each masking setting. The perceptual property predictor 606 may include a feature level enhancer 630, which is configured to generate one or more enhanced soundscape features 632 based on the extracted features 624 of the ambient soundscape data, the extracted features 628 of each masking setting, and a masking gain input 604b, such as a digital gain level. The perceptual property predictor 606 may include a prediction block 620 arranged to generate a prediction 610 based on one or more enhanced soundscape features 632, extracted features 624 of the ambient soundscape data, and extracted features 628 of each masking setting. In summary, the perceptual property predictor 606 may be arranged to generate a prediction 610 based on the extracted features 624 of the ambient soundscape data and the extracted features 628 of each masking setting.
[0090] Figure 77 is a schematic diagram of a perceptual attribute predictor 706 according to various embodiments. In various embodiments, the perceptual attribute predictor 706 may include an objective feature block 734, which is configured to extract objective features 736 from the ambient soundscape data and each masking setting, that is, to extract objective features 736 from the ambient soundscape waveform 712 of the ambient soundscape data and the masking waveform 704a of each masking setting. The perceptual attribute predictor 706 may be further configured to extract the objective features 736 based on the masking gain input 704b. The perceptual attribute predictor 706 may include a subjective feature block 738, which is configured to extract subjective features from the group statistics or contextual data 714. The subjective feature block 738 may also be configured to process the extracted subjective features and the extracted objective features 736 to generate the prediction 710. The perceptual attribute predictor 706 may be configured to generate the prediction 710 based on the extracted objective features 736 and the extracted subjective features.
[0091] Refer to Figure 3 , the predictions 316 provided by the perceptual attribute predictor 306 may be input into the masking setting ranking system 308 to generate a single optimal masking setting 340. The masking setting ranking system 308 may determine optimality by any suitable metric, which may be, but is not limited to, perceived loudness, sound quality, sharpness, roughness, Figure 4 The maximum or minimum value of one or more of the emotional quality attributes and / or any combination thereof is shown. Alternatively, it is contemplated that the predictions 316 provided by the perceptual attribute predictor 306 can be input into the masking setting ranking system 308 to generate multiple optimal masking settings. For a given metric, there may be multiple optimal masking settings (e.g., if multiple masking settings are tied to the metric determined by the user).
[0092] The optimal masking setting 340 may be used as an input to the playback system 310. The playback system 310 may implement actual playback or reproduction of the optimal masking setting 340 in an acoustic environment. The playback system 310 may include, but is not limited to, one or more speakers, one or more virtual reality or augmented reality headsets, one or more headphones, one or more earbuds, and / or any combination thereof.
[0093] Various embodiments may be used in outdoor urban areas with poor sound quality (e.g., a park next to a highway) to mitigate the perceived harmful effects of noise in the surrounding acoustic environment, or in indoor areas (e.g., inside a nursing home) to improve the acoustic comfort and overall quality of the surrounding acoustic environment. Since the system effectively aims to change the perception of the surrounding acoustic environment, any commercial entity with an indoor or outdoor area whose acoustic environment needs to be maintained under fixed or desired conditions may be interested in the system.
[0094] Various embodiments can select and add maskers to both real and imaginary ambient soundscapes. Various embodiments can also add maskers to create abstract "experience zones" if arbitrary audio is input into the data acquisition system instead of real-time recording from a microphone. This will be of interest to any commercial entity interested in making or modifying soundscapes in virtual reality, mixed reality, or augmented reality based on listener perception.
[0095] Various embodiments may involve a system that selects and places audio tracks (i.e., maskers) to be introduced into an existing soundscape (e.g., played through speakers or digitally added through headphones, for example) and into a mixed / enhanced soundscape based on optimizing predicted perceptual responses (e.g., perceived loudness, ISO12913-3 circular eighths - pleasantness, annoyance, tranquility, calmness, activity, eventfulness, etc.).
[0096] Various embodiments may select a masking setting or masker to add to an ambient acoustic environment based on a prediction of human perception by a perceptual attribute predictor, which outputs a value or distribution of a perceptual attribute indicator based on ambient soundscape data (real or hypothetical ambient soundscape data), data of candidate masking settings, and optionally group statistics or contextual data of listeners of the ambient soundscape. This reflects the ISO12913 definition of "soundscape", which considers the acoustic environment as perceived by one or more people in a context. In contrast, existing methods may select a masking setting or masker based on objective acoustic parameters of the ambient acoustic environment (e.g., spectrum, sound pressure level, sound source type and direction), and may not consider the subjective impact of human perception on masking selection or masking setting selection.
[0097] In various embodiments, the perceptual property predictor can output perceptual predictions in near real time (less than 10 seconds of latency between input and prediction), and can therefore automatically make suggestions in real time for optimal masking settings that change over time for acoustic environments that change over time. In other words, various embodiments can autonomously change the masker (i.e., track) and / or settings (e.g., gain level, spatialization) being played as the acoustic environment changes over time. In contrast, existing systems only allow for static masking settings (with only potential for dynamic volume / filter effects to be applied) or user-controlled masking settings (i.e., one must manually change the settings based on one's own personal / expert opinion of optimality).
[0098] In various embodiments, the user can set the system to automatically select the best masker from a number of selection schemes. The selection scheme can be arbitrary, but various embodiments can allow the following selection schemes that are not available in conventional solutions: (1) Probabilistic top-k: rank the predicted perceptual response values of the combined masker and the existing soundscape among different maskers, and randomly select a masking setting from the k masking settings given the top (or bottom) k ranked values; (2) Probabilistic distribution: draw the prediction of the perceptual response value of the combined masker and the existing soundscape as a random sample from a probability distribution generated by the system.
[0099] Existing systems and masking selection strategies may not allow for the selection of the best masker through a dynamic strategy and may have one of the following characteristics: (1) manual selection: the system user or administrator selects the best masker based on his or her own opinion; (2) deterministic: the best masker is chosen from different maskers as a deterministic, predefined function of the combined masker and the perceptual response value of the existing soundscape.
[0100] Various embodiments may include one or more audio transducers (e.g., speakers, headphones) and one or more sensors (e.g., microphones, environmental sensors), the one or more audio transducers and the one or more sensors being coupled to one or more computing devices. Various embodiments may have advantages not available in existing systems / solutions.
[0101] For example, various embodiments may employ non-energy masking. Energy masking exploits the physical limitations of peripheral human auditory processes. The amplitude and frequency bandwidth of the masking sound are adjusted so that the target noise is completely or partially inaudible at the inner ear. Therefore, objective indicators such as sound pressure level or psychoacoustic parameters are usually adopted to optimize or select energy maskers. Various embodiments may refer to (but not limited to) the concept of soundscape defined in ISO12913-1. In the literature, soundscape enhancement using additional sounds is also often referred to as masking. Masking techniques may include energy masking and information masking (or perceptual masking), where information masking consists of an impact on the perceptibility of the sound, which is related to the audio in the higher brain centres. Therefore, the masker may be determined by perceptual factors (e.g., perceptual emotional attributes, perceived annoyance in ISO12913-2), and is not based solely on objective indicators in existing systems / solutions.
[0102] The advantage of non-energetic masking can be demonstrated in the literature on masking of traffic noise using natural sounds, where the reduction in perceived loudness of energetic masking was nullified as the target noise level increased. This is in contrast to the statistically significant reduction in perceived loudness and improvement in soundscape quality at higher levels of traffic noise when the masker providing informative masking was presented at levels up to 6 dB lower. In addition, similar significant reductions in perceived loudness and improvements in soundscape quality were observed with the use of birdsong as a masker, where the use of birdsong as a masker was unable to energetically mask traffic noise, further demonstrating the effectiveness of non-energetic masking.
[0103] The ability to contextualize audio inputs through auxiliary inputs can allow various embodiments to select maskers for arbitrary target soundscapes (instead of being limited to office environments as described in the prior art). Maskers according to various embodiments may also not be limited to tracks generated from random noise, nor to the sounds of natural environments as described in the prior art. As long as the desired perceptual properties are optimized (optimization may mean "maximizing" or "minimizing"), the number of maskers that can be added may be unlimited.
[0104] In various embodiments, the audio input data may be in any format and may be real-time data from a physical microphone or an audio track accessed from a storage medium. Similarly, maskers according to various embodiments may be streamed to speakers (or headphones) or digitally mixed with a target soundscape from a storage medium and played as a single combined audio track in a virtual environment. Thus, various embodiments are capable of masking soundscapes in real-world environments as well as in virtual environments (e.g., in a metaverse application).
[0105] Dynamic masking described in the prior art is based on objective indicators, such as sound level, audio spectrum, or is achieved by proximity to an identified target noise source. Various embodiments may dynamically or adaptively adjust masking settings based on predicted perceptual attributes. Masking settings may not be limited to sound level or spectrum adjustments.
[0106] Example Study 1
[0107] introduce
[0108] It is known that indications based solely on the sound pressure level (SPL) of a given acoustic environment are often inadequate to reflect the level of annoyance and the impact on quality of life caused by excessive noise. This has given rise to the soundscape approach to noise control, which focuses on interventions that improve the perceived properties of noise, instead of simply reducing the SPL. The ISO 12913 series of international standards on soundscapes aims to formalize this approach by providing a circular model of pleasantness and eventfulness, against which interventions can be compared based on a subjective assessment of the surrounding acoustic environment.
[0109] Therefore, many studies have used soundscape enhancement techniques to modify the perception of soundscapes by adding maskers to urban or indoor acoustic environments, thereby optimizing metrics such as subjectively rated pleasantness or calmness. However, the selection of maskers is often arbitrary, expert-guided, or based on post hoc analysis. Arbitrary selection of maskers may be unreliable in achieving the desired perceptual change, expert-guided selection is laborious and time-consuming, and post hoc analysis may not generalize to unseen maskers and soundscapes in unobserved contexts.
[0110] An approach to overcome these limitations is to train a prediction model to predict the value of some perceptual attribute or attributes as subjectively assessed by a person given the original auditory stimulus, based on acoustically diverse choices of soundscapes and maskers. Once trained, simulated additions of maskers to unseen soundscapes can be fed as input to the model to obtain a prediction of the perceptual attribute. The masker that produces the most boost or the most degradation of the attribute can then be selected as the best masker for real-time enhancement.
[0111] Various embodiments may relate to a neural approach for an automatic masker selection system (AMSS) to enhance urban soundscapes by optimizing the pleasantness of the acoustic environment. By using a probabilistic output scheme, the model can predict a pleasantness distribution for each soundscape and masker combination instead of a single determined value, which allows explicit access to the confidence of the prediction as well as the predicted pleasantness.
[0112] Related Work
[0113] In the study of soundscapes, research on developing predictive models for perceptual attributes has focused primarily on simpler machine learning models such as linear regression, support vector machines (SVM), and shallow multilayer perceptrons. The machine learning models use input features based on acoustic measurements, psychoacoustic parameters, environmental characteristics, or some linear combination of machine learning models elucidated by principal component analysis. Scaling metrics for Likert scales have also been proposed to account for nonlinearities in the scales used to develop these systems.
[0114] On the other hand, a recent systematic review showed that, despite the prevalence of deep neural networks in more “objective” tasks such as sound event localization, detection, and classification, there is little research on using deep neural networks to predict the values of perceptual properties of soundscapes. In this regard, the most important research seems to be the one comparing the performance of SVM, convolutional neural network (CNN), long short-term memory network (LSTM), and fine-tuned VGGish network on the Emo-Soundscapes dataset. The Emo-Soundscapes dataset contains 1213 audio clips ranked by valence (pleasantness) and arousal (eventfulness), according to subjective paired comparisons performed with a Self-Assessment Manikin. The fine-tuned VGGish model and the CNN trained from scratch were shown to have the smallest mean squared errors (MSE) in predicting the valence and arousal of the audio clips, respectively. A more recent study used a similar model to classify music into one of four quadrants using a dataset of about a thousand audio clips, using a circular model similar to ISO12913, and classified music into the categories of neutral, calm, happy, sad, angry, and fearful. However, soundscapes often contain more than just music, so it remains to be investigated whether their results can be generalized to a wider range of acoustic stimuli.
[0115] Furthermore, these existing models in the literature are deterministic models, i.e., the same acoustic environment always maps to the same predicted value without taking into account the uncertainty inherent in the subjective ratings used as the baseline true value. However, these ratings given by individuals are inherently random given factors that cannot be reasonably controlled, such as the person's current stress level or noise sensitivity. Therefore, various embodiments may involve a probabilistic approach in which a neural network is trained to predict a distribution of possible ratings for the output, and then draw from this distribution, which allows the model to account for different degrees of uncertainty in each unique soundscape.
[0116] Proposed method
[0117] Figure 8 A schematic diagram of training and inference of an automatic masker selection system (AMSS) according to various embodiments is shown.
[0118] Consider the soundscape and the set of K candidate maskers that can be added to the soundscape {M k}.by Represents a masker M k In other words, Yes Add The soundscape after , which takes into account the response of the playback system and the real-world environment. Finding the optimal To maximize the evaluation of a given perceptual (or emotional attribute) f of the enhanced soundscape, it is equivalent to finding , where f can be, for example, a score on a numerical metric for pleasantness, eventfulness, or other perceptual attributes. We can then use Enhancement , to obtain .
[0119] For example, a naive implementation of AMSS could use Some approximation of Output the predicted value of the perceptual attribute , and select the one with the highest However, as mentioned above, this deterministic output scheme ignores the inherent uncertainty associated with subjective ratings on perceptual attribute indicators. For example, one or a type of soundscape may be very pleasant to some people, but very disturbing to others, while another soundscape may be almost universally slightly pleasant to any listener, depending on the context.
[0120] To collect this uncertainty, can be considered as having a "random" soundscape with some The joint distribution of A random variable where the observed enhanced soundscape Considered as Then, uncertainty will be In this case, the prediction model will try to transform the conditional distribution Modeling for enhanced soundscape A function or some function For brevity, Abbreviated as .
[0121] The neural approximator 806 can be used to output The neural approximator 806 may be referred to as a probabilistic perceptual attribute predictor (PPAP). Figure 8 In the proposed AMSS shown, each masker can be passed to the aforementioned PPAP 806, which outputs one or more parameters of a certain predefined distribution family. PPAP 806 can be trained to output a normal distribution The average and log standard deviation ,average value and log standard deviation Used to output Once sufficient (or all) predictive distributions have been calculated, , the "best" masker can be selected based on some predetermined criteria. For example, Select the masker from The sampling is similar to or similar to A set of determined values to motivate the masker to explore, or one can use considerations more complex standards.
[0122] Although only Observations of are available, and The true distribution of is unknown, but it is still possible to optimize the model given the output distribution by maximizing the log probability of the ground truth in a manner inspired by Bayesian optimization. , a collection of maskers and the benchmark true value In the case of , the contribution to the loss function of the soundscape can be given by:
[0123] (1)
[0124] ,(2)
[0125] in is the output distribution The logarithmic density function of , where the additive constant is omitted. During training, batches of soundscape-masker pairs with available ground truth values can be passed to optimize the model.
[0126] As shown in formula (2), the loss function using normal distribution can be considered as the loss of weighted MSE regularized by the logarithmic standard deviation. is inherently stable because the first encouragement is larger , meanwhile, the second incentive is smaller Of course, some other choices of the output distribution can be similarly reduced to other deviation measures, such as the regularized Laplace distribution of the mean absolute error. Note that it is also possible to k is set to a predetermined constant and uses the reference (ground) and The model is trained with a pure MSE loss between 0 and 1 to force the model to learn deterministically. For the purpose of ablation study, the following formula can be used
[0127] (3)
[0128] As the deterministic counterpart of equation (2), the deterministic loss in equation (3) can be considered as a static =1 in the calculation formula (2).
[0129] Verification experiment
[0130] To verify the proposed system, we set The normalized pleasure measure defined in the ISO12913-3 standard, It can also be called ISO pleasure. Specifically,
[0131] (4)
[0132] in The participants' opinions on enhancing the soundscape were expressed on a 5-point Likert scale. The degree of pleasure, annoyance, calm, confusion, activity, and monotony. For each , which can predict the distribution , where the “ground truth” label It is given by the participants. As an ablation study, the model trained using the proposed method is compared with the Direct prediction The validation experiments are conducted on a dataset of subjective responses to various enhanced soundscapes.
[0133] Dataset
[0134] The dataset contains 12,600 unique sets of subjective responses to enhanced soundscapes in a 5-fold cross-validation set, and 48 additional sets of responses in an independent test set, for a total of 12,648 samples. Each sample maps an enhanced soundscape (as a raw audio recording) to an ISO pleasantness value given a set of responses to the following 6-item subset of the ISO12913-2 emotional response questionnaire: "To what extent do you agree or disagree that the current ambient sound environment is {pleasant, chaotic, active, calm, disturbing, monotonous}?"
[0135] Participants answered on a 5-point scale with multiple labels "strongly disagree", "disagree", "neither agree nor disagree", "agree" and "strongly agree", which were coded as 1, 2, 3, 4 and 5 respectively. The coded answers were then used to calculate the ground truth label of ISO pleasantness for each enhanced soundscape.
[0136] Enhanced Soundscape
[0137] The enhanced soundscapes in the 5-fold cross-validation set were created by adding 30-second excerpts of Freesound and xeno-canto recordings as "maskers" to 30-second excerpts of binaural recordings of soundscapes from the Urban Soundscapes of the World (USotW), a comprehensive dataset of urban soundscapes. Unenhanced soundscapes were also included as controls. All recordings used were sampled at 44.1kHz.
[0138] Each fold has a set of 56 maskers divided into the following categories: birds (16), construction (8), traffic (8), water (16), and wind (8). These categories were chosen to cover the range of sound types that were generally assessed as pleasant and disturbing. The maskers and soundscapes in each fold were disjoint.
[0139] The augmented soundscapes in the test set were made in a similar way, with 7 maskers (8 maskers in total including a control masker that was not augmented) independent of the maskers in the cross-validation set, and 6 stereo recordings of the soundscapes independent of the USotW dataset, recorded using the same Soundscape Indices Protocol. The 7 maskers were extracted from xeno-canto track IDs 640568 (bird "B1") and 568124 (bird "B2"), and Freesound track IDs 586168 (construction "Co"), 587219 (traffic "Tr"), 587000 (water "W1"), 587759 (water "W2"), and 587205 (wind "Wi"). The maskers were added exhaustively to all soundscapes in the test set, which resulted in 48 samples in the test set. All soundscapes were calibrated to the real-time measured A-weighted equivalent SPL ( ) according to the method described by K. Ooi et al. (“Automation of stereo headphone audio calibration on an artificial head”, MethodsX, vol. 8, issue 2, p. 101288, 2021) before adding maskers at a constant soundscape-to-masker ratio of 0 dB for the test set and at a value randomly selected in dB from {-6, -3, 0, 3, 6} for the cross-validation set. .
[0140] Subjective reaction
[0141] To obtain subjective responses to the enhanced soundscapes, 300 participants were recruited to rate 42 unique enhanced soundscapes randomly selected from one compromise of the validation set. Another five participants were recruited to rate 48 enhanced soundscapes in the test set. Thus, each enhanced soundscape in the validation set was rated by one participant and in the test set by five participants. All participants used a pair of over-ear headphones (Beyerdynamic Custom One Pro) driven by an external sound card (Creative Sound Blaster E5) to listen to the calibrated enhanced soundscapes. After listening to each enhanced soundscape, they answered the 6-item questionnaire described above (under the “Dataset” heading), and ISO pleasantness was calculated from their responses according to Equation 4.
[0142] Model and training
[0143] Log-mel spectrograms of the enhanced soundscape are used as input. Log-mel spectrograms are extracted using a 4096-sample Hann window with 50% overlap and compressed to 64 mel bins. Fig. 9 (a) A basic convolutional recurrent neural network (CRNN) architecture used according to various embodiments is shown; (b) a possible implementation of the feature mapping block in (a) according to various embodiments; and (c) other possible implementations of the feature mapping block in (a) according to various other embodiments. Four different feature mapping blocks are studied, namely a vanilla mapping block using a bidirectional gated recurrent unit (BiGRU), an additive attention block, a dot-product attention block, and a multi-head attention block with 4 heads. The base mapping block is shown in FIG. Fig. 9 (b), while all attention-based blocks share the same general workflow as Fig. 9 (c) as shown. Fig. 9 The last few layers of the model in (a) are fully connected layers, which ultimately output μ k and In the deterministic ablation model, is ignored.
[0144] All models were trained using a 5-fold cross-validation scheme, with 10 models in each fold, for a total of 50 models. The same 10 seeds were used for multiple models in each fold. All models were trained using the Adam optimizer with a 5×10 −5 The training was performed for up to 100 epochs with a learning rate of . For each model, the model weight with the best validation loss was used for evaluation in both the cross validation set and the test set.
[0145] Results and Discussion
[0146] Fig.10is a graph showing the mean fold mean squared errors (MSEs) (± standard deviation) of the probability-aware attribute predictor (PPAP) according to various embodiments in 10 runs tested for each setting (top: cross-validation set, bottom: test set). Each run may include five models, each on a different fold of the cross-validation set, but with the same initial conditions. For all models, the contribution to the MSE is calculated as Asterisks (*) indicate statistically significant improvements (p<0.05). Fig.10 The results of the validation experiments described above are summarized on both the 5-fold cross validation set and the independent test set. For reference, the results of a trivial deterministic "label mean model" are also provided. In this "label mean model", the label mean in the training set is used as the prediction for all stimuli in the validation and test sets. All other studied models perform better than the label mean model, thus indicating that the feature extraction performed by the other studied models is beneficial. This may be because the test set is smaller and less diverse (48 samples, 7 maskers) compared to the validation set (12,600 samples, 280 maskers).
[0147] Furthermore, for the four architectures tested, models with probabilistic outputs outperformed equivalent models with deterministic outputs on both the validation and test sets. Fig.10 The percentage of MSE reduction for each model is also shown. It can be observed that CRNN with multi-head attention block and base CRNN present the largest improvements of 1.1% and 7.8% in MSE on validation set and test set, respectively.
[0148] To quantify the significance of these reductions, two-sided Wilcoxon signed-rank tests were performed between the deterministic and probabilistic models on both the validation and test set MSEs. Fig.10 In the , significant reductions at the 5% significance level are marked with asterisks. The reductions in the test set MSE for both the base CRNN and PPAP with additive attention are statistically significant and provide evidence to support the generalizability of the proposed method to unseen data. This can be attributed to the inherent randomness of the test set participants’ contributions being different from those in the training set, and PPAP generalizing better to this randomness. However, further ablation studies are needed to verify this hypothesis.
[0149] However, the lowest improvement in test set MSE is actually observed in PPAP with multi-head attention block. This can be attributed to the fact that PPAP with multi-head attention block has about 50% more parameters than the other three models (about 120K vs. about 80K), which may cause the trained models (both deterministic and probabilistic) to overfit to the relatively small dataset of 12,600 training and validation samples.
[0150] Fig.11 50 models (dot product variants) on the test set are shown for mask selection, using the naive maximum Selection scheme (left) and random sampling scheme according to various embodiments to stimulate masker exploration (right). The selection criteria for each base soundscape are shown on the top of each sub-figure. The cell color represents the average ISO pleasantness rated by the test set participants. The circle size represents the number of models that selected the masker. The masker description has been provided previously (under the heading "Enhanced Soundscape").
[0151] In the naive selection scenario, most models selected the first birdsong masker (B1) across all base soundscapes. This is consistent with the consensus in previous soundscape studies, which generally observed higher pleasantness responses to birdsong maskers. However, the second birdsong masker (B2) may have received lower ratings from participants overall, and accordingly, the model voted lower because it was perceived as less pleasant in the context of the test set soundscapes, which has been observed in previous studies.
[0152] In the random selection scenario, the system still tends to choose B1, which provides good pleasantness scores for most soundscapes, but now the system explores other maskers much more than the naive scenario. This can be beneficial in real-time systems, where human feedback can be obtained in real time to adaptively adjust masking selection or improve future models. It can also be seen that in the case of soundscape 5, the pleasantness of the unenhanced soundscape is already relatively high, and the exploration rate is higher than other basic soundscapes, which allows for a more diverse acoustic experience with only a small compromise in pleasantness level.
[0153] in conclusion
[0154] Various embodiments may be directed to an automatic masking selection system (AMSS) for human-centric urban soundscape enhancement that uses a probabilistic perceptual attribute predictor (PPAP). The proposed PPAP is implemented using a convolutional recurrent neural network and is trained to output predictions in a probabilistic manner. This allows the system to predict perceptual attributes of soundscapes while accounting for the stochasticity inherent in human subjective perception of acoustic stimuli. Through a large-scale listening test with more than 300 participants and more than 12,000 unique soundscapes, we validated the effectiveness of our PPAP in predicting pleasantness of enhanced soundscapes, including soundscapes generated from unseen soundscapes and maskers. Future work on AMSS may include real-time implementation to evaluate its ecological validity, as well as studies of the proposed method on other perceptual attributes (f), such as eventfulness or calmness, since the proposed method is not specific to any particular attribute. In fact, since the main assumption underlying the PPAP is that of random ground truth labels, one can also envision its application to any situation where prediction of subjective ratings is required.
[0155] Soundscape enhancement, which involves adding sounds called "maskers" to a given soundscape, is a human-centered urban noise mitigation measure that aims to improve the overall soundscape quality. However, the selection of maskers is usually predicted based on a cumbersome process and is inflexible to the time-varying nature of real-world soundscapes. Due to the perceptual uniqueness of each soundscape and the inherent subjectivity of human perception, this application proposes a probabilistic perceptual attribute predictor (PPAP) that predicts randomly distributed parameters as output instead of a single determined value as output. Using PPAP, this application develops an automatic masking selection system (AMSS) that selects the best masker candidate based on the predicted distribution of ISO12913-3 pleasantness ratings for a given soundscape. Through large-scale listening tests with 300 participants, 12,600 subjective responses were collected, each for a unique enhanced soundscape, to train the PPAP model with a 5-fold cross-validation scheme. Using a convolutional recurrent neural network backbone and testing several variations of the attention mechanism for PPAP, an evaluation of the proposed system was performed using a blind test set with 48 unseen augmented soundscapes to assess the effectiveness of the probabilistic output scheme over conventional deterministic systems.
[0156] Example Study 2
[0157] introduce
[0158] The reduction of urban noise pollution is a complex and multifaceted problem, in which the elimination or reduction of noise sources is usually not a practical option. Unlike conventional noise control methods that aim to reduce sound pressure levels or sound energy, the soundscape approach proposes a more comprehensive, people-oriented strategy based on perceived acoustic comfort. This technique, commonly referred to as soundscape enhancement, involves adding "desired" sounds to the acoustic environment to "mask" the noise and improve the overall perceived acoustic quality, and this technology has achieved satisfactory results for various types of noise sources in both virtual and real environments.
[0159] However, the selection of "maskers" and playback levels is conventionally arbitrary, expert-guided, or based on post hoc analysis. These masking selection processes are not only typically time-consuming and labor-intensive, but also inflexible to the dynamic characteristics of real-world soundscapes. As a result, these static selections of maskers and playback levels are generally unlikely to always achieve optimal acoustic comfort. To address these limitations, Example Study 1 shows a first attempt at adaptive soundscape enhancement through the use of a masking selection system driven by a probabilistic deep learning model that predicts a probability distribution of "pleasantness" for a particular enhanced soundscape.
[0160] Although the above-mentioned Probabilistic Perceptual Attribute Predictor (PPAP) model provides a compelling proof of concept for deep learning-driven automatic soundscape enhancement, the model may not be designed for real-time deployment. The implementation of the PPAP model requires a pre-mixed track of the enhanced soundscape as input. The mixing process is done at a known real-time SPL level relative to the recorded non-enhanced soundscape track, with a pre-calibrated soundscape-to-masker ratio. This may result in a model that is blind to the masking gain level, which is a critical playback parameter for real-time deployment. In addition, requiring a mix of the base soundscape recording and the masker as input adds unnecessary complexity to the overall system. First, all candidate maskers require a lookup table of digital gains at a specific soundscape-to-masker ratio (SMR) required for playback. Second, during the selection process, each candidate masker-SMR pair requires the enhancement soundscape track to be computed, which increases the computational overhead of the system.
[0161] Various embodiments may present a PPAP model with a modified computational formula that will allow for a more efficient computational and bandwidth system in real-world deployments. The proposed model may decouple the base soundscape, masker, and gain level from each other at the input stage, and instead introduce separate feature extractor branches for the base soundscape and masker. By using a digital gain level of the masker that is independent of the real-time sound pressure level (SPL) as an adjustment input, the "enhancement" process occurs only in the feature space instead of the waveform space. Then, based on the base soundscape features, the masking features, and the gain adjustment features, an attention mechanism is used to "query" the distribution of the target perceptual attributes. The proposed approach results in reduced complexity in multiple components of the system. First, there may be no need to send the original waveform of the soundscape to the inference engine to mix with the masker, because less bandwidth-consuming spectral data can be used as model input. For cloud inference, this means a significant reduction in the data egress rate at the edge. Second, the masking features can be pre-computed independently of the gain level, which reduces the inference time and, therefore, reduces the overall latency. Third, it is now possible to predict properties for multiple gain levels of the same masker faster without having to recalculate the enhanced soundscape or even the soundscape and masking features. Finally, it is now possible to add new maskers to the system without having to create a lookup table.
[0162] Proposed method
[0163] Preparation
[0164] Consider digitizing soundscape recordings , where C ≥ 1 is the number of channels, and is the sample index. Indicates i Real-time A-weighted equivalent sound pressure level in dBA For simplicity, all references to sound pressure level (SPL) refer to 30 seconds. , unless otherwise stated. Assuming that all soundscapes are recorded by the same real-time recording setup, and assuming approximate linearity, the sound pressure-to-digital level ratio (SPDR), i.e. the ratio of the relative sound pressure level on a linear scale to the digital fullscale (DFS) of the setup, can be considered roughly constant and expressed as Indicates that its unit is ,in is the reference sound pressure level. Since all soundscape inputs originate from the same recording device, this also means that real-time SPL information is implicitly embedded in the input spectrogram data.
[0165] Consider a single-channel masker For the maskers, it is assumed that they are not all recorded by the same recording device. In practice, this is because the maskers are usually derived from a different recording system than the live recording device, and are often obtained through open content providers such as Freesound and Xeno-canto. Therefore, the SPDR of each masker is usually unknown and cannot be assumed to be the same as the SPDR of any other masker. Representation Masker Under the assumption of approximate linearity (verified by a thorough gain calibration of all maskers used in this paper), the SPDR of the masker by Set the ratio to make the masker Normalized to the same SPDR as the soundscape recording. and real-time SPL If both are known, then soundscape enhancement in dBA with a specific soundscape to masker ratio (SMR) can be performed relatively easily without further calibration. In practice this is often unclear, so the usual solution is to use a In the case of a playback system with a digital level-to-sound pressure ratio (DSPR), calibration is performed to Among the multiple values of Sound pressure level reproduction Required digital gain .get The calibration required for the exact value of may often require specialized equipment, such as a recording device with an artificial head, and may be time-consuming and / or labor-intensive, depending on the calibration method used. It is used explicitly as a regulatory input, thus removing the need for SPL-dependent SMR or the need for masking calibration.
[0166] Model
[0167] Fig. 12A FIG. 1 is a schematic diagram of a probability-aware attribute predictor (PPAP) according to various embodiments. For Soundscape The logarithmic Mel-spectrogram of It is a masker The logarithmic Mel-spectrogram of is the number of spectrogram time frames, and is the number of mel bins. In this paper, , ,and The corresponding signal is 30 seconds, sampled at 44.1kHz, with a short-time Fourier transform (STFT) window size of 4096 samples and a frame overlap rate of 50%. The soundscape and mask embedding are respectively generated by feature extractor 1222 and feature extractor 1226 Calculate so that
[0168] ,(5)
[0169] (6)
[0170] in is the embedding dimension, and is the number of feature time frames. Each feature extractor 1222, 1226 may include 5 convolution blocks, each of which includes a convolution layer with a 3×3 kernel and a stride of 1; batch normalization; dropout with probability 0.2; Swish activation; and a 2×2 average pooling layer. The convolution layer includes 16, 32, 48, 64, and 64 output channels in order.
[0171] In the case of a query-key-value (QKV) model using attention, the perceptual attribute prediction process can be viewed as using a masker query mapping, where the unenhanced soundscape gives the key and the enhanced soundscape gives the value. Instead of performing soundscape enhancement in the audio or spectral domain, we consider performing "enhancement" in the embedding domain (in the feature enhancement block 1230). This can be done using a simple mapping with masked digital gain adjustment so that
[0172] ,(7)
[0173] in is the query gain level, , is a gain adjustment enhancement layer. There are three different implementations of
[0174] (8)
[0175] (9)
[0176] (10)
[0177] where tensor| is tensor concatenation on the last axis, ‖ is the tensor stacking operator that creates additional axes, is a fully connected layer with D output units, is a convolutional layer with D filters that uses a 1D kernel to compress the stacked dimensions into a single axis. Fig. 12B Three feature enhancement methods according to the calculation formulas (8), (9) and (10) in various embodiments are shown. is a silent track (i.e. no enhancement), we will use Randomization, where and is the mean and standard deviation of the log gain of the training samples with non-silent maskers. This is to teach the model to ignore the gain values of the silent maskers. With the query, key, and value embeddings in place, any QKV-compatible attention block 1220a can be used , compute the output embedding so that
[0178] .(11)
[0179] Then, output block 1220b Calculating the Predictive Distribution , output block 1220b Predicted Distribution Parameters and Three types of attention can be considered, namely additive attention (AA), dot-product attention (DPA), and multi-head attention (MHA). In this example, 4-head MHA is used.
[0180] Loss Function
[0181] As in Example Study 1, human subjective responses can be considered inherently random, since the participants are a sample of the population, and each participant’s response during the listening test is also a random sample of their inherently non-deterministic perceptions. Therefore, the labels used for the target attributes are (i.e. ISO pleasantness) can be considered as a random variable representing the distribution of the target attribute The actual distribution of the target attribute is unknown. When using the maximum likelihood formula, the optimization objective of the PPAP model can be given by the following formula:
[0182] ( )(12)
[0183] in are the parameters of the model, and is the logarithmic density of X estimated at x. The subjective response of each acoustic scene is modeled by an independent normal distribution, and the optimization objective in equation (12) is converted into a loss function
[0184] ,(13)
[0185] in is the number of training rounds.
[0186] Optimizing Inference
[0187] Although the model sees multiple batches of different soundscape-masker-gain samples during training, during real-time inference, the model sees only one base soundscape at a time, while going through multiple masker and gain levels to find the most appropriate masker-gain pair for a particular soundscape. Therefore, by eliminating repeated computations, instead of doing a new forward pass for each masker-gain combination, a significant performance improvement can be achieved. , and Indicates execution , and The time required. The number of maskers to query for each soundscape, and The number of gain values to query for each masker. A naive batching scheme would require In contrast, by optimizing the query process, such as Fig.13 As shown in Algorithm 1, the total running time can be reduced to . Fig.13 An algorithm for reducing the total running time according to various embodiments is shown. Additionally, for a fixed masker set, the masking features can be pre-computed , which allows the total runtime to be further reduced to .
[0188] Dataset
[0189] The dataset used in this paper was extended with additional participants from the dataset used in Example Study 1, totaling 18,564 data points from 442 participants (42 stimuli per participant). Each data point represents a participant’s subjective response to an augmented soundscape.
[0190] A. Stimulation
[0191] Enhanced soundscapes were made at 5 mutually exclusive compromises by adding a 30 second "masker" to a 30 second stereo soundscape recording of a soundscape from the Urban Soundscapes of the World (USotW) database. Unenhanced soundscapes (i.e., soundscapes "enhanced" with a silence masker) were also included as controls. All recordings used were sampled at 44.1 kHz. More details can be found in Example Study 1.
[0192] The addition of maskers to the soundscape was made at specific soundscape to masker ratios (SMRs) of {-6,-3,0,3,6}dBA, where the SMR is defined by the ratio of the SPL of the soundscape to the SPL of the masker. While the real-time SPL of the base soundscape track is known, the masker data was sourced from online repositories, notably Freesound and Xeno-canto, and therefore required calibration. The tracks were thoroughly calibrated to SPL values in the closed interval of 46dBA to 83dBA in 1dBA steps on a dummy head (GRAS 45BB KEMAR Head & Torso) to create the digital gain required to amplify the jth masker to a specific SPL. The calibration process is semi-automatic, using the same audio playback device as the participant. Since the real-time SPL of the base soundscape is usually non-integer, non-integer The digital gain is interpolated from a look-up table using
[0193] (14)
[0194] in is dBA SPL for maskers The calibration gain.
[0195] B. Subjective Response
[0196] All participants listened to the calibrated enhanced soundscapes using a pair of surround headphones (Beyerdynamic Custom One Pro) driven by an external sound card (Creative SoundBlaster E5). After listening to each enhanced soundscape, participants answered a 6-item questionnaire based on the ISO12913-2 standard: "To what extent do you agree or disagree that the current surrounding sound environment is [...]?", where [...] is one of pleasant, chaotic, active, calm, disturbing, and monotonous. Each questionnaire item has multiple options of "strongly disagree", "disagree", "neither agree nor disagree", "agree", and "strongly agree", which are numerically coded from 1 to 5. The ground truth label of ISO pleasantness for each enhanced soundscape was then calculated by the following formula:
[0197] (15)
[0198] in , , , , , Numeric ratings of the questionnaire items were pleasant, disturbed, calm, chaotic, active, and monotonous.
[0199] test
[0200] For all model types in this article, each model was trained with 5-fold cross validation using the same 10 seeds for each validation set, for a total of 50 models per model type. -5 Each model was trained for up to 100 epochs with the Adam optimizer using a learning rate of . If the validation mean squared error (MSE) did not decrease for at least 5 epochs, the learning rate was halved, and if the validation MSE did not decrease for at least 10 epochs, training was stopped early. The MSE and mean absolute error (MAE) disclosed in this paper are the averages of the prediction distributions using With the ground truth label Calculated by the difference between .
[0201] In addition to the three attention types mentioned above (under the heading “Models”), the test also includes a “pass-through” attention block as a baseline model. Fig.14 The through block can be defined as
[0202] (16)
[0203] Fig.14 Shown are (top) a graph of mean squared error (MSE) as a function of attention block type, which shows a violin plot of the validation MSE of pleasantness prediction for each attention block type and feature enhancement method according to various embodiments; and (bottom) a graph of mean absolute error (MAE) as a function of attention block type, which shows a violin plot of the validation MAE of pleasantness prediction for each attention block type and feature enhancement method according to various embodiments. Fig.14 The distribution of prediction errors according to MSE and MAE for each attention and enhancement type is shown. Among all attention types, the CONV enhancement method performs best in both MSE and MAE, followed by CAT and ADD. This can be attributed to the CONV method and kernel filtering, which naturally encourages and Compared with the other two methods, kernel filtering allows more flexible enhancement operations in the feature space.
[0204] It can also be seen that using QKV attention allows for better performance overall compared to models without attention. However, the type of attention mechanism does not seem to have a significant impact on model performance as long as some form of attention is used to couple the query, key, and value tensors. While AA and DPA only have one attention head, the four-head MHA does not seem to achieve a perceptible improvement in prediction error. It should also be noted that the choice of feature augmentation blocks seems to have a more significant impact on the accuracy of the model, as seen by the fact that the CONV augmentation model without attention generally performs better than the ADD model with attention.
[0205] Although the model never saw the audio data of the enhanced soundscape presented to the participants, the best average MSE model (Luong attention with CONV augmentation) at 0.122±0.005 is comparable to the results in Example Study 1, which used the log-mel spectrogram of the enhanced soundscape as input. This demonstrates the effectiveness of feature augmentation techniques in modeling soundscape augmentation in feature space instead of waveform space.
[0206] Fig.15A is a graph of ISO pleasantness as a function of log gain, illustrating gain interpolation for masker class “water” using additive attention (AA) with CONV augmentation (seed 5, validation fold 2), according to various embodiments. Fig. 15B is a graph of ISO pleasantness as a function of log gain, illustrating gain interpolation for masker class “traffic” using additive attention (AA) with CONV augmentation (seed 5, validation fold 2), according to various embodiments. Fig. 15C is a graph of ISO pleasantness as a function of log gain, illustrating gain interpolation for the masker class “bird” using additive attention (AA) with CONV augmentation (seed 5, validation fold 2), according to various embodiments. Fig.15D is a graph of ISO pleasantness as a function of log gain, illustrating gain interpolation for masker class “construction” using additive attention (AA) with CONV augmentation (seed 5, validation fold 2), according to various embodiments. Fig.15Eis a graph of ISO pleasantness as a function of log gain, showing gain interpolation for masker class "silence" using additive attention (AA) with CONV augmentation (seed 5, validation fold 2) according to various embodiments. For the purpose of this visualization, all maskers were adjusted to 65.0 dBA at zero log gain. The base soundscape has a real-time SPL level of 65.32 dBA. During training, the model was not able to see the base soundscape and all maskers. FIG. 15A to FIG. 15E The solid line in represents the mean of the forecast distribution. The shaded area represents pleasantness levels as high as one forecast standard deviation above and as low as one forecast standard deviation below the forecast mean.
[0207] FIG. 15A to FIG. 15E The gain-aware nature of the model can be demonstrated by querying the ISO pleasantness distribution over 256 log gain values in the interval [-2,2] for different types of maskers. A silent “masker” track is also included to test the model’s ability to account for the effects of both the masker and the masker’s gain level. Although the pleasantness predictions are not completely constant, it can be seen that the model can discern that the masker is silent and outputs a roughly constant pleasantness prediction regardless of the query gain value. For the other non-silent maskers, it can be seen that the model is aware of the characteristics of each masker. Consistent with findings from previous studies, the model predicts that pleasantness increases with increasing birdsong masker gain until the birdsong masker gain approaches ambient levels (γ≈-0.5), after which pleasantness drops below that of the unenhanced soundscape (i.e., the silent masker prediction). The water masker shows a similar effect, with pleasantness levels for the water masker being higher than for the unenhanced soundscape up to γ≈−0.5, but pleasantness levels start to drop off at lower gains than for the birdsong masker due to the more continuous nature of water sounds. For known unpleasant maskers, such as traffic and construction, the model correctly outputs increasingly unpleasant predictions as gain levels increase.
[0208] in conclusion
[0209] Various embodiments may relate to an improved probabilistic perceptual attribute predictor model that allows gain-aware prediction of subjective responses to enhanced soundscapes. The proposed model decouples maskers from soundscape feature extraction and models the soundscape enhancement process in feature space, eliminating the need for computationally expensive mixing processes in the waveform domain. In addition, the model is reworked to consider numerical gain levels of maskers instead of soundscape-to-masker ratios, which allows the use of maskers from multiple sources without the need for time-consuming calibration. The modular design of the model allows for the reuse and pre-computation of a large number of features at inference time, which reduces the overall latency and computational resources required for deployment. The ability of the model to accurately predict pleasantness ratings associated with soundscape, maskers, and gain has been demonstrated using a large-scale dataset of 18,000 subjective responses from 442 participants.
[0210] In soundscape enhancement systems, the selection of maskers and playback gain levels is crucial to the effectiveness of soundscape enhancement systems in improving the overall acoustic comfort of a given environment. Conventionally, the selection of appropriate maskers and gain levels is determined by expert opinions that may not represent the target population, or by listening tests that may be time-consuming and labor-intensive. Moreover, the resulting static selection of maskers and gains is usually inflexible to the dynamic characteristics of real-world soundscapes. In this paper, a deep learning model is used to jointly select the best masker and the gain level of the best masker for a given soundscape. The proposed model is designed with highly modular building blocks, which allows for an optimized inference process that can quickly search a large number of combinations of maskers and gains. Furthermore, the use of feature-domain soundscape enhancement regulated in terms of digital gain levels is introduced, which eliminates the computationally expensive waveform-domain mixing process during inference, as well as the cumbersome pre-calibration process required for maskers. The proposed system is validated on a large-scale dataset of over 440 participants' subjective responses to enhanced soundscapes, which ensures the model's ability to predict the combined effects of maskers and their gain levels on perceived pleasantness levels.
Claims
1. Soundscape enhancement system, including: A data acquisition system, the data acquisition system being configured to provide ambient soundscape data; a database, the database comprising a plurality of masking settings; a perceptual property predictor coupled to the data acquisition system and the database, the perceptual property predictor being configured to generate, for each masking setting of the plurality of masking settings, a prediction representative of perception on one or more predefined perceptual property indicators based on the ambient soundscape data; a masking setting ranking system configured to determine one or more optimal masking settings based on the predictions generated by the perceptual property predictor; as well as A playback system is configured to play or reproduce the one or more optimal masking settings.
2. The soundscape enhancement system of claim 1, wherein the data acquisition system is further configured to provide group statistics or contextual data.
3. A soundscape enhancement system according to claim 2, wherein the perceptual attribute predictor is arranged to generate the prediction also based on the population demographic or contextual data.
4. The soundscape enhancement system according to any one of claims 2 to 3, wherein the group demographic or contextual data comprises data related to environmental parameters, group demographic data of one or more listeners, psychological data of the one or more listeners, physiological data of the one or more listeners, or any combination thereof.
5. The soundscape enhancement system of claim 4, wherein the data related to the environmental parameters comprises data related to the location, data related to the visual environment, the number of people nearby, air temperature, humidity, wind speed, or any combination thereof.
6. A soundscape enhancement system according to claim 4 or 5, wherein the group demographic data of the one or more listeners comprises data related to age, gender, occupation or any combination thereof.
7. The soundscape enhancement system according to any one of claims 4 to 6, wherein the psychological data of the one or more listeners comprises data related to noise sensitivity, perceived stress, a happiness index score, or any combination thereof.
8. The soundscape enhancement system according to any one of claims 4 to 7, wherein the physiological data of the one or more listeners comprises data related to heart rate, blood pressure, body temperature or any combination thereof.
9. A soundscape enhancement system according to any one of claims 1 to 8, wherein the perceptual property predictor is arranged to generate a prediction based on one or more masking gain inputs.
10. The soundscape enhancement system according to any one of claims 1 to 9, wherein the one or more predefined perceptual attribute indicators include pleasantness, liveliness, eventfulness, calmness, perceived loudness, sound quality, sharpness, roughness, or any combination thereof.
11. A soundscape enhancement system according to any one of claims 1 to 10, wherein the perceptual property predictor is arranged to generate the prediction by combining the ambient soundscape data and each masking setting.
12. The soundscape enhancement system according to any one of claims 1 to 11, wherein the perceptual attribute predictor comprises a soundscape feature extractor configured to extract features of the ambient soundscape data; wherein the perceptual attribute predictor comprises a masked feature extractor arranged to extract features for each masking setting; and Wherein the perceptual property predictor is arranged to generate the prediction based on extracted features of the ambient soundscape data and extracted features of each masking setting.
13. The soundscape enhancement system according to any one of claims 1 to 12, wherein the perceptual property predictor comprises an objective feature block arranged to extract objective features from the ambient soundscape data and each masking setting; wherein the perceptual attribute predictor comprises a subjective feature block arranged to extract subjective features from the group demographic or contextual data; and Wherein the perceptual attribute predictor is configured to generate the prediction based on the extracted objective features and the extracted subjective features.
14. The soundscape enhancement system according to any one of claims 1 to 13, wherein the perceptual property predictor comprises a linear regression model arranged to generate the prediction based on acoustic or psychoacoustic parameters calculated based on the ambient soundscape data and each masking setting.
15. A soundscape enhancement system according to any one of claims 1 to 14, wherein the perceptual property predictor comprises one or more deep neural networks arranged to generate the predictions based on original audio, a spectrogram representation calculated based on the ambient soundscape data and each masking setting, or a combination of the original audio and the spectrogram representation.
16. The soundscape enhancement system according to any one of claims 1 to 15, wherein the data acquisition system is configured to generate the ambient soundscape data based on input from one or more microphones, one or more antennas, a storage medium or any combination thereof.
17. A soundscape enhancement system according to any one of claims 1 to 16, wherein the plurality of masking settings comprises one or more tracks of recorded or synthesized sounds, one or more tracks of silence, or one or more tracks derived from the one or more tracks of recorded or synthesized sounds and the one or more tracks of silence.
18. The soundscape enhancement system according to any one of claims 1 to 17, wherein the playback system comprises one or more speakers, one or more virtual reality or augmented reality headsets, one or more headphones, one or more earplugs, or any combination thereof.
19. A soundscape enhancement system according to any one of claims 1 to 18, wherein the prediction generated is non-deterministic.
20. A method for forming a soundscape enhancement system, the method comprising: Providing a data acquisition system, the data acquisition system being configured to provide ambient soundscape data; providing a database, the database comprising a plurality of masking settings; coupling a perceptual property predictor to the data acquisition system and the database, the perceptual property predictor being configured to generate, for each masking setting of the plurality of masking settings, a prediction representative of perception on one or more predefined perceptual property indicators based on the ambient soundscape data; providing a masking setting ranking system arranged to determine one or more optimal masking settings based on the predictions generated by the perceptual property predictor; and A playback system is provided, the playback system being arranged to play or reproduce the one or more optimal masking settings.
