Intelligent audio scene recognition and adaptive adjustment method and system

Through intelligent audio scene recognition and adaptive adjustment methods, multi-level audio and sensor incremental learning integrate audio and environmental features, the problem that traditional technology cannot adjust audio in real time is solved, the adaptability and naturalness of sound quality adjustment is improved, and the adaptability and nature of sound quality adjustment is reduced. Cost and computing power consumption.

CN118737187BActive Publication Date: 2024-11-19SHENZHEN BBS ELECTRONICS CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411220591.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-19
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

Traditional audio processing technology is difficult to adapt to complex and changeable environmental needs, especially when it is necessary to process audio signals and environmental sensing data at the same time, it is impossible to effectively adjust the audio effect in real time according to environmental changes.

Method used

Using intelligent audio scene recognition and adaptive adjustment methods, feature embedding, multi-level audio incremental learning and sensing incremental learning are performed by obtaining initial audio and environmental sensing data, integrating audio and environmental features, and generating a fusion implicit representation. To optimize sound quality adjustment.

Benefits of technology

It improves the environmental adaptation effect of sound quality adjustment, increases the naturalness of sound quality, reduces computing power consumption, saves costs, and achieves the best auditory experience in different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118737187B_ABST
    Figure CN118737187B_ABST
Patent Text Reader

Abstract

The present application provides an intelligent audio scene recognition and adaptive adjustment method and system. After feature embedding of initial audio and environmental sensor data respectively, multi-level audio incremental learning is performed on the audio embedding representation, and correspondingly, the audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels is obtained, and multi-level sensor incremental learning is performed on the sensor embedding representation one by one to obtain the integrated implicit representation obtained after the multi-level sensor incremental learning. In this way, the integrated implicit representation is restored and mapped to obtain sound quality adjusted audio that integrates the track implicit representation of the target audio track and the environmental implicit representation of the environmental sensor data. The adjusted audio is constructed based on the integrated implicit representation after multi-level sensor incremental learning, which can improve the environmental adaptation effect of the sound quality adjusted audio, make the target audio track more natural in the sound quality adjusted audio, and thus increase the sound quality of the sound quality adjusted audio.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an intelligent audio scene recognition and adaptive adjustment method and system. Background Art

[0002] With the rapid development of artificial intelligence technology, the field of audio processing has ushered in unprecedented changes. In multiple application scenarios such as smart home, entertainment industry, and communication technology, high-quality audio processing has become a key factor in improving user experience. However, traditional audio processing technology often relies on fixed algorithms and preset parameters, and it is difficult to adapt to complex and changing environmental requirements, especially when it is necessary to process audio signals and environmental sensor data at the same time. Its limitations are particularly obvious. Specifically, when facing audio adjustment tasks in different scenarios, traditional audio processing technology often fails to fully consider the impact of environmental factors on audio effects. For example, in scenarios such as family gatherings, watching movies in theaters, or outdoor activities, factors such as environmental noise and spatial reverberation will significantly affect the audio playback effect, and traditional technologies are usually unable to effectively adjust the audio in real time according to these environmental changes. Summary of the invention

[0003] In view of this, the present application provides an intelligent audio scene recognition and adaptive adjustment method and system. The technical solution of the present application is implemented as follows:

[0004] In a first aspect, the present application provides an intelligent audio scene recognition and adaptive adjustment method, the method comprising: obtaining initial audio and environmental sensor data containing a target audio track; performing feature embedding on the initial audio and the environmental sensor data respectively, and obtaining an audio embedding representation and a sensor embedding representation accordingly; performing multi-level audio incremental learning on the audio embedding representation, and correspondingly obtaining an audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels; performing multi-level sensor incremental learning on the sensor embedding representation one by one; wherein the number of levels of the multi-level sensor incremental learning is consistent with the number of levels of the multi-level audio incremental learning; for both Between two adjacent levels of sensory incremental learning, the sensory incremental learning results obtained by the sensory incremental learning of the front level in the pairwise adjacent levels and the implicit representation of the audio semantic annotation of the corresponding level are implicitly integrated; wherein, when performing the sensory incremental learning of the back level in the pairwise adjacent levels, sensory incremental learning is performed on the implicit representation integration result after the sensory embedded representation and the implicit representation are integrated; the integrated implicit representation obtained after the multi-level sensory incremental learning is restored and mapped to obtain the sound quality adjusted audio that integrates the implicit representation of the target audio track and the implicit representation of the environment of the environmental sensor data.

[0005] In a second aspect, the present application provides an audio system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above-mentioned method when executing the program.

[0006] The beneficial effects of the present application include at least: after performing feature embedding on the initial audio and environmental sensor data respectively, the present application performs multi-level audio incremental learning on the audio embedding representation, and accordingly obtains the audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels, and performs multi-level sensor incremental learning on the sensor embedding representation one by one, and obtains the integrated implicit representation obtained after the multi-level sensor incremental learning, so that the integrated implicit representation is restored and mapped to obtain the sound quality adjusted audio that integrates the track implicit representation of the target audio track and the environmental implicit representation of the environmental sensor data, and constructs the adjusted audio based on the integrated implicit representation after multi-level sensor incremental learning, which can improve the environmental adaptation effect of the sound quality adjusted audio, make the target audio track more natural in the sound quality adjusted audio, and thus increase the sound quality of the sound quality adjusted audio. At the same time, because for the sensor incremental learning of two adjacent levels, the sensor incremental learning results obtained by the sensor incremental learning of the front level in the sensor incremental learning of two adjacent levels and the implicit representation of the audio semantic annotation of the corresponding level number will be implicitly represented and integrated. In addition, when performing the sensor incremental learning of the back level in the sensor incremental learning of two adjacent levels, the sensor incremental learning is performed on the implicit representation integration result after the sensor embedding representation and the implicit representation are integrated. Then, the multi-level sensor incremental learning integrates the audio semantic annotation implicit representation of the target audio track mined during the multi-level audio incremental learning at multiple levels. Then, the audio features of the target audio track can be more effectively integrated to increase the sound quality of the sound quality adjusted audio. In addition, the present application processes the loaded initial audio and environmental sensor data, and the loaded data includes the initial audio and environmental sensor data. No other additional network adjustment process is required. The process is simple, the consumption of computing power is low, and the cost is saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of the implementation flow of an intelligent audio scene recognition and adaptive adjustment method provided in an embodiment of the present application.

[0008] Figure 2 A hardware entity schematic diagram of a sound system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0009] The present application embodiment provides an intelligent audio scene recognition and adaptive adjustment method, which can be executed by a processor of an audio system. Figure 1 As shown, the method comprises the following steps:

[0010] Step S100: Acquire initial audio and environmental sensor data including a target audio track.

[0011] In a specific scenario, the user can select a song or music playlist as the playback target through a smartphone app, a smart home control platform, or directly through the physical button on the speaker. The audio system receives the digital audio file of this song through the interface connected to it (such as Wi-Fi, Bluetooth, or a specific audio input port). This file contains the complete audio information of the target audio track and is the basis for subsequent processing. For example, if the target audio track is a song in MP3 format, the audio system will read the binary data of this song, which represents the waveform information of the audio.

[0012] At the same time, the various sensors built into the smart speaker start working to collect various parameters of the current environment. These sensors may include temperature sensors, humidity sensors, noise level detectors, and possible motion sensors. Each sensor generates one or more data points, which together constitute a "snapshot" of the current environment. For example, a temperature sensor may report that the current room temperature is 22°C, and a noise level detector may show that the current ambient noise is 40 decibels.

[0013] The sound system combines these data from different sources (the original audio file and the environmental sensor data) in preparation for subsequent processing steps. This usually involves converting the audio data into a digital format suitable for processing (if it has not already been converted) and organizing the environmental sensor data into a form that is convenient for computation, such as a dictionary containing multiple key-value pairs or a structured data table.

[0014] In some cases, the sound system may pre-process the initial audio and ambient sensor data in order to optimize the performance or accuracy of subsequent processing steps. For example, the audio data may be resampled to match a specific sampling rate, and the ambient sensor data may be simply filtered to remove noise or outliers.

[0015] Step S200: feature embedding is performed on the initial audio and the environmental sensor data respectively, and audio embedding representation and sensor embedding representation are obtained accordingly.

[0016] In the embodiment of the present application, after the sound system receives the initial audio selected by the user (for example, a pop music track) and the current environmental sensor data (such as room temperature, humidity, noise level, etc.), it begins the process of feature embedding. Feature embedding is a key step in converting raw data into a form that can be efficiently processed by the machine learning model.

[0017] For the initial audio, the sound system uses a specialized audio encoder to extract its features. The audio encoder can be a pre-trained neural network model, such as a convolutional neural network (CNN) or a variant of a recurrent neural network (RNN) (such as LSTM or GRU). These models are able to learn complex feature representations from audio signals, which are essential for subsequent audio analysis and processing.

[0018] Assume that the audio encoder is a CNN-based model that splits the input audio waveform (usually one-dimensional time series data) into multiple overlapping frames and applies a series of convolution operations to each frame to extract local features. These features may include frequency distribution, rhythmic patterns, timbre characteristics, etc. Ultimately, the model outputs a fixed-length vector (audio embedding representation) that captures the key audio features of the entire song. For environmental sensor data, the sound system uses another type of encoder to process it. Since environmental sensor data is usually multi-dimensional (one dimension for each sensor) and the nature of the data in each dimension may be different (such as temperature is a continuous value, while some sensors may output binary signals), an encoder that can adapt to this diversity is required. For example, the sound system uses a fully connected neural network (also known as a densely connected layer) as an encoder for environmental sensor data. This network accepts a vector containing all sensor readings as input and learns the relationships and patterns between these readings through a series of hidden layers. Each layer of neurons receives the output of the previous layer as input, calculates a weighted sum (plus a bias term), and then passes it through a nonlinear activation function (such as ReLU). Ultimately, the network outputs a fixed-length vector (the sensor embedding representation) that summarizes the overall state of the current environment.

[0019] Through step S200, the sound system obtains the audio embedding representation and the sensor embedding representation, both of which are the basis for subsequent processing steps (such as scene recognition, adaptive adjustment, etc.). The audio embedding representation helps the sound system understand the emotion, rhythm and structure of the music, while the sensor embedding representation provides contextual information about the playback environment.

[0020] After obtaining these embedded representations, the sound system can further use them to perform more complex tasks, such as automatically adjusting the volume according to the ambient noise level, or selecting the appropriate music style to play based on the user's mood (inferred indirectly through music analysis) and the ambient atmosphere.

[0021] Step S300: performing multi-level audio incremental learning on the audio embedding representation, and correspondingly obtaining an audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels.

[0022] In the embodiment of the present application, multi-level audio incremental learning is a progressive learning strategy that simulates the process of human understanding of complex audio signals, gradually rising from low-level acoustic features to high-level semantic understanding. In this solution, this learning is particularly reflected in updating the audio embedding representation through residual projection (a specific implementation of incremental learning), thereby capturing the implicit representation of the semantic annotation of the audio at different levels.

[0023] Specifically, the sound system receives an audio embedding representation, which is a high-dimensional vector that captures the key features of the audio signal. The system then initiates a multi-level audio incremental learning process, which is usually supported by a pre-trained neural network model that contains multiple levels (or layers), each of which corresponds to a different depth of understanding of the audio signal. For example, assume that the neural network model used is a variant based on the convolutional neural network (CNN) and residual network (ResNet) architecture. In this model, each layer receives the output of the previous layer as input and updates the audio embedding representation through a series of convolution operations and residual connections.

[0024] At the first level, the model may focus on extracting basic acoustic features, such as frequency content, energy distribution, etc. These features are fundamental and necessary for subsequent levels. As the number of layers increases, the model begins to capture more complex audio structures, such as rhythmic patterns, harmonic progressions, etc. These features have gone beyond simple acoustic properties and begin to touch the semantic level of audio. At the highest level, the model strives to understand high-level semantic information such as emotional expression and thematic content of the audio. This information is crucial for scene recognition and adaptive adjustment.

[0025] In each layer, the residual projection mechanism allows the model to learn and add new and useful feature representations while retaining the information of the previous layer, rather than completely replacing the original representation. This incremental learning method helps maintain the continuity of audio information and reduces the gradient vanishing or exploding problems that may be caused by deepening the layers.

[0026] Residual projection is used as a means of implementing incremental learning in this scheme. The core idea is to add a shortcut in the output of each layer, which directly connects the output of the previous layer to the input of the current layer. Mathematically, this can be expressed as: ;in, is the input of the lth layer (also the output of the previous layer), is the nonlinear transformation function of the layer (usually including convolution, activation and other operations), is the weight parameter of this layer, is the output of this layer. The residual term The addition of enables the model to retain old features while learning new features, thus achieving the effect of incremental learning. Through multi-level audio incremental learning, the sound system can generate implicit representations of audio semantic annotations of the target audio track at multiple levels. These representations not only contain the acoustic features of the audio, but also go deep into its semantic and emotional levels.

[0027] Step S400: performing multi-level sensory incremental learning on the sensory embedding representations one by one; wherein the number of levels of the multi-level sensory incremental learning is consistent with the number of levels of the multi-level audio incremental learning.

[0028] In the embodiment of the present application, multi-level sensor incremental learning means that the sound system will perform deep processing of the sensor embedded representation layer by layer, and each layer attempts to understand the environmental data from a more abstract and higher-level perspective. This learning method echoes the multi-level audio incremental learning, ensuring that the audio adjustment strategy can accurately match the multi-dimensional characteristics of the environment. Importantly, the number of levels of sensor incremental learning is consistent with the number of levels of audio incremental learning, which helps to achieve effective fusion of audio and environmental characteristics in subsequent steps.

[0029] Specifically, the sensor embedding representation received by the sound system is a high-dimensional vector that contains encoded information about multiple environmental parameters (such as temperature, humidity, noise level, etc.). In order to deeply mine this information, the sound system uses a pre-trained neural network model that is designed for sensor data processing and contains multiple layers. For example, assume that the neural network model used is a hybrid architecture based on a long short-term memory network (LSTM) and a fully connected layer (Dense Layer). Each LSTM layer is able to capture the temporal dependencies and sequence patterns between environmental parameters, while the fully connected layer is responsible for extracting higher-level abstract features. At the first layer, the LSTM unit receives the sensor embedding representation as input and begins to capture the trend of environmental parameters over time. For example, it can identify the periodic changes in indoor temperature throughout the day. As the data passes through more LSTM layers, the model begins to learn more complex environmental dynamics. These dynamics may include the interactions between different environmental parameters and how they jointly affect the quality of the indoor environment. At the higher level, the fully connected layer receives the output of the LSTM layer and extracts features that highly summarize the environmental state. These characteristics may represent the overall comfort level of the current environment, or the environment configuration specific to a certain scenario (such as a family gathering or individual work).

[0030] In order to ensure a close match between the audio adjustment strategy and the environmental characteristics, the sound system is designed to ensure that the number of levels of multi-level sensor incremental learning is the same as the number of levels of multi-level audio incremental learning. This means that at each corresponding level, the audio features and environmental features are at a similar level of abstraction, which facilitates feature fusion and decision making in subsequent steps. Through multi-level sensor incremental learning, the sound system obtains a deep understanding of the environmental state and generates a series of implicit representations corresponding to the environmental characteristics. These implicit representations will be fused with the audio semantic annotation implicit representations obtained by multi-level audio incremental learning (as described in step S500) to develop a more accurate and personalized audio adjustment strategy. For example, when a high level of ambient noise is detected, the sound system can automatically lower the volume and adjust the equalizer settings to reduce external interference and improve the listening experience.

[0031] Step S500: For sensor incremental learning between two adjacent levels, implicit representation integration is performed on the sensor incremental learning results obtained by the sensor incremental learning of the front level in the sensor incremental learning between two adjacent levels and the implicit representation of the audio semantic annotation of the corresponding level; wherein, when performing sensor incremental learning of the back level in the sensor incremental learning between two adjacent levels, sensor incremental learning is performed on the implicit representation integration result after the sensor embedded representation and the implicit representation are integrated.

[0032] In the embodiment of the present application, implicit representation integration refers to the process of merging feature representations from different sources but with similar abstraction levels. In step S500, the sound system focuses on the sensor incremental learning between two adjacent levels, and integrates the results of the sensor incremental learning of the previous level with the audio semantic annotation implicit representation of the corresponding level. This integration is not a simple superposition, but uses a machine learning model (such as a neural network) to learn and optimize the interaction between the two, thereby generating a composite implicit representation containing richer information.

[0033] For example, suppose the sound system uses a neural network model with three levels when performing multi-level sensor incremental learning and multi-level audio incremental learning. At each level, sensor incremental learning and audio incremental learning will output implicit representations of their respective fields.

[0034] At the first level, sensor incremental learning may capture the basic physical characteristics of the environment (such as the instantaneous values ​​of temperature and humidity), while audio incremental learning extracts the basic acoustic characteristics of the audio (such as frequency distribution). The sound system integrates these two implicit representations through a fusion layer (which can be a fully connected layer or a specific fusion algorithm). The integration process may include operations such as feature concatenation and weighted summation to generate a new implicit representation that contains both environmental information and audio information. Entering the second level, the sound system no longer relies solely on the original sensor embedding representation for sensor incremental learning, but takes the implicit representation after integration at the first level as input. At the same time, audio incremental learning also outputs a higher-level audio semantic annotation implicit representation at the second level. The system performs implicit representation integration again, but this time it integrates the first-level integration result with the second-level audio semantic annotation implicit representation. In this process, the system may learn a more complex mapping relationship to capture the nonlinear interaction between the environment and the audio. The third level is processed in a similar way, but the integrated implicit representation now contains all the information from the first two levels, so that the final audio adjustment strategy can fully consider the multi-dimensional characteristics of the environment and the deep semantics of the audio. The calculation process of implicit representation integration involves the forward propagation of the neural network and possible back-propagation adjustments. Although the specific calculation formula may vary depending on the model architecture, a simplified example can be a combination of weighted summation and activation function: ; where σ is the activation function (such as ReLU), W1 and W2 are learnable weight matrices corresponding to the linear transformation of the environment and audio input, respectively, b is the bias term, and R−1 is the sensor incremental learning result of the previous layer. is the implicit representation of the audio semantic annotation at the corresponding level, and is the integrated implicit representation. Through the implicit representation integration of step S500, the audio system can generate a series of composite implicit representations that combine environmental perception and audio understanding. These representations play a key role in the subsequent formulation of audio adjustment strategies, enabling the audio system to intelligently adjust the audio output according to the current environmental state, providing a more personalized and comfortable listening experience. For example, when it is detected that the user is working attentively, the audio system may automatically lower the volume and adjust the sound settings to reduce interference; in social occasions such as family gatherings, the bass effect may be enhanced to create a more lively atmosphere.

[0035] Step S600: performing restoration mapping on the integrated implicit representation obtained after the multi-level sensor incremental learning to obtain sound quality adjusted audio that integrates the track implicit representation of the target track and the environment implicit representation of the environment sensor data.

[0036] In step S600, the sound system processes the integrated implicit representation obtained after multi-level sensor incremental learning, and generates sound quality adjusted audio that combines the implicit representation of the target track and the implicit representation of the environment of the environmental sensor data through the process of restoration mapping. Restoration mapping is particularly common when processing autoencoders or generating models, and refers to the process of converting the low-dimensional or implicit representation learned by the model back to the original data space or perceptible data space. In an embodiment of the present application, restoration mapping is used to convert the integrated implicit representation (including audio and environmental information) back to an audio signal, but the audio signal at this time has been optimized and adjusted according to the environmental characteristics.

[0037] To achieve the restoration mapping, the sound system can include a decoder network, which is the inverse process of the encoder network (used to generate the implicit representation in the previous step). The decoder receives the integrated implicit representation as input and gradually restores the original audio waveform through a series of transformation operations, but dynamically adjusts according to the environmental characteristics in the process. For example, assume that the integrated implicit representation is a high-dimensional vector that contains both the track implicit representation of the target track (such as rhythm, melody, timbre, etc.) and the environment implicit representation of the environmental sensor data (such as noise level, room reverberation characteristics, etc.). The decoder network can be composed of multiple deconvolutional layers, upsampling layers, and dense layers. The deconvolution layer gradually expands the high-level features in the implicit representation to generate a feature map with higher resolution. In the context of audio generation, this can be regarded as a preliminary reconstruction of the audio waveform. The upsampling layer increases the size of the feature map through interpolation methods to match the temporal resolution of the original audio signal. After the feature map is upsampled to the appropriate size, the fully connected layer maps each point in the feature map back to a sample point of the audio signal to complete the final audio reconstruction.

[0038] During the decoding process, the decoder can dynamically adjust the audio signal based on the information in the implicit representation of the environment. For example, if the implicit representation of the environment indicates that the current ambient noise level is high, the decoder can adjust the gain of the audio signal or apply a specific noise reduction algorithm to optimize the listening experience.

[0039] After restoration mapping, the sound system outputs sound quality adjusted audio. This audio not only retains the essence of the original target audio track (such as melody, rhythm, etc.), but also optimizes and adjusts according to the current environmental characteristics, making the listening experience more natural and comfortable. For example, in a family gathering scene, the sound quality adjusted audio may enhance the bass to create a warmer atmosphere; in a scene that requires quiet reading, it may reduce the sharpness of the treble and lower the overall volume. In this way, the sound system realizes intelligent adjustment of the audio signal, allowing users to get the best listening experience in different scenarios.

[0040] As an implementation manner, the step S300 performs multi-level audio incremental learning on the audio embedding representation, and correspondingly obtains the audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels, which may specifically include:

[0041] Step S310: Based on multiple intermediate layer components in the pre-debugged audio annotation extraction component, multi-level audio incremental learning is performed on the audio embedding representation, and correspondingly, implicit representations of audio semantic annotations of the target audio track in the initial audio at multiple levels are obtained; wherein, when the first-level audio incremental learning is performed on the audio embedding representation, the first-level audio incremental learning is performed on the audio embedding representation based on the first intermediate layer component among the multiple intermediate layer components, and an implicit representation of audio semantic annotations of the target audio track in the initial audio at the first level is obtained.

[0042] In the embodiment of the present application, the sound system uses multiple intermediate layer components (hidden layers) in the pre-debugged audio annotation extraction component to perform multi-level audio incremental learning on the audio embedding representation. In this process, the sound system focuses on the first-level audio incremental learning, that is, the audio embedding representation is processed using the first intermediate layer component to capture the audio semantic features of the target audio track at the lowest level.

[0043] The first intermediate layer component is the most basic layer in the audio annotation extraction component, which is responsible for extracting the most direct and basic acoustic features from the audio embedding representation. These features may include frequency distribution, energy envelope, short-time zero-crossing rate, etc., which constitute a low-level description of the audio signal. For example, assume that the input audio embedding representation is a digitized vector of a popular song, which contains a variety of information such as the rhythm, timbre, and pitch of the song. In step S310, the first intermediate layer component of the sound system may be a convolutional neural network (CNN) layer, which slides on the audio embedding representation through a series of convolution kernels to perform convolution operations. In the convolution operation, when the convolution kernel slides on the audio embedding representation, it performs a dot product operation with the local features of the area it covers, thereby extracting specific acoustic features of the area. These features are then processed by activation functions (such as ReLU) to enhance the nonlinear capabilities of the model. After convolution and activation operations, the first intermediate layer component outputs a series of feature maps that capture the acoustic characteristics of the audio signal at different time points and frequency points. These features are the basis for higher-level semantic understanding in subsequent layers. In the context of incremental learning of the first-layer audio, these feature maps can be regarded as implicit representations of the audio semantic annotations of the target track at the first layer. Although they do not directly correspond to human-understandable semantic labels, they encode key information in the audio signal in a machine-understandable way.

[0044] The results of incremental learning of the first-level audio not only provide key information for the current level of audio processing, but also lay a solid foundation for the learning of subsequent levels. As the learning level goes deeper, the sound system will use more complex intermediate layer components (such as recurrent neural network RNN, long short-term memory network LSTM or Transformer, etc.) to further process and analyze the first-level output to extract higher-level audio semantic features.

[0045] In summary, step S310 performs first-layer audio incremental learning on the audio embedding representation through the first intermediate layer component, capturing the audio semantic features of the target audio track at the lowest level.

[0046] Step S320: When performing x-th layer audio incremental learning on the audio embedding representation, based on the x-th intermediate layer component among the multiple intermediate layer components, perform x-th layer audio incremental learning on the w-th layer audio semantic annotation implicit representation to obtain the x-th layer audio semantic annotation implicit representation of the target audio track in the initial audio; wherein, x≥2, w=x-1.

[0047] In the embodiment of the present application, step S320 uses multiple intermediate layer components (hidden layers) in the audio annotation extraction component to perform in-depth learning of the audio embedding representation layer by layer. In step S320, the sound system further processes the implicit representation of the audio semantic annotation of the wth layer based on the xth intermediate layer component to obtain higher-level audio semantic features of the target audio track at the xth layer. Here, x represents the current learning level, and x≥2, indicating that this is deep learning other than the first layer; w=x-1, indicating that the input of the current level is the output of the previous level (ie, the wth layer).

[0048] As the learning level goes deeper, the xth intermediate layer component is usually more complex than the first layer component, and can capture more abstract and higher-level features in the audio signal. These features may include melodic patterns, harmonic structures, emotional tendencies, etc., which are crucial to understanding the deep meaning of the audio content. For example, suppose that the sound system is processing the audio embedding representation of a piece of classical music, and the goal is to extract the structure and emotional information of the music from it. In step S320, the xth intermediate layer component may be a complex neural network layer that combines a long short-term memory network (LSTM) and an attention mechanism. The LSTM layer can process the temporal dependencies in the audio signal and capture the melodic lines and rhythmic patterns in the music. It controls the flow of information through the forget gate, input gate, and output gate structure, thereby effectively learning long-term dependencies. The attention mechanism allows the LSTM layer to focus more on important parts when processing audio signals. For example, when identifying the theme in music, the attention mechanism can help the LSTM layer pay more attention to the appearance of the theme melody and ignore some unimportant transition parts. The audio semantic annotation implicit representation of the wth layer, as the input of the xth layer, already contains relatively rich audio features. In the xth layer, the LSTM layer combines the attention mechanism to further extract and integrate these features to generate more abstract and advanced audio semantic features. These features may represent deeper meanings such as the structural layout and emotional changes of the music. The implicit representation of the audio semantic annotation obtained after processing by the xth layer not only contains the acoustic features of the music, but also incorporates the emotional and structural information of the music. This information is of great significance for subsequent scene recognition and adaptive adjustment, and can help the sound system better understand the user's environment and emotional state, thereby providing a more personalized audio experience.

[0049] After obtaining the implicit representation of the audio semantic annotations at the xth layer, the sound system can continue to learn at a deeper level (if x is not the last layer), or fuse these high-level features with other information (such as environmental sensor data) (as described in step S500) to formulate more accurate audio adjustment strategies. Ultimately, these strategies will be applied in actual scenarios to realize the adaptive adjustment function of smart speakers.

[0050] As an implementation mode, the intermediate layer component is a first jump connection component, and the first jump connection component includes an incremental projection algorithm. Based on this, the step S310, based on multiple intermediate layer components in the pre-debugged audio annotation extraction component, performs multi-level audio incremental learning on the audio embedding representation, and correspondingly obtains the audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels, which may include:

[0051] Step S311: Based on the incremental projection algorithm in each first jump connection component in the multiple intermediate layer components, multi-level audio incremental learning is performed on the audio embedding representation, and accordingly, an implicit representation of audio semantic annotations of the target audio track in the initial audio at multiple levels is obtained.

[0052] In an embodiment of the present application, step S310 uses multiple intermediate layer components in the pre-debugged audio annotation extraction component, especially the first jump connection component (ResNet style), combined with the incremental projection algorithm (used to implement residual mapping), to analyze and learn the input audio embedding representation layer by layer, thereby gradually revealing the audio semantic features of the target audio track at different levels. The audio system integrates advanced audio processing technology and machine learning algorithms to provide users with a personalized audio experience. In a smart home environment, the audio system is not only responsible for playing music, audiobooks and other content, but also automatically adjusts the sound quality and volume according to the current scene and user behavior to adapt to different usage needs. In order to achieve this goal, the audio system needs to be able to accurately identify the current scene and understand the deep meaning of the audio content.

[0053] The first skip connection component, as a residual network, solves the gradient vanishing or exploding problem in deep neural network training by introducing skip connections, enabling the network to learn deeper feature representations. In the field of audio processing, this structure is widely used in tasks such as audio classification and recognition. As a specific implementation of residual mapping, the incremental projection algorithm calculates the difference between input and output (i.e., residual) and adds the residual to the input to update the feature representation, thereby achieving layer-by-layer accumulation and enhancement of features.

[0054] In step S311, the sound system uses multiple first jump connection components in the audio annotation extraction component to perform multi-level audio incremental learning on the audio embedding representation. This process can be divided into the following key steps:

[0055] 1. Input audio embedding representation: The sound system receives an initial audio containing the target audio track and converts it into an audio embedding representation through an encoder. This representation is a high-dimensional vector that captures the key features of the audio signal, such as rhythm, timbre, pitch, etc.

[0056] 2. First-layer incremental learning: The system starts the first-layer incremental learning and uses the incremental projection algorithm in the first jump connection component to process the audio embedding representation. In this process, the component extracts low-level acoustic features of the audio signal, such as frequency components, short-time energy, etc., through a series of convolution operations. At the same time, the jump connection ensures that the input information can be passed to subsequent layers losslessly, avoiding the gradient vanishing problem. After calculation by the incremental projection algorithm, the system obtains the first-layer audio semantic annotation implicit representation. Although it does not directly correspond to human-understandable semantic labels, it encodes the key information in the audio signal in a machine-understandable way.

[0057] 3. Deep incremental learning: As the layers go deeper, each subsequent first-skip connection component receives the output of the previous layer as input and continues to perform incremental learning. At this stage, the component not only extracts higher-level acoustic features (such as melodic patterns, harmonic structures, etc.), but also further mines the semantic features of the audio signal through an incremental projection algorithm. Skip connections still play a key role here to ensure the effective transfer and accumulation of features between layers. The learning results of each layer will update the implicit representation of the audio semantic annotation, making it more comprehensive and in-depth Reflect the characteristics of the target track.

[0058] 4. Feature fusion and output: After multiple levels of incremental learning, the sound system obtains implicit representations of audio semantic annotations of the target audio track at different levels. These representations not only contain the acoustic features of the audio signal, but also incorporate its semantic and emotional information. The system can use these implicit representations as input data for subsequent processing steps (such as scene recognition, sound quality adjustment, etc.) to achieve more intelligent audio processing functions.

[0059] For example, suppose the sound system is playing a classical symphony as the target track. In step S311, the first-level incremental learning may capture low-level features such as the basic rhythm and instrument timbre of the symphony; as the level goes deeper, the structural layout of the symphony (such as introduction, presentation, development, recapitulation, etc.) and the emotional expressions of different movements (such as excitement, tranquility, melancholy, etc.) are gradually identified. These high-level features are effectively extracted and accumulated into the implicit representation of audio semantic annotations through the incremental projection algorithm, providing a rich information basis for subsequent scene recognition and adaptive adjustment.

[0060] In summary, step S311 uses the incremental projection algorithm in the first skip connection component to perform multi-level audio incremental learning on the audio embedding representation, capturing the audio semantic features of the target audio track at different levels. This process not only enhances the sound system's ability to understand the audio content, but also provides strong support for its subsequent scene recognition and adaptive adjustment functions.

[0061] As an implementation mode, the step S400, performing multi-level sensor incremental learning on the sensor embedding representations one by one, includes:

[0062] Step S410: Based on the pre-debugged semantic enhancement generation network, multi-level sensory incremental learning is performed on the sensory embedding representation one by one; wherein the semantic enhancement generation network includes a plurality of cascaded second jump connection components, and a saliency focus layer is included between each two adjacent second jump connection components; the number of the second jump connection components is consistent with the number of the first jump connection components in the audio annotation extraction component.

[0063] Step S400 is implemented through multi-level sensory incremental learning, especially with the help of a pre-debugged semantic enhancement generation network, which integrates cascaded multiple second jump connection components and a saliency focus layer (such as an attention layer) to simulate the hierarchical understanding and processing process of humans on complex environmental information.

[0064] The semantic enhancement generation network is the core of step S400. It is designed to perform multi-level incremental learning on the sensor embedding representation, so as to mine deep environmental features that are beneficial to the adjustment of the sound system. The network consists of multiple cascaded second jump connection components, each of which is interspersed with saliency focus layers (such as attention layers) to achieve gradual enhancement and focus of features. Similar to the first jump connection component in the audio annotation extraction component, the second jump connection component also adopts the idea of ​​residual learning, and avoids the gradient vanishing problem in deep networks by directly connecting the input and output. These components may include basic structures such as convolutional layers and pooling layers, which are used to extract and transform basic features in environmental sensor data. The output of each component will be used as the input of the next component, and a higher level of environmental feature representation will be constructed layer by layer.

[0065] Between adjacent second-hop connection components, the saliency focus layer plays a key role by simulating the human attention mechanism and dynamically weighting the feature representation, so that the network can focus on the features that are most critical to the current task. For example, in a family gathering scene, the noise level and room reverberation characteristics may be more important than other environmental parameters, so the saliency focus layer will increase the weights of these features accordingly so that the sound system can make more accurate adjustment decisions.

[0066] In step S410, the sound system uses a semantic enhancement generative network to perform multi-level sensory incremental learning on the sensory embedding representation. The sound system collects environmental data (such as temperature, humidity, noise level, etc.) through sensors and converts it into a sensory embedding representation through an encoder. This representation is a high-dimensional vector that captures the key features of the environmental state.

[0067] In the first layer of the semantic enhancement generative network, the second skip connection component performs preliminary processing on the sensor embedding representation to extract basic environmental features (such as instant noise level, current temperature, etc.). Subsequently, the saliency focusing layer weights these features to highlight the most critical features for subsequent layer learning.

[0068] As the layers go deeper, each subsequent second-skip connection component receives the output of the previous layer as input and continues to perform incremental learning. At this stage, the components not only extract higher-level environmental features (such as environmental comfort assessment, activity pattern recognition, etc.), but also further focus these features through the saliency focus layer so that the sound system can understand the current scene more accurately.

[0069] After multiple levels of incremental learning, the sound system obtains deep feature representations that contain rich environmental information. These representations not only capture the basic state of the environment (such as temperature and humidity), but also incorporate higher-level semantic features (such as scene categories, user behavior patterns, etc.). These features will serve as an important basis for subsequent scene recognition and adaptive adjustment steps.

[0070] For example, suppose that a sound system is deployed in a smart home environment to provide background music during family gatherings. In step S400, the sound system collects environmental data through temperature sensors, humidity sensors, and noise detectors and converts them into sensory embedding representations. Subsequently, the representations are input into the semantic enhancement generative network for multi-level sensory incremental learning.

[0071] In the first layer of incremental learning, the network extracts basic environmental features, such as the current room temperature of 25°C, humidity of 60%, and background noise level of 50 decibels. Subsequently, the saliency focus layer increases the weight of the noise level based on the characteristics of the family gathering scene so that the sound system can prioritize reducing the impact of background noise.

[0072] In deep incremental learning, the network further mines higher-level environmental features, such as assessing that the current environment is relatively comfortable and suitable for a relaxed and enjoyable party atmosphere. At the same time, the activity patterns of family members (such as talking, dancing, etc.) are identified through the saliency focus layer so that the sound system can automatically adjust the rhythm and volume of the music to match the rhythm of the activity. Ultimately, the deep feature representation obtained after multi-level sensor incremental learning will serve as an important basis for scene recognition and adaptive adjustment of the sound system. For example, the system may automatically increase the bass output to reduce the impact of background noise based on the current noise level and the activity patterns of family members, and appropriately adjust the volume and equalizer settings to create a more pleasant party atmosphere.

[0073] Step S400 uses the semantic enhancement generative network to perform multi-level sensory incremental learning on the sensory embedding representation, thereby mining deep environmental features that have practical guiding significance for the sound system adjustment decision. This process not only enhances the sound system's ability to understand the environmental state, but also provides strong support for its subsequent scene recognition and adaptive adjustment functions.

[0074] As an implementation mode, the step S410, based on the pre-debugged semantic enhancement generation network, performs multi-level sensory incremental learning on the sensory embedding representations one by one, which may specifically include:

[0075] Step S411: When performing first-layer sensor incremental learning on the sensor embedded representation, based on the first second jump connection component among the multiple second jump connection components, perform first-layer sensor incremental learning on the sensor embedded representation to obtain a first-layer implicit representation of the environmental sensor data after the first-layer sensor incremental learning.

[0076] Step S411 performs first-layer sensory incremental learning on the sensory embedding representation through the first second jump connection component, aiming to extract basic and key feature information from the original environmental sensor data, laying the foundation for subsequent higher-level incremental learning.

[0077] Sensor embedding is the encoded form of environmental sensor data. It is a high-dimensional vector that contains the comprehensive information of the raw data collected from multiple sensors (such as temperature, humidity, noise decibels, etc.). Through the encoder processing, these data are converted into a format suitable for machine learning model processing to facilitate subsequent feature extraction and learning.

[0078] In the semantic enhancement generation network, the first second jump connection component plays a key role. It is similar to the first jump connection component in audio processing, but is specifically used to process environmental sensor data. This component performs preliminary processing on the sensor embedding representation through structures such as convolutional layers and pooling layers to extract the basic features of the environmental state. At the same time, the jump connection mechanism ensures the effective transmission of input information between layers, avoiding the gradient vanishing problem.

[0079] In step S411, the sound system uses the first second jump connection component to perform first-layer sensory incremental learning on the sensor embedding representation. This process mainly includes the following steps: the sensor embedding representation is fed into the first second jump connection component as input data. The convolution layer in the component performs convolution operations on the input data to extract local features; the pooling layer downsamples the feature map to reduce the amount of calculation and extract the main features. The jump connection adds the input data to the feature map processed inside the component to achieve residual learning. This step ensures the lossless transmission of input information between layers, while allowing the component to focus on learning the difference between input and output (i.e., residual). The feature map is further processed by a nonlinear activation function (such as ReLU) to enhance the expressive power of the model. After the above processing, the first second jump connection component outputs the first layer of implicit representation. This representation captures the basic characteristics of the environmental state, such as temperature range, humidity level, basic noise pattern, etc.

[0080] For example, suppose the sound system is deployed in a smart home living room, and the environmental sensors monitor the indoor temperature, humidity and noise level in real time. In step S411, the sensor embedding representation contains the encoded information of these environmental parameters. The first second jump connection component performs preliminary processing on this information:

[0081] The convolutional layer may recognize that the indoor temperature is within a comfortable range (e.g., 20°C-25°C), the humidity is moderate (e.g., 40%-60%), and the current noise level is low (e.g., less than 40 dB). The pooling layer further simplifies these feature representations and removes unnecessary details. The skip connection ensures that these basic features are effectively passed to subsequent layers. Ultimately, the first-layer implicit representation is a vector containing the above key environmental features, such as [23°C, 50%, 35dB].

[0082] Through step S411, the sound system extracts basic and key feature information from the original environmental sensor data and generates a first-level implicit representation. This representation provides a solid foundation for subsequent higher-level sensor incremental learning, helping the sound system to more accurately understand the current environmental state and make adaptive adjustment decisions accordingly.

[0083] Step S412: After performing the yth layer of sensory incremental learning on the sensory embedding representation, based on the yth saliency focusing layer in the semantic enhancement generation network, the yth layer of implicit representation after the yth layer of sensory incremental learning and the audio semantic annotation implicit representation output by the yth first jump connection component in the audio annotation extraction component are integrated to obtain the yth implicit representation integration result, y≥1.

[0084] In step S412 of the embodiment of the present application, the sound system utilizes the saliency focus layer in the semantic enhancement generation network to integrate the multi-level implicit representation of the environmental sensor data with the corresponding level implicit representation of the audio data, thereby generating a composite implicit representation that combines the information of both.

[0085] The saliency focus layer can dynamically adjust the weights of different features according to the needs of the current task, so that the model can focus on the features that are most critical to task completion. In step S412, the saliency focus layer is used to integrate the implicit representation of environmental sensor data and the implicit representation of audio semantic annotations to ensure that the integrated result can accurately reflect the comprehensive information of the two.

[0086] After performing incremental learning of the yth layer of sensors (y ≥ 1), the sound system obtains the yth layer of implicit representation, which captures the key features of the environmental sensor data at the yth level. At the same time, the yth first jump connection component in the audio annotation extraction component also outputs the yth layer of audio semantic annotation implicit representation, which contains the semantic features of the audio data at the corresponding level. Next, the sound system uses the yth saliency focus layer in the semantic enhancement generation network to integrate the two implicit representations. The integration process may include the following steps: The system concatenates the yth layer of implicit representation and the audio semantic annotation implicit representation to form a longer feature vector. This vector contains information about both environmental sensor data and audio data. Then, the saliency focus layer performs attention weighting on this concatenated feature vector. It calculates the importance weight of each feature and allocates more attention to features that are more critical to the current task. The weighted feature vector can more accurately reflect the comprehensive information of the environmental state and audio content. Finally, the system fuses the weighted feature vectors to generate the yth implicit representation integration result. This result not only contains the characteristic information of environmental sensor data, but also incorporates the semantic features of audio data, providing comprehensive data support for subsequent processing.

[0087] In step S412, the sound system uses the second saliency focus layer to integrate the two implicit representations. Through attention weighting, the system may find that the noise level and conversation atmosphere are the most critical features in the current scene recognition, so they are assigned higher weights. The integrated implicit representation contains both the actual noise level information in the room and the active emotions of the people at the party and the cheerful atmosphere of the music.

[0088] Based on this integrated implicit representation, the sound system can more accurately identify the current scene as a family gathering and adjust the sound quality and volume settings accordingly to provide a more appropriate listening experience. For example, the system may increase the bass output to enhance the rhythm of the music and appropriately increase the volume to match the lively atmosphere of the party.

[0089] Step S413: When performing the a-th layer of sensory incremental learning on the sensory embedded representation, based on the a-th second jump connection component among the multiple second jump connection components, perform the a-th layer of sensory incremental learning on the integration result of the sensory embedded representation and the b-th implicit representation output by the b-th saliency focusing layer to obtain the a-th layer of implicit representation of the environmental sensor data after the a-th layer of sensory incremental learning; wherein, a≥2, b=a+1.

[0090] After the sensory incremental learning and implicit representation integration of the first b-1 layers (where b=a+1 and a≥2), the sound system has accumulated rich fusion information of environmental features and audio features. When entering the a-th layer of sensory incremental learning, the system no longer relies solely on the original sensory embedding representation, but combines it with the b-th implicit representation integration result output by the b-th saliency focusing layer as the input for the learning of this layer.

[0091] As the original encoding form of environmental sensor data, the sensor embedding representation contains information on multiple physical quantities such as temperature, humidity, and noise level. The bth implicit representation integration result is obtained by integrating the implicit representation of the b-1th layer sensor incremental learning and the audio semantic annotation implicit representation output by the audio annotation extraction component through the saliency focus layer in the previous layer (b-1th layer). It combines environmental features and audio features and reflects the comprehensive information of the current scene.

[0092] The ath second jump connection component receives the above two types of input data and further processes them through its internal structure (such as convolutional layer, pooling layer, etc.). This component not only extracts the unique environmental features of the ath layer, but also maintains the integrity of the input information through the jump connection mechanism to avoid the gradient vanishing problem. During the processing, the component may learn how to better fuse environmental features with audio features to generate a more representative ath layer implicit representation. After being processed by the ath second jump connection component, the sound system obtains the ath layer implicit representation after the ath layer sensor incremental learning. This representation not only captures the key features of the environmental sensor data at the ath level, but also contains the deep fusion results of audio and environmental information in all previous levels. It provides richer and more accurate data support for subsequent scene recognition and audio adaptive adjustment.

[0093] When entering the third layer, the sound system further processes the sensor embedding representation by combining the implicit representation integration results output by the fourth saliency focus layer (which may emphasize the most prominent sources of ambient noise and the corresponding audio activity types in the current scene). The third second-jump connection component extracts more detailed environmental features (such as the frequency distribution of specific noise) through its internal mechanism and integrates them with the audio features at a deeper level. The final generated third-layer implicit representation not only accurately reflects the subtle changes in the current environment, but is also closely connected to the emotional expression, rhythm changes and other characteristics of the audio content.

[0094] Based on this more precise and comprehensive third-layer implicit representation, the sound system can more accurately identify the activity type of the current scene (such as family gatherings, watching movies, etc.) and adjust the sound quality and volume settings accordingly to provide an auditory experience that is more in line with the needs of the scene.

[0095] Through the implementation of step S413, the sound system combines the previous implicit representation integration result with the current layer of sensor embedding representation to perform deeper environmental sensor incremental learning. This process not only enhances the system's ability to understand the complexity of the environment, but also provides a solid data foundation for the subsequent deep integration of audio and environment.

[0096] Step S414: taking the implicit representation obtained after the last layer of the multi-level sensing incremental learning as the integrated implicit representation obtained after the multi-level sensing incremental learning.

[0097] The integrated implicit representation is the final product of the sound system's multi-level and multi-dimensional analysis and understanding of environmental sensor data. It not only contains the basic characteristics of the environmental state (such as temperature, humidity, noise level, etc.), but also incorporates the interaction and fusion information of these characteristics at different levels. Since the multi-level sensor incremental learning and multi-level audio incremental learning are consistent in the number of levels, the integrated implicit representation is highly consistent with the audio semantic annotation implicit representation in terms of feature dimensions, which facilitates the subsequent deep fusion of the two.

[0098] After obtaining the integrated implicit representation, the sound system will further use this achievement to deeply integrate audio and the environment. By combining it with the implicit representation of audio semantic annotations obtained by multi-level audio incremental learning, the system can generate audio with sound quality adjustment that is consistent with the audio content and adapts to the environmental state. For example, in a family gathering scene, the system may automatically increase the bass component in the audio and adjust the equalizer settings based on the noise level characteristics in the integrated implicit representation to enhance the rhythm and immersion of the music; at the same time, based on the characteristics of frequent personnel activities, the volume is appropriately adjusted to ensure that everyone can hear the music and conversation clearly.

[0099] As an embodiment, the multiple second jump connection components include one or more second jump connection components for performing dimensionality reduction operations on the sensor embedded representation and one or more second jump connection components for performing interpolation processing on the sensor embedded representation; the number of second jump connection components for performing dimensionality reduction operations on the sensor embedded representation is consistent with the number of second jump connection components for performing interpolation processing on the sensor embedded representation; at the same time, the one or more second jump connection components for performing interpolation processing on the sensor embedded representation are located behind the one or more second jump connection components for performing dimensionality reduction operations on the sensor embedded representation.

[0100] In the embodiment of the present application, the second jump connection component plays a vital role as an important part of the semantic enhancement generation network. These components are not only responsible for extracting and converting key features in environmental sensor data, but also optimize feature representation through specific operations (such as dimensionality reduction and interpolation) to adapt to different levels of sensor incremental learning needs. The following is a detailed explanation of the implementation method of the second jump connection component that includes dimensionality reduction and interpolation processing functions, which is explained in conjunction with specific application scenarios. Dimensionality reduction aims to reduce the complexity of data by reducing the number of features while retaining key information in the original data as much as possible. In the embodiment of the present application, the second jump connection component of the dimensionality reduction operation is responsible for preliminary feature compression of the sensor embedding representation.

[0101] The second jump connection component of the dimensionality reduction operation processes the sensor embedding representation through structures such as convolutional layers and pooling layers, gradually reducing the feature dimension. For example, the convolutional layer can use multiple convolution kernels to extract local features from the sensor embedding representation, while the pooling layer reduces the resolution of the feature map by sampling the maximum value or average value of the local area, thereby achieving a dimensionality reduction effect. After the dimensionality reduction process, the redundant information and noise in the sensor embedding representation are effectively removed, and the key environmental features are retained, providing a more compact and robust feature representation for subsequent feature extraction and fusion.

[0102] Interpolation is a mathematical method used to estimate the value of unknown data points based on known data points. In the scene recognition of smart speakers, the second jump connection component of interpolation is responsible for interpolation operations in the feature space after dimensionality reduction to restore or enhance certain key features and ensure the continuity and integrity of feature representations at different levels.

[0103] There are many ways to implement interpolation, the most common ones are linear interpolation, polynomial interpolation, spline interpolation, etc. In the audio system, these interpolation methods may be embedded in the neural network layer to achieve adaptive interpolation through learnable parameters. For example, a fully connected layer can receive the reduced feature representation as input and generate the interpolated feature vector by weighted summation. The weight parameters here are optimized during the training process to minimize the interpolation error and retain key features. The second jump connection component of the interpolation processing is located after the component of the dimensionality reduction operation. They use the dimensionality reduction result as input and restore or enhance the features that may be lost in the dimensionality reduction process through interpolation operations. This process not only helps to maintain the continuity of feature representations between different levels, but also helps to improve the sensitivity and response speed of the audio system to environmental changes.

[0104] In the audio system, the second jump connection component of the dimensionality reduction operation and the second jump connection component of the interpolation processing work together to optimize the feature space of the sensor embedding representation. The dimensionality reduction operation reduces the computational complexity and noise interference by reducing the number of features; while the interpolation processing maintains the continuity and integrity of the feature representation by restoring and enhancing key features. This synergy enables the audio system to more efficiently extract and integrate the feature information of environmental sensor data at different levels. For example, when the audio system is performing family party scene recognition, it first processes the sensor embedding representation through the second jump connection component of the dimensionality reduction operation to extract key environmental features such as temperature and noise level and reduce the feature dimension. Then, in the interpolation processing component, the system uses these reduced dimensionality features for interpolation operations to restore or enhance features closely related to the party atmosphere (such as the density of personnel activities, the intensity of conversation sounds, etc.). The final feature representation retains both key environmental information and continuity, providing strong data support for subsequent audio adaptive adjustment.

[0105] Through this synergy, the sound system can more accurately identify the current scene and make appropriate audio adjustment decisions. For example, in a family gathering scene, the system may automatically adjust the volume based on the restored and enhanced conversation sound intensity characteristics to ensure that every participant can hear the conversation clearly; at the same time, adjust the sound effect settings based on the characteristics of the density of human activities to create a more active and immersive party atmosphere.

[0106] In summary, the second jump connection component including the dimensionality reduction and interpolation processing functions plays an important role in the embodiments of the present application. They improve the sensitivity and response speed of the sound system to environmental changes by optimizing the feature space of the sensor embedding representation; while maintaining the continuity and integrity of the feature representation at different levels. This design not only enhances the scene recognition capability of the sound system, but also improves the accuracy and intelligence level of its audio adaptive adjustment.

[0107] As an implementation manner, in step S411, before performing first-layer sensory incremental learning on the sensory embedding representation, the method may further include:

[0108] Step S411A: generating a generative implicit representation of the semantic enhancement generative network based on a preset random distribution sampling algorithm;

[0109] Step S411B: loading the generated implicit representation and the sensory embedded representation as implicit representations to be loaded into the semantic enhancement generation network;

[0110] The step S411, based on the first second jump connection component among the multiple second jump connection components, performs first-layer sensor incremental learning on the sensor embedding representation to obtain the first-layer implicit representation of the environmental sensor data after the first-layer sensor incremental learning, may specifically include:

[0111] Step S4111: Based on the first second jump connection component, perform first-layer sensory incremental learning on the sensory embedded representation and the generated implicit representation to obtain the first-layer implicit representation.

[0112] Before officially starting the first-layer sensor incremental learning, the audio system performs a pre-step S411A. The purpose of this step is to introduce a certain degree of randomness to help the model jump out of the local optimal solution in the early stage of training and explore a wider solution space. Specifically, the audio system uses a preset random distribution sampling algorithm to generate a random feature, that is, to generate an implicit representation.

[0113] For the random distribution sampling algorithm, the sound system can use Gaussian distribution (normal distribution) as the basis of the random distribution sampling algorithm. It first sets the parameters of the Gaussian distribution (mean μ and standard deviation σ), and then generates a series of random numbers based on these parameters. These random numbers form a vector, that is, the generated implicit representation. For example, if μ=0 and σ=1 are set, the generated random numbers will fluctuate around the value of 0, forming a distribution that is roughly a bell-shaped curve. In machine learning, the introduction of randomness helps prevent the model from falling into overfitting and improves the generalization ability of the model. Generating implicit representations as a "noise" signal, input into the model together with the sensor embedding representation, can prompt the model to learn more robust feature representations.

[0114] After generating the generative implicit representation, the sound system executes step S411B to load the generative implicit representation and the sensory embedded representation as implicit representations to be loaded into the semantic enhancement generative network. This step provides initial input data for subsequent sensory incremental learning.

[0115] The sensor embedding representation is an encoded form of environmental sensor data, which contains comprehensive information of raw data (such as temperature, humidity, noise level, etc.) collected from various sensors. After being processed by the encoder, these raw data are converted into a high-dimensional vector, which is convenient for machine learning models to process and analyze. During the loading process, the audio system concatenates the generated implicit representation and the sensor embedding representation into a longer vector. This concatenated vector contains both randomness (generated implicit representation) and determinism (sensor embedding representation) information, providing rich input data for subsequent incremental learning. After completing the pre-step, the audio system enters step S411, and performs first-layer sensor incremental learning on the sensor embedding representation and the generated implicit representation based on the first second jump connection component in the semantic enhancement generation network. This process aims to extract the basic features of environmental sensor data from the input data and generate the first-layer implicit representation.

[0116] In step S4111, the sound system uses the first second jump connection component to perform convolution, pooling and other operations on the concatenated input vector (including the sensor embedding representation and the generated implicit representation) to extract basic features and maintain the integrity of the information through the residual learning mechanism. Finally, the component outputs the first layer of implicit representation.

[0117] In the first second skip connection component, the convolution layer performs a sliding window operation on the input vector through a series of convolution kernels to extract local features. These convolution kernels can be regarded as feature detectors that can identify specific patterns in the input vector (such as trends in temperature changes, fluctuations in noise levels, etc.). The pooling layer usually follows the convolution layer and is used to reduce the resolution of the feature map (i.e., dimensionality reduction). The pooling operation can reduce the amount of computation and extract the main features while maintaining the invariance of the features (such as translation invariance). In the first-layer sensor incremental learning, the pooling operation helps to remove redundant information and retain key features. The first second skip connection component implements residual learning through skip connections. This means that part (or all) of the input vector will be directly passed to the subsequent layers and added to the feature map after convolution and pooling. This design helps to alleviate the gradient vanishing problem in deep networks and accelerate the convergence of the model. After convolution, pooling, and residual learning, the first second skip connection component outputs the first-layer implicit representation. This representation is a low-dimensional vector that captures the basic features of the environmental sensor data at the first layer. These features may include temperature range, humidity level, basic noise patterns, etc., providing a basis for subsequent higher-level incremental learning.

[0118] For example, suppose that the sound system is deployed in a smart home living room, and the environmental sensors monitor the indoor temperature, humidity, and noise level in real time. In step S411A, the sound system generates a generative implicit representation using a Gaussian distribution sampling algorithm, such as [0.1, -0.3, 0.2]. At the same time, the sensor embedding representation may be a high-dimensional vector containing the encoded information of temperature, humidity, and noise level, such as [22°C, 50%, 45dB] after encoder conversion.

[0119] In step S411B, the sound system concatenates the two vectors into a longer vector, such as [22°C encoding, 50% encoding, 45dB encoding, 0.1, -0.3, 0.2]. This concatenated vector is then loaded into the first second jump connection component in the semantic enhancement generation network.

[0120] In step S4111, the first second jump connection component performs a convolution operation on the concatenated vector. Assume that there is a 3×1 convolution kernel sliding on the vector. Each slide will calculate the dot product of the convolution kernel and the coverage area, and add a bias term (the bias term is omitted here for simplicity). After convolution, a new feature map is obtained. Then, the pooling layer performs a maximum pooling operation on this feature map and selects the maximum value in each pooling area as the output. Finally, a part of the input vector is added to the pooled feature map through the residual learning mechanism to obtain the first-layer implicit representation.

[0121] Assume that after a series of processing, the first-level implicit representation is [Feature 1, Feature 2, ..., Feature N]. This representation captures the basic features of the living room environment on the first level, such as "the current room temperature is moderate, the humidity is comfortable, and the noise level is low." These features will serve as input data for incremental learning in subsequent levels, helping the sound system to understand the current environmental state more deeply.

[0122] Through the detailed explanation and example of step S411 and its preceding steps S411A and S411B, we can see how the sound system uses randomness introduction, data splicing and multi-layer perception technology to perform first-layer sensor incremental learning. This process not only enhances the sound system's basic understanding of the environmental state, but also provides a solid foundation for subsequent higher-level feature extraction and fusion.

[0123] As an implementation method, the feature dimension of the implicit representation of the yth layer is consistent with the feature dimension of the implicit representation of the audio semantic annotation output by the yth first jump connection component. Based on this, the step S412, based on the yth saliency focus layer in the semantic enhancement generation network, performs the implicit representation integration on the yth layer implicit representation after the yth layer sensor incremental learning and the audio semantic annotation implicit representation output by the yth first jump connection component in the audio annotation extraction component to obtain the yth implicit representation integration result, which may specifically include:

[0124] Step S4121: Based on the yth saliency focus layer in the semantic enhancement generation network, the yth layer implicit representation and the audio semantic annotation implicit representation output by the yth first skip connection component are fused in different feature dimensions to obtain a fused implicit representation;

[0125] Step S4122: performing a dimensionality reduction operation on the fused implicit representation to obtain a reduced-dimensional implicit representation;

[0126] Step S4123: filtering the dimension-reduced implicit representation to obtain an array of influence coefficients between the y-th layer implicit representation and the audio semantic annotation implicit representation output by the y-th first skip connection component;

[0127] Step S4124: performing implicit representation integration on the fused implicit representation according to the influence coefficient array to obtain the yth implicit representation integration result.

[0128] In the embodiment of the present application, step S412 is a step to achieve deep fusion of audio and environmental data. This process integrates the implicit representation of the yth layer obtained by sensor incremental learning with the implicit representation of the yth audio semantic annotation output by the audio annotation extraction component through the saliency focus layer in the semantic enhancement generation network, thereby generating a composite implicit representation that combines the information of both.

[0129] In the process of multi-level sensor incremental learning and multi-level audio incremental learning, the sound system generates multi-level implicit representations of environmental sensor data and audio data respectively. These implicit representations capture the characteristic information of their respective fields at different levels. In order to achieve deep fusion of audio and environmental data, the sound system needs to integrate the implicit representations at each corresponding level. Step S412 ensures the consistency of the implicit representation of the y-th layer and the implicit representation of the y-th audio semantic annotation in the feature dimension, and generates the integrated implicit representation through a series of operations.

[0130] Before performing implicit representation integration, the sound system ensures that the y-th layer implicit representation is consistent with the y-th audio semantic annotation implicit representation in terms of feature dimensions. This requirement allows the two implicit representations to be compared and fused in the same dimensional space. Assume that at the third layer (i.e., y=3), the environmental sensor data generates a third layer of implicit representation after sensor incremental learning, which contains information about environmental features such as temperature, humidity, and noise level. At the same time, the third first jump connection component in the audio annotation extraction component outputs the third audio semantic annotation implicit representation, which captures the characteristics of the audio signal in terms of rhythm, melody, emotion, etc. Since the two are consistent in feature dimensions (for example, both are 128-dimensional vectors), the sound system can splice and fuse them.

[0131] Concatenation fusion is a simple feature fusion method that concatenates two implicit representations in the feature dimension to form a longer feature vector. For example, if the third layer implicit representation is [env_feat1,env_feat2,...,env_feat128] and the third audio semantic annotation implicit representation is [audio_feat1,audio_feat2,...,audio_feat128], then the fused implicit representation is [env_feat1,env_feat2,...,env_feat128,audio_feat1,audio_feat2,..., audio_feat128]. This fused implicit representation contains information about both environmental features and audio features.

[0132] Step S4122 performs a dimensionality reduction operation. Directly processing the fused implicit representation may result in excessive computation or feature redundancy. Therefore, the sound system usually performs a dimensionality reduction operation on the fused implicit representation to extract the main features and reduce the data dimension.

[0133] Continuing with the third layer as an example, the sound system may perform dimensionality reduction on the fused 256-dimensional feature vector. There are many ways to reduce dimensionality, the most common of which are max pooling and average pooling. Max pooling selects the maximum value in each pooling window as the output, while average pooling calculates the average of all values ​​in the pooling window as the output. Assume that the sound system uses max pooling for dimensionality reduction, and the pooling window size is 2 and the step size is 2. Then, it will slide the window on the fused implicit representation, and select the maximum value of two adjacent features as the output each time. After dimensionality reduction, the feature dimension is reduced from 256 dimensions to 128 dimensions, and a reduced dimensional implicit representation is obtained. It should be noted that although only max pooling and average pooling are mentioned here to simplify the explanation, in fact the sound system may use more complex dimensionality reduction techniques, such as principal component analysis (PCA), linear discriminant analysis (LDA), etc.

[0134] Step S4123 involves filtering and generating an array of influence coefficients. Although the dimension of the implicit representation after dimensionality reduction is reduced, it still contains a large amount of feature information. In order to further clarify which features have a greater impact on the integration result, the sound system needs to filter the reduced-dimensional implicit representation and generate an array of influence coefficients. Filtering can be regarded as a process of feature selection or weighting. The sound system may use a convolutional layer or a fully connected layer (Dense Layer) in a convolutional neural network (CNN) to filter the reduced-dimensional implicit representation. These layers perform weighted summation operations on the input features through convolution kernels or weight matrices and output new feature representations. In this process, the filtering operation is actually calculating the importance weight of each feature. The sound system can organize these weights into a two-dimensional array (i.e., an array of influence coefficients), each element of which represents the importance of the corresponding feature in the integration result. Assume that the reduced-dimensional implicit representation is [feat1', feat2', ..., feat128'].

[0135] In step S4124, implicit representation integration is performed. The sound system performs implicit representation integration on the fused implicit representation according to the influence coefficient array to generate the yth implicit representation integration result. After obtaining the influence coefficient array, the sound system applies it to the fused implicit representation (or the reduced dimensional implicit representation) and generates the integrated implicit representation by weighted summation or the like. This process can be regarded as a feature reorganization and optimization process, which makes the integrated implicit representation more in line with the needs of the current scene.

[0136] Assume that the fusion is implicitly represented as [feat_fused 1 , feat_fused 2 , ..., feat_fused 256] (here we still use the dimensions before fusion for the sake of convenience), the influence coefficient array has been applied to the implicit representation after dimensionality reduction in some way (in fact, it is more common to apply it directly to the fused implicit representation before dimensionality reduction). Then the integrated implicit representation may be calculated as follows: ;in, represents the i-th eigenvalue in the implicit representation after integration, Represents the influence weight of the j-th fusion feature on the i-th integrated feature.

[0137] Through the detailed explanation of the implementation method of step S412 and the example of application scenarios, we can see how the sound system achieves deep fusion of audio and environmental data through a series of complex operations such as saliency focus layer, feature fusion, dimensionality reduction operation, filtering processing and implicit representation integration. This process not only enhances the sound system's ability to understand complex scenes, but also provides comprehensive data support for its subsequent adaptive adjustment. At the same time, it should be noted that in actual applications, these operations may be adjusted and optimized according to specific needs and technical conditions to achieve the best results.

[0138] As an implementation method, the intelligent audio scene recognition and adaptive adjustment method is performed based on a pre-debugged audio generation network, wherein the audio generation network includes a first subnetwork and a second subnetwork; the first subnetwork includes a sensor data embedding component and a semantic enhancement generation network, and the second subnetwork includes an audio embedding component and an audio annotation extraction component.

[0139] The embodiment of the present application also includes a calibration process of the audio generation network, which may specifically include:

[0140] Step S10: loading the audio training data in the training sample into the second sub-network, and performing audio embedding representation on the audio training data based on the audio embedding component in the second sub-network to obtain a sample audio embedding representation;

[0141] Step S20: performing multi-level audio incremental learning on the sample audio embedding representation based on the audio annotation extraction component in the second sub-network, and correspondingly obtaining implicit representations of sample audio semantic annotations of the sample target audio track in the audio training data at multiple levels;

[0142] Step S30: loading the environmental sensor training data in the training sample into the first sub-network, and performing sensor embedding representation on the environmental sensor training data based on the sensor data embedding component in the first sub-network to obtain a sample sensor embedding representation;

[0143] Step S40: performing multi-level sensory incremental learning on the sample sensory embedding representations one by one based on the semantic enhancement generation network in the first sub-network to obtain the sample integrated implicit representation;

[0144] Step S50: performing error determination on the audio generation network according to the implicit representation of the sample audio semantic annotation and the implicit representation of the sample integration to obtain a calibration error of the audio generation network;

[0145] Step S60: adjusting network parameters in the audio generation network according to the calibration error to obtain a calibrated audio generation network.

[0146] In the embodiment of the present application, the audio generation network consists of two main parts: a first sub-network (responsible for processing environmental sensor data) and a second sub-network (responsible for processing audio data). The first sub-network includes a sensor data embedding component and a semantic enhancement generation network, while the second sub-network includes an audio embedding component and an audio annotation extraction component. This design allows the network to process audio and environmental sensor data simultaneously to achieve more intelligent scene recognition and audio adaptive adjustment.

[0147] During the calibration process, step S10 loads audio training data and performs audio embedding representation.

[0148] For example, suppose there is a set of audio training data containing multiple types of music, such as classical music, pop music, jazz, etc. These audio data will be used to train the audio generation network so that it can recognize different types of music and make corresponding adjustments. In step S10, the sound system loads the audio training data into the second sub-network. The audio embedding component in the second sub-network processes the audio data and converts it into an audio embedding representation. The audio embedding representation is a high-dimensional vector that captures the key features of the audio signal, such as rhythm, timbre, melody, etc. For example, suppose that a piece of audio training data of classical music is loaded into the network, and the audio embedding component processes it through a model such as a convolutional neural network (CNN) or a recurrent neural network (RNN) to generate a fixed-length audio embedding vector, such as [0.2, -0.1, 0.3,...], which contains the key feature information of classical music.

[0149] In step S20, multi-level audio incremental learning is performed to obtain the audio semantic annotation implicit representation. For example, after obtaining the audio embedding representation, the sound system needs to further extract the deep semantic features of the audio data. These features will help the system better understand the audio content and provide a basis for subsequent scene recognition and audio adjustment. In step S20, the audio annotation extraction component in the second subnetwork performs multi-level audio incremental learning on the sample audio embedding representation. This process is similar to the hierarchical feature extraction in deep neural networks, which extracts the semantic features of audio signals layer by layer through multiple convolutional layers, pooling layers, fully connected layers and other structures. For example, taking RNN as an example, the audio annotation extraction component may contain multiple LSTM (long short-term memory network) layers. Each LSTM layer receives the output of the previous layer as input and processes the audio embedding representation through its internal forget gate, input gate and output gate mechanism. As the layers go deeper, each LSTM layer extracts more abstract and advanced audio semantic features and finally generates the sample audio semantic annotation implicit representation. These representations may include advanced features such as music emotions, styles, and rhythm patterns.

[0150] In step S30, the environmental sensor training data is loaded and the sensor embedding representation is performed. For example, in a smart home environment, the sound system not only needs to process audio data, but also needs to perceive and understand the environmental state (such as temperature, humidity, noise level, etc.) in real time. These environmental state information is crucial for scene recognition and audio adaptive adjustment. In step S30, the sound system loads the environmental sensor training data in the training sample into the first sub-network. The sensor data embedding component in the first sub-network processes these environmental sensor data and converts them into sensor embedding representation. The sensor embedding representation is also a high-dimensional vector that captures the key features of the environmental state. For example, assume that the environmental sensor training data contains information in three dimensions: temperature, humidity, and noise level. The sensor data embedding component may process these raw data through a series of fully connected layers or convolutional layers to generate a sensor embedding vector containing all environmental feature information, such as [25°C encoding, 60% encoding, 45dB encoding] (here, the encoding representation is used to simplify the explanation, and it may be a specific value or vector in practice).

[0151] In step S40, multi-level sensory incremental learning is performed to obtain sample integrated implicit representations. For example, after obtaining the sensory embedding representation, the sound system needs to further process this data in order to deeply fuse it with the audio data. Multi-level sensory incremental learning is a key step to achieve this goal. In step S40, the semantic enhancement generation network in the first sub-network processes the sample sensory embedding representations one by one. Through multiple second jump connection components (similar to the residual block in ResNet) and a saliency focus layer (such as an attention layer), the network extracts and enhances the feature representation of the environmental sensor data layer by layer. At the same time, through the implicit representation integration mechanism (such as feature concatenation, weighted summation, etc.), the network deeply fuses the environmental features with the audio features at the corresponding level. For example, it is assumed that the semantic enhancement generation network contains three second jump connection components and a saliency focus layer. In the first jump connection component, the network preliminarily processes the sensory embedding representation and extracts basic environmental features; in the second component, the network further extracts high-level environmental features and integrates them with the audio features output by the first saliency focus layer; in the third component, the network finally generates the sample integrated implicit representation. This representation contains information about both environmental features and audio features.

[0152] In step S50, error determination is performed to obtain a calibration error. For example, after completing the incremental learning of audio and sensor data, the sound system needs to evaluate the performance of the current audio generation network and calibrate the network according to the evaluation results. Error determination is a key step in this process. In step S50, the sound system calculates the calibration error of the audio generation network according to the implicit representation of the sample audio semantic annotation and the implicit representation of the sample integration. This process may involve a variety of error calculation methods, such as mean square error (MSE), cross entropy loss, etc. For example, assume that the system uses MSE as an error calculation method. First, the system performs a reduction mapping (such as through a decoder) on the implicit representation of the sample integration to generate sound quality adjusted audio; then the MSE value between the audio and the original audio training data is calculated as the initial calibration error; at the same time, a certain similarity measure (such as cosine similarity) between the implicit representation of the sample audio semantic annotation and the integrated implicit representation is calculated as the audio semantic annotation calibration error; finally, the two are combined to obtain the final calibration error.

[0153] It should be noted that the calculation formula of MSE is not given directly here because in practical applications MSE is calculated for specific data, but the general form of MSE can be expressed as: ;in is the actual value (such as a characteristic value of the original audio), is the predicted value (such as the corresponding feature value of the sound quality adjusted audio), and n is the number of samples.

[0154] In step S60, the network parameters are adjusted according to the calibration error. For example, after obtaining the calibration error, the sound system adjusts the parameters in the audio generation network to optimize the network performance. This process is achieved through optimization methods such as back propagation algorithm and gradient descent. In step S60, the sound system adjusts the weights, biases and other parameters in the audio generation network according to the calibration error. Through multiple iterations of training until the calibration error is reduced to an acceptable range. For example, assume that the system uses the gradient descent method to update the network parameters. In each iteration, the system calculates the calibration error of the current network output and then calculates the gradient value of each parameter according to the error back propagation algorithm; then updates the value of each parameter according to the learning rate (a hyperparameter) and the gradient value; finally, repeat the above process until the stop condition is met (such as reaching the maximum number of iterations or the calibration error is less than a certain threshold). Through the above detailed explanation and example of the audio generation network calibration process of the intelligent audio scene recognition and adaptive adjustment method, it can be seen that this process involves multiple complex steps and components including audio embedding, multi-level incremental learning, sensor embedding, implicit representation integration, error determination and parameter adjustment. These steps and components work together to enable the sound system to achieve deep integration and understanding of audio and environmental data, thereby realizing intelligent scene recognition and audio adaptive adjustment functions.

[0155] As an implementation manner, the step S50, determining the error of the audio generation network according to the implicit representation of the sample audio semantic annotation and the implicit representation of the sample integration, to obtain the calibration error of the audio generation network, may specifically include:

[0156] Step S51: Determine the initial error of the audio generation network according to the sample integrated implicit representation and the audio training data to obtain an initial calibration error;

[0157] Step S52: determining the audio semantic annotation error of the audio generation network according to the sample audio semantic annotation implicit representation and the sample integrated implicit representation, and obtaining the audio semantic annotation adjustment error;

[0158] Step S53: taking the sum of the initial calibration error and the audio semantic annotation calibration error as the calibration error of the audio generation network.

[0159] In the embodiment of the present application, when executing step S50, the sound system evaluates the performance of the audio generation network based on the implicit representation of the sample audio semantic annotation (representing the deep understanding of the audio content) and the implicit representation of the sample integration (comprehensive feature representation that integrates audio and environmental data), and calculates the corresponding calibration error. This error will guide the subsequent network parameter adjustment process.

[0160] In step S51, the initial error is determined. The sound system calculates the initial calibration error based on the sample integrated implicit representation and the original audio training data. The purpose of this step is to evaluate the ability of the audio generation network to restore the original audio signal. For example, the sound system restores and maps the sample integrated implicit representation through a decoder to generate a sound quality adjusted audio. This audio is an approximate reconstruction of the original audio signal by the audio generation network based on the current network parameters. The calculated MSE value is the initial calibration error, which reflects the accuracy of the audio generation network in restoring the original audio signal. The smaller the error, the stronger the reconstruction ability of the network. For example, suppose that a certain sample value sequence in the original audio training data is ([0.1, 0.2, 0.3, ..., 0.9]), and the corresponding sample value sequence in the sound quality adjusted audio is ([0.11, 0.19,0.31, ..., 0.88]). After substituting these values ​​into the MSE formula for calculation, the initial calibration error obtained will be used for subsequent network parameter adjustment.

[0161] In step S52, the audio semantic annotation error is determined. In order to evaluate the ability of the audio generation network in retaining audio semantics, the sound system calculates the audio semantic annotation calibration error based on the sample audio semantic annotation implicit representation and the sample integrated implicit representation.

[0162] First, the audio system restores the implicit representation of the sample audio semantic annotations and the implicit representation of the sample integration separately (if they are not already in a directly comparable form). This step may involve passing it through different decoders or mapping functions to produce comparable audio feature representations.

[0163] Similarity measurement: Then, the sound system calculates the similarity between the two restored audio feature representations. Similarity measurement can be done using a variety of methods such as cosine similarity, Euclidean distance, Manhattan distance, etc. Taking cosine similarity as an example, its calculation formula is: ; Among them, A and B are two restored audio feature representation vectors, and n is the dimension of the vector.

[0164] Since cosine similarity is a value between -1 and 1, the sound system usually needs to convert it into an error form so that it can be compared and accumulated with other error terms of the network. A simple method is to subtract cosine similarity from 1 to get a new error term (note that it is assumed that the higher the cosine similarity, the smaller the error). However, it is more common to directly use negative cosine similarity as the error term (because the optimization algorithm usually minimizes the error term). The calculated audio semantic annotation calibration error reflects the ability of the audio generation network to preserve the audio semantics. The smaller the error, the stronger the reconstruction ability of the network at the semantic level. For example, suppose that the audio feature vector obtained after the implicit representation of the sample audio semantic annotation is restored is ([0.5, -0.3, 0.2]), and the audio feature vector obtained after the sample integrated implicit representation is restored is ([0.48, -0.28, 0.19]). After substituting these values ​​into the cosine similarity formula (the vector normalization step is omitted here for simplicity), the obtained similarity value will be used to calculate the audio semantic annotation calibration error. If negative cosine similarity is directly used as the error term, the error value will be used for subsequent network parameter adjustment.

[0165] In step S53, a comprehensive calculation of the calibration error is performed. The sound system performs a weighted sum (or a simple sum if the two are already on a comparable scale) of the initial calibration error (such as MSE) and the audio semantic annotation calibration error (such as negative cosine similarity) to obtain the calibration error. The choice of weights can be determined according to specific application scenarios and requirements. For example:

[0166] ;

[0167] Among them, α and β are weight coefficients used to balance the importance of initial training error and audio semantic annotation training error.

[0168] Through the detailed explanation and example of the implementation of step S50, we can see how the sound system comprehensively evaluates the performance of the audio generation network by calculating the initial calibration error and the audio semantic annotation calibration error. These two error terms consider the network from two aspects: the accuracy of signal restoration and the degree of retention of audio semantics, and finally obtain the calibration error by weighted summation. This calibration error will serve as the basis for subsequent network parameter adjustment to help the sound system continuously optimize its scene recognition and adaptive adjustment capabilities.

[0169] As an implementation manner, the step S51, according to the sample integrated implicit representation and the audio training data, determines the initial error of the audio generation network to obtain the initial calibration error, which may specifically include:

[0170] Step S511: performing restoration mapping on the sample integrated implicit representation to obtain a sample sound quality adjusted audio that fuses the track implicit representation of the sample target track and the environment implicit representation of the environment sensor training data;

[0171] Step S512: taking the mean square error between the sample sound quality adjusted audio and the audio training data as the initial calibration error.

[0172] In step S511, the sound system performs restoration mapping on the sample integrated implicit representation. Restoration mapping is an inverse process, which uses a decoder (or generator) network corresponding to the audio generation network to convert the integrated implicit representation back to the audio signal space, thereby generating a sound quality adjusted audio. The sound quality adjusted audio not only contains the track implicit representation of the sample target track (i.e., the restoration of the audio content), but also incorporates the environment implicit representation of the environment sensor training data (i.e., the reflection of the environment information).

[0173] The decoder network is a component of the audio generation network, which is responsible for converting the implicit representation (here, the sample integrated implicit representation) back into an audible audio signal. The decoder network may contain multiple upsampling layers, deconvolution layers, and possible fully connected layers. Through the combined operation of these layers, the low-dimensional implicit representation is gradually expanded to a high-dimensional audio signal. In the process of restoring the mapping, the decoder network not only focuses on restoring the audio signal itself, but also makes appropriate adjustments to the restored audio signal based on the environmental information in the integrated implicit representation. This adjustment may be fine-tuning of the volume, timbre, balance, etc. to adapt to different environmental scenarios.

[0174] After the sample sound quality adjusted audio is generated in step S511, the sound system needs to evaluate the difference between the audio and the original audio training data. This evaluation process is completed by calculating the mean square error (MSE) between the two.

[0175] Mean Squared Error (MSE) is a commonly used loss function that measures the difference between the predicted value and the true value. In the process of tuning the audio generation network, MSE is used as an important indicator to evaluate the network's restoration ability. The MSE value is obtained by calculating the square of the difference between the sample sound quality adjusted audio and the original audio training data at each sampling point, and then averaging it.

[0176] First, you need to ensure that the sample sound quality adjustment audio and the original audio training data are aligned in time, that is, the sampling rate, number of sampling points and other parameters of the two need to be consistent. For each corresponding sampling point in the sample sound quality adjustment audio and the original audio training data, calculate their difference. Square the difference of each sampling point, and then sum the squared differences of all sampling points. Divide the sum by the total number of sampling points (that is, the length of the audio) to obtain the mean square error MSE. The calculation formula for MSE is as follows:

[0177] ;

[0178] Where N is the total number of sampling points in the audio signal, is the value of the i-th sampling point in the original audio training data, is the value of the i-th sampling point corresponding to the sample sound quality adjustment audio.

[0179] Through the detailed explanation and example illustration of the implementation method of step S51, this process not only helps to understand the network performance under the current network parameter settings, but also provides an important reference for subsequent network parameter adjustments.

[0180] As an implementation manner, the step S52, according to the sample audio semantic annotation implicit representation and the sample integrated implicit representation, determines the audio semantic annotation error of the audio generation network to obtain the audio semantic annotation adjustment error, which may specifically include:

[0181] Step S521: performing restoration mapping on the implicit representation of the semantic annotation of the sample audio to obtain restored audio;

[0182] Step S522: performing restoration mapping on the sample integrated implicit representation to obtain a sample sound quality adjusted audio that fuses the track implicit representation of the sample target track and the environment implicit representation of the environment sensor training data;

[0183] Step S523: performing track identification and embedding on the restored audio and the sample sound quality adjusted audio respectively, and obtaining a first embedded implicit representation and a second embedded implicit representation accordingly;

[0184] Step S524: taking the feature distance between the first embedded implicit representation and the second embedded implicit representation as the audio semantic annotation calibration error.

[0185] In the embodiment of the present application, step S52 plays a key role in evaluating the ability of the audio generation network to retain audio semantic information. By calculating the audio semantic annotation calibration error, the sound system can understand whether the current network can still accurately retain the semantic content of the original audio after fusing the environmental sensor data with the audio data. The following is a detailed explanation of the implementation method of step S52, which is explained in conjunction with specific application scenarios and supplemented by examples to assist understanding.

[0186] In step S521, the sound system performs restoration mapping on the sample audio semantic annotation implicit representation to generate restored audio. The purpose of this step is to convert the high-dimensional implicit representation back into an audible audio signal, so as to directly evaluate the ability of these implicit representations in preserving audio semantics.

[0187] The sound system uses a decoder network corresponding to the audio annotation extraction component to perform the restoration mapping. This decoder network is trained to learn the inverse mapping relationship from the implicit representation to the audio signal. The decoder network receives the implicit representation of the sample audio semantic annotation as input and gradually converts it back to the audio waveform through a series of upsampling, deconvolution and other operations. In this process, the network attempts to reconstruct the key semantic features in the original audio signal.

[0188] In step S522, the sound system continues to restore the sample integrated implicit representation to generate the sample sound quality adjusted audio. The sample integrated implicit representation is a comprehensive feature vector that combines audio and environmental data, and it includes the track implicit representation of the target track and the environment implicit representation of the environmental sensor training data. Similar to step S521, the sound system uses another decoder network (for example, a decoder corresponding to the semantic enhancement generation network) to restore the sample integrated implicit representation. This decoder network is also trained to learn the inverse mapping relationship from the integrated implicit representation to the audio signal. During the restoration mapping process, the decoder network considers both the track implicit representation and the environment implicit representation. It may make appropriate adjustments to the track implicit representation (such as adjusting the volume, timbre, etc.) according to the information in the environment implicit representation to generate sound quality adjusted audio adapted to a specific environmental scene. After being processed by the decoder network, the sample integrated implicit representation is converted into a sound quality adjusted audio that combines the target track and environmental information. This audio not only retains the semantic features of the original audio, but also reflects the impact of the environment on the audio. For example, consider a sample-integrated implicit representation of a jazz music sample in a family gathering setting. During the restoration mapping process, the decoder network adjusts the implicit representation of the jazz music sample track based on the environmental sensing training data (such as noise level, room reverberation, etc.). The resulting sound-adjusted audio may have a higher volume, stronger low-frequency response, and richer reverberation effects to fit the lively atmosphere of a family gathering setting.

[0189] In step S523, the sound system performs track identification and embedding operations on the restored audio and the sample sound quality adjusted audio. The purpose of this step is to convert the audio signal into a form that is easier to compare and analyze - embedding implicit representation.

[0190] The sound system uses a pre-trained track recognition network to identify and analyze audio signals. This network can identify different tracks in the audio (such as vocals, musical instruments, etc.) and encode them. For restored audio and sample quality-adjusted audio, the track recognition network will identify the tracks separately and convert these tracks into corresponding embedded implicit representations. These embedded implicit representations are usually of low dimensionality but retain the key semantic information in the audio signal.

[0191] Specifically, the track recognition network extracts features from the audio signal through a series of convolutional layers, pooling layers, and other structures. These features may include spectral features, rhythmic features, timbre features, etc., which together describe the semantic content of the audio signal. The extracted features are then sent to a fully connected layer or embedding layer for encoding. In this process, the network learns a mapping relationship from feature vectors to embedded implicit representations, converting high-dimensional feature vectors into low-dimensional embedded implicit representations.

[0192] For example, suppose both the restored audio and the sample quality-adjusted audio contain information about jazz tracks. The track recognition network will identify these tracks separately and convert them into embedded implicit representations. Although these embedded implicit representations have low dimensions, they can retain key semantic features in the original audio signal (such as the rhythmic pattern and harmonic structure of jazz), which facilitates subsequent comparison and analysis.

[0193] In step S524, the sound system calculates the characteristic distance between the first embedded implicit representation (from the restored audio) and the second embedded implicit representation (from the sample sound quality adjusted audio) as the audio semantic annotation adjustment error. This error reflects the ability of the audio generation network to retain the audio semantic information after fusing the environmental data. The sound system uses a suitable distance measurement method (such as Euclidean distance, cosine similarity, etc.) to calculate the distance between the two embedded implicit representations. The smaller the distance value, the more similar the two embedded implicit representations are, that is, the restored audio and the sample sound quality adjusted audio are semantically closer.

[0194] Based on the calculated feature distance value, the sound system can determine the audio semantic annotation calibration error. This error will serve as one of the bases for subsequent network parameter adjustments to help optimize the audio generation network's ability to retain audio semantic information.

[0195] In the specific implementation, the sound system can select a suitable distance measurement method according to the specific characteristics and requirements of the embedded implicit representation. For example, if the embedded implicit representation is normalized and distributed on the unit sphere, cosine similarity can be used as the distance measurement; otherwise, Euclidean distance or Manhattan distance can be used. Regardless of the distance measurement method selected, the sound system will calculate the specific distance value based on the first embedded implicit representation and the second embedded implicit representation, and use it as the audio semantic annotation calibration error. This error value will directly reflect the performance of the current network in retaining audio semantic information.

[0196] For example, suppose the first embedded implicit representation (from the restored audio) is [0.5, -0.3, 0.2], and the second embedded implicit representation (from the sample sound quality adjusted audio) is [0.48, -0.29, 0.19]. If the sound system chooses Euclidean distance as the distance metric, the feature distance d can be calculated according to the following formula:

[0197] This feature distance value (audio semantic annotation adjustment error) reflects the slight semantic difference between the restored audio and the sample sound quality adjusted audio. The sound system can evaluate the current network's ability to preserve audio semantic information based on this error value and adjust the network parameters accordingly to optimize performance.

[0198] Through the detailed explanation and example of the implementation of step S52, we can see how the sound system evaluates the ability of the audio generation network to retain audio semantic information through restoration mapping, track recognition embedding and feature distance calculation. This process not only helps to understand whether the current network can accurately retain the semantic content of the original audio after integrating the environmental data; it also provides an important reference for subsequent network parameter adjustments. In practical applications, the sound system can optimize the network parameter settings according to the size and direction of the audio semantic annotation adjustment error, thereby improving the overall performance of the audio generation network in scene recognition and adaptive adjustment.

[0199] An embodiment of the present application provides an audio system, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0200] Figure 2 A hardware entity diagram of a sound system provided in an embodiment of the present application, such as Figure 2 As shown, the hardware entity of the sound system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0201] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An intelligent audio scene recognition and adaptive adjustment method, characterized in that: The method comprises: Obtaining initial audio and environmental sensor data including the target audio track; Performing feature embedding on the initial audio and the environmental sensor data respectively, and obtaining an audio embedding representation and a sensor embedding representation accordingly; Performing multi-level audio incremental learning on the audio embedding representation, and correspondingly obtaining audio semantic annotation implicit representations of the target audio track in the initial audio at multiple levels; Performing multi-level sensory incremental learning on the sensory embedding representations one by one; wherein the number of levels of the multi-level sensory incremental learning is consistent with the number of levels of the multi-level audio incremental learning; For sensor incremental learning between two adjacent levels, the sensor incremental learning results obtained by the sensor incremental learning of the front level in the sensor incremental learning of the two adjacent levels and the implicit representation of the audio semantic annotation of the corresponding level are integrated with each other; wherein, when the sensor incremental learning of the back level in the sensor incremental learning of the two adjacent levels is performed, the sensor incremental learning is performed on the implicit representation integration result after the sensor embedding representation and the implicit representation are integrated; The integrated implicit representation obtained after the multi-level sensor incremental learning is restored and mapped to obtain the sound quality adjusted audio that integrates the track implicit representation of the target track and the environment implicit representation of the environment sensor data.

2. The method according to claim 1, characterized in that The step of performing multi-level audio incremental learning on the audio embedding representation to correspondingly obtain an audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels includes: Based on multiple intermediate layer components in the pre-debugged audio annotation extraction component, multi-level audio incremental learning is performed on the audio embedding representation, and correspondingly, the audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels is obtained; wherein, when the first-level audio incremental learning is performed on the audio embedding representation, the first-level audio incremental learning is performed on the audio embedding representation based on the first intermediate layer component among the multiple intermediate layer components, and the audio semantic annotation implicit representation of the target audio track in the initial audio at the first level is obtained; When performing x-th layer audio incremental learning on the audio embedding representation, based on the x-th intermediate layer component among the multiple intermediate layer components, performing x-th layer audio incremental learning on the w-th layer audio semantic annotation implicit representation, obtains the x-th layer audio semantic annotation implicit representation of the target audio track in the initial audio; wherein, x≥2, w=x-1.

3. The method according to claim 2, characterized in that The intermediate layer component is a first jump connection component, and the first jump connection component includes an incremental projection algorithm; the multiple intermediate layer components in the pre-debugged audio annotation extraction component perform multi-level audio incremental learning on the audio embedding representation, and correspondingly obtain the audio semantic annotation implicit representation of the target audio track in the initial audio at multiple levels, including: Based on the incremental projection algorithm in each first jump connection component in the multiple intermediate layer components, multi-level audio incremental learning is performed on the audio embedding representation, and accordingly, an implicit representation of audio semantic annotations of the target audio track in the initial audio at multiple levels is obtained.

4. The method according to claim 3, characterized in that The step of performing multi-level sensory incremental learning on the sensory embedding representations one by one includes: Based on a pre-debugged semantic enhancement generation network, multi-level sensory incremental learning is performed one by one on the sensory embedding representation; wherein the semantic enhancement generation network includes a plurality of cascaded second jump connection components, and a saliency focus layer is included between each two adjacent second jump connection components; the number of the second jump connection components is consistent with the number of the first jump connection components in the audio annotation extraction component.

5. The method according to claim 4, characterized in that The semantic enhancement generation network based on the pre-debugging performs multi-level sensory incremental learning on the sensory embedding representations one by one, including: When performing first-layer sensory incremental learning on the sensory embedded representation, based on the first second jump connection component among the multiple second jump connection components, performing first-layer sensory incremental learning on the sensory embedded representation to obtain a first-layer implicit representation of the environmental sensor data after the first-layer sensory incremental learning; After performing the yth layer of sensory incremental learning on the sensory embedding representation, based on the yth saliency focusing layer in the semantic enhancement generation network, the yth layer of implicit representation after the yth layer of sensory incremental learning and the audio semantic annotation implicit representation output by the yth first jump connection component in the audio annotation extraction component are integrated to obtain the yth implicit representation integration result, y≥1; When performing the a-th layer sensory incremental learning on the sensory embedding representation, based on the a-th second jump connection component among the multiple second jump connection components, the a-th layer sensory incremental learning is performed on the integration result of the sensory embedding representation and the b-th implicit representation output by the b-th saliency focusing layer to obtain the a-th layer implicit representation of the environmental sensor data after the a-th layer sensory incremental learning; wherein a≥2, b=a+1; Using the implicit representation obtained after the last layer of the multi-level sensor incremental learning as the integrated implicit representation obtained after the multi-level sensor incremental learning; The multiple second jump connection components include: one or more second jump connection components for performing dimensionality reduction operations on the sensor embedded representation and one or more second jump connection components for performing interpolation processing on the sensor embedded representation; the number of second jump connection components for performing dimensionality reduction operations on the sensor embedded representation is consistent with the number of second jump connection components for performing interpolation processing on the sensor embedded representation; at the same time, the one or more second jump connection components for performing interpolation processing on the sensor embedded representation are located behind the one or more second jump connection components for performing dimensionality reduction operations on the sensor embedded representation.

6. The method according to claim 5, characterized in that Before performing first-layer sensory incremental learning on the sensory embedding representation, the method further includes: Based on a preset random distribution sampling algorithm, generating a generative implicit representation of the semantic enhancement generative network; Loading the generated implicit representation and the sensory embedded representation as implicit representations to be loaded into the semantic enhancement generation network; The step of performing first-layer sensory incremental learning on the sensory embedding representation based on a first second jump connection component among the multiple second jump connection components to obtain a first-layer implicit representation of the environmental sensor data after the first-layer sensory incremental learning includes: Based on the first second skip connection component, performing first-layer sensory incremental learning on the sensory embedding representation and the generated implicit representation to obtain the first-layer implicit representation; The feature dimension of the implicit representation of the yth layer is consistent with the feature dimension of the implicit representation of the audio semantic annotation output by the yth first jump connection component; the yth saliency focusing layer based on the semantic enhancement generation network performs the implicit representation integration on the yth layer implicit representation after the yth layer sensor incremental learning and the audio semantic annotation implicit representation output by the yth first jump connection component in the audio annotation extraction component to obtain the yth implicit representation integration result, including: Based on the yth saliency focusing layer in the semantic enhancement generation network, the yth layer implicit representation and the audio semantic annotation implicit representation output by the yth first skip connection component are fused in different feature dimensions to obtain a fused implicit representation; Performing a dimensionality reduction operation on the fused implicit representation to obtain a reduced-dimensional implicit representation; Filtering the reduced-dimensional implicit representation to obtain an array of influence coefficients between the y-th layer implicit representation and the audio semantic annotation implicit representation output by the y-th first skip connection component; The fused implicit representation is implicitly integrated according to the influence coefficient array to obtain the yth implicit representation integration result.

7. The method according to any one of claims 1 to 6, characterized in that: The intelligent audio scene recognition and adaptive adjustment method is performed based on a pre-debugged audio generation network; the audio generation network includes a first sub-network and a second sub-network; the first sub-network includes a sensor data embedding component and a semantic enhancement generation network, and the second sub-network includes an audio embedding component and an audio annotation extraction component; The tuning process of the audio generation network includes: Loading the audio training data in the training sample into the second sub-network, performing audio embedding representation on the audio training data based on the audio embedding component in the second sub-network, and obtaining a sample audio embedding representation; Based on the audio annotation extraction component in the second sub-network, multi-level audio incremental learning is performed on the sample audio embedding representation, and accordingly, implicit representations of sample audio semantic annotations of the sample target audio track in the audio training data at multiple levels are obtained; Loading environmental sensor training data in the training sample into the first sub-network, performing sensor embedding representation on the environmental sensor training data based on the sensor data embedding component in the first sub-network to obtain a sample sensor embedding representation; Based on the semantic enhancement generation network in the first sub-network, multi-level sensory incremental learning is performed on the sample sensory embedding representation one by one to obtain the sample integrated implicit representation; Determine the error of the audio generation network according to the implicit representation of the sample audio semantic annotation and the implicit representation of the sample integration to obtain a calibration error of the audio generation network; The network parameters in the audio generation network are adjusted according to the calibration error to obtain a calibrated audio generation network.

8. The method according to claim 7, characterized in that The step of determining the error of the audio generation network according to the implicit representation of the sample audio semantic annotation and the implicit representation of the sample integration to obtain the calibration error of the audio generation network includes: According to the sample integrated implicit representation and the audio training data, an initial error is determined for the audio generation network to obtain an initial calibration error; According to the sample audio semantic annotation implicit representation and the sample integrated implicit representation, determining the audio semantic annotation error of the audio generation network to obtain the audio semantic annotation adjustment error; The sum of the initial calibration error and the audio semantic annotation calibration error is used as the calibration error of the audio generation network.

9. The method according to claim 8, characterized in that The step of integrating the implicit representation and the audio training data according to the sample, determining an initial error of the audio generation network, and obtaining an initial calibration error, comprises: Performing restoration mapping on the sample integrated implicit representation to obtain a sample sound quality adjusted audio that fuses the track implicit representation of the sample target track and the environment implicit representation of the environment sensor training data; Taking the mean of the square errors between the sample sound quality adjusted audio and the audio training data as the initial calibration error; The step of determining the audio semantic annotation error of the audio generation network according to the sample audio semantic annotation implicit representation and the sample integrated implicit representation to obtain the audio semantic annotation adjustment error includes: Performing restoration mapping on the implicit representation of the semantic annotation of the sample audio to obtain restored audio; Performing restoration mapping on the sample integrated implicit representation to obtain a sample sound quality adjusted audio that fuses the track implicit representation of the sample target track and the environment implicit representation of the environment sensor training data; Performing track identification and embedding on the restored audio and the sample sound quality adjusted audio respectively, and obtaining a first embedded implicit representation and a second embedded implicit representation accordingly; The feature distance between the first embedded implicit representation and the second embedded implicit representation is used as the audio semantic annotation calibration error.

10. An audio system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

Citation Information

Patent Citations

  • Two-channel audio playing method and playing apparatus using same

    CN106412672A

  • Sound energy-saving regulation and control method and system based on intelligent control

    CN117596524A

  • System for speech recognition text enhancement fusing multi-modal semantic invariance

    US11488586B1