Audio processing method and related apparatus

Abstract

The application discloses an audio processing method and related device, and relates to the technical field of mobile terminal audio processing, and comprises the following steps: acquiring a fine-grained content type identifier of a first audio stream being played, wherein the fine-grained content type identifier represents a semantic scene category to which the first audio stream belongs; acquiring a continuous attention state representation generated based on non-audio sensor data fusion, wherein the continuous attention state representation reflects the degree of attention input of a user to a current device; and determining an audio processing strategy of the first audio stream based on the fine-grained content type identifier and the continuous attention state representation. Through the construction of a two-dimensional joint decision mechanism of content semantics and user state, the application realizes the dynamic adaptation of an audio processing strategy to a real scene of a user.

Description

Technical Field

[0001] This application relates to the field of mobile terminal audio processing technology, and in particular to an audio processing method and related apparatus. Background Technology

[0002] As mobile terminals become increasingly feature-rich in multimedia functions, it is becoming more and more common for users to run multiple audio applications simultaneously on the same device. As a result, the system needs to manage and coordinate the various audio streams in a reasonable manner.

[0003] Existing audio management systems mainly rely on the audio purpose or content type declared by the application for priority arbitration, or perform coarse classification based on the acoustic characteristics of the audio signal and then switch preset sound effect modes. Some solutions also combine single sensor data such as screen brightness to adjust volume or notification strategies.

[0004] However, the above management methods have the following shortcomings. First, the content recognition granularity is too coarse. Existing solutions can only distinguish broad categories such as music and speech, and cannot identify fine-grained content types such as podcasts, audiobooks, and ASMR. This results in the system being unable to make reasonable processing decisions based on the actual semantic context of the content. Second, existing systems completely disregard the user's current actual state when processing audio focus. The same notification sound should be processed differently in scenarios where the user is attentively listening and when it is playing in the background, but existing solutions cannot distinguish between these. The root of these problems lies in the fact that existing solutions lack a fine perception of the semantics of audio content and an objective inference of the user's attention state. Furthermore, they lack a mechanism to jointly model these two aspects to drive audio processing decisions, leading to a systematic deviation between audio processing strategies and the user's true intentions. Summary of the Invention

[0005] In view of the problem that existing audio processing strategies do not match the user's true intentions, this application provides an audio processing method and related apparatus to achieve joint decision-making based on fine-grained content semantics and user attention state, so as to enable the audio processing strategy to dynamically adapt to the user's real usage scenario. The specific solution is as follows:

[0006] The first aspect of this application provides an audio processing method, including:

[0007] Obtain the fine-grained content type identifier of the first audio stream that is currently playing, wherein the fine-grained content type identifier represents the semantic scene category to which the first audio stream belongs;

[0008] Obtain a continuous attention state representation generated based on non-audio sensor data fusion, the continuous attention state representation reflecting the user's level of attention to the current device;

[0009] Based on the fine-grained content type identifier and the continuous attention state representation, the audio processing strategy for the first audio stream is determined.

[0010] In one possible implementation, obtaining the fine-grained content type identifier of the first audio stream being played includes:

[0011] Extract the spectral features of the audio frames of the first audio stream;

[0012] The audio frame spectral features are classified using an edge-side classification model, and a multi-dimensional content type label vector is output.

[0013] Temporal smoothing is applied to the multidimensional content type label vector of consecutive audio frames to obtain the fine-grained content type identifier.

[0014] In one possible implementation, the multidimensional content type tag vector contains confidence scores for at least ten fine-grained categories, including voice content, music content, and cue and sound effects categories.

[0015] The time-series smoothing process includes: performing a sliding average or majority vote on the multidimensional content type tag vector within a preset time window; and updating the fine-grained content type identifier when the dominant category changes after smoothing and continues for more than a preset retention time.

[0016] In one possible implementation, obtaining the continuous attention state representation generated based on non-audio sensor data fusion includes:

[0017] Collect multidimensional non-audio sensor data, which includes device interaction status data and context status data;

[0018] The multidimensional non-audio sensing data is weighted and fused, and the weighted fusion includes a first-order weighting term and a cross term between the multidimensional non-audio sensing data.

[0019] The fusion result is normalized by nonlinear mapping to generate the continuous attention state representation with values ​​within a preset continuous interval.

[0020] In one possible implementation, the device interaction state data includes screen state and touch interaction frequency within a preset time window, and the context state data includes foreground application type, accelerometer output, and ambient light intensity.

[0021] The weighted fusion uses a nonlinear activation function to nonlinearly map the weighted sum of the first-order weighted terms and the cross terms. The first-order weighted terms are the sum of the products of each normalized non-audio sensor data and its corresponding weight, and the cross terms are the sum of the products of any two non-audio sensor data and their corresponding weights.

[0022] The weights of the first-order weighted term and the weights of the cross term are updated incrementally online based on the correlation between historical user behavior and audio processing effects.

[0023] In one possible implementation, the method further includes:

[0024] In response to the detection of a playback request for the second audio stream, a content compatibility relationship between the first and second audio streams is determined based on their respective fine-grained content type identifiers. When the content compatibility relationship is complementary, the first and second audio streams are configured as a primary audio stream and a secondary audio stream, respectively, and frequency-domain selective dynamic attenuation is performed on the secondary audio stream based on the real-time energy distribution of the primary audio stream.

[0025] In one possible implementation, performing frequency-domain selective dynamic attenuation on the secondary audio stream based on the real-time energy distribution of the primary audio stream includes:

[0026] Extract the active frequency bands of the main audio stream in the current frame, where the active frequency bands are a set of frequency points whose energy exceeds an adaptive threshold;

[0027] Dynamic notch attenuation is applied to the auxiliary audio stream at the frequency point corresponding to the active frequency band, and the attenuation depth of the dynamic notch attenuation is positively correlated with the energy of the main audio stream at the corresponding frequency point;

[0028] When the main audio stream is in a silent state in the current frame, the full-band gain of the auxiliary audio stream is restored to a non-attenuated state.

[0029] In one possible implementation, extracting the active frequency bands of the main audio stream in the current frame includes: performing a short-time Fourier transform on the main audio stream to obtain the energy of each frequency point, and determining the frequency points whose energy exceeds the product of the average energy of the frame and a preset multiplier factor as active frequency points;

[0030] The step of applying dynamic notch attenuation to the auxiliary audio stream at the frequency point corresponding to the active frequency band includes: calculating the gain of the auxiliary audio stream at each frequency point based on the energy of the main audio stream at the corresponding frequency point, the energy of the auxiliary audio stream at the corresponding frequency point, and the maximum attenuation coefficient, wherein the gain is negatively correlated with the energy of the main audio stream at the corresponding frequency point.

[0031] In one possible implementation, determining the content compatibility relationship between the first and second audio streams based on their respective fine-grained content type identifiers includes:

[0032] Based on preset mutual exclusion and complementary matrices, the relationship between the fine-grained content type identifiers of the first audio stream and the second audio stream is determined;

[0033] When a mutual exclusion relationship is determined, the playback request of the second audio stream is delayed until the first audio stream finishes playing or is added to the queue.

[0034] When a complementary relationship is determined, the first audio stream and the second audio stream are respectively assigned as the main audio stream and the auxiliary audio stream based on a preset priority rule.

[0035] In one possible implementation, the method further includes:

[0036] In response to changes in the fine-grained content type identifier or the continuous attention state representation that cause a change in the audio processing strategy, the audio processing parameters are interpolated and gradually switched.

[0037] In one possible implementation, the method further includes:

[0038] Collect user feedback on the automatic application of the audio processing strategy, and record the user’s reverse manual operation within a preset time as a negative feedback signal when the user is detected to have performed a reverse manual operation.

[0039] Based on the negative feedback signal, the fusion weights for generating the continuous attention state representation and the decision model for determining the audio processing strategy are updated incrementally online.

[0040] In one possible implementation, the method further includes:

[0041] When the fine-grained content type identifier indicates that the currently playing content is immersive content, and the continuous attention state representation is greater than the preset attention threshold, the output mode of the system notification is switched from audio playback to haptic feedback or screen visual prompts; the reverse manual operation includes the user manually increasing the volume that has been automatically reduced by the system, or manually restoring the notification prompt sound that has been automatically suppressed by the system.

[0042] A second aspect of this application provides an audio processing apparatus, comprising:

[0043] A fine-grained content type identifier acquisition unit is used to acquire a fine-grained content type identifier of the first audio stream that is being played, wherein the fine-grained content type identifier represents the semantic scene category to which the first audio stream belongs;

[0044] A continuous attention state representation acquisition unit is used to acquire a continuous attention state representation generated based on non-audio sensor data fusion, wherein the continuous attention state representation reflects the user's level of attention to the current device.

[0045] The processing strategy determination unit is used to determine the audio processing strategy of the first audio stream based on the fine-grained content type identifier and the continuous attention state representation.

[0046] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the audio processing method of the first aspect or any implementation thereof.

[0047] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0048] The memory is used to store computer programs;

[0049] The processor is used to execute the computer program so that the electronic device can implement the audio processing method of the first aspect or any implementation thereof.

[0050] The fifth aspect of this application provides a computer-readable storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the audio processing method of the first aspect or any implementation thereof.

[0051] By employing the aforementioned technical solutions, the audio processing method and related apparatus provided in this application construct a two-dimensional joint decision-making mechanism of content semantics and user state by acquiring fine-grained content type identifiers representing the semantic scene category to which the first audio stream belongs, and combining them with continuous attention state representations reflecting the user's attention input generated based on non-audio sensor data fusion. This mechanism enables the determination of audio processing strategies to no longer rely on coarse-grained acoustic classification or a single application declaration, but rather to accurately perceive the actual meaning of the current audio content to the user and the user's current true listening intention, thereby achieving dynamic adaptation of audio processing strategies to the user's real-world scenario, effectively overcoming the technical defects of traditional solutions where strategies and scenarios are separated and processing logic is applied in a one-size-fits-all manner. Attached Figure Description

[0052] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0053] Figure 1 A flowchart illustrating an audio processing method provided in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application;

[0055] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0057] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0058] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0059] As mobile devices become increasingly feature-rich in multimedia capabilities, it's becoming more common for users to run multiple audio applications simultaneously on the same device. Examples include listening to music while receiving navigation voice prompts, watching video tutorials while background music plays in the background, or receiving system notifications while listening to ASMR before bed. In these concurrent audio stream scenarios, the system needs to manage and coordinate the various audio streams effectively to balance the clarity of information delivery with the user's overall listening experience.

[0060] Existing audio management systems primarily rely on applications' declared audio purpose or content type for priority arbitration. Represented by Android's AudioFocus mechanism, applications request audio focus and declare the audio purpose and content type before playback. The system then decides whether to grant focus, reject the request, or require the current user to lower the volume based on preset priority rules. A typical mixing method is global cutting, which attenuates the volume of secondary audio streams to a fixed percentage of their original volume. Another approach analyzes the audio signal itself in real time, classifying it into a few broad categories such as music, speech, or ambient sound, and switching preset sound effect modes accordingly. For example, music mode enables bass enhancement, and speech mode enables vocal enhancement. Other solutions utilize single or limited sensor data, such as screen brightness and foreground application type, to determine the device usage scenario and adjust volume, brightness, or notification strategies. Additionally, some solutions collect ambient noise through microphones and adjust playback volume or dynamic range based on noise levels to maintain appropriate loudness in different environments.

[0061] However, the above management methods have the following shortcomings. First, the content recognition granularity is too coarse. Existing solutions can only distinguish broad categories such as music and voice, and cannot identify fine-grained content types such as podcasts, audiobooks, instructional video dialogues, ASMR, and game voice chat. This results in the system being unable to make reasonable processing decisions based on the actual acoustic characteristics of the content and the user's listening scenario. For example, educational podcasts and entertainment music are both classified as media playback, but users usually need to listen attentively to the former and should not be easily interrupted, while the latter can be played in the background. Second, existing systems do not consider the user's current actual state when processing audio focus. The user may be listening attentively to podcast content (screen on, frequent interaction), or they may simply be playing music in the background (screen off, device idle). The same notification sound should be suppressed or converted into haptic feedback in the former scenario, but can be played normally in the latter scenario. The root of the above problems lies in the fact that existing solutions lack a fine perception of the semantics of audio content and an objective inference of the user's attention state, and even more so, they lack a mechanism to jointly model the two to drive audio processing decisions, resulting in a systematic deviation between audio processing strategies and the user's true intentions.

[0062] To address the aforementioned problems, this application provides an audio processing method. The audio processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0063] Example 1

[0064] Reference Figure 1 , Figure 1This is a flowchart illustrating an audio processing method provided in an embodiment of this application. This method addresses the mismatch between existing audio processing strategies and the user's true intent through a two-dimensional joint decision-making process involving content semantics and user attention. Figure 1 As shown in the embodiment of this application, an audio processing method may include steps S101 to S103, which will be described in detail below.

[0065] Step S101: Obtain the fine-grained content type identifier of the first audio stream that is being played. The fine-grained content type identifier represents the semantic scene category to which the first audio stream belongs.

[0066] In this application, the fine-grained content type identifier represents the semantic scene category to which the first audio stream belongs, which differs from the coarse classification based solely on acoustic features in existing technologies (such as simply distinguishing between speech and music). Semantic scene categories focus on the actual meaning and purpose of the audio content for the user at present, such as distinguishing between educational podcasts for learning and entertainment music for relaxation, or between ASMR (Autonomous Sensory Meridian Response) content requiring an immersive experience and ordinary system prompts. By identifying these semantic-level differences, the system can understand the value of the content in a specific scene, thus providing the necessary prerequisites for subsequently developing differentiated audio processing strategies and avoiding unreasonable interruptions or mixing caused by overly coarse content type identification.

[0067] It should be understood that the specific granularity of semantic scene classification can be adjusted according to actual application needs, as long as it can reflect the differences in user intent for different audio content.

[0068] Step S102: Obtain a continuous attention state representation generated based on non-audio sensor data fusion. The continuous attention state representation reflects the user's level of attention to the current device.

[0069] In this application, the continuous attention state representation is generated based on the fusion of non-audio sensor data. This means that the judgment of user attention does not depend on the characteristics of the audio signal itself, but is inferred by perceiving the user's interaction with the device and the environmental context. Using continuous numerical values ​​rather than discrete states (such as simple binary judgments of focus or inattention) to represent the degree of attention engagement can more subtly reflect the gradual process of user state changes. This design solves the problem of frequent strategy switching that is easily caused by single threshold judgment, and also avoids the misjudgment that may occur when judging attention solely based on audio features. For example, a user listening quietly to high-value content may be misjudged as not paying attention, ensuring the objectivity and stability of attention perception. By introducing non-audio dimension perception data, the system can accurately capture the user's true listening state without interrupting the user's audio experience.

[0070] Step S103: Determine the audio processing strategy for the first audio stream based on fine-grained content type identification and continuous attention state representation.

[0071] In this application, this step constructs a two-dimensional joint decision-making mechanism based on content semantics and user state. The determination of audio processing strategies no longer depends solely on content type or user state, but rather considers both together. For example, when playing background music, it may be considered a distraction and have its priority reduced when the user is highly focused, but can be retained as the main ambient sound when the user is in the background. Conversely, even when the user is focused, instructional dialogue should be given enhanced protection, while ordinary sound effects may remain unchanged.

[0072] This joint decision-making enables audio processing strategies to dynamically adapt to the combination of the user's true intent and the current scenario, effectively overcoming the technical defects of traditional solutions where strategies and scenarios are separated and processing logic is applied in a one-size-fits-all manner, and realizing the transformation of audio management from passive response to proactive adaptation.

[0073] Furthermore, the above audio processing strategy is determined based on a two-dimensional decision space of preset content type and attention. Using content type categories as rows and attention score ranges as columns, the following decision mapping table can be constructed. The attention score range can be divided into focused state (e.g., attention score greater than 0.7), semi-attentive state (e.g., attention score between 0.3 and 0.7), and background state (e.g., attention score less than 0.3).

[0074]

[0075] The table above is merely an example of decision mapping. The actual mapping is established through an offline-trained decision model, rather than a simple static table lookup. The input to the decision model is a joint feature vector composed of a multi-dimensional content type label vector and an attention score, and the output is an audio processing chain configuration. This configuration includes at least three types of parameters: the first type is a set of audio effect parameters, including equalizer curves, dynamic range, and vocal enhancement switches; the second type is audio stream priority, representing the preemption or being preempted level of the current audio stream relative to other audio streams; and the third type is mixing behavior rules, including whether concurrency is allowed, ducking depth, and whether frequency-domain selective mixing is enabled. This decision model can be updated online based on user feedback to adapt to individual differences among users.

[0076] It should be understood that all of the above modules can be implemented at the Android Framework layer, and the neural network classifier can run on the edge NPU, DSP or CPU, without depending on specific audio hardware, and has good portability.

[0077] Example 2

[0078] In one possible implementation, obtaining the fine-grained content type identifier of the first audio stream being played includes: extracting the audio frame spectral features of the first audio stream; classifying the audio frame spectral features using an edge-side classification model to output a multi-dimensional content type label vector; and applying temporal smoothing processing to the multi-dimensional content type label vectors of consecutive audio frames to obtain the fine-grained content type identifier.

[0079] In this application, the audio frame spectral features utilize Mel spectrograms, and the edge classification model employs a lightweight convolutional neural network. Mel spectrograms, by simulating the nonlinear perception characteristics of the human ear for different frequencies, can more accurately reflect the auditory semantic information of audio content than traditional Fourier spectrograms, making them ideal input features for semantic scene classification. In practical processing, the playing audio stream can be divided into frames with windows ranging from 20ms to 50ms, and the Mel spectrogram of each frame can be calculated as input to the classifier. The typical number of Mel filter banks can be 40 to 64 to achieve a balance between feature representation capability and computational overhead. The edge classification model specifically refers to a lightweight neural network deployed on the local processor of a mobile terminal (such as an NPU, DSP, or CPU). Its design goal is to minimize model size and inference latency while ensuring classification accuracy, thereby meeting the real-time requirements of audio processing and protecting user data privacy. For example, this lightweight convolutional neural network can be specifically modified and optimized based on the MobileNet architecture. After quantization and pruning, the model size can be controlled at the level of hundreds of KB, and the single-frame inference latency is less than 10ms, thereby ensuring that real-time content recognition is completed without affecting the performance of the main audio playback thread.

[0080] The multi-dimensional content type label vector includes confidence scores for at least ten fine-grained categories, including audio content, music content, and cues and sound effects. This fine-grained category division is not based solely on acoustic features (such as distinguishing only between audio and music), but rather on the semantic attributes of the program type to which the audio content belongs and the user's usage scenario. Specifically, the audio content category can be further subdivided into podcasts, audiobooks, instructional video dialogues, game voice chat, and call voice, etc.; the music content category can be subdivided into pop music, classical music, background music, and ASMR, etc.; and the cues and sound effects category can be subdivided into game sound effects, system notification sounds, and short video background music, etc.

[0081] This multi-dimensional tag vector output, encompassing at least ten subcategories, enables the system to distinguish between different semantic scenarios, such as educational podcasts and entertainment music, immersive ASMR and ordinary notification sounds. This fine-grained semantic differentiation forms a necessary prerequisite for implementing differentiated audio processing strategies, avoiding misjudgments caused by overly coarse content type identification.

[0082] The time-series smoothing process includes: performing a sliding average or majority vote on the multi-dimensional content type tag vector within a preset time window; and updating the fine-grained content type identifier when the dominant category changes after smoothing and continues for more than a preset retention time.

[0083] Because audio signals are highly time-varying, single-frame classification results are easily affected by transient noise or brief silences, causing jitter. Directly switching audio processing strategies based on single-frame results would lead to frequent jumps and instability in the listening experience. Therefore, this embodiment introduces a dual anti-jitter mechanism. First, by performing a moving average or majority vote within a preset time window (typically 2 to 3 seconds), transient classification outliers are filtered out to obtain a statistically stable content type tendency. Second, a preset hold time is introduced as a hysteresis threshold for state switching. Only when the smoothed dominant category changes and this new category remains stable for more than the preset hold time is the fine-grained content type identifier officially updated and the corresponding strategy adjustment triggered. This mechanism effectively filters classification uncertainties during content switching transitions or in complex audio segments, ensuring the smoothness and robustness of audio processing strategy changes and preventing the system from repeatedly oscillating in critical states.

[0084] Example 3

[0085] In one possible implementation, obtaining a continuous attention state representation generated based on the fusion of non-audio sensor data includes: collecting multi-dimensional non-audio sensor data, which includes device interaction state data and context state data; performing weighted fusion on the multi-dimensional non-audio sensor data, which includes a first-order weighting term and a cross term between the multi-dimensional non-audio sensor data; and normalizing the fusion result through a non-linear mapping to generate a continuous attention state representation with values ​​within a preset continuous interval.

[0086] In this application, device interaction state data is primarily used to reflect the user's current direct control behavior on the device, and may include screen status and touch interaction frequency within a preset time window. Screen status can be further subdivided into modes such as on screen, off screen, or always-on display, used to initially determine whether the user is looking at the device; touch interaction frequency is calculated by counting the density of touch events over a past period (e.g., 60 seconds), used to quantify the user's interaction activity with the device. Context state data is mainly used to supplement the description of the user's environment and usage scenario, and may include foreground application type, accelerometer output, and ambient light intensity. Foreground application type provides semantic context for audio consumption, such as distinguishing between music players, video applications, games, or e-readers; accelerometer output can be used to determine whether the device is stationary, handheld, walking, or in a moving vehicle; ambient light intensity helps distinguish between bright and dim usage environments.

[0087] This multi-dimensional non-audio sensor data acquisition method is completely independent of the audio signal itself, avoiding potential misjudgments when judging attention solely based on audio features. For example, a user may be misjudged as not paying attention when quietly listening to high-value content, thus ensuring the objectivity and privacy of attention perception.

[0088] When fusing the aforementioned multidimensional non-audio sensor data, this embodiment introduces cross terms to capture the nonlinear relationships between the sensor data. This is because a user's attention state is often not a linear superposition of data from a single sensor, but rather the result of the synergistic effect of multiple factors. For example, simply having the screen on cannot distinguish whether a user is listening to audio (they may be reading text), and frequent touches alone cannot distinguish whether the user is focused on audio content (they may be operating other applications). Only when the screen is on and accompanied by high-frequency touches, while an audio-related application is running in the foreground, can it be inferred with high confidence that the user is in a focused interactive state. By introducing cross terms, the model can learn the deep semantics implied by these combined features, thereby significantly improving the accuracy and robustness of attention inference.

[0089] This three-dimensional joint judgment method based on screen status, touch interaction frequency, and foreground application type avoids the unreliability of traditional audio focus management that relies on application-driven declarations, inferring the user's true state through objective non-audio sensor data. In traditional solutions, audio processing and user state perception are two independent subsystems, lacking a mechanism for joint modeling. This embodiment, through weighted fusion and cross-term introduction of multi-dimensional sensor data, introduces user attention scores as a direct input to the audio processing chain for the first time, achieving joint modeling of audio processing and user state perception. Simultaneously, the entire attention inference process is entirely based on local sensor data on the terminal, without involving user identification, biometric recognition, or the collection and uploading of personal privacy data. The fusion and processing of sensor data is completed in real time on the terminal, fully ensuring user privacy and security.

[0090] Furthermore, the weighted fusion uses the Sigmoid function to perform a nonlinear mapping on the weighted sum of the first-order weighted terms and the cross terms, and its specific calculation formula is as follows:

[0091]

[0092] in, As a representation of continuous attention state, its value range is normalized to a preset continuous interval (e.g., 0 to 1), where 0 represents pure background playback state and 1 represents user focused listening state. The sigmoid function is used to map the fused weighted sum to this continuous interval, realizing a non-linear transformation from discrete sensor data to continuous attention scores; For the normalized first Non-audio sensor data, such as standardized screen brightness values ​​or touch frequency values; The weights of the corresponding first-order weighted terms reflect the basic contribution of a single sensor feature to attention judgment. In practical applications, the weight of the screen-on state is usually the highest, followed by the frequency of touch interaction. and The first The and the first Non-audio sensor data; The weights of the interaction terms are used to characterize the first term. The and the first The combined effect of individual sensor data on attention; This is a bias term used to adjust the baseline activation level of the model.

[0093] This formula constructs an attention evaluation model that can capture both basic features and complex interaction patterns by combining first-order weighting terms and cross-terms. Compared to traditional single-threshold judgments or simple linear weighting, this model can more delicately depict the gradual process of user attention and effectively solve the problem of policy jitter caused by discrete state switching. At the same time, the continuous scalar output format enables subsequent audio processing strategies to achieve a smooth transition, avoiding auditory abrupt changes caused by sudden state changes.

[0094] To adapt to the dynamic changes in the personalized habits and usage scenarios of different users, the weights of the first-order weighted terms... Weights of interaction terms Instead of remaining fixed, the system performs online incremental updates based on the correlation between historical user behavior and audio processing effects. In this application, the system continuously monitors user feedback to the automatically applied audio processing strategies. When the system adjusts the volume or notification method based on the current attention score, if the user performs a reverse manual operation within a short period (such as manually adjusting the volume or restoring the notification sound), it is considered a negative feedback signal; conversely, if the user makes no adjustment or shows acceptance, it is considered a positive feedback signal. The system uses these feedback signals as supervision labels and employs online learning algorithms (such as single-step updates using stochastic gradient descent) to fine-tune the fusion weights. This online incremental update mechanism allows the attention inference model to gradually adapt to the user's personal preferences over time, effectively solving the adaptation inaccuracy problem that may occur in the cold start phase of general models or when facing individual differences, and realizing the system's continuous evolution and personalized service capabilities.

[0095] Example 4

[0096] When multiple audio streams run concurrently, existing solutions employ fixed priority preemption or global cutting, failing to provide fine-grained processing based on the acoustic characteristics and contextual relationships between content. For example, background music and navigation voice could coexist as primary and secondary elements, but traditional cutting continuously lowers the volume of the music across all frequency ranges throughout the navigation process, resulting in a loss of rhythm and a disjointed listening experience for the background music. To address these issues, this embodiment provides a fine-grained processing method for concurrent multi-audio streams.

[0097] In one possible implementation, in response to the detection of a playback request for the second audio stream, a content compatibility relationship between the first and second audio streams is determined based on their respective fine-grained content type identifiers. When the content compatibility relationship is complementary, the first and second audio streams are configured as the main audio stream and the auxiliary audio stream, respectively, and frequency-domain selective dynamic attenuation is performed on the auxiliary audio stream based on the real-time energy distribution of the main audio stream.

[0098] In this application, in the scenario of multiple audio streams running concurrently, this embodiment first determines whether two audio streams can coexist based on content compatibility. This constitutes a prerequisite for performing subsequent fine-grained mixing, which is fundamentally different from the traditional technology that simply preempts or lowers the global volume based on application priority.

[0099] When a complementary relationship is determined, such as the coexistence of navigation voice and background music, the system does not continuously attenuate the auxiliary audio stream across the entire frequency band. Instead, based on the real-time energy distribution of the main audio stream, it dynamically attenuates the auxiliary audio stream only in specific frequency bands and time periods where the two conflict spectrally. This frequency-selective dynamic attenuation mechanism ensures that the auxiliary audio stream accurately avoids the main audio stream in its critical frequency bands, while maintaining its original sound quality in the silent or non-conflicting frequency bands of the main audio stream. Compared to traditional global attenuation methods, this mechanism ensures clear delivery of key information while maximizing the integrity and coherence of the auxiliary audio, significantly improving the user experience when multiple streams are running concurrently. It should be noted that the determination of content compatibility is based on the semantic scenario represented by the fine-grained content type identifier, rather than the application package name or the system's preset audio usage tag. This ensures consistency between the negotiation logic and the content actually listened to by the user.

[0100] Before describing the specific implementation of the frequency-domain selective dynamic attenuation described above, the overall system architecture adopted in this application will be explained first.

[0101] The audio processing system of this application consists of five core modules, adopting an overall architecture combining dual-channel parallel inference and two-dimensional joint decision-making. The PCM data of the audio playback stream is fed into channel A and channel B for parallel processing. Channel A is the content type identification module, responsible for Mel spectrum extraction of the audio playback stream, classification using a lightweight convolutional neural network, and temporal smoothing, outputting a multi-dimensional content type label vector S containing at least ten fine-grained category confidences. Channel B is the user attention inference module, responsible for parallel acquisition of multi-source non-audio sensor data such as screen state, foreground application type, touch interaction frequency, accelerometer output, and ambient light intensity, performing weighted fusion and nonlinear mapping, outputting a continuous attention score A ranging from 0 to 1, where 0 represents pure background playback and 1 represents the user's focused listening state. The strategy decision-making module receives the content type label vector S output from channel A and the attention score A output from channel B, determining the optimal audio processing chain configuration C through a two-dimensional mapping between content type and attention. Configuration C is applied in real-time by the processing chain control module, which performs gradual parameter switching when configuration changes occur. Simultaneously, in the case of multiple concurrent audio streams, the content type negotiation module performs compatibility checks, constructs primary and secondary channels, and performs frequency-domain selective mixing. Channel A and Channel B operate in parallel and independently, ensuring the system's real-time performance and robustness.

[0102] The following section details the specific implementation of frequency-domain selective dynamic attenuation of the auxiliary audio stream based on the real-time energy distribution of the main audio stream.

[0103] In one possible implementation, based on the real-time energy distribution of the main audio stream, frequency-selective dynamic attenuation is performed on the secondary audio stream, including: extracting the active frequency bands of the main audio stream in the current frame, where the active frequency bands are a set of frequency points whose energy exceeds an adaptive threshold; applying dynamic notch attenuation to the frequency points of the secondary audio stream corresponding to the active frequency bands, where the attenuation depth of the dynamic notch attenuation is positively correlated with the energy of the main audio stream at the corresponding frequency point; and restoring the full-band gain of the secondary audio stream to a non-attenuated state when the main audio stream is in a silent state in the current frame.

[0104] In this application, the implementation of frequency-domain selective dynamic attenuation first relies on real-time analysis of the spectral characteristics of the main audio stream. The system performs a short-time Fourier transform (STFT) on the main audio stream to obtain its energy distribution in the time and frequency domains. In practical processing, the typical frame length of the STFT can be 20ms to 40ms to ensure sufficient time and frequency resolution. Through the transform, the system can accurately determine the energy distribution of the main audio stream in the [missing information - likely a specific time or frequency range]. Frame number Energy at each frequency point Based on this, in order to accurately identify the frequency bands that truly carry key information in the main audio stream, rather than treating all energetic frequency points as interference sources, this embodiment introduces an adaptive threshold determination mechanism.

[0105] Specifically, it will satisfy The frequency points were identified as active frequency points, among which For the first Average energy of the frame This is a preset multiplier factor. The typical value range for this is 1.5 to 3. Its function is to filter out low-energy background noise or minor harmonic components in the main audio stream, retaining only the core frequency bands with significantly higher energy than the average level as active frequency bands. For example, for speech-based main audio streams, active frequency bands are typically concentrated in the 300Hz to 3400Hz fundamental frequency and formant region of the human voice; while for instrumental solos, the active frequency bands follow the fundamental frequency changes of the instrument. This adaptive extraction method based on relative energy ratio allows the determination of active frequency bands to automatically adjust with the dynamic changes of the main audio content, avoiding the limitations of fixed-band filtering.

[0106] After determining the active frequency bands, the system applies dynamic notch attenuation to the auxiliary audio stream at these frequency points. The gain of the auxiliary audio stream at each frequency point is calculated based on the energy of the main audio stream at the corresponding frequency point, the energy of the auxiliary audio stream at the corresponding frequency point, and the maximum attenuation coefficient. The specific calculation formula is as follows:

[0107]

[0108] in, The energy of the secondary audio stream at the corresponding frequency point. The maximum attenuation coefficient, To prevent division by zero by a pre-defined positive number. In this formula, the maximum attenuation coefficient... The value ranges from 0 to 1, with typical values ​​set to 0.6 to 0.9. It determines the maximum attenuation depth that the auxiliary audio can achieve at a given frequency when the energy of the primary audio is much greater than that of the auxiliary audio. (Preset positive number) It is a minimum value used to ensure that the denominator is not zero, thus ensuring the numerical stability of the calculation.

[0109] As can be seen from the formula structure, the gain Size and main audio energy There is a negative correlation: the stronger the energy of the main audio at a certain frequency, the lower the gain and the deeper the attenuation of the auxiliary audio at that frequency; conversely, if the energy of the main audio at a certain frequency is weak or comparable to that of the auxiliary audio, the auxiliary audio retains more of its original energy. This positively correlated dynamic attenuation characteristic means that the mixing effect is no longer a mechanical global compression, but rather presents an intelligent, content-driven attenuation effect. This is especially true when the main audio stream is silent or at extremely low energy. Approaching zero, gain Automatic convergence to 1 means that the gain of the secondary audio stream across all frequencies is restored to a non-attenuated state, allowing background music or ambient sound effects to instantly return to normal loudness, thus ensuring the continuity and integrity of the listening experience. Compared to the global attenuation method in traditional technologies that continuously reduces the volume across all frequencies throughout the existence of secondary audio, the solution in this embodiment precisely avoids attenuation only in necessary frequency bands and time periods, significantly improving the user experience in multi-stream concurrent scenarios.

[0110] After determining the gain of each frequency point of the auxiliary audio stream, the system superimposes the spectrum of the amplified auxiliary audio stream with the spectrum of the main audio stream, and then restores it to the time-domain signal output through inverse short-time Fourier transform. The formula for calculating the mixed output is:

[0111]

[0112] in, and The main audio stream and the secondary audio stream are respectively in the first... Frame number The complex spectrum at each frequency point. To avoid sound coloration caused by abrupt gain changes, the gain... Smooth transitions are applied between adjacent frames to ensure a natural and consistent sound in the mixed output.

[0113] The frequency-selective dynamic attenuation mechanism described above has significant technical advantages over traditional global cutting. Traditional cutting continuously and across the entire frequency range reduces the volume during the existence of the secondary audio stream, typically attenuating the average loudness of background music to about 40% of its original value, resulting in a loss of rhythm and a broken listening experience. This embodiment precisely avoids the primary audio stream only in the frequency range and time period where it is actually emitted, maintaining the original loudness of non-human voice frequency bands (such as low-frequency rhythms and high-frequency overtones in music). Taking a typical scenario of concurrent navigation voice and background music as an example, the active frequency band of navigation voice is usually concentrated in the 300Hz to 3400Hz human voice region, accounting for about 30% of the entire spectrum. This embodiment only applies attenuation to this 30% frequency band (attenuating it to about 50% of its original value), while the remaining 70% remains unchanged. The overall loudness retention rate is 0.3×0.5+0.7×1.0=0.85, or 85%, which is 112% higher than the 40% of the traditional solution. This result means that the background music can maintain most of its auditory integrity during navigation announcements, significantly improving the user's subjective auditory continuity.

[0114] The frequency-domain selective attenuation scheme in this embodiment breaks a long-standing technical bias among those skilled in the art, namely, that to ensure the clarity of the primary audio, secondary audio must be continuously and across the entire frequency range suppressed. This embodiment demonstrates that through frequency-domain selective attenuation, it is possible to preserve the auditory integrity of secondary audio to the maximum extent while ensuring the clarity of the primary audio, thereby achieving a better user experience in scenarios with multiple concurrent audio streams.

[0115] It should be understood that although the above embodiments provide specific gain calculation formulas based on energy ratio, this is not the only way to achieve frequency-domain selective dynamic attenuation. In other possible implementations, this dynamic attenuation can also be based on an auditory masking model, that is, calculating the psychoacoustic masking threshold of the primary audio signal on the secondary audio signal based on its spectral characteristics, and only attenuating or suppressing secondary audio components below the masking threshold, thereby preserving the perceptible components in the secondary audio to the greatest extent while ensuring the clarity of the primary audio. Alternatively, a sub-band energy compression approach can be used, dividing the spectrum into several sub-bands, independently comparing the energy of the primary and secondary audio in each sub-band, and applying corresponding compression gains. These alternative solutions can also achieve the technical effect of frequency-domain selective processing of the secondary audio based on the real-time energy distribution of the primary audio, and should all be covered within the scope of protection of this application.

[0116] Example 5

[0117] In one possible implementation, the content compatibility relationship between the first audio stream and the second audio stream is determined based on their respective fine-grained content type identifiers, including: determining the relationship between the fine-grained content type identifiers of the first audio stream and the second audio stream based on a preset mutual exclusion matrix and a complementary matrix.

[0118] In this application, the determination of content compatibility is not based on application package names, system-preset audio usage tags, or simple priority numerical comparisons, but rather directly on matching based on fine-grained content type identifiers that represent semantic scene categories. Preset mutual exclusion and complementary matrices constitute the logical hub of multi-stream collaborative decision-making, essentially a knowledge base that maps content semantic combinations to specific collaborative strategies. In this knowledge base, the rows and columns of the matrix correspond to different fine-grained content type identifiers, and the values ​​of the matrix elements indicate the relationship attributes when two content types are concurrent. This content semantic-based determination mechanism enables the system to understand the conflicts or collaborative potential of different audio streams at the user's cognitive level, thereby avoiding erroneous interruptions or improper mixing caused by a lack of understanding of content meaning in traditional solutions.

[0119] It should be understood that the specific contents of the mutual exclusion matrix and the complementary matrix can be configured and updated according to the actual application scenario, user preferences or system version, and this application does not limit their specific entries.

[0120] When a mutually exclusive relationship is determined, the playback request for the second audio stream will be delayed until the first audio stream finishes playing or is added to the queue.

[0121] Mutually exclusive relationships typically refer to scenarios where two audio streams have strong semantic conflicts, or where simultaneous playback would lead to cognitive overload and loss of key information for the user. In a mutual exclusion matrix, podcasts, audiobooks, and instructional video dialogues are marked as mutually exclusive with their own categories. This means that two streams of the same type of content (such as two podcasts) or different types of content requiring focused listening (such as a podcast and an audiobook) cannot be played concurrently. For example, if the first audio stream is an educational podcast, and the second audio stream is another podcast, audiobook, or instructional video dialogue, both belong to the category of audio content requiring high user focus. Simultaneous playback would cause semantic confusion, and in this case, the mutual exclusion matrix marks them as mutually exclusive. Similarly, if the first audio stream is a game voice chat, and the second audio stream is a phone call, both are highly interactive real-time audio content. Concurrent playback would severely disrupt communication efficiency, and thus also fall under the category of mutual exclusion.

[0122] To address such mutually exclusive relationships, the system does not resort to forced preemption or volume reduction. Instead, it delays the playback request of the newly arrived second audio stream until the current first audio stream finishes playing naturally, or adds it to a playback queue for later scheduling. This approach fully respects the integrity and continuity of high-value semantic content, preventing users from missing important information or being forced to interrupt their immersive experience due to mechanical priority arbitration, thus reflecting an audio management philosophy centered on content value.

[0123] When a complementary relationship is determined, the first audio stream and the second audio stream are assigned as the main audio stream and the auxiliary audio stream respectively based on preset priority rules.

[0124] Complementary relationships refer to two audio streams that are semantically independent or even complementary, and possess the potential for frequency domain separation in acoustic characteristics, making them suitable for coexistence through refined mixing. Typical complementary combinations include background music and navigation voice, background music and game voice chat, and environmental sound effects and system prompts. In these combinations, one audio stream typically carries core information or has higher timeliness and security requirements (such as navigation commands or game teammate voices), making it suitable as the main audio stream; the other audio stream mainly serves to enhance the atmosphere or provide background accompaniment (such as music or ambient sounds), making it suitable as a secondary audio stream.

[0125] The preset priority rules comprehensively consider multiple factors such as information density of content type, user attention needs, security relevance, and real-time performance, automatically and dynamically assigning primary and secondary roles. For example, when navigation voice and background music coexist, navigation voice, due to its key path guidance information and high real-time performance, is automatically assigned as the primary audio stream, while background music is assigned as the secondary audio stream. This primary-secondary allocation is a prerequisite for subsequent frequency-domain selective dynamic attenuation, ensuring that the mixing processing is targeted. That is, while ensuring the primary audio stream is clearly distinguishable, the auditory integrity of the secondary audio stream is preserved to the maximum extent, thereby optimizing the user experience in multi-stream concurrent scenarios.

[0126] It should be noted that the above methods for determining and handling mutually exclusive and complementary relationships are merely illustrative. In practical applications, in addition to delayed playback and adding to the queue, other equivalent processing methods can be adopted for mutually exclusive relationships depending on the specific scenario, such as silently buffering the second audio stream or converting it into a non-auditory form of cues (such as vibration or visual icons). Similarly, for the primary and secondary allocation of complementary relationships, a mechanism for manual user specification or dynamic adjustment based on historical behavioral habits can be introduced.

[0127] Furthermore, the mutual exclusion matrix and the complementary matrix do not necessarily have to be two independent physical storage structures; they can also be merged into a unified compatibility mapping table, or the determination result can be dynamically generated through a real-time inference model. As long as its core logic is based on fine-grained content type identification to determine the cooperation strategy between audio streams, it should be covered within the scope of protection of this application.

[0128] Example 6

[0129] In one possible implementation, the method further includes: interpolating and gradually switching audio processing parameters in response to changes in the audio processing strategy caused by changes in fine-grained content type identifiers or continuous attention state representations.

[0130] In this application, since adjusting audio processing strategies often involves simultaneous changes to multiple parameters such as equalizer curves, dynamic range compression thresholds, and mixing gain, directly using a hard switching method can easily produce noticeable abrupt noise or volume jumps in the listening experience, severely damaging the user's listening experience. Therefore, this embodiment introduces a parameter interpolation gradual transition mechanism. That is, after detecting the strategy change trigger condition, the target configuration is not immediately applied. Instead, within a preset transition time window, the current parameters are smoothly transitioned to the target parameter value using a linear or nonlinear interpolation algorithm. The typical range of this transition time window is 100 milliseconds to 500 milliseconds, ensuring the smoothness of parameter switching to eliminate auditory abruptness while avoiding strategy response lag caused by excessively long transition times. This gradual switching mechanism ensures that the system maintains a consistent and natural listening experience throughout the dynamic adaptation process, which is a key guarantee for achieving seamless intelligent audio management.

[0131] Furthermore, the method also includes collecting user feedback on the automatic application of audio processing strategies. When a user performs a reverse manual operation within a preset time period, it is recorded as a negative feedback signal. Here, a reverse manual operation refers to a user's proactive intervention in the opposite direction of the automatic audio adjustment performed by the system within a short period. For example, when the system automatically lowers the background music volume based on the current attention state, the user manually adjusts the volume back to the original level or higher; or when the system automatically suppresses the playback of a notification sound, the user manually triggers the resumption of playback or views the collapsed notification details. These behaviors clearly express the user's dissatisfaction or rejection of the current automatic strategy, and are therefore captured and marked as valid negative feedback signals by the system. Compared to traditional feedback collection methods that require users to actively rate or fill out questionnaires, this feedback mechanism based on implicit behavior capture is not only zero-intrusion and low-burden, but also more realistically and instantly reflects the user's subjective preferences in specific scenarios, providing high-value supervisory data for subsequent model optimization.

[0132] Based on this, the method also includes: online incremental updates of the fusion weights for generating continuous attention state representations and the decision model for determining audio processing strategies based on negative feedback signals.

[0133] In this application, the system uses the collected negative feedback signal as input to the loss function and employs an online learning algorithm (such as single-step update of stochastic gradient descent) to fine-tune the model parameters. For the generation of continuous attention state representations, the updated objects mainly include the first-order weighting terms in the aforementioned weighted fusion formula. and cross term weights This allows the model to correct attention evaluation biases for specific sensor combinations. For determining audio processing strategies, the updated model maps content type and attention score to processing configuration, gradually converging to the optimal solution that aligns with the user's individual habits. This online incremental update mechanism enables the system to continuously evolve, effectively solving the adaptation inaccuracies caused by a lack of individual data in the cold start phase of general models. It also adapts to the dynamic changes in the same user's preferences at different times, truly achieving personalized audio services tailored to each individual user.

[0134] In addition to online incremental updates based on single-user local data, federated learning mechanisms can be used to further improve the model's generalization ability and cold start performance.

[0135] In this application, while protecting user privacy, each terminal device uploads the gradient or parameter increment of its local model update to the cloud server. The cloud server aggregates the local model updates from multiple users to obtain better general model parameters, and then distributes the aggregated general model to each terminal device as the initial model. This approach allows new users to obtain a general model optimized with a large amount of user data from the very beginning, significantly shortening the adaptation period during the cold start phase. At the same time, the user's original sensor data is always kept locally and is not uploaded to the cloud, fully protecting user privacy and security.

[0136] In one possible implementation, the method further includes: when the fine-grained content type identifier indicates that the currently playing content is immersive content, and the continuous attention state representation is greater than a preset attention threshold, the output mode of the system notification is switched from audio playback to haptic feedback or screen visual cues.

[0137] In this application, immersive content specifically refers to content types that require extremely high purity of the auditory environment and where users are typically in a highly focused or relaxed state. This includes at least ASMR (Autonomous Sensory Meridian Response) content, and may also encompass guided meditation, white noise for sleep aid, and similar scenarios. In these scenarios, any sudden audio notification will instantly shatter the immersive atmosphere, causing a strong sense of disturbance. Therefore, when the system recognizes that such content is currently playing, and infers through non-audio sensor data fusion that the user's level of attention is high (i.e., continuous attention state representation exceeds a preset threshold, indicating that the user is in a deep immersive state rather than simply having the content playing), it will automatically intercept system notifications that should be output in audio form and convert them into non-auditory modal prompts. Haptic feedback can be achieved by driving the device's vibration motor to convey the notification information through specific vibration patterns (such as gentle pulse vibrations); screen visual prompts can be implemented by illuminating the screen edge breathing light, displaying floating icons, or showing silent message cards on the lock screen.

[0138] This design fully demonstrates the system's ability to protect high-value immersive experiences. While ensuring that users do not miss important information, it maximizes the auditory purity and psychological comfort in specific scenarios, which is an important manifestation of the intelligence and humanization of audio processing.

[0139] The above embodiments will be further explained below with reference to several typical application scenarios.

[0140] Scenario 1: Blocking game push notifications while focused on podcast listening: A user is listening to an educational podcast with headphones. Channel A identifies it as a podcast, the screen lights up and there are frequent touches, and the attention score in Channel B is A=0.85, indicating focused listening. At this time, a background game triggers a push notification, and the new request semantics are game sound effects. The content type relationship is determined to be mutually exclusive, and the negotiation module downgrades the game sound effects to silent or very short notifications, so that the podcast is not interrupted.

[0141] Scenario 2: Notification replacement when listening to ASMR before bed: When a user listens to ASMR before bed, channel A identifies it as ASMR. The ambient light is dim, the device is idle, the screen is off, and the attention score indicates a relatively focused and immersive state. When a system notification is received at this time, the system does not play a notification sound, but instead switches the notification output method to haptic feedback or screen visual prompts, so as not to disrupt the immersive feeling of ASMR.

[0142] Scenario 3: Frequency Domain Avoidance and Coexistence of Navigation and Background Music: When a user is driving, background music is played and navigation is used simultaneously. The content types of the two are determined to be complementary, and a primary and secondary dual-channel is constructed. When the navigation is actually broadcast, frequency domain selective mixing applies notch attenuation to the music only in the human voice frequency band, making the navigation voice clear and distinguishable. During the pause of the navigation voice, the music automatically returns to normal volume across the entire frequency band.

[0143] Scenario 4: Background music dialogue avoidance when watching video tutorials: When a user watches a video tutorial with narration dialogue, channel A identifies it as the teaching video dialogue, which scores highly in attention, while background music plays in the background; the system applies notch filtering to the secondary channel music in the active frequency band of the dialogue to make the narration clear, rather than simply reducing the music volume globally.

[0144] The above describes an audio processing method provided by an embodiment of this application. The following will describe the apparatus for performing the above audio processing method.

[0145] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application. Figure 2 As shown, the audio processing device includes:

[0146] The fine-grained content type identifier acquisition unit 11 is used to acquire the fine-grained content type identifier of the first audio stream that is being played, wherein the fine-grained content type identifier represents the semantic scene category to which the first audio stream belongs;

[0147] The continuous attention state representation acquisition unit 12 is used to acquire a continuous attention state representation generated based on non-audio sensor data fusion, wherein the continuous attention state representation reflects the user's level of attention to the current device.

[0148] The processing strategy determination unit 13 is used to determine the audio processing strategy of the first audio stream based on the fine-grained content type identifier and the continuous attention state representation.

[0149] Each unit in the aforementioned audio processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the corresponding operations of each unit.

[0150] This application also provides an electronic device in its embodiments. (See reference...) Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0151] like Figure 3 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0152] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0153] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the audio processing methods provided in this application.

[0154] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the audio processing methods provided in this application.

[0155] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0158] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An audio processing method, characterized in that, include: Obtain the fine-grained content type identifier of the first audio stream that is currently playing, wherein the fine-grained content type identifier represents the semantic scene category to which the first audio stream belongs; Obtain a continuous attention state representation generated based on non-audio sensor data fusion, the continuous attention state representation reflecting the user's level of attention to the current device; Based on the fine-grained content type identifier and the continuous attention state representation, the audio processing strategy for the first audio stream is determined.

2. The method according to claim 1, characterized in that, The step of obtaining the fine-grained content type identifier of the currently playing first audio stream includes: Extract the spectral features of the audio frames of the first audio stream; The audio frame spectral features are classified using an edge-side classification model, and a multi-dimensional content type label vector is output. Temporal smoothing is applied to the multidimensional content type label vector of consecutive audio frames to obtain the fine-grained content type identifier.

3. The method according to claim 2, characterized in that, The multidimensional content type tag vector contains confidence scores for at least ten fine-grained categories, including voice content, music content, and prompts and sound effects. The time-series smoothing process includes: performing a sliding average or majority vote on the multidimensional content type tag vector within a preset time window; and updating the fine-grained content type identifier when the dominant category changes after smoothing and continues for more than a preset retention time.

4. The method according to claim 1, characterized in that, The acquisition of the continuous attention state representation generated based on non-audio sensor data fusion includes: Collect multidimensional non-audio sensor data, which includes device interaction status data and context status data; The multidimensional non-audio sensing data is weighted and fused, and the weighted fusion includes a first-order weighting term and a cross term between the multidimensional non-audio sensing data. The fusion result is normalized by nonlinear mapping to generate the continuous attention state representation with values ​​within a preset continuous interval.

5. The method according to claim 4, characterized in that, The device interaction status data includes screen status and touch interaction frequency within a preset time window; the context status data includes foreground application type, accelerometer output, and ambient light intensity. The weighted fusion uses a nonlinear activation function to nonlinearly map the weighted sum of the first-order weighted terms and the cross terms. The first-order weighted terms are the sum of the products of each normalized non-audio sensor data and its corresponding weight, and the cross terms are the sum of the products of any two non-audio sensor data and their corresponding weights. The weights of the first-order weighted term and the weights of the cross term are updated incrementally online based on the correlation between historical user behavior and audio processing effects.

6. The method according to claim 1, characterized in that, The method further includes: In response to the detection of a playback request for the second audio stream, a content compatibility relationship between the first and second audio streams is determined based on their respective fine-grained content type identifiers. When the content compatibility relationship is complementary, the first and second audio streams are configured as a primary audio stream and a secondary audio stream, respectively, and frequency-domain selective dynamic attenuation is performed on the secondary audio stream based on the real-time energy distribution of the primary audio stream.

7. The method according to claim 6, characterized in that, The frequency-domain selective dynamic attenuation of the secondary audio stream based on the real-time energy distribution of the main audio stream includes: Extract the active frequency bands of the main audio stream in the current frame, where the active frequency bands are a set of frequency points whose energy exceeds an adaptive threshold; Dynamic notch attenuation is applied to the auxiliary audio stream at the frequency point corresponding to the active frequency band, and the attenuation depth of the dynamic notch attenuation is positively correlated with the energy of the main audio stream at the corresponding frequency point; When the main audio stream is in a silent state in the current frame, the full-band gain of the auxiliary audio stream is restored to a non-attenuated state.

8. The method according to claim 7, characterized in that, The step of extracting the active frequency bands of the main audio stream in the current frame includes: performing a short-time Fourier transform on the main audio stream to obtain the energy of each frequency point, and determining the frequency points whose energy exceeds the product of the average energy of the frame and a preset multiplier factor as active frequency points; The step of applying dynamic notch attenuation to the auxiliary audio stream at the frequency point corresponding to the active frequency band includes: calculating the gain of the auxiliary audio stream at each frequency point based on the energy of the main audio stream at the corresponding frequency point, the energy of the auxiliary audio stream at the corresponding frequency point, and the maximum attenuation coefficient, wherein the gain is negatively correlated with the energy of the main audio stream at the corresponding frequency point.

9. The method according to claim 6, characterized in that, The step of determining the content compatibility relationship between the first audio stream and the second audio stream based on their respective fine-grained content type identifiers includes: Based on preset mutual exclusion and complementary matrices, the relationship between the fine-grained content type identifiers of the first audio stream and the second audio stream is determined; When a mutual exclusion relationship is determined, the playback request of the second audio stream is delayed until the first audio stream finishes playing or is added to the queue. When a complementary relationship is determined, the first audio stream and the second audio stream are respectively assigned as the main audio stream and the auxiliary audio stream based on a preset priority rule.

10. The method according to claim 1, characterized in that, The method further includes: In response to changes in the fine-grained content type identifier or the continuous attention state representation that cause a change in the audio processing strategy, the audio processing parameters are interpolated and gradually switched.

11. The method according to claim 1, characterized in that, The method further includes: Collect user feedback on the automatic application of the audio processing strategy, and record the user’s reverse manual operation within a preset time as a negative feedback signal when the user is detected to have performed a reverse manual operation. Based on the negative feedback signal, the fusion weights for generating the continuous attention state representation and the decision model for determining the audio processing strategy are updated incrementally online.

12. The method according to claim 11, characterized in that, The method further includes: When the fine-grained content type identifier indicates that the currently playing content is immersive content, and the continuous attention state representation is greater than the preset attention threshold, the output mode of the system notification is switched from audio playback to haptic feedback or screen visual prompts; the reverse manual operation includes the user manually increasing the volume that has been automatically reduced by the system, or manually restoring the notification prompt sound that has been automatically suppressed by the system.

13. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the audio processing method as described in any one of claims 1 to 12.

14. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the audio processing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the audio processing method as described in any one of claims 1 to 12.

Images