Audio playing processing method and device based on multi-modal data, equipment and medium

Abstract

The application provides an audio playing processing method and device based on multi-modal data, equipment and medium, relates to the technical field of audio processing, and can finely adjust the audio playing parameters combined with multi-dimensional environment data, and improve the adjustment accuracy of the audio playing parameters. The method comprises the following steps: obtaining original data, the original data comprising at least two of environment audio data, visual perception data, sensor data and device state data of an environment where an electronic device is located; performing feature extraction on the original data to obtain a plurality of data feature vectors; performing fusion processing on the plurality of data feature vectors to obtain a fusion feature vector; predicting an audio control parameter according to the fusion feature vector to obtain a plurality of target audio control parameters; and controlling the electronic device to play audio according to the plurality of target audio control parameters.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio playback processing method, apparatus, device and medium based on multimodal data. Background Technology

[0002] During audio playback, the terminal device can adjust the audio playback parameters to adapt to the playback environment of the terminal device.

[0003] As an example, the process of adjusting audio playback parameters is as follows: The terminal device collects ambient sound through the microphone and calculates the sound pressure level of the ambient noise (such as 50dB, 70dB, etc.). When the sound pressure level of the noise exceeds the preset threshold (such as 65dB), the playback volume is automatically increased. When the sound pressure level of the noise is lower than the preset threshold, the playback volume is automatically decreased.

[0004] It is evident that the above-mentioned process of adjusting audio playback parameters relies on monitoring a single ambient noise source, resulting in low adjustment accuracy. Summary of the Invention

[0005] This application provides an audio playback processing method, apparatus, device, and medium based on multimodal data, which can combine multidimensional environmental data to finely adjust audio playback parameters and improve the adjustment accuracy of audio playback parameters.

[0006] The first aspect of this application provides an audio playback processing method based on multimodal data, applied to an electronic device, the method comprising: Acquire raw data, which includes at least two types of data from the environment in which the electronic device is located, including ambient audio data, visual perception data, sensor data, and device status data. The visual perception data is data collected by the camera of the electronic device from the user's face. The device status data includes at least one of the electronic device's battery level, temperature, and operating mode. Feature extraction is performed on the raw data to obtain multiple data feature vectors, which include at least two of the following: environmental audio feature vectors, visual feature vectors, sensor feature vectors, and device status feature vectors. The multiple data feature vectors are fused to obtain a fused feature vector. Based on the fused feature vector, the audio control parameters are predicted to obtain multiple target audio control parameters; The electronic device is controlled to play audio according to the plurality of target audio control parameters.

[0007] In one possible implementation, the step of predicting audio control parameters based on the fused feature vector to obtain multiple target audio control parameters includes: The fused feature vector is input into the target neural network model to obtain the output data of the target neural network model, and the output data is the plurality of target audio control parameters; The target neural network model includes a preprocessing module and multiple output sub-modules. The preprocessing module preprocesses the fused feature vector to obtain a processed fused feature vector. Each output sub-module predicts the processed fused feature vector to obtain at least some of the multiple target audio control parameters. The target neural network model is obtained by training a preset neural network model based on sample data, which includes sample fused features and sample audio control parameters.

[0008] In one possible implementation, the plurality of target audio control parameters include target equalizer parameters, which are multi-dimensional vectors. Controlling the electronic device to play audio according to the plurality of target audio control parameters includes: Based on the value of each element in the target equalizer parameters and the preset coefficient, the target gain coefficient of the filter band that matches the corresponding element is determined, and multiple target gain coefficients of multiple filter bands are obtained. The equalizer of the electronic device includes multiple sub-filters, and the multiple sub-filters include at least one operating frequency band. One element in the target equalizer parameters corresponds to one operating frequency band of one sub-filter. The plurality of sub-filters are controlled to operate according to the plurality of target gain coefficients in order to play the audio.

[0009] In one possible implementation, the plurality of target audio control parameters include target dynamic range compression parameters, wherein the target dynamic range compression parameters are two-dimensional vectors, and controlling the electronic device to play audio according to the plurality of target audio control parameters includes: Based on the value of the first element in the target dynamic range compression parameter and the first calculation formula, the target value of the compression threshold is determined. The first calculation formula is as follows: compression threshold = A1 - B1 × element A, where A1 and B1 are set values, and element A is the first element in the target dynamic range compression parameter. Based on the value of the second element in the target dynamic range compression parameter and the second calculation formula, the target value of the compression ratio is determined. The second calculation formula is as follows: Compression ratio = A2 + B2 × element B, where A2 and B2 are set values, and element B is the second element in the target dynamic range compression parameter. The audio signal amplitude for playback is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

[0010] In one possible implementation, before processing the signal amplitude of the audio signal for audio playback according to the target value of the compression threshold and the target value of the compression ratio before playback, the method further includes: Determine whether the signal amplitude is greater than a preset amplitude threshold; The step of processing the signal amplitude of the audio signal to be played according to the target value of the compression threshold and the target value of the compression ratio before playback includes: When the signal amplitude is greater than the preset amplitude threshold, the signal amplitude of the audio signal to be played is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

[0011] In one possible implementation, the process of fusing the multiple data feature vectors to obtain a fused feature vector includes: Based on the modality type to which each of the various data feature vectors belongs, a modality type vector corresponding to each of the data feature vectors is obtained; Each data feature vector and its corresponding modality type vector are mapped to a preset multidimensional space to obtain multiple projection vectors corresponding to the various data feature vectors respectively. After performing multimodal fusion processing on the multiple projection vectors, a fused sequence is obtained; The fused sequence is subjected to mean pooling to obtain the fused feature vector.

[0012] In one possible implementation, the step of performing multimodal fusion processing on the plurality of projection vectors to obtain the fused sequence includes: The multiple projection vectors are concatenated to obtain a vector sequence; The vector sequence is input into the target Transformer encoder. The multiple projection vectors in the vector sequence are aligned and fused using the target attention weight matrix in the target Transformer encoder to obtain the output sequence of the target Transformer encoder. The output sequence is the fused sequence. The number of layers in the target Transformer encoder is the same as the number of modal types in the various data feature vectors. The target Transformer encoder is obtained by training the initial Transformer encoder based on the sample sequence, which includes the sample input sequence and the sample fusion sequence.

[0013] A second aspect of this application provides an audio playback processing apparatus based on multimodal data, the apparatus comprising: The acquisition module is used to acquire raw data, which includes at least two types of data from the environment in which the electronic device is located, such as ambient audio data, visual perception data, sensor data, and device status data. The visual perception data is data collected by the camera of the electronic device from the user's face. The device status data includes at least one of the electronic device's battery level, temperature, and operating mode. The feature extraction module is used to extract features from the raw data and obtain multiple data feature vectors, which include at least two feature vectors among environmental audio feature vectors, visual feature vectors, sensor feature vectors, and device status feature vectors. The fusion processing module is used to fuse the multiple data feature vectors to obtain a fused feature vector; The control module is configured to predict audio control parameters based on the fused feature vector to obtain multiple target audio control parameters; and to control the electronic device to play audio based on the multiple target audio control parameters.

[0014] A third aspect of this application provides an electronic device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of the first aspects.

[0015] A fourth aspect of this application provides a storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method as described in any one of the first aspects.

[0016] In the above technical solution, by comprehensively collecting at least two types of data from environmental audio, visual perception, sensors, and device status data, multimodal feature extraction and fusion are performed, and multi-dimensional audio control parameters are predicted based on the fused features. This enables fine-grained adjustment of audio playback, effectively solving the problem of single perception dimension in existing technologies and improving the accuracy of audio control parameter adjustment. Furthermore, since the audio control parameters are adjusted by combining multimodal data, the audio playback parameters can be dynamically optimized according to complex scenarios, which can improve the matching with the scenario in which the terminal device is located. Attached Figure Description

[0017] Figure 1 A flowchart illustrating an example of an audio playback processing method based on multimodal data provided in this application embodiment; Figure 2 A schematic diagram illustrating an example of an electronic device provided in an embodiment of this application; Figure 3 A structural block diagram of an example audio coding subnetwork provided in an embodiment of this application; Figure 4 A structural block diagram of an example of a visual coding subnetwork provided in an embodiment of this application; Figure 5 This is a structural block diagram of an example of a sensor coding subnetwork provided in an embodiment of this application; Figure 6 This is a structural block diagram of an example of a device state coding subnetwork provided in an embodiment of this application; Figure 7 A flowchart of an example of an audio playback processing method provided in an embodiment of this application. Figure 2 ; Figure 8 A flowchart of an example of an audio playback processing method provided in an embodiment of this application. Figure 3 ; Figure 9 A flowchart of an example of an audio playback processing method provided in an embodiment of this application. Figure 4 ; Figure 10 A structural block diagram of an example of an audio playback processing apparatus provided in an embodiment of this application; Figure 11 A structural block diagram of another example of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0019] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. For example, "first instruction" and "second instruction" are used to distinguish different user instructions and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0020] It should be noted that, in this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] Furthermore, the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0022] This application proposes an audio playback processing method based on multimodal data, applied to electronic devices, the method comprising: Acquire raw data, which includes at least two types of data from the environment in which the electronic device is located, including ambient audio data, visual perception data, sensor data, and device status data. The visual perception data is data collected by the camera of the electronic device from the user's face. The device status data includes at least one of the electronic device's battery level, temperature, and operating mode. Feature extraction is performed on the raw data to obtain multiple data feature vectors, including environmental audio feature vectors, visual feature vectors, sensor feature vectors, and device status feature vectors. The multiple data feature vectors are fused to obtain a fused feature vector; Based on the fused feature vector, the audio control parameters are predicted to obtain multiple target audio control parameters; The electronic device is controlled to play audio according to the multiple target audio control parameters.

[0023] In the above technical solution, by comprehensively collecting at least two types of data from environmental audio, visual perception, sensors, and device status data, multimodal feature extraction and fusion are performed, and multi-dimensional audio control parameters are predicted based on the fused features. This enables fine-grained adjustment of audio playback, effectively solving the problem of single perception dimension in existing technologies and improving the accuracy of audio control parameter adjustment. Furthermore, since the audio control parameters are adjusted by combining multimodal data, the audio playback parameters can be dynamically optimized according to complex scenarios, which can improve the matching with the scenario in which the terminal device is located.

[0024] The audio playback processing method based on multimodal data provided in this application embodiment will be described below with reference to the accompanying drawings. It is understood that the method provided in this application embodiment can be applied to electronic devices with computing power and audio playback functions, such as smartphones, tablets, and smart speakers. This electronic device can integrate various sensors for collecting multimodal data required for adjusting audio control parameters. These sensors may include cameras, microphones, etc., which will not be listed here.

[0025] Of course, electronic devices may also include other components, such as Bluetooth modules, etc., without limitation.

[0026] Please refer to Figure 1 The flowchart below is an example of an audio playback processing method based on multimodal data provided in this application embodiment. The flowchart includes the following steps: S101. Obtain the raw data.

[0027] The raw data includes at least two types of data from the environment in which the electronic device is located: ambient audio data, visual perception data, sensor data, and device status data.

[0028] This raw data can be acquired by a data acquisition device in an electronic device. For example, please refer to... Figure 2 This is a schematic diagram illustrating an example of an electronic device in an embodiment of this application. For example... Figure 2 As shown, the electronic device 100 may include a camera 110, a microphone 120, a speaker 130, a sensor 140, and a processor 150.

[0029] The camera 110 can be positioned on the top of the electronic device 100. The number of cameras 110 can be one or more. When multiple cameras 110 are included, they can be positioned on one side of the display screen to form a front-facing camera; then, other cameras can be positioned on the side opposite the display screen to form a rear-facing camera. There are no limitations on the number or position of the cameras 110. The microphone 120 and speaker 130 can both include multiple units, for example, in... Figure 2The device includes two microphones 120 and two speakers 130, respectively located at the top and bottom of the electronic device 100. In cases where multiple microphones 120 are included, these microphones may also include a main microphone and a noise-canceling microphone, with functions similar to those described in related technologies, and will not be elaborated further here. The sensor 140 may include various types of sensors, such as ambient light sensors, inertial sensors, ultrasonic sensors, and temperature sensors, etc., which will not be listed individually here. The processor 150 may be a central processing unit or a graphics processing unit, etc. The processor 150 may contain target applications or target program code for monitoring the operating status data of the electronic device, or the processor 150 may call and query the operating system interface; no limitations are imposed here.

[0030] The raw data in this embodiment may include at least two of the following: audio data of the electronic device's environment collected by microphone 120, visual perception data collected by camera 110, sensor data collected by sensor 140, and device status data monitored by a preset target application or target program code in processor 150 or by calling and querying the operating system interface. Audio data may include human voices or background sounds; visual perception data may include data collected from the user's face; sensor data may include information from ultrasonic or light sensors detecting when a face approaches the phone to control the screen's on / off state, or screen brightness detected by a light sensor, information from an inertial unit (such as a gyroscope) determining the phone's posture (handheld, hands-free, flat on a table), whether a wired headset is plugged in or a Bluetooth headset is connected, information from the connected Bluetooth device monitoring heart rate, whether it is in sports mode, and ambient temperature. The device status data includes at least one of the following: battery level, temperature, operating mode (e.g., power-saving mode or normal operating mode), and data from the currently running application. The currently running application may include a foreground application or a background application, etc., without limitation.

[0031] The acquisition frequency and accuracy of these raw data can be configured according to the actual application scenario. For example, environmental audio data can be collected at a rate of tens of frames per second, visual perception data can be collected at a rate of several frames per second, and device status data can be queried once every few seconds.

[0032] As an example, the microphone array 120 built into the electronic device 100 collects ambient sounds inside the subway car in real time, such as the rumble of the train and the conversations of passengers, forming ambient audio data. Simultaneously, the front-facing camera 110 of the electronic device periodically captures images of user A's face to obtain visual perception data such as whether user A is looking at the screen and their head posture. Furthermore, the device's inertial measurement unit detects the device's motion state (e.g., whether it is vibrating), and the proximity sensor detects the distance between the device and user A's face; these constitute sensor data. The processor 150 of the electronic device also queries its own battery level, processor temperature, and whether it is currently in power-saving mode, among other device status data.

[0033] S102. Perform feature extraction on the original data to obtain multiple data feature vectors, wherein the multiple data feature vectors include at least two of the following: environmental audio feature vector, visual feature vector, sensor feature vector, and device status feature vector.

[0034] After obtaining the raw data, feature extraction is performed on the raw data to obtain various data feature vectors.

[0035] It should be noted that the various data feature vectors correspond one-to-one with the data types contained in the original data. For example, if the original data includes ambient audio data, visual perception data, sensor data, and device status data of the electronic device's environment, then the corresponding various data feature vectors may include audio feature vectors, visual feature vectors, sensor feature vectors, and device status feature vectors. Similarly, if the original data includes ambient audio data, visual perception data, and sensor data, then the various data feature vectors are correspondingly audio feature vectors, visual feature vectors, and sensor feature vectors. It should be understood that the data types covered by the original data can be flexibly set according to actual application needs, and this application embodiment does not specifically limit this.

[0036] For example, for environmental audio data, techniques such as short-time Fourier transform and Mel-frequency cepstral coefficients (MFCC) can be used to extract audio feature vectors reflecting environmental noise intensity, spectral distribution, and acoustic scene classification. For visual perception data, image processing algorithms or pre-trained convolutional neural networks can be used to extract visual feature vectors such as facial key points, eye opening and closing states, head posture, and gaze direction from user facial images. For sensor data, statistical analysis or time-domain feature extraction can be performed on time-series data from accelerometers and gyroscopes, and combined with readings from proximity sensors and ambient light sensors to generate sensor feature vectors. For equipment status data, discrete variables (such as operating modes) can be one-hot encoded, and continuous variables (such as battery level and temperature) can be normalized to obtain equipment status feature vectors.

[0037] S103. The multiple data feature vectors are fused to obtain a fused feature vector.

[0038] For example, all extracted feature vectors can be combined to form a multi-dimensional vector as a fused feature vector. Alternatively, a weight can be assigned to each feature vector, and then a weighted sum or weighted average can be performed to highlight the importance of certain modalities.

[0039] S104. Based on the fused feature vector, the audio control parameters are predicted to obtain multiple target audio control parameters.

[0040] For example, the fused feature vector can be input into a pre-trained machine learning model, which could be a support vector machine, random forest, or a simple multilayer perceptron. This pre-trained machine learning model is trained to predict the values ​​of multiple audio control parameters based on the input features, such as volume gain, equalizer parameters, and dynamic range compression parameters. These predicted parameters are optimized for the currently perceived environment and user state, aiming to provide a more comfortable, clearer, or more immersive listening experience. The training process of this pre-trained machine learning model is similar to that in related technologies. For example, it involves obtaining the fused feature vector and reference audio control parameters from the sample scene. These reference audio control parameters can be audio control parameters that allow the user to obtain sufficient clarity in the sample environment. Then, an initial model is trained based on the fused feature vector and reference audio control parameters from the sample scene to obtain the pre-trained machine learning model. Further details are omitted here.

[0041] For example, after receiving this fused feature vector, a pre-trained model might predict the following target audio control parameters: target volume gain of 75% (relative to the original set volume), target equalizer parameters indicating that the mid-frequency band (e.g., 1kHz-4kHz) needs to be boosted by 3dB, and target dynamic range compression parameters indicating a compression threshold of -25dB and a compression ratio of 3:1.

[0042] S105. Control the electronic device to play audio according to the plurality of target audio control parameters.

[0043] The processor 150 of the electronic device can receive these target audio control parameters and send them to the audio rendering engine, so that the audio rendering engine can adjust the processing method of the audio signal according to the target audio control parameters. Of course, the processor 150 can also directly adjust the audio playback parameters according to the target audio control parameters, which is not limited here.

[0044] If the predicted target volume gain is a specific value, the audio playback volume will be adjusted accordingly. If the predicted target equalizer parameters indicate that a certain frequency band needs to be boosted or attenuated, the spectral characteristics of the audio signal will be modified in real time. To avoid abrupt auditory changes, these parameter adjustments can be made in a smooth transition manner, such as through linear interpolation or exponential decay algorithms, or by using a fixed step size, so that the parameter values ​​gradually change to the target value over a certain period of time. For example, if the volume changes gradually in 0.5dB / second, and it is confirmed whether the app is closed, if the app is closed, the electronic device can enter sleep mode to save power and continue to the next monitoring cycle. The cycle can be set to 200-500ms, forming a continuous adaptive closed loop.

[0045] Using the example above, the audio rendering engine of an electronic device receives a 75% volume gain instruction, thereby increasing the overall playback volume of the music. Simultaneously, the equalizer enhances the mid-frequency range of the music signal by 3dB according to the instruction, making vocals and instrumental details more penetrating in noisy environments. The dynamic range compressor processes the audio signal with a -25dB threshold and a 3:1 compression ratio, ensuring that loud parts of the music are not overloaded, while quiet parts are still clearly audible, thus maintaining overall loudness while avoiding noise masking and auditory fatigue.

[0046] This method achieves deep perception of the environment and user status of electronic devices by acquiring multimodal raw data, including environmental audio data, visual perception data, sensor data, and device status data. For example, in a subway scenario, this method not only perceives the intensity of environmental noise but also uses visual perception data to determine that user A is looking at the screen (indicating an active willingness to listen), uses sensor data to identify that the device is vibrating (further confirming the subway environment), and uses device status data to understand the device's operating status. This comprehensive acquisition of multimodal data overcomes the problem of single-dimensional perception in existing technologies.

[0047] Furthermore, this method extracts features from these multimodal raw data and performs fusion processing to obtain a fused feature vector. This fused feature vector provides a more comprehensive representation of the complex context. Through fusion processing, information from different modalities is complementary and enhanced. For example, visual data can confirm user intent, and sensor data can help determine the type of environment, thereby improving the accuracy of identifying the current scene of the electronic device. Based on this, audio control parameters are predicted according to the fused feature vector to obtain multi-dimensional and refined audio control parameters, including volume gain, equalizer parameters, and dynamic range compression parameters.

[0048] As an example, the implementation process of step S102 will be described in detail below.

[0049] Electronic devices can incorporate multimodal feature extraction encoders, whose coding sub-networks correspond to the data types of the original data. For example, if the original data includes environmental audio data, visual perception data, sensor data, and device status data, then the corresponding feature vectors include audio feature vectors, visual feature vectors, sensor feature vectors, and device status feature vectors. In this example, the multimodal feature extraction encoder comprises four parts: an audio coding sub-network, a visual coding sub-network, an auxiliary sensor sub-network, and a device status coding sub-network.

[0050] The system comprises several sub-networks: an audio coding sub-network for analyzing environmental audio data and extracting noise intensity, spectral characteristics, and acoustic scene classification; a visual coding sub-network for analyzing visual perception data and acquiring information including facial key points, eye opening / closing status, head posture, gaze direction, ambient lighting, and environmental information; an auxiliary sensor sub-network for analyzing sensor data and acquiring information on device posture, motion status, ear-to-ear distance, and ambient brightness; and a device status coding sub-network for analyzing device status data and monitoring the current battery level, processor temperature, operating mode (performance / power saving), current foreground application type (music player app, video call app, game app), and headphone status (none / wired / Bluetooth in-ear / Bluetooth over-ear). A multimodal feature extraction encoder converts the raw sensor data into high-dimensional feature vectors.

[0051] The specific construction of each coding subnetwork is explained below.

[0052] 1. Audio coding sub-network: In this embodiment of the application, taking the acquisition of ambient audio data by a microphone array in a 2-second sliding window as an example, the input of the audio coding sub-network is the ambient audio data acquired by the microphone array.

[0053] Please refer to Figure 3 Here is a block diagram of an example audio coding subnetwork, such as... Figure 3As shown, the audio coding sub-network includes three modules: a first processing module 301, a first convolutional network 302, and a first output module 303. The first processing module 301 preprocesses the environmental audio data. For example, using short-time Fourier transform (SFT) as the preprocessing method, the first processing module 301 performs SFT on the environmental audio data to generate a Mel spectrogram. As an example, the size of this Mel spectrogram is 128 frequency bands × 100 frames. The first convolutional network 302 can be a four-layer two-dimensional convolutional network. The Mel spectrogram is input into the first convolutional network to obtain its output. Then, the output of the first convolutional network is input into the second convolutional network to obtain its output, and so on, until the output of the third convolutional network is obtained. The output of the third convolutional network is then input into the fourth convolutional network, which performs global average pooling and finally outputs the first multi-dimensional feature.

[0054] As an example, the dimensions of the output data of a four-layer two-dimensional convolutional network are as follows: Layer 1: 3×3 convolution, 64 channels, stride 2 → output dimension (64, 50, 64); Layer 2: 3×3 convolution, 128 channels, stride 2 → Output dimensions (128, 25, 32); Layer 3: 3×3 convolution, 256 channels, stride 2 → output dimension (256, 13, 16); Layer 4: Global average pooling → Output 256-dimensional features.

[0055] As shown in the example above, the output of the four-layer two-dimensional convolutional network is the first multidimensional feature, which has 256 dimensions. This first multidimensional feature is then input into the first output module 303. The first output module 303 is used to perform dimensionality reduction on the first multidimensional feature; for example, the fully connected layer reduces the dimensionality to 128 dimensions, outputting an audio feature vector F_audio. This audio feature vector contains information such as environmental noise intensity, spectral distribution, and acoustic scene classification (e.g., traffic / office / indoor).

[0056] 2. Visual coding subnetwork: In this embodiment of the application, taking the video frame sequence (e.g., 10 frames) captured by the front-facing camera 110 in a 2-second sliding window as an example, the input of the visual coding sub-network is the visual perception data captured by the camera.

[0057] Please refer to Figure 4 Here is a block diagram of an example visual coding subnetwork, such as... Figure 4As shown, the visual encoding sub-network includes five modules: a second processing module 401, a second convolutional network 402, a first dimensionality reduction module 403, an aggregation module 404, and a second output module 405. The second processing module 401 scales each frame of the image to a preset pixel range, which can be 224×224 pixels, to obtain scaled visual perception data. This scaled visual perception data is then input to the second convolutional network 402 to extract and output multi-dimensional image features. As an example, the MobileNetV3 lightweight convolutional network outputs a 1280-dimensional image feature vector. Then, the multi-dimensional image features are input to the first dimensionality reduction module 403 to perform dimensionality reduction processing, obtaining dimensionality-reduced multi-dimensional image features. For example, global average pooling can reduce the 1280-dimensional image feature vector to 512 dimensions. The dimensionality-reduced multidimensional image features are input to the aggregation module 404, which aggregates temporal information to obtain aggregated multidimensional image features. For example, the aggregation module 404 can be a two-layer Long Short-Term Memory (LSTM) network, where the hidden layer has 256 dimensions. The aggregated multidimensional image features are then input to the second output module 405, where the fully connected layer reduces the dimensionality to 256 dimensions and outputs a visual feature vector F_visual. This visual feature vector contains information such as facial key points, eye opening and closing status, head pose, gaze direction, ambient lighting, and environmental information.

[0058] 3. Sensor coding sub-network: In this embodiment, taking sensor data acquired by sensor 140 at a 2-second sliding window and a sampling frequency of 100Hz as an example, the sensor data may include time-series data from an accelerometer + gyroscope (inertial measurement unit), an ultrasonic / optical sensor, and an ambient light sensor, without limitation. As an example, the input of the sensor coding sub-network is (batch, 200, 6), that is, the number of independent samples in one training process, such as batch=32 representing 32 segments of sensor data. Each sample in a batch is a time series of length 200, containing 6 sensor features at each moment, specifically: data acquired from 3 accelerometer axes (usually linear acceleration in the x, y, and z directions) and data acquired from 3 gyroscope axes (usually angular velocity in the x, y, and z directions). For example, each segment contains 200 consecutive sampling points, and each sampling point has 6 feature dimensions.

[0059] Please refer to Figure 5 Here is a block diagram of an example sensor coding subnetwork, such as... Figure 5As shown, the sensor coding sub-network includes four modules: a third convolutional network 501, a first pooling module 502, a second dimensionality reduction module 503, and a third output module 504. The third convolutional network 501 can be a three-layer one-dimensional convolutional network: Layer 1: convolutional kernel 5, 32 channels, stride 1 → output (batch, 200, 32); Layer 2: convolutional kernel 3, 64 channels, stride 2 → output (batch, 100, 64); Layer 3: convolutional kernel 3, 128 channels, stride 2 → output (batch, 50, 128). Then, the output vector of the third convolutional network 501 is input into the first pooling module 502, which performs global average pooling and outputs a 128-dimensional feature vector. Then, through the second dimensionality reduction module 503, the fully connected layer reduces the dimensionality to 64 dimensions and outputs the sensor initial vector. The sensor initial vector is input into the third output module 504, which concatenates the proximity sensor distance value (1-dimensional) and the ambient light value (1-dimensional) with the transmitter initial vector to output the sensor feature vector F_sensor. This sensor feature vector is a 66-dimensional vector (64-dimensional motion features + 1-dimensional distance + 1-dimensional illumination), which contains information such as device posture, motion state, contact distance, and ambient brightness.

[0060] 4. Device Status Coding Sub-network: Please refer to Figure 6 Here is a block diagram of an example device state coding subnetwork, such as... Figure 6 As shown, the device status coding sub-network includes a third processing module 601, a splicing module 602, and a fourth output module 603. The third processing module 601 includes two methods: embedding discrete data or embedding discrete data after normalization. For example, application type (music / video / game / call) → 8-dimensional embedding; headphone status (none / wired / Bluetooth in-ear / Bluetooth headband) → 8-dimensional embedding; operating mode (performance / balanced / power saving) → 4-dimensional embedding; audio output (earpiece / speaker / other peripherals) → 4-dimensional embedding; battery percentage (0-100) → normalized to 0-1 → 1-dimensional embedding; processor temperature (0-100℃) → normalized to 0-1 → 1-dimensional embedding; screen brightness (0-1) → 1-dimensional embedding; current system volume (0-1) → 1-dimensional embedding. The embedded vector is then input into the splicing module 602 for splicing to obtain a spliced ​​vector, such as a 28-dimensional vector. The spliced ​​vector is then input into the fourth output module 603, which, through a fully connected layer, outputs a 32-dimensional vector from the 28-dimensional input, resulting in a device state feature vector F_context, which includes information such as device battery level, temperature, operating mode, application type, headphone connection status, audio output, screen brightness, and current volume.

[0061] The above describes one implementation of step S102. Of course, those skilled in the art can also perform step S102 in other ways, and no restrictions are imposed here.

[0062] The implementation method of step S104 will be explained below.

[0063] In one possible implementation, step S104 may include the following steps: The fused feature vector is input into the target neural network model to obtain the output data of the target neural network model, and the output data is the plurality of target audio control parameters; The target neural network model includes a preprocessing module and multiple output sub-modules. The preprocessing module is used to preprocess the fused feature vector to obtain a processed fused feature vector. Each output sub-module is used to predict the processed fused feature vector to obtain at least some of the audio control parameters among the multiple target audio control parameters. The target neural network model is obtained by training a preset neural network model based on sample data, and the sample data includes sample fused features and sample audio control parameters.

[0064] The target neural network model is a machine learning model that can predict audio control parameters by learning the complex mapping relationship between input and output data. This model can employ various architectures, such as deep learning-based multilayer perceptrons, recurrent neural networks, or convolutional neural networks, or even more advanced Transformer models. Its implementation can include, but is not limited to, using deep learning frameworks such as TensorFlow and PyTorch; or using ensemble models built with machine learning libraries such as Scikit-learn.

[0065] In this embodiment, the target neural network model comprises two parts: a preprocessing module and multiple output sub-modules. The preprocessing module further optimizes and transforms the input fused feature vector to improve its effectiveness in subsequent prediction tasks. This processing may include, but is not limited to: normalizing or standardizing the feature vector to eliminate dimensional differences; reducing dimensionality through methods such as Principal Component Analysis (PCA) or t-SNE to reduce redundancy; or enhancing features through nonlinear transformations, thereby improving the model's convergence speed and prediction accuracy. The multiple output sub-modules are modules in the target neural network model used to predict different types of audio control parameters. For example, one sub-module may focus on predicting equalizer parameters, while another sub-module is responsible for predicting dynamic range compression parameters. This modular design allows the model to optimize learning for the characteristics of each parameter, thereby improving the specificity and accuracy of predictions. Each output sub-module can be a simple fully connected layer or a small neural network with a multi-layered structure.

[0066] Sample data is the dataset used to train the target neural network model. It contains all the information needed for the model to learn the input-output mapping relationship. Sample data typically consists of pairs of sample fusion features and corresponding sample audio control parameters. The sample fusion features have the same structure as the fusion feature vector in actual applications, while the sample audio control parameters are the corresponding ground truth values ​​or expert-annotated values. This data can be obtained through manual annotation, expert experience, or collection in a controlled environment. A pre-defined neural network model refers to a neural network model whose basic architecture (such as the number of network layers, the number of neurons per layer, and the type of activation function) and initial weight parameters have been determined before training. This model can be designed based on domain knowledge or can be a model that has undergone preliminary or pre-training. By training the pre-defined neural network model, it can be adapted to a specific prediction task, thereby obtaining the final target neural network model.

[0067] As an example, the preprocessing module can be a multi-layered shared network, the functions of which are shown in Table 1.

[0068] Table 1

[0069] As shown in Table 1, the three layers of the multilayer shared network are connected sequentially, with the output of the previous layer serving as the input of the next layer, and finally outputting a 64-dimensional vector.

[0070] Each of the multiple output submodules is connected to the preprocessing module. If the preprocessing module is the multi-layer shared network described in Table 1, then each output submodule is connected to the S3 layer of the shared network and processes the 64-dimensional vector output by the S3 layer to obtain the corresponding output result.

[0071] As an example, multiple output submodules can include 6 output submodules, labeled as branch 1 to branch 6. Each branch corresponds to an audio control parameter. For example, the 6 branches can correspond to volume gain parameter, equalizer coefficient parameter, dynamic range compression parameter, spatial audio intensity parameter, sound effect mode parameter, and APP status parameter, as shown in Table 2.

[0072] Table 2

[0073] In other words, multiple output submodules can be used to obtain target values ​​for multiple audio control parameters. The outputs of different output submodules can include fully connected layers and an activation function. For example, the outputs of branches 1, 3, and 4 include fully connected layers and the sigmoid activation function; the output of branch 2 includes a fully connected layer and the Tanh activation function; and the outputs of branches 5 and 6 include a fully connected layer and the softmax activation function. Different activation functions may correspond to different ranges of output values. For example, the Tanh activation function has an output range of -1 to 1, while other activation functions have an output range of 0 to 1.

[0074] Since the output values ​​of multiple output submodules are in the range of 0-1 or -1 to 1, while the value range of audio control parameters may not be within this range, after obtaining the output values ​​of multiple output submodules, it is necessary to map the output values ​​of multiple submodules to the value range of audio control parameters. For example, for branch 1, the output value range of the sigmoid function is 0~1, which corresponds to the volume bar, where 0 represents mute and 1 represents 100% maximum volume; for branch 5, its output dimension is 3 dimensions, with different dimensions corresponding to the probabilities of three sound effects: music, movie, and speech; for branch 4, similar to branch 1, the output value range of the sigmoid function is 0~1, where 0 represents minimum intensity and 1 represents 100% maximum intensity.

[0075] By employing the aforementioned technical solution, the fused feature vector is input into a target neural network model containing a preprocessing module and multiple output sub-modules for audio control parameter prediction. This effectively addresses the potential accuracy limitations and tuning difficulties of single prediction models when handling complex and diverse audio control parameters. The preprocessing module's optimization of the fused feature vector improves feature quality and model learning efficiency. The parallel prediction mechanism of multiple output sub-modules enables the model to learn and predict specific types of audio control parameters, significantly enhancing the accuracy and precision of predictions for various audio control parameters. Furthermore, training the pre-defined neural network model with sample data ensures that the model learns optimal parameter adjustment strategies from real-world scenarios. This allows electronic devices to more intelligently and adaptively adjust audio playback based on real-time multimodal environmental information, providing users with a more personalized and immersive listening experience. This modular and specialized prediction method makes audio playback control more flexible and efficient.

[0076] The equalizer parameters and dynamic range compression parameters are explained below.

[0077] As an example, the equalizer of an electronic device does not employ a single filter type. Instead, it includes multiple sub-filters, each with at least one operating frequency band. One element in the target equalizer parameters corresponds to one operating frequency band of a sub-filter. Therefore, the optimal filter combination can be selected and the gain values ​​of different filters adjusted based on the characteristics of different frequency bands. The frequency band division of the equalizer is shown in Table 3.

[0078] Table 3

[0079] In this case, please refer to Figure 7 The implementation method of step S105 is as follows: S701. Based on the value of each element in the target equalizer parameters and the preset coefficient, determine the target gain coefficient of the filter band that matches the corresponding element, and obtain multiple target gain coefficients for multiple filter bands.

[0080] S702. Control the plurality of sub-filters to operate according to the plurality of target gain coefficients in order to play the audio.

[0081] In this scheme, the equalizer parameters are multi-dimensional vectors, as shown in Table 2. They can be 9-dimensional vectors, that is, the filter coefficients of the output equalizer frequency band division. Each dimension is mapped to a value between -1 and 1 through the Tanh function, and multiplied by a preset coefficient, such as a constant of 12, to determine the gain coefficient Q of the filter frequency band corresponding to each element. When the preset coefficient is a constant of 12, the gain coefficient Q of each filter frequency band is -12dB to 12dB. The multimodal neural network adjusts the equalizer Q according to the scene, that is, the effect of playing sound.

[0082] The equalizer in an electronic device can be a 9-band parametric equalizer integrated into an audio DSP chip. After predicting the target equalizer parameters, the target gain coefficients for the 9 filter bands can be calculated using the method described above. These target gain coefficients are then written into the registers of the corresponding sub-filters in the DSP chip, allowing each sub-filter to process the audio signal in real time according to the new gain value, thereby changing the timbre of the audio. As shown in Table 3, band 1 (frequency range 20~50Hz) uses a low-frequency shelving filter, bands 2~8 use peak filters, and band 9 uses a high-frequency shelving filter. These equalizers adjust the filter gain Q value based on the center frequency. As an example, in a scenario where someone is listening to music with headphones while commuting on the subway, when a multimodal audio neural network processes the noise from the subway headphones, the equalizer performs the following adjustments: a 3dB boost in the mid-frequency band (1kHz~4kHz) to enhance the penetration of musical details; a 2dB boost in the low-frequency band (64Hz) to compensate for subway noise masking; and slight adjustments to other frequency bands.

[0083] Through the above technical solution, this application can specifically apply the abstract audio control parameters predicted from multimodal data to the equalizer of electronic devices, achieving refined and adaptive adjustment of the audio playback timbre. This method overcomes the limitations of traditional audio playback control, where equalizer parameter settings typically rely on preset modes or manual adjustments. It allows audio playback to dynamically optimize the audio frequency response based on real-time information such as the user's environment, visual perception, and device status. For example, in noisy environments, the system can automatically increase the gain in the mid-to-high frequency bands to enhance speech clarity; when the user's facial expression is relaxed, it can adjust to a softer timbre. This significantly improves the user experience, making audio playback more intelligent and personalized, thus providing a high-quality listening experience even in complex and ever-changing usage scenarios.

[0084] As an example, multiple target audio control parameters include target dynamic range compression parameters, which are two-dimensional vectors. In this case, please refer to... Figure 8 The implementation method of step S105 is as follows: S801, based on the value of the first element in the target dynamic range compression parameter and the first calculation formula, determine the target value of the compression threshold. The first calculation formula is as follows: compression threshold = A1 - B1 × element A, where A1 and B1 are set values, and element A is the first element in the target dynamic range compression parameter. S802, based on the value of the second element in the target dynamic range compression parameter and the second calculation formula, determine the target value of the compression ratio. The second calculation formula is as follows: Compression ratio = A2 + B2 × element B, where A2 and B2 are set values, and element B is the second element in the target dynamic range compression parameter. S803, the signal amplitude of the audio signal to be played is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

[0085] As an example, A1 and B1 can be determined based on the range of compression threshold values; A2 and B2 can be determined based on the range of compression ratio values. For example, Table 4 below shows an example of a method for calculating the compression threshold and compression ratio provided in an embodiment of this application.

[0086] Table 4

[0087] As shown in Table 4, the compression threshold ranges from -20dB to -60dB. The compression threshold (Threshold) = -20 - 40 × element A (param_threshold), where A1 is -20 and B1 is 40. Element A ranges from 0 to 1. In this example, when element A is 0, the compression threshold is -20dB; when element A is 1, the compression threshold is -60dB. The compression ratio ranges from 1:1 to 10:1. The compression ratio (ratio) = 1 + 9 × element B (param_ratio), where A2 is 1 and B2 is 9. Element B ranges from 0 to 1. In this example, when element B is 0, the compression ratio is 1:1; when element B is 1 / 3, the compression ratio is 4:1; and when element B is 1, the compression ratio is 10:1.

[0088] It should be noted that the range of values ​​for the compression threshold and compression ratio can be set according to actual needs, and this application embodiment does not limit this. Correspondingly, the values ​​of A1, A2, B1, and B2 can also be determined according to the range of values ​​for the compression threshold and compression ratio, and this application embodiment does not limit this.

[0089] Different compression ratios have different effects. For example, when the compression ratio is 1:1, there is no compression, and the original dynamic range is preserved; when the compression ratio is 2:1, there is light compression, which is suitable for music listening and other scenarios, preserving the musicality; when the compression ratio is 3:1-4:1, there is medium compression, which is suitable for movies, phone calls and other scenarios, improving intelligibility; when the compression ratio is 5:1-8:1, there is heavy compression, which is suitable for noisy environments and ensures clarity; when the compression ratio is 10:1 or higher, there is extreme compression / limiting, which is suitable for night mode and prevents popping sounds.

[0090] These calculated target values ​​for compression thresholds and compression ratios are then passed to the digital signal processor (DSP) or audio processing module within the electronic device. This module monitors the input amplitude of the audio signal in real time and dynamically adjusts the audio signal gain based on the received target compression thresholds and compression ratios. For example, when the instantaneous amplitude of the audio signal exceeds the target compression threshold, the DSP attenuates the signal according to the target compression ratio, ensuring that the amplitude of the output audio signal remains within an acceptable range, thus providing a smoother and more perceptible listening experience during playback.

[0091] Through the above implementation methods, this application can adaptively predict and apply dynamic range compression parameters based on the complex environment and user state reflected by multimodal data. This refined dynamic range control effectively solves the problem of auditory discomfort or information loss caused by excessive dynamic range in traditional audio playback. Specifically, by dynamically adjusting the compression threshold and compression ratio, the system can avoid the masking of quiet parts in noisy environments or the excessive harshness of loud parts in quiet environments, significantly improving the clarity, comfort, and overall auditory quality of audio playback. This allows electronic devices to provide users with a more personalized and optimized audio experience in various usage scenarios, thereby improving user satisfaction.

[0092] As an example, in practical applications, indiscriminately compressing the dynamic range of all audio signals may lead to unnecessary processing of signals with inherently low amplitudes. This could not only introduce undesirable auditory effects, such as loss of detail or unnatural sound, but also waste computational resources. Therefore, this application embodiment proposes, before processing the signal amplitude of the audio signal to be played according to the target value of the compression threshold and the target value of the compression ratio, determining whether the signal amplitude is greater than a preset amplitude threshold. In this case, step S803 is implemented as follows: When the signal amplitude is greater than the preset amplitude threshold, the signal amplitude of the audio signal to be played is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

[0093] This judgment can be achieved by monitoring the instantaneous amplitude or root mean square (RMS) amplitude of the audio signal in real time and comparing it with a preset amplitude threshold. Alternatively, a level detector can be set in the digital signal processing chain. This detector continuously analyzes the peak or average level of the input audio stream and outputs a Boolean signal indicating whether the current level exceeds the preset threshold. The preset amplitude threshold is a pre-determined reference value used to determine whether the audio signal amplitude needs dynamic range compression processing. It defines the minimum signal strength required to trigger dynamic range compression. This threshold can be set as a fixed decibel (dB) value based on principles of auditory psychology or the needs of specific application scenarios, through experimentation or experience. Alternatively, the threshold can be dynamically adjusted based on factors such as ambient noise levels, user preferences, or the maximum output capability of the device. For example, a lower threshold can be set in a quiet environment, while a higher threshold can be set in a noisy environment.

[0094] When the signal amplitude exceeds a preset amplitude threshold, the audio signal amplitude is adjusted in real time according to the target compression threshold and target compression ratio, and then the processed signal is sent to the playback unit. Alternatively, in a digital audio workstation (DAW) or embedded audio processor, the activation of the compression algorithm can be controlled by conditional statements or state machines. Once the signal amplitude is detected to meet the condition, the gain of the audio signal is attenuated according to the compression ratio so that its peak value does not exceed the compression threshold, thereby achieving dynamic range control.

[0095] If the signal amplitude does not reach the threshold, dynamic range compression will not be performed, and the audio signal will be played with its original dynamic range, or only other uncompressed processing will be performed.

[0096] By employing this conditional dynamic range compression process, unnecessary compression of already weak signals is avoided, thus preventing potential signal distortion or unnatural sound, while also reducing unnecessary computational burden. In this way, the proposed solution can more intelligently adapt to the characteristics of audio signals, optimizing dynamic range when needed and maintaining the original signal state when not required, thereby improving processing accuracy and efficiency while ensuring audio quality.

[0097] As an example, please refer to Figure 9 The implementation method of step S103 will be explained below.

[0098] S901, based on the modality type to which each of the multiple data feature vectors belongs, obtain the modality type vector corresponding to each of the data feature vectors.

[0099] In some implementations, a learnable modality type identifier vector can be pre-set for each modality type, and the identifier vectors of all modality types form a set of vectors with a dimension of 4×256, where 4 represents the total number of modality types and 256 represents the dimension of the uniform embedding space.

[0100] For any data feature vector, first determine its modality type, and then select the corresponding identifier vector from the set of learnable modality type identifier vectors according to the modality type, as the modality type vector of the data feature vector.

[0101] As an example, the preset modal types include four types: audio modality, visual modality, sensor modality, and context modality. Correspondingly, four learnable modality type identifier vectors are set, denoted as audio modality identifier vector, visual modality identifier vector, sensor modality identifier vector, and context modality identifier vector, respectively, with each identifier vector having a dimension of 256. When an audio modality feature vector is included among the multiple data feature vectors, the audio modality identifier vector is selected as the modality type vector corresponding to that feature vector, based on its audio modality type. Similarly, if a data feature vector belongs to the visual modality, the visual modality identifier vector is selected; if it belongs to the sensor modality, the sensor modality identifier vector is selected; and if it belongs to the context modality, the context modality identifier vector is selected.

[0102] In this way, each data feature vector can obtain a modality type vector that matches its modality source, which can be used for subsequent modality identification and fusion processing.

[0103] S902, map each data feature vector and its corresponding modality type vector to a preset multidimensional space to obtain multiple projection vectors corresponding to the various data feature vectors respectively.

[0104] The preset multidimensional space refers to a unified embedded space, whose dimension is denoted as dmodel. In this application embodiment, dmodel=256 is used as an example.

[0105] For each modality's data feature vector, a linear projection layer corresponding to that modality type can be pre-defined to map the modality's feature vector from the original input dimension to the unified embedding space. The linear projection layer is a fully connected layer with no bias or with learnable bias, and its weight matrix has a size of dmodel×din, where din is the original dimension of the modality's feature vector.

[0106] For any data feature vector, perform the following mapping operation: First, the data feature vector is input into the linear projection layer corresponding to its mode, and an intermediate projection vector is obtained through matrix multiplication. The dimension of the intermediate projection vector is equal to dmodel (i.e., 256 dimensions). Then, the mode type vector (also 256 dimensions) corresponding to the data feature vector obtained in step S901 is added element-wise to the intermediate projection vector to obtain the final projection vector.

[0107] As an example, the original input dimensions for the four modalities are set as follows: audio modality feature vector is 128-dimensional, visual modality is 256-dimensional, sensor modality is 66-dimensional, and context modality is 32-dimensional. Correspondingly, four independent linear projection layers are set: audio projection layer (input 128, output 256), visual projection layer (input 256, output 256), sensor projection layer (input 66, output 256), and context projection layer (input 32, output 256). When processing an audio modality feature vector, it is first input into the audio projection layer to obtain a 256-dimensional audio intermediate projection vector; then, the corresponding audio modality type vector (256-dimensional) is obtained from step S901; the two are added element-wise to obtain the audio modality projection vector. Similarly, for the visual modality feature vector, after passing through the visual projection layer, it is added to the visual modality type vector to obtain the visual projection vector; the same applies to the sensor modality and context modality.

[0108] In this way, each data feature vector is mapped to a projection vector in a unified embedding space, and the vector carries the identification information of the modality source for subsequent sequence stacking and cross-modal interaction.

[0109] S903, after performing multimodal fusion processing on the multiple projection vectors, a fused sequence is obtained.

[0110] As an example, one implementation of step S903 is as follows: The multiple projection vectors are concatenated to obtain a vector sequence; The vector sequence is input into the target Transformer encoder. The multiple projection vectors in the vector sequence are aligned and fused using the target attention weight matrix in the target Transformer encoder to obtain the output sequence of the target Transformer encoder. The output sequence is the fused sequence. The number of layers in the target Transformer encoder is the same as the number of modal types in the various data feature vectors. The target Transformer encoder is obtained by training the initial Transformer encoder based on the sample sequence, which includes the sample input sequence and the sample fusion sequence.

[0111] For example, the projection vectors of each modality obtained in step S902 are stacked in a preset order (e.g., audio, vision, sensor, context) along the sequence length dimension to form a vector sequence of length L, where L equals the total number of modality types (in this embodiment, L=4 is used as an example). The dimension of this vector sequence is (L, dmodel), i.e., (4, 256). Each position in this sequence corresponds to a projection vector of a modality, and since the projection vectors already contain modality type identification information, the sequence itself retains the source attributes of each modality.

[0112] The target Transformer encoder consists of multiple identical stacked coding layers. Each coding layer includes: a multi-head self-attention sublayer, a residual connection and layer normalization sublayer, a feedforward network sublayer, and another residual connection and layer normalization sublayer. The multi-head self-attention sublayer has eight attention heads. Each attention head generates a query, key, and value for each position in the input sequence using a learnable linear transformation matrix, and then calculates the attention weights: Attention(Q,K,V)=softmax(QK^T / sqrt(dk))×V, where dk=dmodel=256.

[0113] For example, the calculation process for attention weights is as follows: First, in the Transformer encoder, the embedding vector of each modality (which has been projected and aligned, with a dimension of 256) is first transformed by three different linear transformation matrices to generate Query, Key, and Value, respectively. These three matrices are parameters learned by the model itself.

[0114] Secondly, given a query (denoted as Q) for a given modality, keys (denoted as K) for all modalities, and corresponding values ​​(denoted as V), calculate the attention weight matrix. This is achieved by using the query matrix Q and the transpose of the key matrix K. T Perform matrix multiplication to obtain the attention score matrix: scores = Q × K T The larger the dot product, the better the query and key match (i.e., the more relevant the content that the current modality wants to focus on is to the content that the other modality can provide).

[0115] Then, the score matrix is ​​divided by the scaling factor sqrt(dk) (dk is the dimension of the key, which is 256 here) to prevent the dot product value from being too large and causing softmax gradient saturation.

[0116] Next, a Softmax operation is performed on the scaled score matrix row by row to convert the scores into a probability distribution (summing to 1), thus obtaining the attention weight α.

[0117] Finally, the attention weight matrix is ​​multiplied by the value matrix V to obtain the output of the current attention head, Attention(Q,K,V)=α×V. The output dimension is (4,256), which represents the updated representation of each modality after weighted fusion.

[0118] Eight attention heads perform the above calculations in parallel, each head having an independently learnable W. Q W K W V Parameter matrix. The outputs of the 8 heads are concatenated along the feature dimension to obtain a tensor of dimension (4,256×8)=(4,2048). Then, a linear transformation matrix is ​​used to map it back to dimension (4,256), which is used as the final output of the multi-head self-attention sublayer.

[0119] The feedforward network sublayer first expands the feature dimension from 256 to 1024, and then maps it back to 256 to enhance the non-linear expressive power of the model.

[0120] The target Transformer encoder receives a vector sequence of length 4 as input, and after layer-by-layer interaction and transformation through 4 encoding layers, outputs an output sequence with dimensions still of (4, 256). Each position in this output sequence not only contains the original information of the corresponding modality, but also incorporates contextual information from other modalities, achieving deep alignment and fusion of multimodal features. This output sequence is the fused sequence.

[0121] As an example, for the sequence of input audio projection vectors, visual projection vectors, sensor projection vectors, and context projection vectors, the first 1-2 layers of the target Transformer encoder mainly achieve shallow alignment, such as timestamp synchronization or direct correlation matching between modalities; the latter 3-4 layers achieve deep semantic alignment, such as comprehensively judging the high-level semantic meaning of "the user is about to fall asleep". Through the above process, the fused sequence provides a joint representation rich in cross-modal correlations for subsequent feature aggregation or prediction tasks.

[0122] Different modalities have completely different data structures, distributions, and information densities. Transformer's serialization and self-attention provide a unified computational paradigm, allowing them to be compared and interact in the same space. Furthermore, it solves the problem of temporal / spatial misalignment; for example, in a video, audio, subtitles, and visuals do not perfectly correspond frame-by-frame. Multi-head self-attention can automatically find cross-modal alignment relationships through weights, eliminating the need for manual annotation. In simpler terms, multi-head attention allows each modal position to "attention" to the positions of all other modalities. Self-attention weights are essentially correlation scores between modal elements. Through training, the model can automatically learn which text segments correspond to which image regions and which audio segments. In summary, cross-modal fusion ensures that the features of each modality are no longer isolated but integrated with the contextual information of other modalities. For example, blurry image features can be enhanced by clear text descriptions; noisy audio can be corrected by visual lip movement information.

[0123] Through the above implementation methods, this application effectively solves the technical problem of how to deeply capture the complex correlations between different modal information and achieve effective alignment and fusion in the process of multimodal data fusion. By concatenating multiple projection vectors into a vector sequence and using the target Transformer encoder and its target attention weight matrix for alignment and fusion processing, the model can dynamically identify and integrate key information in different modal data, thereby generating a more representative and discriminative fusion sequence. This deep fusion method based on a self-attention mechanism enables electronic devices to more accurately understand the complex environment and state of the user. For example, when the user's visual perception data indicates changes in the user's emotions, the volume, sound effects, or dynamic range of audio playback can be adjusted more finely by combining environmental audio data and sensor data. In particular, matching the number of layers of the target Transformer encoder with the number of modal types and performing supervised training further ensures that the model can fully learn and optimize multimodal fusion strategies, thereby significantly improving the quality of the fused feature vectors and thus improving the accuracy of audio control parameter prediction and the adaptability of intelligent audio playback. This enables electronic devices to provide a more personalized, immersive, and context-appropriate audio experience.

[0124] S904, the fused sequence is subjected to mean pooling to obtain the fused feature vector.

[0125] The fused sequence is a dataset containing multiple position vectors, each corresponding to a fused representation of a modality. All position vectors have the same dimension, i.e., the dimension of the unified embedding space (such as the 256-dimensional example mentioned earlier). For example, the length of the fused sequence is equal to the total number of modality types (4 in this embodiment), corresponding to the fused outputs of audio modality, visual modality, sensor modality, and context modality, respectively.

[0126] The mean pooling process refers to calculating the arithmetic mean of all position vectors in the fused sequence. That is, the values ​​of each position vector in the sequence along the same dimension are added together, and then the sum is divided by the length of the sequence (i.e., the number of position vectors) to obtain a single vector. This vector has the same dimension as the individual position vector (256 dimensions in this embodiment), and this vector is the fused feature vector.

[0127] As an example, following the four modalities described in the preceding steps, the fused sequence contains four 256-dimensional vectors, denoted as the audio fusion vector, visual fusion vector, sensor fusion vector, and context fusion vector, respectively. During mean pooling, each dimension of these four vectors is added together to obtain 256 sums. Each sum is then divided by 4 to obtain 256 averages. These 256 averages are sequentially arranged to form a 256-dimensional vector, which is the fused feature vector.

[0128] The mean pooling process described above compresses a fixed-length sequence into a global vector. This fused feature vector integrates key information from all modalities and their deep semantic relationships, serving as a global context representation for subsequent prediction tasks.

[0129] Through the above implementation methods, this application can effectively solve the problems of heterogeneity and semantic differences in the representation of different modal data during multimodal data fusion. By introducing modality type vectors, the system can clearly distinguish and process data from different sources; by mapping feature vectors and modality type vectors to a unified preset multidimensional space, semantic alignment of different modal data is achieved; through multimodal fusion processing, the correlation and complementarity between various modal data are deeply explored, generating a more information-rich sequence representation; finally, through mean pooling, a concise and comprehensive fusion feature vector is obtained. This fusion strategy enables the fusion feature vector to more accurately and comprehensively reflect the complex state of the electronic device's environment and the user, thereby significantly improving the accuracy and robustness of audio control parameter prediction. This allows electronic devices to provide a more intelligent, personalized, and immersive audio playback experience based on the actual context, avoiding the problem of inaccurate audio control caused by insufficient single-modal information or improper multimodal information fusion.

[0130] In some implementations, to achieve adaptive optimization of the personalized auditory experience, a user identifier embedding layer can be introduced into the target neural network model. For example, a unique user ID embedding vector can be assigned to each user, the dimension of which is the same as or aligned with the dimension of the aforementioned fused feature vector through a linear transformation. The user ID embedding vector is concatenated or element-wise added to the fused feature vector obtained in step S904, serving as the joint input for subsequent prediction tasks (such as audio content recommendation, volume adjustment, scene mode switching, etc.). Simultaneously, an online learning mechanism is used to incrementally fine-tune the deployed model. During the model's online operation, feedback data from user interactions with the system is collected in real time. This feedback data includes, but is not limited to, explicit user evaluations of the currently playing content (such as likes and dislikes) and implicit behavioral indicators (such as playback duration, skip counts, repeated listening ratios, and manual volume adjustment). Based on the above feedback data, a loss function is constructed, and the model parameters are updated asynchronously or in small batches with a small preset learning rate. The parameters of the user ID embedding layer and its subsequent fully connected layers are given priority in the update, while the parameters of the multimodal feature extraction part can be selectively frozen or adjusted at a lower update frequency.

[0131] In this way, the target neural network model can gradually learn and adapt to the user's personalized auditory preferences as the user's usage time increases. This results in different prediction results for different users even if they input similar multimodal feature vectors, thus achieving a personalized adaptive experience.

[0132] The following are examples of scenarios in which the methods provided in the embodiments of this application are applied.

[0133] Scenario 1: A user is wearing in-ear headphones and playing music in a subway car.

[0134] Multimodal perception: The audio encoder detected continuous high-intensity (75dB), low-frequency abundant ambient noise, which was classified as "traffic noise".

[0135] The visual encoder detects that the user is looking at the phone screen (via the front-facing camera), and the closed-eye state is marked as "open-eye".

[0136] The sensor encoder detected that the mobile phone was vibrating regularly with the subway (acceleration spectrum characteristics), and the handheld posture indicated that the proximity sensor showed a distance of >30cm.

[0137] The context encoder detected that the earphones (Bluetooth in-ear type) are connected, the foreground application is a music app, and the battery is fully charged.

[0138] Joint representation generation: The cross-modal Transformer fuses the above information to generate a joint representation, which is used to represent the scenario of "high-intensity traffic noise environment, user actively listening, wearing in-ear headphones, and music playing".

[0139] Parameter prediction and execution: Volume gain: Increased to 75% (original setting 45%).

[0140] Equalizer: Mid-frequency range (1kHz-4kHz) boosted by 3dB to enhance the penetration of musical details; Low-frequency range (64Hz) boosted by 2dB to compensate for subway noise masking.

[0141] Dynamic compression: Threshold -25dB, compression ratio 3:1, mid-frequency emphasis, ensuring clarity of the main music.

[0142] Spatial audio: Maintain stereo mode.

[0143] Scenario 2: The user places their phone flat on the table and uses the speaker to participate in a video conference.

[0144] Multimodal perception: The audio encoder detected ambient noise (45dB), mainly from the air conditioner.

[0145] The visual encoder detects multiple faces using the front-facing camera, but no one is looking at the phone.

[0146] The sensor encoder detected that the phone was stationary and lying flat, with no obstructions.

[0147] The context encoder detected that the foreground application is a video conferencing app, and no headset is connected.

[0148] Joint Representation Generation: The cross-modal Transformer fuses the above information to generate a joint representation, which is used to represent the scenario of "quiet office environment, hands-free speaker mode, multiple people present, and video conference in progress".

[0149] Parameter prediction and execution: Volume gain: Set to 45% (medium volume, to avoid disturbing colleagues).

[0150] Equalizer: Mid-frequency range (500Hz-2kHz) boosted by 3dB to optimize vocal clarity; Low-frequency range attenuated by 3dB to reduce air conditioner noise interference.

[0151] Dynamic compression: Threshold -30dB, compression ratio 4:1, mid-frequency emphasis, to prevent popping sounds when participants suddenly raise their pitch.

[0152] Spatial Audio: Enables slight spatial enhancement, making the sound field wider and more natural.

[0153] Scenario 3: The user lies down in the bedroom and plays soothing music to prepare for sleep.

[0154] Multimodal perception: The audio encoder detected a quiet environment (28dB) with no sudden noise.

[0155] The visual encoder detected that the user had closed their eyes for more than 30 seconds, their head was still, and they were not looking at anything.

[0156] The sensor encoder detected that the phone was lying flat next to the pillow with no movement, and the proximity sensor showed a distance of >50cm.

[0157] The context encoder detected that the foreground application was a music app, playing a sleep aid playlist at 23:00.

[0158] Temporal state evolution: T1 (22:35): The user closes their eyes and enters the relaxation period.

[0159] T2 (22:50): The user turns over less frequently and enters a light sleep period.

[0160] T3 (23:10): No body movement for more than 5 minutes, entering deep sleep.

[0161] Joint representation generation: Cross-modal Transformer fuses information across time periods to output a state machine evolution of "user from awake → relaxed → light sleep → deep sleep".

[0162] Parameter prediction and execution (gradual adjustment) are shown in Table 5 below.

[0163] Table 5

[0164] In summary, the technical solution provided in this application, by comprehensively collecting environmental audio, visual perception, sensor and device status data, extracting and fusing multimodal features, and predicting multi-dimensional audio control parameters based on the fused features, achieves fine-grained adjustment of audio playback, effectively solving the problem of single perception dimension in the prior art, and improving the adjustment accuracy of audio control parameters; furthermore, since the audio control parameters are adjusted by combining multimodal data, the audio playback parameters can be dynamically optimized according to complex scenarios, which can improve the matching with the scenario in which the terminal device is located.

[0165] Based on the same inventive concept, embodiments of this application also provide an audio playback processing apparatus based on multimodal data, which is used to implement any of the above-described audio playback processing methods, for example... Figure 1 The audio processing method shown is illustrated, and the device can also perform the functions of the electronic device mentioned above.

[0166] Please refer to Figure 10 This is a schematic diagram of the structure of an audio playback processing device provided in an embodiment of this application. Figure 10 As shown, the audio playback processing device 1000 includes an acquisition module 1001, a feature extraction module 1002, a fusion processing module 1003, and a control module 1004.

[0167] The acquisition module 1001 is used to acquire raw data, which includes at least two types of data from the environment audio data, visual perception data, sensor data and device status data of the electronic device. The visual perception data is data collected by the camera of the electronic device from the user's face. The device status data includes at least one of the electronic device's battery level, temperature and operating mode. Feature extraction module 1002 is used to extract features from the original data and obtain multiple data feature vectors, wherein the multiple data feature vectors include at least two feature vectors among environmental audio feature vectors, visual feature vectors, sensor feature vectors and device status feature vectors; The fusion processing module 1003 is used to fuse the multiple data feature vectors to obtain a fused feature vector; The control module 1004 is used to predict audio control parameters based on the fused feature vector to obtain multiple target audio control parameters; and to control the electronic device to play audio based on the multiple target audio control parameters.

[0168] In one possible implementation, the control module 1004 is specifically used for: The fused feature vector is input into the target neural network model to obtain the output data of the target neural network model, and the output data is the plurality of target audio control parameters; The target neural network model includes a preprocessing module and multiple output sub-modules. The preprocessing module is used to preprocess the fused feature vector to obtain a processed fused feature vector. Each output sub-module is used to predict the processed fused feature vector to obtain at least some of the audio control parameters among the multiple target audio control parameters. The target neural network model is obtained by training a preset neural network model based on sample data, and the sample data includes sample fused features and sample audio control parameters.

[0169] In one possible implementation, the plurality of target audio control parameters include target equalizer parameters, which are multi-dimensional vectors. The control module 1004 is specifically used for: Based on the value of each element in the target equalizer parameters and the preset coefficient, the target gain coefficient of the filter band that matches the corresponding element is determined, and multiple target gain coefficients of multiple filter bands are obtained. The equalizer of the electronic device includes multiple sub-filters, and the multiple sub-filters include at least one operating frequency band. One element in the target equalizer parameters corresponds to one operating frequency band of one sub-filter. The plurality of sub-filters are controlled to operate according to the plurality of target gain coefficients in order to play the audio.

[0170] In one possible implementation, the plurality of target audio control parameters include target dynamic range compression parameters, wherein the target dynamic range compression parameters are two-dimensional vectors, and the control module 1004 is specifically used for: Based on the value of the first element in the target dynamic range compression parameter and the first calculation formula, the target value of the compression threshold is determined. The first calculation formula is as follows: compression threshold = A1 - B1 × element A, where A1 and B1 are set values, and element A is the first element in the target dynamic range compression parameter. Based on the value of the second element in the target dynamic range compression parameter and the second calculation formula, the target value of the compression ratio is determined. The second calculation formula is as follows: Compression ratio = A2 + B2 × element B, where A2 and B2 are set values, and element B is the second element in the target dynamic range compression parameter. The audio signal amplitude for playback is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

[0171] In one possible implementation, the control module 1004 is further configured to: determine whether the signal amplitude is greater than a preset amplitude threshold before playing the audio signal after processing the signal amplitude of the audio signal for audio playback according to the target value of the compression threshold and the target value of the compression ratio; The control module 1004 is specifically used for: When the signal amplitude is greater than the preset amplitude threshold, the signal amplitude of the audio signal to be played is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

[0172] In one possible implementation, the fusion processing module 1003 is specifically used for: Based on the modality type to which each of the various data feature vectors belongs, a modality type vector corresponding to each of the data feature vectors is obtained; Each data feature vector and its corresponding modality type vector are mapped to a preset multidimensional space to obtain multiple projection vectors corresponding to the various data feature vectors respectively. After performing multimodal fusion processing on the multiple projection vectors, a fused sequence is obtained; The fused sequence is subjected to mean pooling to obtain the fused feature vector.

[0173] In one possible implementation, the fusion processing module 1003 is specifically used for: The multiple projection vectors are concatenated to obtain a vector sequence; The vector sequence is input into the target Transformer encoder. The multiple projection vectors in the vector sequence are aligned and fused using the target attention weight matrix in the target Transformer encoder to obtain the output sequence of the target Transformer encoder. The output sequence is the fused sequence. The number of layers in the target Transformer encoder is the same as the number of modal types in the various data feature vectors. The target Transformer encoder is obtained by training the initial Transformer encoder based on the sample sequence, which includes the sample input sequence and the sample fusion sequence.

[0174] Based on the same inventive concept, embodiments of this application provide an electronic device for implementing any of the above-described audio playback processing methods, for example... Figure 1 The audio playback processing method shown is illustrated, and the device can also perform the functions of the electronic device described above.

[0175] Please refer to Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 1100 includes at least one processor 1101 and a memory 1102 communicatively connected to the at least one processor 1101. The processor 1101 can be a general-purpose processor or a dedicated processor. For example, the processor 1101 may include a baseband processor or a central processing unit (CPU). The baseband processor can be used to process communication protocols and communication data. The CPU can be used to control the electronic device 1100, execute software programs, and / or process data. Different processors can be independent devices or can be integrated into one or more processing circuits, for example, integrated onto one or more application-specific integrated circuits (ASICs). In one embodiment, memory 1102 stores instructions that can be executed by at least one processor 1101. At least one processor 1101 implements the functions of the aforementioned electronic device by executing the instructions stored in memory 1102, and correspondingly, can also implement the steps performed by the aforementioned electronic device. In this embodiment, the electronic device 1100 can also perform the functions of the preceding audio playback processing device 1000, and at least one processor 1101 in the electronic device 1100 can also perform the functions of the preceding text acquisition module 1001, the feature extraction module 1002, the fusion processing module 1003 and the control module 1004.

[0176] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform any of the audio playback processing methods described above, for example... Figure 1 The audio playback processing method shown. Based on the same inventive concept, embodiments of this application provide a computer program product containing computer instructions that, when run on a computer, cause any of the audio playback processing methods described above to be implemented, for example... Figure 1 The audio playback processing method shown.

[0177] It should be noted that the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0178] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio playback processing method based on multimodal data, characterized in that, Applied to electronic devices, the method includes: Acquire raw data, which includes at least two types of data from the environment in which the electronic device is located: ambient audio data, visual perception data, sensor data, and device status data. Feature extraction is performed on the raw data to obtain various data feature vectors; The multiple data feature vectors are fused to obtain a fused feature vector. Based on the fused feature vector, the audio control parameters are predicted to obtain multiple target audio control parameters; The electronic device is controlled to play audio according to the plurality of target audio control parameters.

2. The method of claim 1, wherein, The audio control parameters are predicted based on the fused feature vector to obtain multiple target audio control parameters, including: The fused feature vector is input into the target neural network model to obtain the output data of the target neural network model, and the output data is the plurality of target audio control parameters; The target neural network model includes a preprocessing module and multiple output sub-modules. The preprocessing module preprocesses the fused feature vector to obtain a processed fused feature vector. Each output sub-module predicts the processed fused feature vector to obtain at least some of the multiple target audio control parameters. The target neural network model is obtained by training a preset neural network model based on sample data, which includes sample fused features and sample audio control parameters.

3. The method according to claim 2, characterized in that, The plurality of target audio control parameters include target equalizer parameters, which are multi-dimensional vectors. Controlling the electronic device to play audio according to the plurality of target audio control parameters includes: Based on the value of each element in the target equalizer parameters and the preset coefficient, the target gain coefficient of the filter band that matches the corresponding element is determined, and multiple target gain coefficients of multiple filter bands are obtained. The equalizer of the electronic device includes multiple sub-filters, and the multiple sub-filters include at least one operating frequency band. One element in the target equalizer parameters corresponds to one operating frequency band of one sub-filter. The plurality of sub-filters are controlled to operate according to the plurality of target gain coefficients in order to play the audio.

4. The method according to claim 2, characterized in that, The plurality of target audio control parameters include target dynamic range compression parameters, wherein the target dynamic range compression parameters are two-dimensional vectors. Controlling the electronic device to play audio according to the plurality of target audio control parameters includes: Based on the value of the first element in the target dynamic range compression parameter and the first calculation formula, the target value of the compression threshold is determined. The first calculation formula is as follows: compression threshold = A1 - B1 × element A, where A1 and B1 are set values, and element A is the first element in the target dynamic range compression parameter. Based on the value of the second element in the target dynamic range compression parameter and the second calculation formula, the target value of the compression ratio is determined. The second calculation formula is as follows: Compression ratio = A2 + B2 × element B, where A2 and B2 are set values, and element B is the second element in the target dynamic range compression parameter. The audio signal amplitude for playback is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

5. The method according to claim 4, characterized in that, Before playing the audio signal after processing its amplitude according to the target value of the compression threshold and the target value of the compression ratio, the method further includes: Determine whether the signal amplitude is greater than a preset amplitude threshold; The step of processing the signal amplitude of the audio signal to be played according to the target value of the compression threshold and the target value of the compression ratio before playback includes: When the signal amplitude is greater than the preset amplitude threshold, the signal amplitude of the audio signal to be played is processed according to the target value of the compression threshold and the target value of the compression ratio before playback.

6. The method according to any one of claims 1-5, characterized in that, The process of fusing the multiple data feature vectors to obtain a fused feature vector includes: Based on the modality type to which each of the various data feature vectors belongs, a modality type vector corresponding to each of the data feature vectors is obtained; Each data feature vector and its corresponding modality type vector are mapped to a preset multidimensional space to obtain multiple projection vectors corresponding to the various data feature vectors respectively. After performing multimodal fusion processing on the multiple projection vectors, a fused sequence is obtained; The fused sequence is subjected to mean pooling to obtain the fused feature vector.

7. The method according to claim 6, characterized in that, The step of performing multimodal fusion processing on the multiple projection vectors to obtain the fused sequence includes: The multiple projection vectors are concatenated to obtain a vector sequence; The vector sequence is input into the target Transformer encoder. The multiple projection vectors in the vector sequence are aligned and fused using the target attention weight matrix in the target Transformer encoder to obtain the output sequence of the target Transformer encoder. The output sequence is the fused sequence. The number of layers in the target Transformer encoder is the same as the number of modal types in the various data feature vectors. The target Transformer encoder is obtained by training the initial Transformer encoder based on the sample sequence, which includes the sample input sequence and the sample fusion sequence.

8. An audio playback processing device based on multimodal data, characterized in that, The device includes: The acquisition module is used to acquire raw data, which includes at least two types of data from the environment audio data of the electronic device's environment, visual perception data, sensor data, and device status data. The feature extraction module is used to extract features from the original data and obtain various data feature vectors; The fusion processing module is used to fuse the multiple data feature vectors to obtain a fused feature vector; The control module is configured to predict audio control parameters based on the fused feature vector to obtain multiple target audio control parameters; and to control the electronic device to play audio based on the multiple target audio control parameters.

9. An electronic device, characterized in that, include: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-7.

10. A storage medium storing computer program instructions thereon, characterized in that, The method as described in any one of claims 1-7 is implemented when the computer program instructions are executed by the processor.

Images