Multi-user adaptive dynamic sound field optimization system

Abstract

The invention provides a multi-user adaptive dynamic sound field optimization system. The multi-user adaptive dynamic sound field optimization system comprises a multi-user position sensing module, a room acoustic modeling module and a dynamic sound field optimization module, the multi-user position sensing module is used for preprocessing position data acquired by the microphone array, the millimeter wave radar and the camera and then fusing the position data to generate a user position; the room acoustic modeling module is used for performing room acoustic modeling and analyzing to obtain acoustic performance of a room, and the acoustic performance comprises an acoustic model and reflection and absorption characteristics; the dynamic sound field optimization module is set to adjust audio parameters in real time according to the position of a user, acoustic performance of a room and other environmental factors, and the other environmental factors comprise user preference setting and temperature and humidity in environmental monitoring data. The multi-user adaptive dynamic sound field optimization system can provide a flexible, efficient and personalized sound effect optimization solution for a multi-user and multi-device environment.

Description

Technical Field

[0001] The present invention belongs to the field of audio processing, and in particular relates to a multi-user adaptive dynamic sound field optimization system. Background Art

[0002] In the field of audio technology, as people's demands for sound quality and audio experience continue to increase, a variety of technologies have emerged to meet these needs. Multi-user location awareness technology was initially used in the military and communications fields, such as radar positioning and tracking of multiple targets. It has since expanded to civilian applications, such as multi-user positioning in intelligent conferencing systems.

[0003] However, the existing technology has many deficiencies in the audio system.

[0004] Existing technologies, such as the intelligent metasurface-assisted integrated communication and perception system, can achieve multi-user positioning and beamforming gain, but their application in audio systems is not yet mature, the perception accuracy of user positions is limited, and their adaptability in complex environments is poor. Specifically, in terms of multi-user position perception, existing technologies such as UWB positioning, Bluetooth positioning, and Wi-Fi positioning, although they can achieve basic positioning functions, still need to improve their positioning accuracy and stability in complex environments. For example, UWB positioning requires the deployment of a positioning network, which is expensive and needs to be redeployed when encountering wall obstructions. Bluetooth positioning is greatly affected by noise signals and has slightly poor stability. Wi-Fi positioning requires a large workload for location mapping and is easily interfered with by other signals, affecting its accuracy.

[0005] The development of dynamic sound field adjustment technology stems from people's higher pursuit of sound quality. Room acoustic modeling technology optimizes sound parameters by analyzing the acoustic characteristics of the room. In terms of dynamic sound field adjustment, although the existing technology can optimize sound parameters through room acoustic modeling, it has shortcomings in dynamically optimizing sound parameters and cannot be adjusted in real time according to the user's activity area.

[0006] Cross-device audio synchronization technology aims to meet the need for consistent audio in multi-device collaboration scenarios. For example, Huawei's cross-device audio and video synchronization technology uses distributed hardware to achieve audio and video synchronization. While it can achieve basic synchronization, there is still room for improvement in synchronization accuracy and latency. For example, issues such as codec latency, device performance differences, and clock skew can all affect synchronization.

[0007] Intelligent management and optimization technologies have emerged with the development of artificial intelligence. Deep learning and other technologies are used to intelligently manage audio systems. Existing technologies, such as smart speaker permission design and management methods, can achieve basic permission management, but they are insufficient in terms of dynamism and intelligence, and cannot adapt to the complex needs of multiple users and multiple scenarios.

[0008] In summary, existing technologies still have some problems and shortcomings in multi-user position perception, dynamic sound field adjustment, cross-device sound synchronization, and intelligent management and optimization, which require further improvement and innovation. Summary of the Invention

[0009] The purpose of the present invention is to provide a multi-user adaptive dynamic sound field optimization system to provide a flexible, efficient and personalized sound optimization solution for a multi-user, multi-device environment.

[0010] In order to achieve the above-mentioned objectives, the present invention provides a multi-user adaptive dynamic sound field optimization system, including a multi-user position perception module, a room acoustic modeling module and a dynamic sound field optimization module; the multi-user position perception module is used to preprocess the position data collected by the microphone array, millimeter-wave radar and camera, and then fuse them to generate the user position; the room acoustic modeling module is configured to perform room acoustic modeling and analyze the acoustic performance of the room, which includes the acoustic model and reflection and absorption characteristics; the dynamic sound field optimization module is configured to adjust the audio parameters in real time according to the user position, the acoustic performance of the room and other environmental factors, and other environmental factors include user preference settings and temperature and humidity in environmental monitoring data.

[0011] The multi-user location perception module includes a data acquisition module, a data preprocessing module for outputting preprocessed data, and a location information fusion module connected in sequence; the data acquisition module includes a sound signal acquisition branch, a radar signal acquisition branch, and an image acquisition branch connected in parallel with each other; the sound signal acquisition branch uses a microphone array, the radar signal acquisition branch uses a millimeter wave radar, and the image acquisition branch uses a camera; the data preprocessing module is used to preprocess the collected data of the data acquisition module, and the preprocessing includes filtering, denoising, and feature extraction; the location information fusion module is configured to receive preprocessed data and fuse different types of preprocessed data to generate user locations.

[0012] The multi-user location perception module also includes a user activity trajectory analysis module, which is configured to: input the fused location results together with the user's behavior pattern and appearance characteristics into a machine learning algorithm to analyze and obtain the real-time user location and predicted activity trajectory, from which the predicted user location is obtained as the final output user location; then, based on the user's voice characteristics, behavior pattern or appearance characteristics, the user location is associated with the user identity to achieve user identity binding.

[0013] The multi-user location perception module also includes an anomaly detection module, which sets an anomaly threshold based on the user's behavior pattern. When at least one of the speed and acceleration obtained from the user's historical location data exceeds the anomaly threshold, it is judged as an anomaly; otherwise, it is judged as normal.

[0014] The room acoustic modeling module includes an acoustic model construction module, the input end of the acoustic model construction module is connected to the acoustic characteristic measurement module and the sound source modeling module; the acoustic characteristic measurement module receives the three-dimensional model of the room, measures the material at each position in the three-dimensional model of the room, and measures the acoustic characteristics of each material, and finally outputs the three-dimensional model of the room and the acoustic characteristics of each material in the room; the sound source modeling module is configured to measure the sound source characteristics of each sound source and establish a sound source characteristic model based on the sound source characteristics of the sound source; the acoustic model construction module receives the three-dimensional model of the room and the acoustic characteristics of each material in the room output by the acoustic characteristic measurement module and the sound source characteristic data output by the sound source modeling module, and then performs acoustic simulation to construct an acoustic model based on the three-dimensional model of the room, the acoustic characteristics of each material in the room, and the sound source characteristic model, and obtains the acoustic performance of the room based on the acoustic model analysis, and the acoustic performance of the room includes the sound pressure distribution, sound intensity distribution, frequency response, acoustic model and reflection and absorption characteristics of the sound wave in the room; the acoustic simulation uses a finite element analysis method or a boundary element analysis method.

[0015] The input end of the acoustic model construction module is also connected to the background noise modeling module, and the output end of the acoustic model construction module is connected to the model calibration and optimization module; the background noise modeling module is configured to measure the background noise in the room and input the background noise into the acoustic model construction module to add the background noise to the acoustic model; the model calibration and optimization module is configured to compare the acoustic model with the actual measured acoustic data to adjust the parameters of the acoustic model.

[0016] The dynamic sound field optimization module is configured to perform the following steps:

[0017] S301: Dividing the room into multiple independent sound field areas using a sound field area division module according to the user's position;

[0018] S302: Inputting the divided sound field areas into a sound field parameter calculation module, so that the sound field parameter calculation module calculates the sound field parameters of each sound field area according to the acoustic performance of the room;

[0019] S303: determining a calibration amount of the sound field parameter according to the environmental parameter, and calibrating the sound field parameter according to the calibration amount of the sound field parameter;

[0020] S304: Determine target sound field parameters based on the user's satisfaction evaluation of the sound effect and the user's preference settings;

[0021] S305: Adjust the audio signal according to the sound field parameters of step S303 and the target sound field parameters of step S303.

[0022] Collecting user satisfaction evaluations on sound effects includes: asking users to rate the sound effects or put forward specific opinions and suggestions, or indirectly evaluating user satisfaction with the sound effects by analyzing user historical behavior data; the dynamic sound field optimization module is further configured to execute step S306: using a microphone to detect a sound feedback signal, and adjusting the audio signal in real time according to the difference between the sound field parameters of the sound feedback signal and the target sound field parameters.

[0023] The multi-user adaptive dynamic sound field optimization system further includes a high-precision cross-device audio synchronization module, which is triggered in response to the user's satisfaction rating of the sound effect; the high-precision cross-device audio synchronization module is configured as follows:

[0024] S401: Acquire audio signals to be output by the audio device from multiple audio devices and pre-process the audio signals;

[0025] S402: Select a clock synchronization reference device; then distribute the clock information of the clock synchronization reference device as clock synchronization reference information to all audio devices, ensuring that all audio devices can receive a unified time reference;

[0026] S403: Using a high-precision clock synchronization algorithm based on the clock synchronization reference information to accurately adjust the clocks of each audio device to keep them consistent with the clock of the clock synchronization reference device;

[0027] S404: Establishing an optimized audio transmission protocol. The optimization of the audio transmission protocol at least includes using a forward error correction mechanism and a packet loss retransmission mechanism to improve data transmission reliability during network transmission.

[0028] S405: Synchronizing the audio signals of the various devices according to frames using an optimized audio transmission protocol, and then synthesizing the synchronized audio signals to generate a multi-channel synchronized audio stream, where each channel corresponds to an audio device;

[0029] S406: monitoring the synchronization error of the synchronous audio stream in real time, and determining whether to perform error compensation according to the synchronization error;

[0030] S407: Send the multi-channel synchronous audio streams to all corresponding audio devices respectively, so that the audio output remains synchronized in time.

[0031] The multi-user adaptive dynamic sound field optimization system also includes an intelligent management and optimization module, which is configured to generate an intelligent optimization strategy based on user feedback data, environmental monitoring data and system operation data. The intelligent optimization strategy includes the automatic adjustment results of the parameters of the room acoustic modeling module, the dynamic sound field optimization module and the high-precision cross-device audio synchronization module; the user feedback data includes the user's satisfaction evaluation of the sound effect and the user's preference settings, which are used to automatically adjust the target sound field parameters in the dynamic sound field optimization module; the environmental monitoring data includes temperature, humidity and background noise, and the temperature and humidity are used for the sound field parameter calibration in the dynamic sound field optimization module, and the background noise is used as the background noise of the background noise modeling module of the room acoustic modeling module; the system operation data includes device performance parameters and the quality of the audio signal, and the system operation data is used to optimize the audio transmission protocol of the high-precision cross-device audio synchronization module.

[0032] The multi-user adaptive dynamic sound field optimization system of the present invention has the following advantages:

[0033] ① Achieving multi-user location awareness: Combining acoustics, millimeter-wave radar, and vision technologies, the system achieves centimeter-level positioning and user identity binding. Through multi-sensor data fusion, the system can identify the specific locations of multiple users and their predicted movement trajectories in real time, precisely adjusting the direction and intensity of each user's sound output. It can also predict future changes in a user's activity area or location and adjust sound field settings in advance. This predictive capability not only improves the user's listening experience but also reduces the latency of real-time adjustments, ensuring a smooth transition in sound output.

[0034] ② Dynamic Sound Field Optimization: Combining room acoustic modeling and user location, the system dynamically optimizes the reverberation, gain, and delay parameters of the sound based on the room's reflection and absorption characteristics and the user's activity area. This system can adjust the sound effects during user activity to provide an optimal audio experience, especially in complex environments (such as rooms with multiple users and multiple obstacles), rather than relying on fixed, preset sound field settings. Furthermore, the environmental modeling system in the present invention is tightly integrated with the sound field optimization algorithm, enabling it to quickly analyze the acoustic characteristics of a room or space (such as reflection, absorption coefficient, surface material, etc.) and generate sound field parameters in real time. This close integration of environmental modeling and sound field adjustment significantly improves the accuracy of the system, making the optimization effect even more pronounced in highly dynamic environments.

[0035] ③ Cross-device audio synchronization: Using GPS-based synchronization or network time protocols, this solution addresses the latency issues inherent in existing cross-device audio output technologies, ensuring that multiple devices can simultaneously output precisely synchronized audio signals. Through optimized audio transmission protocols, this reduces inter-device audio lag and improves audio output consistency in multi-device collaboration scenarios.

[0036] ④ Intelligent Management and Optimization: Automatically adjust parameters based on user feedback, environmental monitoring data, and system operation data to provide personalized sound effects. System performance can be optimized based on user feedback and environmental changes, providing a more accurate and convenient audio experience. Furthermore, this invention innovatively implements intelligent sound field optimization through deep learning algorithms, combined with user feedback and environmental changes. This intelligent optimization capability ensures that the system's sound effects continuously improve with usage and feedback, meeting the personalized needs of different users.

[0037] In summary, the present invention realizes multi-user position perception, dynamic sound field adjustment, and cross-device sound synchronization. It aims to provide flexible, efficient, and personalized sound optimization solutions for multi-user, multi-device environments by comprehensively using microphone arrays, AI analysis, millimeter-wave radar, visual technology and other means. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a system block diagram of the multi-user adaptive dynamic sound field optimization system of the present invention.

[0039] Figure 2 It is a structural block diagram of a multi-user position perception module of the multi-user adaptive dynamic sound field optimization system of the present invention.

[0040] Figure 3 It is a structural block diagram of the room acoustic modeling module of the multi-user adaptive dynamic sound field optimization system of the present invention.

[0041] Figure 4 It is a structural block diagram of a high-precision cross-device audio synchronization module of the multi-user adaptive dynamic sound field optimization system of the present invention.

[0042] Figure 5 It is a structural block diagram of the intelligent management and optimization module of the multi-user adaptive dynamic sound field optimization system of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to specific examples. It should be understood that the following examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0044] like Figure 1 As shown, a multi-user adaptive dynamic sound field optimization system of the present invention includes five modules: a multi-user position perception module 10, a room acoustic modeling module 20, a dynamic sound field optimization module 30, a high-precision cross-device audio synchronization module 40, and an intelligent management and optimization module 50. In some embodiments, the high-precision cross-device audio synchronization module 40 and the intelligent management and optimization module 50 can be omitted.

[0045] like Figure 2As shown, the multi-user location perception module 10 is used to pre-process the location data collected by the microphone array, millimeter wave radar and camera, and then fuse them to generate the user location.

[0046] like Figure 2 As shown, the multi-user location perception module 10 includes a data acquisition module, a data preprocessing module 12 for outputting preprocessed data, a location information fusion module 13 and a user activity trajectory analysis module 14 which are connected in sequence.

[0047] The data acquisition module includes a parallel-connected sound signal acquisition branch, a radar signal acquisition branch, and an image acquisition branch. The sound signal acquisition branch uses a microphone array to collect sound signals, the radar signal acquisition branch uses a millimeter-wave radar to collect radar signals, and the image acquisition branch uses a camera to collect image data. Thus, the multi-user location sensing module 10 integrates multiple advanced sensing technologies, including microphone arrays, millimeter-wave radar, and visual sensing, to monitor user locations and activity trajectories in real time.

[0048] The principle that the sound signal of the microphone array can help determine the user's position is that it takes a certain amount of time for sound to propagate in the air. When the sound source is not on the central axis of the microphone array, the time it takes for the sound to reach different microphones in the array will be different. Based on the sound propagation speed and time difference, the position of the sound source in space can be calculated, thereby helping to determine the user's position.

[0049] The data preprocessing module 12 is used to preprocess the data collected by the data acquisition module, such as filtering, denoising, and feature extraction, to output preprocessed data, improving the accuracy and efficiency of subsequent processing. The collected data includes sound signals captured by the microphone array, radar signals from the millimeter-wave radar, and images captured by the camera. The preprocessed data includes the preprocessed sound signals, radar signals, and image signals.

[0050] The position information fusion module 13 is configured to receive preprocessed data and fuse different types of preprocessed data to generate the user's position. For example, by combining the sound signal of the microphone array with the radar signal of the millimeter-wave radar, the position information fusion module 13 can more accurately determine the user's position. The combination of the image provided by the camera and the radar signal of the millimeter-wave radar further enhances the accuracy of position perception, especially in complex environments. The multi-sensor data fusion algorithm used by the position information fusion module 13 is an existing technology, for example, see the literature [Jin Lu, Gao Liang, Research on Multi-Sensor Data Fusion Technology Algorithm [J]. Digital Design, 2023(7): 62-65. DOI: 10.3969 / j.issn.1672-9129.2023.7.smsj-shang202307018].

[0051] The user activity trajectory analysis module 14 includes a machine learning algorithm. This module is configured to input the fused location results, along with the user's behavioral patterns and appearance characteristics, into the machine learning algorithm to analyze the real-time user location and predicted activity trajectory (including the predicted user location). The predicted user location is then output as the final user location. Subsequently, the module associates the user location with their identity based on the user's voice, behavioral, or appearance characteristics, achieving user identity binding. This allows for early prediction of the user's location, enabling preparation for optimal audio output and rapid response and adjustment during user activity.

[0052] The input data for machine learning algorithms includes: fused location results, behavioral patterns, and appearance features. Behavioral patterns include time-series data of the user's location, velocity, and acceleration, which can be determined based on the user's historical behavior data and the typical activity patterns of a person in the current environment. Appearance features include the user's height and body type (BMI or body type categories can be used). When preprocessing the input data, the time-series data of the user's location, velocity, and acceleration needs to be sliced ​​into time windows. Height and body type are used as global features, which are then combined with the time-series features through splicing and normalization.

[0053] The output data includes real-time user location and a sequence of locations for the next T time steps, where T is a positive integer. The machine learning algorithm aims for multi-task learning. Real-time user location is achieved through regression, and activity trajectory predicts the next trajectory based on the current user location, achieved through time series prediction.

[0054] The architecture of the machine learning algorithm can adopt convolutional neural network (CNN) or recurrent neural network, or CNN-RNN hybrid architecture.

[0055] In this embodiment, when the machine learning algorithm adopts a recurrent neural network, a long short-term memory network (LSTM) can be used. The input layer receives time series data, one branch of the output layer outputs the real-time user location, and the other branch outputs the location sequence of the next T time steps.

[0056] The loss function adopts a joint loss function, whose value is equal to the mean squared error (MSE) of the real-time position prediction plus the smooth L1 loss of the future sequence prediction.

[0057] The user activity trajectory analysis module 14 also associates each user's location with a user identity by analyzing the user's voice characteristics, behavioral patterns, or appearance, achieving identity binding and obtaining the user preferences corresponding to that identity. This enables the system to provide customized sound effects for each user based on their preferences. Specifically, if the user's location successfully matches an existing user identity (identity binding), the system continues to operate according to the user preferences corresponding to that identity. If the match fails (no binding), the system is treated as a new user and the user manually adjusts the equalizer to obtain the user preferences corresponding to that identity.

[0058] The user preferences corresponding to each user identity include the user's preferences for sound effects (volume, sound effect mode, etc.), the user's preferences for different types of audio content, and the user's historical behavior data (previous location information, activity tracks, sound effect adjustment records, etc.). This information can be used to adjust the sound effect settings to ensure that the user always has the best listening experience.

[0059] Furthermore, the multi-user location awareness module 10 also includes an anomaly detection module, which is configured to implement anomaly detection based on user behavior patterns and historical location data. Specifically, the anomaly detection module sets anomaly thresholds (e.g., maximum movement speed, acceleration, etc.) based on the user's behavior patterns. If at least one of the speed or acceleration derived from the user's historical location data exceeds the anomaly threshold, the module determines the situation as abnormal; otherwise, the situation is considered normal. Thus, by monitoring the user's location and behavior in real time, it can identify abnormal situations, such as sudden movement or immobility.

[0060] Finally, the multi-user position perception module 10 outputs the user position (from the predicted user position), user identity, and detected anomalies, thereby providing data support for the subsequent dynamic sound field optimization module 30 .

[0061] Among them, the predicted user position can be used to locate the output direction and intensity of the sound effect, which is the user position required by subsequent modules; the user identity can be used to determine the user preference settings, and then set the sound effects according to the user preference settings. The user preference settings corresponding to the user identity can guide the style and intensity adjustment of the sound effects; abnormal information can be used for reminders or security purposes, and generally has little impact on the adjustment of sound effects, but can also be used in some cases to adjust the sound effects to adapt to the user's current state.

[0062] The room acoustic modeling module 20 is configured to perform room acoustic modeling and analyze the acoustic performance of the room. The acoustic performance includes the sound pressure distribution, sound intensity distribution, frequency response, acoustic model, and reflection and absorption characteristics of sound waves in the room, thereby providing data support for the subsequent dynamic sound field optimization module 30.

[0063] like Figure 3 As shown, the room acoustic modeling module 20 includes an acoustic model construction module 21, the input end of the acoustic model construction module 21 is connected to the acoustic characteristic measurement module 22, the sound source modeling module 23 and the background noise modeling module 24, and the output end of the acoustic model construction module 21 is connected to the model calibration and optimization module 25.

[0064] In some embodiments, the background noise modeling module 24 and the model calibration and optimization module 25 can be omitted as needed.

[0065] The acoustic characteristics measurement module 22 receives the 3D model of the room output by the 3D room modeling module 26, measures the materials at each location in the 3D model, and measures the acoustic properties of each material. Ultimately, it outputs the 3D model of the room and the acoustic properties of each material in the room. The 3D model of the room includes the room's geometry and materials. The acoustic properties of the materials include sound velocity, density, damping coefficient, and reflection, absorption, and diffraction characteristics. The simulation results from the acoustic characteristics measurement module 22 help the system understand the reflection and absorption of sound at various materials within the room.

[0066] The 3D room modeling module 26 is configured to obtain the physical parameters of the room, including its dimensions, shape, wall material, and furniture layout, and to generate a 3D model of the room based on the room's physical parameters. The room's physical parameters can be obtained through user input or automatic measurement. For example, a user can input the room's dimensions via a mobile app, or the system can automatically measure the room's dimensions and layout using a laser rangefinder. The 3D model of the room includes not only the room's geometry but also the location and materials of various objects within the room. For example, the system records the location and materials of furniture such as sofas, tables, and curtains, as these factors can affect sound propagation.

[0067] The sound source modeling module 23 is configured to measure the sound source characteristics of each sound source and establish a sound source characteristic model according to the sound source characteristics of the sound source. The sound source characteristic model is used to describe the sound generation behavior and sound radiation law of the sound source.

[0068] Among them, the sound source characteristics of a sound source refer to parameters such as frequency characteristics, intensity and phase. Measuring the sound source characteristics of each sound source includes measuring the frequency characteristics emitted by the sound source, the sound pressure level generated at different positions and directions, the total sound power radiated by the sound source to the surrounding space, the time domain characteristics of the emitted sound, etc. Establishing a sound source characteristic model specifically includes: establishing a corresponding sound source physical model based on the actual physical structure and sound generation principle of the sound source; based on a large amount of sound source measurement data, establishing a sound source empirical model to describe the characteristics of the sound source through statistical analysis and curve fitting methods, wherein mathematical tools and methods such as differential equations, Fourier series, wavelet transform, etc. are used to establish a sound source empirical model based on the acoustic characteristics of the sound source.

[0069] The acoustic model construction module 21 receives the three-dimensional model of the room and the acoustic characteristics of each room material output by the acoustic characteristic measurement module 22, as well as the sound source characteristic data output by the sound source modeling module 23. Based on the three-dimensional model of the room, the acoustic characteristics of each room material, and the sound source characteristic model, it performs acoustic simulation to construct an acoustic model. The acoustic performance of the room is analyzed based on the acoustic model. The acoustic performance of the room includes the sound pressure distribution, sound intensity distribution, frequency response, acoustic model, and reflection and absorption characteristics of sound waves within the room. Acoustic simulation can use methods such as finite element analysis or boundary element analysis to determine the propagation path and intensity of sound waves within the room.

[0070] The acoustic model building module 21 is configured to perform the following steps:

[0071] Step S201: Integrate the three-dimensional model of the room and the location of the sound source; that is, import the three-dimensional model of the room output by the acoustic characteristic measurement module 22 into the acoustic model construction module 21, and accurately mark the location of the sound source in the three-dimensional model of the room.

[0072] For example, if the sound source is located in a corner of the room, the sound source point needs to be set at the corresponding position in the 3D model of the room.

[0073] Step S202: setting the acoustic properties of the material; that is, assigning the acoustic properties of the corresponding material to each part of the three-dimensional model according to the material of each position in the three-dimensional model of the room (such as the measured materials of the walls, furniture, etc.).

[0074] For example, for a wooden wall, set its sound velocity to 5000m / s, density to 550kg / m³, and damping coefficient to 0.05.

[0075] Step S203: setting boundary conditions and initial conditions; that is, setting boundary conditions according to the three-dimensional model of the room, including the reflection characteristics of the walls, floor and ceiling of the room, and setting the initial sound pressure and phase of the sound source as initial conditions.

[0076] For a closed room, the walls can be set as reflecting boundary conditions.

[0077] Step S204 : defining sound source characteristics; that is, defining a sound source characteristic model for each sound source according to the type of each sound source and the sound source characteristic model output by the sound source modeling module 23 .

[0078] Sound source characteristics refer to parameters such as frequency characteristics, intensity and phase.

[0079] For example, if the sound source is a speaker playing music, the sound intensity output at different frequencies needs to be defined according to the speaker's frequency response curve as its sound source characteristics.

[0080] Step S205: running an acoustic simulation; that is, selecting a suitable simulation solver and calculation parameters, and running an acoustic simulation to construct an acoustic model.

[0081] Step S206: analysis and evaluation; that is, obtaining the acoustic performance of the room based on the acoustic model analysis, including the sound pressure distribution, sound intensity distribution, frequency response, acoustic model, and reflection and absorption characteristics of the sound waves in the room.

[0082] The acoustic performance of a room includes the sound pressure distribution, sound intensity distribution, frequency response, acoustic model, and reflection and absorption characteristics of sound waves in the room, thus providing data support for the dynamic sound field optimization module.

[0083] The model calibration and optimization module 25 is configured to compare the acoustic model with actual measured acoustic data to adjust the parameters of the acoustic model so that it more accurately reflects the actual acoustic characteristics of the room.

[0084] For example, the model calibration and optimization module 25 may adjust the absorption coefficient of the wall in the acoustic model (ie, the acoustic properties of each material in the room) to match the actual measured reflection intensity.

[0085] The background noise modeling module 24 is configured to measure the background noise in the room (such as air conditioning and fan sounds) and input this background noise into the acoustic model building module 21, thereby incorporating the background noise into the acoustic model. This helps to better suppress noise and improve sound clarity during subsequent sound field optimization. The background noise is derived from environmental monitoring data.

[0086] The dynamic sound field optimization module 30 is configured to adjust audio parameters in real time based on the user's location, the room's acoustic properties, and other environmental factors to provide the best possible sound experience. Room acoustic properties include the sound pressure distribution, sound intensity distribution, frequency response, acoustic model, and reflection and absorption characteristics of sound waves within the room. Other environmental factors include user preferences and temperature and humidity from environmental monitoring data.

[0087] The dynamic sound field optimization module 30 is configured to perform the following steps:

[0088] Step S301: Divide the room into multiple independent sound field areas using a sound field area division module according to the user's position.

[0089] Each sound field zone corresponds to one or more users, ensuring that the audio signal is optimized for that specific user. For example, if two users are sitting diagonally opposite each other in the room, the system will divide the room into two independent sound field zones, one for each user.

[0090] In one embodiment, the division rule is: use the user's spatial coordinates (such as (xi, yi)) as the "seed point"; divide the room into multiple regions through the Voronoi diagram (Thyssen polygon); each region contains all points closest to its corresponding user.

[0091] All sound field areas combine to cover the entire room, giving all users the best possible sound experience.

[0092] Step S302: The divided sound field regions are input into the sound field parameter calculation module, which calculates the sound field parameters for each sound field region based on the room's acoustic properties. Sound field parameters include reverberation time, gain, and delay. For example, the system calculates reverberation time based on the room's reflective properties, delay based on the distance between the user and the audio device, and gain based on the room's absorption properties.

[0093] Among them, the gain parameters of each sound field area are calculated to ensure that users have a consistent sound experience in different positions.

[0094] The specific gain parameters for calculating each sound field area include:

[0095] First, determine the distance attenuation coefficient between the i-th audio device and user u; where the system sets the distance between the i-th audio device and user u to be , the speed of sound is c, then the distance attenuation coefficient of this path in the free sound field is: .

[0096] Then, the absorption attenuation coefficient between the i-th audio device and user u is determined The room acoustic modeling module also provides the sound absorption characteristics of the material segments traversed by the path. Assume that the path traverses N different room material segments, and the sound energy absorption coefficient of the jth segment material is , the path length of the jth segment of material is , the system calculates the material's absorption attenuation coefficient for sound energy propagation according to the following formula :

[0097] ,

[0098] Finally, the gain parameter G of the i-th audio device at the user u position is determined i,u ; Gain parameter G of the i-th audio device at the user u position i,u for:

[0099] ,

[0100] in, is the reference gain value of the i-th audio device (e.g., the standard gain under unobstructed propagation at a reference distance of 1 meter), is the distance attenuation coefficient, is the absorption attenuation coefficient, is the distance between the i-th audio device and user u, The acoustic energy absorption coefficient of the material in the jth section is, is the path length of the jth segment of material.

[0101] The gain parameter G i,u It is used to adjust the amplitude of the audio signal in the subsequent steps to achieve directional sound optimization for each user and each sound field area. By combining the user's current position, the sound absorption characteristics of the room material and the position of the device, the gain parameter G of the i-th audio device at the user u position is calculated. i,u It can be dynamically updated when the user moves, ensuring that the volume of the sound heard by the user changes smoothly during movement, and the sound effect experience is continuous and stable. In one embodiment, the target gain parameter in the target sound field parameter remains unchanged, so the gain parameter G of the i-th audio device at the user u position can be adjusted by adjusting the baseline gain value of the i-th audio device (i.e., adjusting its volume). i,u Remains constant as the user moves, to keep it equal to the target sound field parameters.

[0102] What is calculated here are the sound field parameters of the sound field area rather than the sound field parameters of the user's position. The reason is: if the sound field parameters at the user's position are calculated, they need to be recalculated as soon as the user moves. The sound field parameters of the sound field area are still valid when the user moves, and can take into account multi-user and generalized processing, providing continuous spatial sound field performance, and can have stable parameters even when the user's position moves slightly.

[0103] The delay of each point in the sound field area is different, so the delay of the sound field area is considered as follows: dynamic weighted center point + threshold range compensation.

[0104] Specifically, the delay of the sound field area is calculated as follows: a weight is assigned to each user location (i.e., point of interest (POI)) within the sound field area. , according to the weight of the user's position, calculate the weighted center point of the entire sound field area; for each sound field area, calculate the delay of all audio devices with respect to the weighted center point of the sound field area.

[0105] Weight The distribution method is as follows: the closer to the center or the audio device, the higher the weight; the speaker, the main listener, and the main controller are given priority; those who use remote controls, gestures, microphones, etc. to interact more actively are given priority; user positions that are more frequently used in certain sound field areas within a certain period of time are given priority.

[0106] The calculation formula for the weighted center point of the entire sound field area is:

[0107]

[0108] in, is the location coordinate of the i-th user; is the weight; is the weighted center point of the current frame / period.

[0109] In this way, the main users are taken care of through the dynamic central point.

[0110] Step S303: determining a calibration amount of the sound field parameter according to the environmental parameter, and calibrating the sound field parameter according to the calibration amount of the sound field parameter to further optimize the sound effect.

[0111] For example, if the temperature and humidity in a room change, the system adjusts the reverberation time and gain to compensate for the effect these changes have on the sound.

[0112] First, temperature and humidity sensors monitor environmental parameters in real time (increased temperature accelerates the movement of air molecules, while increased humidity changes the air's sound wave absorption characteristics). Then, sound field parameters are calibrated. These calibrations include delay, reverberation time (high temperature or high humidity shortens reverberation time), and gain (temperature and humidity changes can cause attenuation in specific frequency bands, such as high humidity weakening high-frequency gain).

[0113] The mathematical model of the sound speed c changing with temperature and humidity is as follows:

[0114] c=331.4+0.6T+0.0124H,

[0115] Where T is the current ambient temperature, H is the current ambient relative humidity, and c is the calculated speed of sound in air.

[0116] In this embodiment, to correct for the sound attenuation caused by the change in reverberation time due to temperature and humidity changes, the Sabine formula can be used to estimate the reverberation time after calibration using the reverberation time calibration amount for temperature and humidity changes. This can be directly used as the calculation result of the reverberation time in step S302, without the need to calculate the reverberation time calibration amount for temperature and humidity changes.

[0117] In this embodiment, the reverberation time after calibration of the reverberation time calibration value of temperature and humidity changes is for:

[0118] , where V is the room volume, For the equivalent sound absorption area, the sound absorption coefficient can be corrected by establishing a function based on the influence of humidity on the material.

[0119] Specifically, when the sound field parameter is delay, a calibration amount of the sound field parameter is determined according to the environmental parameter, and the sound field parameter is calibrated according to the calibration amount of the sound field parameter, specifically including:

[0120] Step S3031: performing delay calibration on the delays of all audio devices with respect to the weighted center point of the sound field area calculated in step S302, to obtain calibrated delays of the audio devices with respect to the weighted center point of the sound field area;

[0121] When the user is d meters away from the speaker, the delay of the audio device relative to the user's position is: , compared to the standard speed of sound c rcf =343m / s, and the change in sound velocity caused by temperature and humidity changes will further lead to the delay calibration amount: Therefore, the delay calibration amount caused by temperature and humidity changes can be used to calibrate the delay calculation result of step S302, and the delay of the calibrated audio device for the weighted center point of the sound field area is , which serves as the basis for dynamic adjustment of delay compensation parameters in the system.

[0122] Step S3032: When the audio device delay difference exceeds a threshold, an equivalent delay module is added to the audio signal output path according to the threshold compensation strategy to prevent delay jumps caused by user movement. This implements the threshold compensation strategy, thereby reducing the delay impact of other users in the sound field area.

[0123] In this embodiment, if the delay difference range of all audio devices with respect to the weighted center point of the sound field area is less than the delay threshold, no compensation is performed; if the delay difference range of any audio device is greater than or equal to the delay threshold, an equivalent delay module is added to the audio signal output path according to the threshold compensation strategy. Since for each audio device, the delay of the calibrated audio device with respect to the weighted center point of the sound field area is Therefore, for the same weighted center point, the calibrated delay of each audio device is calculated in step S3031. After that, the calibration delay of different audio devices Compare and find the maximum propagation delay and minimum propagation delay And the difference between the two As the delay difference range, in the presence of the delay difference range of audio equipment When the delay threshold is greater than or equal to the threshold, an equivalent delay module is added to the audio signal output path according to the threshold compensation strategy.

[0124] The delay threshold is usually 20ms, and in high-precision mode, the delay threshold can be 10ms.

[0125] According to the threshold compensation strategy, an equivalent delay module is added to the audio signal output path, specifically including:

[0126] Step A1: For each sound field area, obtain the delay of each audio device to reach the weighted center point of the sound field area, and find the maximum propagation delay of all current audio devices to the user position; where the maximum propagation delay among all audio devices is for: , is the delay of the i-th audio device;

[0127] Therefore, the present invention uses the weighted center point of the sound field area to represent the representative position of a sound field area. Instead of processing each user individually, the present invention calculates the delay and compensation time of all audio devices based on the weighted center point, and finally performs delay compensation on each audio device separately in the audio output path.

[0128] Step A2: For each audio device, calculate its compensation time;

[0129] Among them, the compensation time of the i-th audio device for: ;in, is the delay of the i-th audio device, The maximum propagation delay among all audio devices;

[0130] Step A2: If the compensation time of the i-th audio device If the delay threshold is exceeded, the system defaults to no compensation to reduce the processing burden; if the compensation time of the i-th audio device exceeds the delay threshold, an equivalent delay module is added to the audio signal output path of the audio device.

[0131] In one embodiment, the delay time of the output signal of the equivalent delay module is the compensation time of the audio device. .

[0132] In another embodiment, to avoid delay jumps caused by user movement, the equivalent delay module adopts a smooth update mechanism. The delay time of the output signal of the equivalent delay module is the delay adjustment value of the current frame of the audio device: ,in, is the delay adjustment value of the current frame of the i-th audio device; is the delay adjustment value of the i-th audio device for the previous frame, It is the delay update rate (such as 0.1~0.3), which effectively controls the sound "jumping" or "distortion" during the delay adjustment process.

[0133] The Equivalent Delay Module can be used as an independent sub-module in the audio output chain, supporting dynamic parameter updates, user-configurable delay threshold settings, and status feedback mechanisms, making it suitable for different room layouts and user needs.

[0134] Step S304: performing personalized sound effect configuration, that is, determining target sound field parameters according to the user's satisfaction evaluation of the sound effect and the user's preference settings.

[0135] The target sound field parameters also include reverberation time, gain and delay.

[0136] In this embodiment, user preferences are obtained by manually adjusting the equalizer and audio content, and analyzing the user's historical behavior data. As mentioned above, the user's historical behavior data includes previous location information, activity tracks, sound effect adjustment records, etc.

[0137] Collecting user satisfaction ratings on sound effects can involve asking users to rate the sound effects or provide specific opinions and suggestions, or indirectly assessing user satisfaction with the sound effects by analyzing historical user behavior data (such as the frequency and magnitude of adjustments made to sound effect settings). In this embodiment, natural language processing and sentiment analysis techniques can be used to extract user satisfaction ratings on sound effects. For example, if a user comments that "the bass is too weak," the system will identify the user's dissatisfaction with the bass effect and optimize the target sound field parameters. Furthermore, historical user behavior data includes the frequency with which users adjust sound effect settings. Frequent adjustments may indicate that the current sound effects do not meet their expectations.

[0138] For example, if the user likes the bass effect, the system will increase the bass gain in the target sound field parameters.

[0139] If the user prefers bass, the system might increase the gain in the 80-250Hz frequency band and employ dynamic range compression (DSP algorithms) to prevent distortion. It also adjusts the width of the virtual sound field based on preferences, and superimposes personalized settings based on environmental adaptation (for example, prioritizing high-frequency attenuation in humid environments, then applying user-defined bass boost). Ultimately, the system outputs optimized audio signals, which have been adjusted to provide the best possible sound experience.

[0140] Step S305: adjusting the audio signal according to the sound field parameters of step S303 and the target sound field parameters of step S304.

[0141] Sound field parameters include reverberation time, gain, and delay. For example, the system adjusts the reverberation effect of the audio signal by extending or shortening the reverberation through an artificial reverberator to simulate an ideal acoustic environment. The system also adjusts the volume of the audio signal based on the required baseline gain value, and adjusts the playback time of the audio signal based on the required delay. These adjustments ensure that every user receives the best sound experience.

[0142] In addition, the dynamic sound field optimization module 30 is further configured to execute step S306: using a microphone to detect a sound feedback signal, and adjusting the audio signal in real time according to the difference between the sound field parameters of the sound feedback signal and the target sound field parameters, thereby forming a closed-loop optimization.

[0143] The dynamic sound field optimization module 30 uses a low-latency DSP chip or GPU acceleration algorithm to ensure real-time audio signal processing (delay <20ms).

[0144] like Figure 4 As shown, the high-precision cross-device audio synchronization module 40 is configured to achieve high-precision audio synchronization to ensure that the audio signals output by each audio device are completely consistent in time, thereby providing a seamless sound experience.

[0145] The high-precision cross-device audio synchronization module 40 is triggered to work in response to the user's satisfaction evaluation of the sound effect to achieve satisfaction monitoring. In other words, user satisfaction monitoring is achieved by collecting the user's satisfaction evaluation of the sound effect.

[0146] Collecting user satisfaction ratings on sound effects can include asking users to rate the sound effects or provide specific opinions and suggestions, or indirectly assessing user satisfaction with the sound effects by analyzing historical user behavior data (such as the frequency and amplitude of user adjustments to sound effect settings). In this embodiment, natural language processing and sentiment analysis techniques can be used to extract user satisfaction ratings on sound effects. For example, if a user comments that "the bass is too weak," the system will identify the user's dissatisfaction with the bass effect and optimize the target sound field parameters. Furthermore, historical user behavior data includes the frequency of user adjustments to sound effect settings. If a user frequently adjusts sound effect settings, this may indicate that the current sound effects do not meet their expectations and that the user's satisfaction rating is low, triggering the high-precision cross-device audio synchronization module 40 to operate.

[0147] The high-precision cross-device audio synchronization module 40 is configured to perform the following steps:

[0148] Step S401: obtaining audio signals to be output by the audio device from multiple audio devices and preprocessing the audio signals;

[0149] The audio signal may be in digital format or in analog format after analog-to-digital conversion.

[0150] To ensure accuracy in subsequent processing, these audio signals undergo pre-processing, including filtering and sampling. Filtering removes noise and interference from the signal, improving its purity. Sampling, on the other hand, unifies the audio signal format across devices, ensuring they have the same sampling rate for synchronized processing.

[0151] Step S402: Select a clock synchronization reference device; then distribute the clock information of the clock synchronization reference device as clock synchronization reference information to all audio devices, ensuring that all audio devices can receive a unified time reference.

[0152] Step S402 is crucial for subsequent clock synchronization, providing a common time reference point for all devices. Clock synchronization reference information includes timestamp sequence, frequency correction parameters, synchronization status indicators, fault tolerance check data, etc.

[0153] The clock synchronization reference device has a high-precision clock and can provide accurate time reference for other devices. The clock synchronization reference device can be an audio device, a GPS satellite, etc.

[0154] Step S403: Based on the clock synchronization reference information, a high-precision clock synchronization algorithm is used to accurately adjust the clock of each audio device to keep it consistent with the clock of the clock synchronization reference device.

[0155] High-precision clock synchronization algorithms include GPS-based synchronization or Network Time Protocol synchronization. GPS-based synchronization utilizes the precise time information provided by satellite signals, while NTP (Network Time Protocol) synchronization uses network communication to obtain the timestamp of the reference clock and perform corresponding clock adjustments.

[0156] High-precision clock synchronization algorithms can include using a delay error algorithm to compensate for network delay. Network delay is the time delay caused by network transmission and processing during audio transmission. When using a delay error algorithm to compensate for network delay, the network delay is measured by measuring the time difference between the audio signal's transmission and reception. This delay is then used to compensate the audio signal for time, reducing synchronization errors caused by network delay.

[0157] In order to further improve synchronization accuracy, the system will also optimize the algorithm parameters and hardware design of the high-precision clock synchronization algorithm, reduce clock drift, increase the frequency of clock synchronization, and thus reduce clock errors.

[0158] Step S404: establishing an optimized audio transmission protocol to ensure that audio data can be transmitted quickly and reliably;

[0159] The optimized audio transmission protocol of this invention is based on the existing Real-Time Transport Protocol (RTP). This optimization includes: 1) using a high-efficiency audio codec (such as Opus) to reduce data volume and transmission latency; 2) using forward error correction (FEC) and negative acknowledgement (NACK) mechanisms during network transmission to improve data transmission reliability; and 3) optimizing the audio data transmission path based on the network topology, using redundant links and intelligent routing to avoid network congestion and latency.

[0160] Among them, efficient audio codecs are used for sending and receiving data of all audio devices and are suitable for real-time transmission, including AAC, MP3 or Opus. These audio codecs can reduce data volume and transmission delay while ensuring sound quality.

[0161] The FEC mechanism adds redundant information to the audio data, allowing the receiver to restore the original audio data even if some data packets are lost, thereby reducing synchronization errors caused by data loss. The NACK mechanism allows the receiver to request the sender to resend lost data packets, ensuring the integrity of the audio data and improving synchronization accuracy.

[0162] Step S405: Synchronous audio stream generation; that is, using an optimized audio transmission protocol, the audio signals of each device are synchronized according to the frame, and then the synchronized audio signals are synthesized to generate a multi-channel synchronous audio stream, where each channel corresponds to an audio device.

[0163] Frames are generated during the execution of the audio transmission protocol. During audio transmission, the audio signal is divided into small data blocks, each of which is a frame. Each frame of audio data is accompanied by a timestamp, indicating the time when the frame should be played. Frame synchronization is achieved by comparing the timestamps of the frames on different devices to ensure that each frame has the same timestamp.

[0164] Step S406: monitoring the synchronization error of the synchronous audio stream in real time, and determining whether to perform error compensation according to the synchronization error (including time error and phase difference).

[0165] Specifically, if the synchronization error exceeds a certain threshold, the system compensates for the error by fine-tuning the audio signal's playback time or inserting a compensation signal, ensuring that the audio signals of different channels are aligned in time and phase. Error compensation can compensate for the error by fine-tuning the audio signal's playback time or inserting a compensation signal, thus ensuring the accuracy of the synchronization effect.

[0166] Synchronization errors include time difference and phase difference. Synchronization errors are detected by comparing the timestamps of the audio signals of each device. The phase difference of the audio signals is also monitored to ensure that the audio signals of different channels are consistent in time and phase.

[0167] Step S407: Send the multi-channel synchronous audio streams to all corresponding audio devices respectively, so that the audio output remains synchronized in time.

[0168] Thus, by precisely controlling the audio playback timing of each device, the system ensures that the audio output of all devices is completely synchronized in time.

[0169] like Figure 5 As shown, the intelligent management and optimization module 50 is configured to generate an intelligent optimization strategy based on user feedback data, environmental monitoring data, and system operation data. The intelligent optimization strategy includes automatic adjustment results of parameters of the room acoustic modeling module 20, the dynamic sound field optimization module 30, and the high-precision cross-device audio synchronization module 40.

[0170] Specifically, the intelligent management and optimization module 50 receives user feedback data, environmental monitoring data and system operation data.

[0171] The user feedback data includes the user's satisfaction rating on the sound effects and the user's preference settings, which are used to automatically adjust the target sound field parameters in the dynamic sound field optimization module 30 and trigger the operation of the high-precision cross-device audio synchronization module 40.

[0172] The environmental monitoring data includes temperature, humidity and background noise. The temperature and humidity are used for sound field parameter calibration in the dynamic sound field optimization module 30 , and the background noise is used as the background noise of the background noise modeling module 24 of the room acoustic modeling module 20 .

[0173] The system operation data includes device performance parameters and the quality of the audio signal. The system operation data is used to optimize the audio transmission protocol of the high-precision cross-device audio synchronization module 40.

[0174] For example, if device performance degrades, the system adjusts the audio codec parameters in the audio transmission protocol to ensure sound stability. If audio signal quality degrades, the system optimizes the audio transmission protocol to reduce transmission delay and errors. Specifically, when the system detects increased CPU or DSP load or memory constraints on the device, it automatically switches the current audio codec from a high-complexity codec to a low-complexity codec. It also dynamically adjusts the bitrate (e.g., from 320kbps to 160kbps), sampling rate (e.g., from 96kHz to 48kHz), and frame length (e.g., from 10ms to 20ms) to reduce processing overhead and stabilize audio playback. If, during transmission, the system detects an increase in audio packet loss rate, latency jitter, or synchronization errors exceeding a preset threshold, it optimizes the audio transmission protocol. This includes enabling a stronger packet loss concealment (PLC) mechanism, increasing the buffer size, adopting a more robust UDP transmission redundancy strategy (e.g., repeated frame transmission and forward error correction (FEC)), or switching to a low-latency adaptive bitrate encoding channel. This effectively improves audio transmission quality and reduces the perceived audio degradation on the user side.

[0175] Device performance parameters and audio signal quality are evaluated by connecting equipment such as audio analyzers, signal generators, and oscilloscopes to the input and output of the audio device. For example, when testing an audio amplifier, a signal generator outputs a known test signal to the amplifier's input, and an oscilloscope or audio analyzer is connected to the amplifier's output to analyze the amplified signal.

[0176] like Figure 5 As shown, the intelligent management and optimization module 50 is also configured to collect user feedback data before automatic adjustment, environmental monitoring data and system operation data, the corresponding room acoustic modeling module 20, the dynamic sound field optimization module 30, the initial parameter values ​​of the high-precision cross-device audio synchronization module 40 and the automatic adjustment results of the parameters, and the corresponding user feedback data and the change in system operation data, these time series data; and then use a deep learning algorithm to train the collected time series data to obtain an intelligent optimization strategy that allows users to evaluate their satisfaction with the sound effects and optimize the system operation data.

[0177] Taking a typical training sample of a deep learning algorithm as an example, the input feature vector of the deep learning algorithm can be expressed as:

[0178] X=[P CPU , M free ,T,H, , f resp， , bitrate, S volume， , U en ],

[0179] The parameters are defined as follows:

[0180]

[0181] The corresponding training output vector is: y=[S sat ,E sys ],

[0182] Where: S sat The user's satisfaction rating for the current sound effect is in the range of 0-100, which can be estimated through subsequent subjective scoring questionnaires or emotion recognition systems (such as facial expressions and voice emotions); E sys The system resource efficiency score is a comprehensive function of system load and sound quality per unit time. It has no units and is automatically calculated by the system based on running time, power consumption, and processing load.

[0183] The training goal of the deep learning algorithm is to find a set of policy functions f:x→y such that S sat Maximize, while E sys Minimize or optimize within a certain range. Through continuous optimization and iterative training, a localized optimization strategy model is formed under specific room types and user configurations.

[0184] In other embodiments, the step of using the deep learning algorithm to train the intelligent management and optimization module 50 to obtain the intelligent optimization strategy can be omitted, and only the adjustment amount of each parameter needs to be manually set based on experience.

[0185] In this embodiment, the deep learning algorithm uses a long short-term memory (LSTM) or Transformer model, which is suitable for processing multi-source data and has strong learning capabilities. Preprocessed time series data is input into the model, and the model parameters are adjusted using an optimization algorithm (such as Adam) to minimize the error between the target value and the true value. During the training process, cross-validation and other techniques are used to evaluate model performance and avoid overfitting.

[0186] The deep learning model in the present invention focuses on the fusion and feature extraction of multi-source data in terms of training data preprocessing; in terms of model selection, it focuses on the processing ability of time series data; in the training process, it focuses on the accuracy and personalization of audio parameter adjustment.

[0187] Example 1: Optimizing Multi-User Audio Experience in a Home Theater System

[0188] In a home theater system, users desire an immersive audio experience when watching movies. However, achieving this goal is challenging due to the acoustic characteristics of the room and the varying positions of multiple users. This embodiment addresses this issue by integrating a multi-user position sensing module 10, a room acoustics modeling module 20, and a dynamic sound field optimization module 30.

[0189] First, the multi-user location awareness module 10 monitors each user's location and activity in real time using a microphone array, millimeter-wave radar, and cameras. The system captures sound signals, reflected signals, and visual images. After preprocessing through filtering, denoising, and feature extraction, it integrates the data using a multi-sensor data fusion algorithm. Using deep learning algorithms (such as CNN or RNN), the system accurately identifies each user's location and activity trajectory, tying their location to their identity, and providing data support for subsequent sound optimization.

[0190] Next, the room acoustic modeling module 20 acquires the physical parameters of the room, including its size, shape, wall material, and furniture layout. The system generates a three-dimensional model of the room, recording the position and material of the furniture. It then calculates the room's acoustic properties, such as reflection, absorption, and diffraction, by simulating sound wave propagation. By comparing the model parameters with actual measured acoustic data, the system improves modeling accuracy. Furthermore, the system considers ambient noise modeling, incorporating background noise into the acoustic model.

[0191] Finally, the dynamic sound field optimization module 30 divides the room into multiple sound field zones based on the user's position and the room's acoustic characteristics. The system calculates the sound field parameters for each zone, such as reverberation time, gain, and delay, and applies a real-time dynamic adjustment algorithm to adjust the audio signal based on these calculated sound field parameters. Through environmental impact analysis and personalized sound effect configuration, the system further optimizes the sound effects, ensuring every user receives the best possible sound experience.

[0192] Through the above steps, this embodiment achieves the goal of providing an immersive audio experience in a home theater system based on the position of each user and the acoustic characteristics of the room.

[0193] Example 2: Intelligent Audio Management and Optimization in a Conference System

[0194] In a conference system, multiple participants sit in different locations for a discussion. The system needs to dynamically adjust audio parameters based on the participants' locations and voice activity to ensure that every participant can clearly hear the conference content. This embodiment achieves this goal by integrating a multi-user position sensing module 10, a room acoustics modeling module 20, a dynamic sound field optimization module 30, and an intelligent management and optimization module 50.

[0195] First, the multi-user location awareness module uses a microphone array and camera to monitor attendees' locations and movements in real time. The system captures audio signals and visual images, pre-processes them, and integrates the data using a multi-sensor data fusion algorithm. Using deep learning algorithms, the system accurately identifies each attendee's location and movement trajectory, tying their location to their identity, providing data support for subsequent audio optimization.

[0196] Secondly, the room acoustic modeling module 20 is used to establish an acoustic model of the conference room.

[0197] Secondly, the dynamic sound field optimization module 30 divides the conference room into multiple sound field zones based on participant location information. The system calculates the sound field parameters for each zone, such as reverberation time, gain, and delay, and applies a real-time dynamic adjustment algorithm to adjust the audio signal based on these calculated sound field parameters. Through environmental impact analysis and personalized sound effect configuration, the system further optimizes the sound quality, ensuring that every participant can clearly hear the meeting content.

[0198] Finally, the intelligent management and optimization module 50 receives participant feedback, environmental monitoring data, and system operation data. Using natural language processing and sentiment analysis, the system extracts participant evaluations and requirements for sound effects and generates intelligent optimization strategies based on environmental changes and system performance. The system adjusts audio signal parameters based on participant preferences, providing personalized sound effects.

[0199] Through the above steps, this embodiment achieves the goal of dynamically adjusting audio parameters in a conference system based on the location and voice activity of the participants, ensuring that each participant can clearly hear the conference content.

[0200] Example 3: High-precision cross-device audio synchronization in a gaming console

[0201] In a gaming system, multiple players use different audio devices (such as headphones and speakers) to play games. The system needs to achieve high-precision audio synchronization to ensure that the audio signals output by each device are completely consistent in time, thereby providing a seamless sound experience. This embodiment achieves this goal by activating the high-precision cross-device audio synchronization module 40 in the multi-user adaptive dynamic sound field optimization system.

[0202] First, the high-precision cross-device audio synchronization module 40 obtains audio signals from multiple audio devices. These signals may be in digital format or analog signals after analog-to-digital conversion. The system pre-processes these audio signals, including filtering and sampling, to improve signal purity and unify the format.

[0203] Next, the system selects an audio device with a high-precision clock as a reference device and distributes this device's clock information to other devices. Using a synchronization algorithm based on GPS or NTP, the system precisely adjusts the clocks of each device to align with the reference device's clock. To further improve synchronization accuracy, the system optimizes algorithm parameters and hardware design to reduce clock drift and increase the frequency of clock synchronization.

[0204] Next, the system optimizes the audio transmission protocol, adopting either the Real-Time Transport Protocol or the High-Efficiency Audio Transport Protocol to ensure fast and reliable transmission of audio data. The system selects an audio codec suitable for real-time transmission, such as AAC, MP3, or Opus, and optimizes the audio data transmission path based on the network topology, employing redundant links and intelligent routing to avoid network congestion and delays.

[0205] During the synchronized audio stream generation phase, the system synchronizes the audio signals from each device on a frame-by-frame basis. Each audio frame is accompanied by a timestamp, indicating the time when the frame should be played. By comparing the audio frame timestamps of each device, the system ensures that each frame's timestamps are consistent, thus achieving frame synchronization. The system then synthesizes the synchronized audio signals to generate a multi-channel synchronized audio stream.

[0206] Finally, the system monitors the accuracy of the synchronized audio streams in real time, detecting synchronization errors by comparing the timestamps of the audio signals from each device. If the synchronization error exceeds a certain threshold, the system uses an error compensation algorithm to adjust and ensure accurate synchronization. During the audio playback phase, the system sends the synchronized audio stream to each audio device for coordinated playback, ensuring that the audio output of all devices is completely synchronized in time.

[0207] Through the above steps, this embodiment achieves high-precision audio synchronization between multiple audio devices in a gaming system, providing players with a seamless and consistent audio experience.

[0208] The multi-user adaptive dynamic sound field optimization system of the present invention has the following advantages:

[0209] ① Achieving multi-user location awareness: Combining acoustics, millimeter-wave radar, and vision technologies, the system achieves centimeter-level positioning and user identity binding. Through multi-sensor data fusion, the system can identify the specific locations of multiple users and their predicted movement trajectories in real time, precisely adjusting the direction and intensity of each user's sound output. It can also predict future changes in a user's activity area or location and adjust sound field settings in advance. This predictive capability not only improves the user's listening experience but also reduces the latency of real-time adjustments, ensuring a smooth transition in sound output.

[0210] ② Dynamic Sound Field Optimization: Combining room acoustic modeling and user location, the system dynamically optimizes the reverberation, gain, and delay parameters of the sound based on the room's reflection and absorption characteristics and the user's activity area. This system can adjust the sound effects during user activity to provide an optimal audio experience, especially in complex environments (such as rooms with multiple users and multiple obstacles), rather than relying on fixed, preset sound field settings. Furthermore, the environmental modeling system in the present invention is tightly integrated with the sound field optimization algorithm, enabling it to quickly analyze the acoustic characteristics of a room or space (such as reflection, absorption coefficient, surface material, etc.) and generate sound field parameters in real time. This close integration of environmental modeling and sound field adjustment significantly improves the accuracy of the system, making the optimization effect even more pronounced in highly dynamic environments.

[0211] ③ Cross-device audio synchronization: Using GPS-based synchronization or network time protocols, this solution addresses the latency issues inherent in existing cross-device audio output technologies, ensuring that multiple devices can simultaneously output precisely synchronized audio signals. Through optimized audio transmission protocols, this reduces inter-device audio lag and improves audio output consistency in multi-device collaboration scenarios.

[0212] ④ Intelligent Management and Optimization: Automatically adjust parameters based on user feedback, environmental monitoring data, and system operation data to provide personalized sound effects. System performance can be optimized based on user feedback and environmental changes, providing a more accurate and convenient audio experience. Furthermore, this invention innovatively implements intelligent sound field optimization through deep learning algorithms, combined with user feedback and environmental changes. This intelligent optimization capability ensures that the system's sound effects continuously improve with usage and feedback, meeting the personalized needs of different users.

[0213] In summary, the present invention realizes multi-user position perception, dynamic sound field adjustment, and cross-device sound synchronization. It aims to provide flexible, efficient, and personalized sound optimization solutions for multi-user, multi-device environments by comprehensively using microphone arrays, AI analysis, millimeter-wave radar, visual technology and other means.

[0214] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Various modifications are possible. Any simple, equivalent changes and modifications made in accordance with the claims and description of the present invention are within the scope of protection of the patent claims. Anything not fully described in this invention is conventional technology.

Claims

1. A multi-user adaptive dynamic sound field optimization system, characterized in that: Includes multi-user position perception module, room acoustic modeling module and dynamic sound field optimization module; The multi-user location perception module is used to pre-process the location data collected by the microphone array, millimeter-wave radar, and camera, and then fuse them to generate the user location; The room acoustic modeling module is set to perform room acoustic modeling and analyze the acoustic performance of the room, which includes the acoustic model and reflection and absorption characteristics; The dynamic sound field optimization module is set to adjust audio parameters in real time based on the user's position, the room's acoustic performance and other environmental factors. Other environmental factors include temperature and humidity from user preferences and environmental monitoring data.

2. The multi-user adaptive dynamic sound field optimization system according to claim 1, characterized in that: The multi-user location perception module includes a data acquisition module, a data preprocessing module for outputting preprocessed data, and a location information fusion module connected in sequence; The data acquisition module includes a sound signal acquisition branch, a radar signal acquisition branch and an image acquisition branch connected in parallel with each other; The sound signal acquisition branch uses a microphone array, the radar signal acquisition branch uses a millimeter-wave radar, and the image acquisition branch uses a camera; The data preprocessing module is used to preprocess the collected data of the data acquisition module, and the preprocessing includes filtering, denoising and feature extraction; The location information fusion module is configured to receive pre-processed data and fuse different types of pre-processed data to generate a user location.

3. The multi-user adaptive dynamic sound field optimization system according to claim 2, characterized in that: The multi-user location perception module also includes a user activity trajectory analysis module, which is configured to: input the fused location results together with the user's behavior pattern and appearance characteristics into a machine learning algorithm to analyze and obtain the real-time user location and predicted activity trajectory, from which the predicted user location is obtained as the final output user location; then, based on the user's voice characteristics, behavior pattern or appearance characteristics, the user location is associated with the user identity to achieve user identity binding.

4. The multi-user adaptive dynamic sound field optimization system according to claim 1, characterized in that: The multi-user location perception module also includes an anomaly detection module, which sets an anomaly threshold based on the user's behavior pattern. When at least one of the speed and acceleration obtained from the user's historical location data exceeds the anomaly threshold, it is judged as an anomaly; otherwise, it is judged as normal.

5. The multi-user adaptive dynamic sound field optimization system according to claim 1, characterized in that: The room acoustic modeling module includes an acoustic model construction module, and the input end of the acoustic model construction module is connected to the acoustic characteristic measurement module and the sound source modeling module; The acoustic characteristic measurement module receives the 3D model of the room, measures the materials at each location in the 3D model, measures the acoustic characteristics of each material, and finally outputs the 3D model of the room and the acoustic characteristics of each material in the room; The sound source modeling module is configured to measure the sound source characteristics of each sound source and establish a sound source characteristic model according to the sound source characteristics of the sound source; The acoustic model construction module receives the three-dimensional model of the room and the acoustic characteristics of each material in the room output by the acoustic characteristic measurement module and the sound source characteristic data output by the sound source modeling module, and then performs acoustic simulation to construct an acoustic model based on the three-dimensional model of the room, the acoustic characteristics of each material in the room, and the sound source characteristic model. The acoustic performance of the room is obtained based on the acoustic model analysis. The acoustic performance of the room includes the sound pressure distribution, sound intensity distribution, frequency response, acoustic model, and reflection and absorption characteristics of the sound wave in the room. The acoustic simulation uses a finite element analysis method or a boundary element analysis method.

6. The multi-user adaptive dynamic sound field optimization system according to claim 5, characterized in that: The input end of the acoustic model construction module is also connected to the background noise modeling module, and the output end of the acoustic model construction module is connected to the model calibration and optimization module; The background noise modeling module is configured to measure background noise in a room and input the background noise into the acoustic model building module to incorporate the background noise into the acoustic model; The model calibration and optimization module is configured to compare the acoustic model with actual measured acoustic data to adjust the parameters of the acoustic model.

7. The multi-user adaptive dynamic sound field optimization system according to claim 1, characterized in that: The dynamic sound field optimization module is configured to perform the following steps: Step S301: Divide the room into multiple independent sound field areas using a sound field area division module according to the user's position; Step S302: inputting the divided sound field areas into a sound field parameter calculation module, so that the sound field parameter calculation module calculates the sound field parameters of each sound field area according to the acoustic performance of the room; Step S303: determining a calibration amount of the sound field parameter according to the environmental parameter, and calibrating the sound field parameter according to the calibration amount of the sound field parameter; Step S304: determining target sound field parameters based on the user's satisfaction rating of the sound effect and the user's preference settings; Step S305: adjusting the audio signal according to the sound field parameters of step S303 and the target sound field parameters of step S303.

8. The multi-user adaptive dynamic sound field optimization system according to claim 7, characterized in that: Collecting user satisfaction evaluations on sound effects includes: asking users to rate the sound effects or provide specific opinions and suggestions, or indirectly evaluating user satisfaction with the sound effects by analyzing user historical behavior data; The dynamic sound field optimization module is further configured to execute step S306: detecting a sound feedback signal using a microphone, and adjusting the audio signal in real time according to a difference between a sound field parameter of the sound feedback signal and a target sound field parameter.

9. The multi-user adaptive dynamic sound field optimization system according to claim 8, characterized in that: It also includes a high-precision cross-device audio synchronization module, which is triggered in response to the user's satisfaction rating of the sound effect; The high-precision cross-device audio synchronization module is configured as follows: Step S401: obtaining audio signals to be output by the audio device from multiple audio devices and preprocessing the audio signals; Step S402: Select a clock synchronization reference device; The clock information of the clock synchronization reference device is then distributed to all audio devices as clock synchronization reference information, ensuring that all audio devices can receive a unified time reference; Step S403: Using a high-precision clock synchronization algorithm based on the clock synchronization reference information to accurately adjust the clock of each audio device to keep it consistent with the clock of the clock synchronization reference device; Step S404: establishing an optimized audio transmission protocol. The optimization of the audio transmission protocol at least includes using a forward error correction mechanism and a packet loss retransmission mechanism to improve the reliability of data transmission during network transmission. Step S405: synchronizing the audio signals of the various devices on a frame-by-frame basis using an optimized audio transmission protocol, and then synthesizing the synchronized audio signals to generate a multi-channel synchronized audio stream, where each channel corresponds to one audio device. Step S406: monitoring the synchronization error of the synchronous audio stream in real time, and determining whether to perform error compensation according to the synchronization error; Step S407: Send the multi-channel synchronous audio streams to all corresponding audio devices respectively, so that the audio output remains synchronized in time.

10. The multi-user adaptive dynamic sound field optimization system according to claim 1, characterized in that: The system also includes an intelligent management and optimization module, which is configured to generate an intelligent optimization strategy based on user feedback data, environmental monitoring data, and system operation data. The intelligent optimization strategy includes automatic adjustment results of parameters of the room acoustic modeling module, the dynamic sound field optimization module, and the high-precision cross-device audio synchronization module; User feedback data includes user satisfaction ratings on sound effects and user preferences, which are used to automatically adjust target sound field parameters in the dynamic sound field optimization module; Environmental monitoring data includes temperature, humidity, and background noise. Temperature and humidity are used for sound field parameter calibration in the dynamic sound field optimization module, and background noise is used as background noise in the background noise modeling module of the room acoustic modeling module. System operation data includes device performance parameters and audio signal quality. System operation data is used to optimize the audio transmission protocol of the high-precision cross-device audio synchronization module.

Images