Earphone adaptive audio equalization method based on deep reinforcement learning
By applying deep reinforcement learning technology in headphones, combining A3 algorithm and LSTM network, optimizing the headphone equalizer parameters, solving the problems of personalized needs and long-term dependencies in the existing technology, and achieving efficient and intelligent audio equalization effect.
Patent Information
- Application Number
- CN202411688531.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-04-25
AI Technical Summary
The existing headphone balance technology is difficult to meet the personalized needs of different users in various scenarios, and it is too difficult to deal with complex audio scenarios, neglecting the long-term dependence in audio processing.
Adaptive audio equalization method based on deep reinforcement learning is adopted to learn and optimize the balance strategy by building a reinforcement learning environment by using the headphone equalizer parameters as actions, using instant reward and long-term reward mechanisms, combining the improved A3 algorithm and long-term memory network (LSTM).
It realizes personalized balance, environmental adaptability and real-time response, and can capture long-term dependencies in audio signals, provide a more intelligent and coherent balance effect, and improves the user's listening experience.
Smart Images
Figure CN119893358A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of headphone adaptive audio equalization methods, and in particular to a headphone adaptive audio equalization method based on deep reinforcement learning. Background Art
[0002] With the continuous development of audio technology, headphones have become an indispensable electronic device in people's daily lives. However, as users' requirements for sound quality continue to increase, traditional audio equalization technology has gradually revealed its limitations. At present, the mainstream headphone equalization technology on the market mainly relies on preset equalizer parameters. Although this method is simple and direct, it is difficult to meet the personalized needs of different users in various scenarios.
[0003] The closest existing technologies usually use fixed parameter equalizers or simple adaptive equalization algorithms. These methods can improve the sound quality to a certain extent, but there are still many problems. First, the fixed parameter equalizer cannot be adjusted in real time according to different audio content and user preferences, resulting in unsatisfactory sound quality in some cases. Secondly, although the simple adaptive equalization algorithm can make limited adjustments based on audio characteristics, it lacks the ability to learn users' long-term listening habits and is difficult to provide a truly personalized listening experience.
[0004] In addition, existing technologies are often unable to cope with complex audio scenarios. For example, in a noisy environment, traditional equalizers are difficult to effectively highlight important audio information, which greatly reduces the user's listening experience. At the same time, for rapidly changing music styles or mixed audio types, the response speed and accuracy of existing technologies are difficult to be satisfactory.
[0005] Most importantly, existing technologies generally ignore the long-term dependencies in audio processing. Music is a signal with a strong temporal nature. Equalization based solely on the audio features of the current moment often ignores the overall structure and emotional expression of the music, making it impossible to truly restore the intention of the music creator. Summary of the invention
[0006] The present invention aims to solve these problems in the prior art and proposes a new headphone adaptive audio equalization method based on deep reinforcement learning. The method aims to solve a series of technical problems such as personalized equalization, environmental adaptability, real-time response, and long-term audio feature learning. By introducing deep reinforcement learning technology, the present invention can continuously learn and optimize the equalization strategy to provide users with a more intelligent and personalized listening experience.
[0007] The present invention proposes a headphone adaptive audio equalization method based on deep reinforcement learning, comprising the following steps:
[0008] (1) Construct a reinforcement learning environment and use the filter parameters in the headphone equalizer as the action a in reinforcement learning t , the error between the sound output by the sound source after the sound source is adjusted by the headphones and EQ parameters and the real undistorted sound is used as the immediate reward r of reinforcement learning t and the long-term reward R t ;
[0009] (2) Synchronize the EQ parameter adjustment process, sound playback, and listener feedback process;
[0010] (3) The EQ adjustment strategy learning process is divided into a training phase and an online adjustment phase;
[0011] (4) In the training phase, user audio data and listener feedback are obtained, and the improved A3 algorithm is used to extract audio acoustic features and determine the audio type;
[0012] (5) Introduce the long-term memory network LSTM to learn the action a and state s of reinforcement learning to obtain the neural network Q function;
[0013] (6) EQ parameter adjustment is defined as a Markov decision process (MDP), and a set of EQ parameter adjustment strategies corresponding to each audio type is output;
[0014] (7) In the online adjustment stage, the audio features during EQ adaptive adjustment are input into the trained neural network Q function to obtain the strategy value and normalize it;
[0015] (8) Use the EQ parameter adjustment strategy to perform EQ adaptive filtering and adjust the parameters of the headphones.
[0016] Preferably, the instant reward r t The calculation formula is:
[0017] r t = -MSE(y t ,y′ t )
[0018] Where MSE represents the mean square error function, y t represents the true undistorted sound at time t, y′ t Represents the sound after EQ adjustment at time t.
[0019] Preferably, the calculation formula of the long-term reward R_t is:
[0020] k from 0 to ∞
[0021] Among them, γ represents the discount factor, which ranges from [0,1] and is used to balance the importance of immediate rewards and future rewards.
[0022] Preferably, the core formula of the improved A3 algorithm is:
[0023]
[0024] Where x represents the input audio feature vector, c i represents the i-th audio category, P(c i |x) means that given feature x, it belongs to category c i The posterior probability, P(x|c i ) indicates category c i The likelihood probability of , P(x) represents the prior probability of feature x.
[0025] Preferably, the update formula of the neural network Q function is:
[0026]
[0027] Among them, α represents the learning rate, which ranges from (0,1] and is used to control the step size of Q value update; s t represents the state at time t, a t represents the action at time t, s t +1 indicates the state at time t+1.
[0028] Preferably, the core update formula of the LSTM is:
[0029] j t =v(W f ·[u t-1 ,x t ]+b f )
[0030] i t =σ(W i ·[h t-1 ,x t ]+b i )
[0031] o t =σ(W o ·[h t-1 ,x t ]+b o )
[0032]
[0033] h t =o t *tanh(c t )
[0034] Among them, f t 、i t , ot They represent the forget gate, input gate and output gate respectively, and c t Indicates the unit state, h t represents the hidden state, σ represents the sigmoid function, W and b represent the weight matrix and bias vector respectively.
[0035] Preferably, the strategy value normalization formula is:
[0036]
[0037] Among them, V(s t ,a t ) means in state s t Take action a t The value of a V(s t ,a) and max a V(s t , a) respectively represent the state s t The minimum and maximum value of all possible actions.
[0038] Preferably, it also includes using a simulated listener feedback function, which is expressed as:
[0039] F(x,θ)=w 1 *f 1 (x,θ)+w 2 *f 2 (x,θ)+…+w n *f n (x,θ)
[0040] Among them, x represents the input audio, θ represents the current EQ parameter, and f i (x,θ) represents the i-th evaluation index function, w i represents the weight of the i-th evaluation index, and ∑w i =1.
[0041] Preferably, it also includes calculating the strategy value score π(a|s), and the calculation formula is:
[0042]
[0043] Among them, τ represents the temperature parameter, which is used to control the degree of exploration of the strategy. The larger τ is, the more the strategy tends to be evenly distributed, and the smaller τ is, the more the strategy tends to be greedy.
[0044] Preferably, it also includes using a response function H(f) of a human auditory model, which is expressed as:
[0045]
[0046] Where, f represents the frequency, f 0 represents the characteristic frequency, f q represents the Q value frequency, k, α, and γ are constant parameters used to adjust the shape and sensitivity of the curve.
[0047] The beneficial effects of the present invention are mainly reflected in the following aspects:
[0048] The core of the present invention is to cleverly transform the complex audio equalization problem into a decision-making process that can be solved by deep reinforcement learning. This innovative approach not only solves many problems faced by traditional technologies, but also achieves multiple breakthroughs in algorithmic mechanisms. First, by using the headphone equalizer parameters as reinforcement learning actions, the present invention establishes a direct and efficient parameter optimization mechanism. Secondly, the dual reward mechanism of immediate rewards and long-term rewards is introduced to effectively balance the relationship between short-term sound quality improvement and long-term listening optimization.
[0049] More importantly, the present invention successfully captures the long-term dependencies in audio signals by introducing the long short-term memory network (LSTM). This innovation not only solves the problem that traditional methods ignore the overall structure of music, but also enables the system to "understand" the emotional and style changes of music, thereby providing a more intelligent and coherent equalization effect. At the same time, the application of the improved A3 algorithm greatly improves the accuracy of audio type recognition, laying a solid foundation for the subsequent selection of equalization strategies.
[0050] The present invention also cleverly solves multiple contradictions in the algorithm mechanism. For example, in terms of the balance between exploration and utilization, by introducing a dynamic adjustment strategy for the temperature parameter τ, it is ensured that the system has sufficient exploration space in the early stage and can converge to the optimal strategy in the later stage. In terms of the trade-off between computational efficiency and optimization effect, the present invention uses a carefully designed neural network structure and optimization algorithm to ensure high-quality balancing effects while controlling the response time to the millisecond level, achieving near real-time adjustment.
[0051] In addition, the present invention achieves complementary and synergistic effects in multiple core technical points. For example, the long-term dependency features captured by the LSTM network and the audio type information identified by the A3 algorithm complement each other, and together provide a comprehensive basis for the selection of equalization strategies. At the same time, the combination of the reinforcement learning framework and the human auditory model enables the system to not only optimize objective audio indicators, but also better meet the subjective feelings of the human ear.
[0052] In general, this invention successfully brings audio equalization technology to a new level through the innovative application of deep reinforcement learning technology. It not only solves the problems of personalization, environmental adaptability and real-time performance faced by traditional methods, but also achieves comprehensive improvement in sound quality, user experience and computing efficiency through innovation and optimization of algorithm mechanisms. This all-round technological breakthrough has opened up a brand new path for the development of future intelligent audio processing technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 This is the improved A3 algorithm logic block diagram of the present invention.
[0054] Figure 2 Update the logic block diagram of the LSTM network of the present invention.
[0055] Figure 3 Update the logic block diagram for the Q function of the present invention.
[0056] Figure 4 It is a logic block diagram of the strategy value calculation and normalization of the present invention. DETAILED DESCRIPTION
[0057] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0058] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0059] Example 1
[0060] See also Figure 1-4 The present invention relates to a method for adaptive audio equalization of headphones based on deep reinforcement learning. The method realizes intelligent and personalized equalization adjustment of headphone audio by combining deep learning and reinforcement learning technology, significantly improving the user's listening experience.
[0061] First, the present invention proposes a headphone adaptive audio equalization method based on deep reinforcement learning, which includes the following steps:
[0062] (1) Construct a reinforcement learning environment and use the filter parameters in the headphone equalizer as the action a in reinforcement learning t , the error between the sound output by the sound source after the sound source is adjusted by the headphones and EQ parameters and the real undistorted sound is used as the immediate reward r of reinforcement learningt and the long-term reward R t ;
[0063] (2) Synchronize the EQ parameter adjustment process, sound playback, and listener feedback process;
[0064] (3) The EQ adjustment strategy learning process is divided into a training phase and an online adjustment phase;
[0065] (4) In the training phase, user audio data and listener feedback are obtained, and the improved A3 algorithm is used to extract audio acoustic features and determine the audio type;
[0066] (5) Introduce the long-term memory network LSTM to learn the action a and state s of reinforcement learning to obtain the neural network Q function;
[0067] (6) EQ parameter adjustment is defined as a Markov decision process (MDP), and a set of EQ parameter adjustment strategies corresponding to each audio type is output;
[0068] (7) In the online adjustment stage, the audio features during EQ adaptive adjustment are input into the trained neural network Q function to obtain the strategy value and normalize it;
[0069] (8) Use the EQ parameter adjustment strategy to perform EQ adaptive filtering and adjust the parameters of the headphones.
[0070] When constructing a reinforcement learning environment, the present invention innovatively uses the filter parameters in the headphone equalizer as the action a in reinforcement learning. t This design enables the system to directly learn and optimize the parameter settings of the equalizer, thereby adjusting the audio effect more accurately. Preferably, the present invention uses the error between the sound output by the sound source after the sound is adjusted by the headphones and EQ parameters and the real undistorted sound as the instant reward r of reinforcement learning. t and the long-term reward R t This reward mechanism can effectively guide the system to optimize towards the best sound quality.
[0071] Secondly, the present invention proposes an instant reward r t Specifically, r t = -MSE(y t ,y′ t ), where MSE represents the mean square error function y t represents the true undistorted sound at time t, y′ tRepresents the sound after EQ adjustment at time t. By minimizing the mean square error, the system can continuously adjust the EQ parameters to make the output sound closer to the real undistorted sound. In practical applications, we found that when the MSE value is less than 0.01, the human ear can hardly perceive the distortion, so 0.01 can be used as an important threshold.
[0072] Again, the present invention introduces the concept of long-term reward R_t, which is calculated as follows:
[0073] k from 0 to ∞
[0074] Wherein, γ represents a discount factor, and its value range is [0,1], which is used to balance the importance of immediate rewards and future rewards. In one embodiment of the present invention, we found that when γ is 0.95, a good balance can be achieved between short-term effects and long-term effects. This long-term reward mechanism allows the system to not only focus on the improvement of sound quality at the current moment, but also take into account the overall listening experience in the future.
[0075] Next, the present invention uses the improved A3 algorithm to extract audio acoustic features and determine the audio type. The core formula of the A3 algorithm is:
[0076]
[0077] Where x represents the input audio feature vector, c i represents the i-th audio category, P(c i |x) means that given feature x, it belongs to category c i The posterior probability, P(x|c i ) indicates category c i The likelihood probability of feature x is expressed as P(x), and P(x) represents the prior probability of feature x. This improved A3 algorithm can more accurately identify different types of audio and provide an important basis for subsequent EQ parameter adjustment.
[0078] Then, the present invention introduces the neural network Q function, and its update formula is:
[0079]
[0080] Among them, α represents the learning rate, which ranges from (0,1] and is used to control the step size of Q value update; s t represents the state at time t, a t represents the action at time t, s t +1 represents the state at time t+1. In practice, we found that when α is 0.1, the system can converge to the optimal strategy quickly. The introduction of this Q function enables the system to learn the strategy of selecting the optimal action in different states.
[0081] In order to further improve the learning ability of the system, the present invention also introduces a long short-term memory network (LSTM). The core update formula of LSTM includes multiple components such as forget gate, input gate, output gate, etc., which enables the system to capture long-term dependencies in audio signals.
[0082] The core update formula of the LSTM is:
[0083] j t =v(W f ·[u t-1 ,x t ]+b f )
[0084] i t =σ(W i ·[h t-1 ,x t ]+b i )
[0085] o t =σ(W o ·[h t-1 ,x t ]+b o )
[0086]
[0087] h t =o t *tanh(c t )
[0088] Among them, f t 、i t , o t They represent the forget gate, input gate and output gate respectively, and c t Indicates the unit state, h t represents the hidden state, σ represents the sigmoid function, W and b represent the weight matrix and bias vector respectively. For example, when processing a piece of music with strong rhythm changes, LSTM can predict the subsequent equalization needs based on the previous rhythm pattern and make adjustments in advance.
[0089] In the calculation of the strategic value, the present invention adopts normalization processing, and the formula is:
[0090]
[0091] Among them, V(s t ,a t ) means in state s t Take action a t The value of aV(s t ,a) and max a V(s t , a) respectively represent the state s t This normalization process makes the policy values in different states comparable, thus better guiding the decision-making of the system.
[0092] In addition, the present invention also introduces a simulated listener feedback function:
[0093] F(x,θ)=w 1 *f 1 (x,θ)+w 2 *f 2 (x,θ)+…+w n *f n (x,θ)
[0094] Among them, x represents the input audio, θ represents the current EQ parameter, and f i (x,θ) represents the i-th evaluation index function, w i represents the weight of the i-th evaluation index, and ∑w i = 1. This function takes into account multiple evaluation indicators, such as sound clarity, bass fullness, treble brightness, etc. In practical applications, we can dynamically adjust the weights of each indicator according to the preferences of different users. i .
[0095] In order to balance exploration and exploitation, the present invention introduces the calculation formula of the strategy value score π(a|s):
[0096]
[0097] Where τ is a temperature parameter, which is used to control the degree of exploration of the strategy. The larger the τ, the more the strategy tends to be evenly distributed, and the smaller the τ, the more the strategy tends to be greedy. Where τ is a temperature parameter, which controls the degree of exploration of the strategy. We found that setting τ to a higher value (such as 1.0) at the beginning of training is conducive to the system's full exploration, while gradually reducing the value of τ (such as to 0.1) as training progresses can make the system more inclined to choose the known optimal strategy.
[0098] Finally, the present invention also takes into account the human auditory model and introduces a response function:
[0099]
[0100] Where, f represents the frequency, f 0 represents the characteristic frequency, f qrepresents the Q value frequency, k, α and γ are constant parameters used to adjust the shape and sensitivity of the curve. This function simulates the sensitivity of the human ear to sounds of different frequencies, making the system's equalization adjustment more in line with the perceptual characteristics of the human ear. For example, when processing high-frequency sounds, the system will appropriately reduce the gain according to this model to avoid harshness.
[0101] Through the organic combination of the above steps, the present invention realizes an intelligent and personalized headphone adaptive audio equalization method. This method can not only automatically adjust the EQ parameters according to different types of audio content, but also perform real-time optimization according to the user's personal preferences and listening environment. For example, when the user switches from pop music to classical music, the system can quickly adjust the EQ parameters to highlight the mid-frequency details of classical music; when the ambient noise increases, the system will appropriately increase the gain of certain frequency bands to ensure the clarity of key audio information.
[0102] In general, this invention successfully transforms the complex audio equalization problem into an intelligent system that can learn and optimize autonomously through deep reinforcement learning technology. This not only greatly improves the user's listening experience, but also opens up a new research direction in the field of audio processing. In the future, we can further optimize the algorithm and introduce more contextual information, such as the user's hearing health status, current emotional state, etc., to make the system's equalization effect more accurate and personalized.
[0103] In order to verify the superiority of the headphone adaptive audio equalization method based on deep reinforcement learning proposed in the present invention, we conducted a series of comparative experiments. In the embodiment, we adopted the complete solution of the present invention, including core technologies such as deep reinforcement learning, LSTM network, and improved A3 algorithm. In the comparative example, we used the traditional fixed parameter equalizer method.
[0104] Embodiment: In this embodiment, we selected a common headset on the market as the test object. We collected a variety of audio content including pop music, classical music, jazz, electronic music and audiobooks, totaling 500 samples, each sample lasting 3 minutes. We invited 50 volunteers of different ages and different music preferences to participate in the test. During the test, volunteers need to listen to these audio samples in different environmental noise levels (quiet environment, medium noise environment and high noise environment) and rate the sound quality.
[0105] Comparative Example: In the comparative example, we used the same headphones, but adopted the traditional fixed parameter equalizer method. This method uses preset equalizer parameters and simply switches according to the type of music, without considering personal preferences and environmental factors. The test process and volunteer group are the same as those in the embodiment.
[0106] Detection indicators and methods:
[0107] 1. Sound quality satisfaction (MOS): Using the Mean Opinion Score method, volunteers are asked to rate the sound quality on a scale of 1-5, with 5 being the highest.
[0108] 2. Equalizer response time: measures the time from when the audio type changes to when the equalizer parameters are adjusted, in milliseconds.
[0109] 3. Environmental adaptability: Test the system's performance in different noise environments. The scoring criteria are the same as the sound quality satisfaction.
[0110] 4. Personalization level: Evaluate the system's ability to adapt to different user preferences, using a 1-5 point scale.
[0111] 5. Battery Life: Measure the battery life of the headset in hours under continuous use.
[0112] The test results are as follows:
[0113] index Embodiment (the present invention) Comparative Example (Traditional Method) Sound quality satisfaction (MOS) 4.7 3.8 Balanced response time (ms) 50 500 Environmental adaptability 4.6 3.2 Degree of personalization 4.8 2.5 Battery life (h) 18 20
[0114] It can be clearly seen from the test results in the above table that the present invention is significantly superior to the traditional method in multiple key indicators. First, in terms of sound quality satisfaction, the method of the present invention obtained a high score of 4.7, while the traditional method only scored 3.8. This result fully demonstrates that the deep reinforcement learning method can better optimize the audio effect and provide a better listening experience.
[0115] Secondly, in terms of equalization response time, the method of the present invention can complete parameter adjustment in just 50 milliseconds, while the traditional method requires 500 milliseconds. This near real-time response speed makes it almost impossible for users to feel the adjustment process when switching between different types of audio content, greatly improving the smoothness of the user experience.
[0116] In terms of environmental adaptability, the method of the present invention scored 4.6, which is much higher than the 3.2 of the traditional method. This result verifies that our system can effectively adjust in real time according to the environmental noise level and maintain good sound quality performance in different usage scenarios.
[0117] The comparison of personalization degree further highlights the advantages of the present invention. The method of the present invention scored a high score of 4.8 on this indicator, while the traditional method only scored 2.5. This huge gap shows that our deep reinforcement learning method can capture and adapt to the preferences of different users very well and provide a truly personalized listening experience.
[0118] It is worth noting that in terms of battery life, the method of the present invention (18 hours) is slightly inferior to the traditional method (20 hours). This is because the operation of the deep learning algorithm does consume more computing resources. However, considering the significant improvement in sound quality and user experience, this slight reduction in battery life is a completely acceptable trade-off. Moreover, with the advancement of chip technology and further optimization of the algorithm, we are confident that this gap will be narrowed in future versions.
[0119] Based on the above test results, we believe that the current implementation has reached a very ideal balance point and can be regarded as the best implementation. In particular, the present invention has demonstrated excellent performance in several core indicators, including sound quality satisfaction, balanced response time, and personalization.
[0120] These test results strongly demonstrate the innovation and practical value of this invention. By applying deep reinforcement learning technology to the field of headphone audio equalization, we not only significantly improved the sound quality, but also achieved unprecedented personalization and environmental adaptability. This intelligent audio processing method provides users with a richer, more accurate and personalized listening experience, representing an important development direction of audio technology.
[0121] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A headphone adaptive audio equalization method based on deep reinforcement learning, characterized in that: The following steps are involved: (1) Construct a reinforcement learning environment and use the filter parameters in the headphone equalizer as the action a in reinforcement learning t , the error between the sound output by the sound source after the sound source is adjusted by the headphones and EQ parameters and the real undistorted sound is used as the immediate reward r of reinforcement learning t and the long-term reward R t ; (2) Synchronize the EQ parameter adjustment process, sound playback, and listener feedback process; (3) The EQ adjustment strategy learning process is divided into a training phase and an online adjustment phase; (4) In the training phase, user audio data and listener feedback are obtained, and the improved A3 algorithm is used to extract audio acoustic features and determine the audio type; (5) Introduce the long-term memory network LSTM to learn the action a and state s of reinforcement learning to obtain the neural network Q function; (6) EQ parameter adjustment is defined as a Markov decision process (MDP), and a set of EQ parameter adjustment strategies corresponding to each audio type is output; (7) In the online adjustment stage, the audio features during EQ adaptive adjustment are input into the trained neural network Q function to obtain the strategy value and normalize it; (8) Use the EQ parameter adjustment strategy to perform EQ adaptive filtering and adjust the parameters of the headphones.
2. The method according to claim 1, characterized in that The instant reward t The calculation formula is: r t =-MSE(y t ,y′ t ) Where MSE represents the mean square error function, y t represents the true undistorted sound at time t, y′ t Represents the sound after EQ adjustment at time t.
3. The method according to claim 2, characterized in that The calculation formula of the long-term reward R_t is: k from 0 to ∞ Among them, γ represents the discount factor, which ranges from [0,1] and is used to balance the importance of immediate rewards and future rewards.
4. The method according to claim 1, characterized in that The core formula of the improved A3 algorithm is: Where x represents the input audio feature vector, c i represents the i-th audio category, P(c i |x) means that given feature x, it belongs to category c i The posterior probability, P(x|c i ) indicates category c i The likelihood probability of , P(x) represents the prior probability of feature x.
5. The method according to claim 1, characterized in that The update formula of the neural network Q function is: Among them, α represents the learning rate, which ranges from (0,1] and is used to control the step size of Q value update; s t represents the state at time t, a t represents the action at time t, s t +1 indicates the state at time t+1.
6. The method according to claim 1, characterized in that The core update formula of the LSTM is: j t =v(W f ·[u t-1 ,x t ]+b f ) i t =σ(W i ·[h t-1 ,x t ]+b i ) the t =σ(W o ·[h t-1 ,x t ]+b o ) h t =o t *tanh(c t ) Among them, f t 、i t , o t They represent the forget gate, input gate and output gate respectively, and c t Indicates the unit state, h t represents the hidden state, σ represents the sigmoid function, W and b represent the weight matrix and bias vector respectively.
7. The method according to claim 1, characterized in that The normalization formula of the strategy value is: Among them, V(s t ,a t ) means in state s t Take action a t The value of a V(s t ,a) and max a V(s t , a) respectively represent the state s t The minimum and maximum value of all possible actions.
8. The method according to claim 1, characterized in that It also includes the use of a simulated listener feedback function, which is expressed as: F(x,θ)=w1*f1(x,θ)+w2*f2(x,θ)+…+w n *f n (x,θ) Among them, x represents the input audio, θ represents the current EQ parameter, and f i (x,θ) represents the i-th evaluation index function, w i represents the weight of the i-th evaluation index, and ∑w i =1.
9. The method according to claim 1, characterized in that It also includes calculating the strategy value score π(a|s), which is calculated as follows: Among them, τ represents the temperature parameter, which is used to control the degree of exploration of the strategy. The larger τ is, the more the strategy tends to be evenly distributed, and the smaller τ is, the more the strategy tends to be greedy.
10. The method according to claim 1, characterized in that It also includes the response function H(f) using the human auditory model, which is expressed as: Among them, f represents frequency, f0 represents characteristic frequency, and f q represents the Q value frequency, k, α, and γ are constant parameters used to adjust the shape and sensitivity of the curve.