An audio steganography method for vehicle networking security communication based on reinforcement learning

Abstract

This invention discloses an audio steganography method for secure communication in vehicle-to-everything (V2X) networks based on reinforcement learning. The method involves windowing and framing the in-vehicle audio signal and performing a short-time Fourier transform to construct time-domain and frequency-domain state vectors. A time-domain agent decides whether to embed information in the current frame. If embedding is selected, a frequency-domain agent is activated to determine the frequency band, embedding strength, and encoding method, and encoded ciphertext noise is superimposed on the amplitude spectrum. By combining multi-dimensional reward functions considering capacity, robustness, concealment, latency, and detection risk, the agent's policy convergence is driven. This invention enables secure, concealed transmission with high capacity, low latency, and high robustness while ensuring audio perception quality and real-time communication. It is suitable for concealed audio carrier communication in vehicle-to-vehicle and vehicle-to-infrastructure (V2I) scenarios.

Description

Technical Field

[0001] This invention relates to the field of modern communication network security technology, and in particular to an audio steganography method based on reinforcement learning for secure communication in the Internet of Vehicles. Background Technology

[0002] In the connected vehicle environment, the network environment is characterized by rapid changes. When large-capacity, covert information transmission is required, traditional covert communication technologies have high computational and communication costs and slow response speeds, making it difficult to meet the needs of connected vehicles for efficient and secure communication.

[0003] In the field of vehicle-to-everything (V2X) communication, audio, as an important information carrier, is not only used in human-machine interaction and entertainment systems, but is also beginning to play a unique role in communication between vehicles and between vehicles and infrastructure. However, with the continuous development of V2X, ensuring the security and anonymity of the data transmitted via audio has become particularly critical. Currently, existing methods have not yet enabled the application of audio steganography technology in V2X communication scenarios. Summary of the Invention

[0004] This invention provides an audio steganography method for secure communication in vehicle-to-everything (V2X) networks based on reinforcement learning. Using audio as a carrier, it employs a heterogeneous multi-agent reinforcement learning algorithm to optimize audio steganography technology, enabling dynamic adjustment of information embedding strategies, thereby achieving high-capacity, low-latency, secure transmission of audio information in V2X scenarios.

[0005] This invention provides an audio steganography method based on reinforcement learning for secure communication in vehicle-to-everything (V2X) networks, comprising the following steps: Step 1: Obtain the vehicle network system. Set up a vehicle network system containing P vehicles and Q base stations, and conduct covert communication through audio signals; Step 2, for the first The audio signals collected from the vehicles are preprocessed, and the first... The continuous analog signals collected from the vehicles are converted from analog to digital to obtain discrete-time series, and the discrete-time series is then windowed and framed to obtain the first... Signal for adding windows to vehicles; Step 3: Perform Discrete Fourier Transform on each windowed signal frame, decompose the complex spectrum vector output by the Discrete Fourier Transform into amplitude spectrum and phase spectrum, and take the positive frequency part. Step 4, in the vehicle The temporal agent is deployed at the edge to perform embedding decisions, and the frequency agent is deployed to determine the embedding parameters. The parameters of the temporal agent policy network are initialized, the parameters of the frequency agent policy network are initialized, and the learning rate and discount factor are initialized. Step 5, in the vehicle The time-domain state vector of the time-domain agent and the frequency-domain state vector of the frequency-domain agent are constructed at the end. Step 6, vehicle The temporal agent selects whether to embed based on the temporal state vector; Step 7, vehicle The frequency domain agent is activated when the time domain agent selects embedding, and information is embedded according to the actions of the frequency domain agent. Step 8, vehicle Calculate the rewards for the time-domain agent and the frequency-domain agent separately; Step 9: After each time slot, the state, action, reward, and next state of the time-domain agent and the frequency-domain agent are combined into an experience tuple and stored in the experience pool. During training, a batch of experiences is randomly sampled, and the Q network parameters of the time-domain agent and the frequency-domain agent are updated using the temporal difference error, respectively. The Adam optimizer is used to update the parameters. Step 10: Repeat steps 6 to 9 until the time-domain agent and frequency-domain agent policies converge, the reward function stabilizes, and the vehicle-to-everything (V2X) system adaptively selects the optimal embedding strategy based on the audio content features and the vehicle communication network status.

[0006] Optionally, in one embodiment of the present invention, in step 2, let the first The audio signals collected by the vehicles are continuous analog signals. Each audio segment is [length missing] Seconds, corresponding number of sampling points , For the sampling rate, the discrete-time series Windowing is performed for frame division, using a Hamming window function, denoted as... , No. The windowing signal for the frame is: ; in, This indicates element-wise multiplication.

[0007] Optionally, in one embodiment of the present invention, step 3 includes: Perform a Discrete Fourier Transform on each windowed signal frame: ; in, This represents the Discrete Fourier Transform operator. For the first The first vehicle Frame windowing signal, output Let be a complex frequency spectrum vector, corresponding to the frequency of... , Indicates the number of sampling points. For the length of the window, Sampling rate; The output complex spectrum vector Decomposed into amplitude spectrum and phase spectrum After taking the positive frequency portion, ,common There are points, with the shape of... , .

[0008] Optionally, in one embodiment of the present invention, in step 5, when the first The first signal for adding windows to a vehicle Frame, calculate the energy of the current frame. Frame energy is defined as: ; in, For the length of the window, For the first The first vehicle Frame windowing signal; Calculate the average gradient of the current frame Defined as: ; Where n is the sampling index at discrete time points, For vehicles In the The first frame of the windowed signal The amplitude of each sampling point; Get the current frame index and embedding the start frame index And record the current embedding progress. Construct the state vector of the time-domain intelligent agent for: ; The frequency-domain agent is responsible for determining the embedding parameters and constructing the state vector at the frequency domain level. for: ; in, This is the amplitude spectrum.

[0009] Optionally, in one embodiment of the present invention, in step 6, the current time-domain state vector is... The input is fed into a pre-trained temporal agent policy network, and the actions of the temporal agent are... Defined as a discrete binary set ,in, Indicates "skip" Indicating "embed", the temporal agent policy network outputs the probability distribution of selecting "embed" in the current state: ; in, For feature extractors, For the parameters of the temporal intelligent agent policy network, The function is sigmoid; the probability distribution of the temporal agent policy network output choosing "skip" in the current state is as follows: ; Sampling is performed based on the probability distribution to obtain the final action. This determines whether to execute the embedded decision.

[0010] Optionally, in one embodiment of the invention, in step 7, the vehicle Frequency domain intelligent agents in When activated, information is embedded based on the actions of the frequency domain agent. The action space of the frequency domain agent is: ; in, These are the normalized start and stop frequencies of the embedding band. For embedding strength coefficient, For encoding schemes; First, the secret message is encoded into a frequency domain noise signal. The coding scheme is based on the action. Select, then modify the amplitude spectrum within the selected frequency band. Internally, according to the embedding strength coefficient The frequency domain noise signal of the secret message is superimposed onto the amplitude spectrum: ; in, For the first The amplitude spectrum of the original embedding at frame time, This is a normalization factor used to control the maximum intensity of the noise; Finally, time-domain reconstruction is performed, combined with the original phase spectrum. and the modified amplitude spectrum Perform inverse Fourier transform to reconstruct time-domain dense audio frames .

[0011] Optionally, in one embodiment of the present invention, in step 8, a temporal agent reward function is defined. for: ; in, The capacity reward is positively correlated with the number of embeddable bits; The robust reward for the temporal agent is positively correlated with signal similarity. As a delay penalty, it is applied when a frame is skipped; These are weight parameters; Frequency domain agent reward function for: ; in, The reward is concealed and is positively correlated with the signal-to-noise ratio; The robustness reward for the temporal agent is positively correlated with decoding similarity. To detect risk penalties, the energy distribution difference between the original and dense audio is measured based on JS divergence. These are the weight parameters.

[0012] This invention presents an audio steganography method based on reinforcement learning for secure communication in the Internet of Vehicles (IoV). Addressing the problems of high computational overhead, slow response, and rigid embedding strategies in covert audio communication within IoVs, this invention constructs a heterogeneous multi-agent reinforcement learning framework. It deploys temporal and frequency-domain agents for collaborative decision-making, achieving dynamic adaptive optimization of the embedding location, embedding strength, encoding scheme, and noise parameters of secret information. Specifically, it includes: windowing and framing the in-vehicle audio signal and performing short-time Fourier transform to construct temporal and frequency-domain state vectors; the temporal agent decides whether to embed information in the current frame; if embedding is selected, the frequency-domain agent is activated to determine the frequency band, embedding strength, and encoding method, and coded ciphertext noise is superimposed on the amplitude spectrum; by combining multi-dimensional reward functions considering capacity, robustness, concealment, latency, and detection risk, the agent's strategy converges. This invention can achieve high-capacity, low-latency, and highly robust secure covert transmission while ensuring audio perception quality and real-time communication, and is suitable for covert audio carrier communication in vehicle-to-vehicle and vehicle-to-infrastructure (V2I) scenarios.

[0013] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an audio steganography method for secure communication in vehicle-to-everything (V2X) networks based on reinforcement learning, according to an embodiment of the present invention. Figure 2 This is a flowchart of an audio steganography method based on reinforcement learning for secure communication in vehicle-to-everything (V2X) networks. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0016] Figure 1 This is a flowchart illustrating an audio steganography method for secure communication in vehicle-to-everything (V2X) networks based on reinforcement learning, according to an embodiment of the present invention.

[0017] like Figure 1 As shown, this reinforcement learning-based audio steganography method for secure communication in vehicle-to-everything (V2X) includes the following steps: Step 1: Obtain the vehicle network system. Set up a vehicle network system containing P vehicles and Q base stations, and conduct covert communication through audio signals; Step 2, for the first The audio signals collected from the vehicles are preprocessed, and the first... Continuous analog signals collected by vehicles The discrete time series is obtained after analog-to-digital conversion. and for discrete time series Perform windowing and frame splitting to obtain the first frame. Signal for adding windows to vehicles; Step 3: Perform a Discrete Fourier Transform (DFT) on each windowed signal frame, and output the complex spectrum vector after the DFT. Decomposed into amplitude spectrum and phase spectrum Take the positive frequency portion; Step 4, in the vehicle A temporal agent is deployed at the endpoint to perform embedding decisions, and a frequency-domain agent is deployed to determine embedding parameters. The parameters of the temporal agent policy network are initialized. Initialize the parameters of the frequency domain agent policy network. and initialize the learning rate. and discount factor ; Step 5, in the vehicle Constructing the temporal state vector of a temporal agent at the edge and the frequency domain state vector of the frequency domain agent ; Step 6, vehicle The temporal agent selects whether to embed based on the temporal state vector; Step 7, vehicle The frequency domain agent is activated when the time domain agent selects embedding, and information is embedded according to the actions of the frequency domain agent. Step 8, vehicle Calculate the reward of the time-domain agent separately Rewards for frequency domain agents ; Step 9: After each time slot, combine the state, action, reward, and next state of the time-domain agent and the frequency-domain agent into an experience tuple. The experience is stored in the experience pool D. During training, a batch of experience is randomly sampled, and the Q network parameters of the time-domain agent and the frequency-domain agent are updated using the temporal difference (TD) error, respectively. The Adam optimizer is used to update the parameters. Step 10: Repeat steps 6 to 9 until the time-domain agent and frequency-domain agent policies converge, the reward function stabilizes, and the vehicle-to-everything (V2X) system adaptively selects the optimal embedding strategy based on the audio content features and the vehicle communication network status.

[0018] Optionally, in one embodiment of the present invention, in step 2, let the first The audio signals collected by the vehicles are continuous analog signals. Each audio segment is [length missing] Seconds, corresponding number of sampling points , For the sampling rate, the discrete-time series Windowing is performed for frame division, using a Hamming window function, denoted as... , No. The windowing signal for the frame is: ; in, This indicates element-wise multiplication.

[0019] Optionally, in one embodiment of the present invention, step 3 includes: Perform a Discrete Fourier Transform on each windowed signal frame: ; in, This represents the Discrete Fourier Transform operator. For the first The first vehicle Frame windowing signal, output Let be a complex frequency spectrum vector, corresponding to the frequency of... , Indicates the number of sampling points. For the length of the window, Sampling rate; The output complex spectrum vector Decomposed into amplitude spectrum and phase spectrum After taking the positive frequency portion, ,common There are points, with the shape of... , .

[0020] Optionally, in one embodiment of the present invention, in step 5, when the first The first signal for adding windows to a vehicle Frame, calculate the energy of the current frame. Frame energy is defined as: ; in, For the length of the window, For the first The first vehicle Frame windowing signal; Calculate the average gradient of the current frame Defined as: ; Where n is the sampling index at discrete time points, For vehicles In the The first frame of the windowed signal The amplitude of each sampling point; Get the current frame index and embedding the start frame index And record the current embedding progress. Construct the state vector of the time-domain intelligent agent for: ; The frequency-domain agent is responsible for determining the embedding parameters and constructing the state vector at the frequency domain level. for: ; in, This is the amplitude spectrum.

[0021] Optionally, in one embodiment of the present invention, in step 6, the current time-domain state vector is... The input is fed into a pre-trained temporal agent policy network, and the actions of the temporal agent are... Defined as a discrete binary set ,in, Indicates "skip" Indicating "embed", the temporal agent policy network outputs the probability distribution of selecting "embed" in the current state: ; in, For feature extractors, For the parameters of the temporal intelligent agent policy network, The function is sigmoid; the probability distribution of the temporal agent policy network output choosing "skip" in the current state is as follows: ; Sampling is performed based on the probability distribution to obtain the final action. This determines whether to execute the embedded decision.

[0022] Optionally, in one embodiment of the invention, in step 7, the vehicle Frequency domain intelligent agents in When activated, information is embedded based on the actions of the frequency domain agent. The action space of the frequency domain agent is: ; in, These are the normalized start and stop frequencies of the embedding band. For embedding strength coefficient, For encoding schemes; First, the secret message is encoded into a frequency domain noise signal. The coding scheme is based on the action. Select, then modify the amplitude spectrum within the selected frequency band. Internally, according to the embedding strength coefficient The frequency domain noise signal of the secret message is superimposed onto the amplitude spectrum: ; in, For the first The amplitude spectrum of the original embedding at frame time, This is a normalization factor used to control the maximum intensity of the noise; Finally, time-domain reconstruction is performed, combined with the original phase spectrum. and the modified amplitude spectrum Perform inverse Fourier transform to reconstruct time-domain dense audio frames .

[0023] Optionally, in one embodiment of the present invention, in step 8, a temporal agent reward function is defined. for: ; in, The capacity reward is positively correlated with the number of embeddable bits; The robust reward for the temporal agent is positively correlated with signal similarity. As a delay penalty, it is applied when a frame is skipped; These are weight parameters; Frequency domain agent reward function for: ; in, The reward is concealed and is positively correlated with the signal-to-noise ratio (SNR). The robustness reward for the temporal agent is positively correlated with decoding similarity. To detect risk penalties, the energy distribution difference between the original and dense audio is measured based on JS divergence. These are the weight parameters.

[0024] Combination Figure 2 As shown, in order to better understand the technical content of the present invention, the following embodiments are provided in detail.

[0025] The audio steganography method based on reinforcement learning for secure communication in vehicle-to-everything (V2X) networks includes the following steps: Step 1: The vehicle-to-everything (V2X) system consists of two vehicles and one base station, which communicate covertly via audio signals.

[0026] Step 2: Vehicle 1 collects audio signals; the original audio is a continuous analog signal. Discrete-time series are obtained through analog-to-digital conversion sampling. Sampling rate Set to 16 kHz. Set the processing time for each audio segment. Seconds, corresponding number of sampling points .right Perform windowed frame division, window length Frame shift The window function uses the Hamming window, denoted as . . No. The windowing signal for the frame is: .

[0027] Step 3: Perform DFT on each windowed signal frame to obtain the complex spectrum. And decomposed into amplitude spectrum and phase spectrum Take the positive frequency portion. The spectral shape is .

[0028] Step 4: Deploy a temporal agent and a frequency-domain agent in vehicle 1, respectively for performing embedding decisions and determining embedding parameters. Both the temporal and frequency-domain agents use fully connected neural networks as the backbone of a Q-network, with input layer dimensions of 5 (temporal domain) and 130 (frequency domain), two hidden layers of 64 neurons each, and output layers of 2 (temporal domain actions) and 4 (frequency domain actions), respectively. The parameters for initializing the temporal agent policy network are as follows: The parameters of the frequency domain agent policy network are: The learning rate is set to 0.001, and the discount factor is... .

[0029] Step 5: The specific process of constructing the two-state vector is as follows: In the first... Frame, calculate the energy of the current frame. Frame energy is defined as: ; Calculate the average gradient of the current frame Defined as: ; Get the index of the current processing frame and embedding the start frame index And record the current embedding progress. Based on the above information, the state vector of the time-domain agent is constructed as follows: ; The frequency-domain agent is responsible for determining the embedding parameters at the frequency domain level and constructing its state vector as follows: .

[0030] Step 6: The time-domain agent of vehicle 1 first determines the time-domain state. To select whether to embed, first, the current time-domain state vector is... The input is fed into its pre-trained temporal agent policy network. The actions of the temporal agent are... Defined as a discrete binary set ,in, Indicates "skip" Indicates "embed", the network outputs the probability distribution of choosing "embed" in the current state: ; in For feature extractors, For network parameters, This is the sigmoid function. The system then samples according to this probability distribution to obtain the final action. .

[0031] Step 7: If Then, information embedding is performed based on the actions of the frequency-domain agent. First, the secret message is encoded into a frequency-domain noise signal. The coding scheme is based on the action. Select. Then modify the amplitude spectrum within the selected frequency band. Internally, according to the strength coefficient Add message noise to the amplitude spectrum: ; in, The normalization factor is set to 0.1 to control the maximum noise intensity. Finally, time-domain reconstruction is performed, combined with the original phase spectrum. and the modified amplitude spectrum Perform inverse short-time Fourier transform to reconstruct time-domain dense audio frames .

[0032] Step 8: Temporal agent reward: ; in, The capacity reward is positively correlated with the number of embeddable bits; Robust rewards are positively correlated with signal similarity. As a latency penalty, it is applied when skipping frames; the weight is set to... ; Frequency domain agent rewards: ; in, The reward is concealed and is positively correlated with the signal-to-noise ratio (SNR). The reward is robust and positively correlated with decoding similarity. To detect risk penalties, JS divergence is used; the weight is set to... .

[0033] Step 9: After each time slot, combine the states, actions, rewards, and next states of the two agents into an experience tuple. Store in experience pool During training, a batch of experience samples is randomly sampled, and the network parameters are updated using the TD error and the Adam optimizer. The learning rate is 0.001.

[0034] Step 10: Repeat steps 6-9, processing the entire audio frame by frame and continuously updating the agent's policy. Set the total number of training rounds to 1000, and evaluate whether the average reward tends to stabilize after each round. When the reward function converges and the fluctuation is less than a preset threshold (e.g., 0.01), the system is considered to have learned a stable embedding policy.

[0035] The audio steganography method for secure communication in vehicle-to-everything (V2X) networks proposed in this embodiment of the invention utilizes reinforcement learning to optimize existing audio steganography techniques. This allows the method to adaptively select the optimal embedding position, embedding strength, encoding scheme, and noise parameters based on the current network status and audio content characteristics. As a result, it can be applied to V2X communication environments and achieve high-capacity, covert information transmission.

[0036] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0037] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0038] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

Claims

1. An audio steganography method based on reinforcement learning for secure communication in vehicle-to-everything (V2X) networks, characterized in that, Includes the following steps: Step 1: Obtain the vehicle network system. Set up a vehicle network system containing P vehicles and Q base stations, and conduct covert communication through audio signals; Step 2, for the first The audio signals collected from the vehicles are preprocessed, and the first... The continuous analog signals collected from the vehicles are converted from analog to digital to obtain discrete-time series, and the discrete-time series is then windowed and framed to obtain the first... Signal for adding windows to vehicles; Step 3: Perform Discrete Fourier Transform on each windowed signal frame, decompose the complex spectrum vector output by the Discrete Fourier Transform into amplitude spectrum and phase spectrum, and take the positive frequency part. Step 4, in the vehicle The temporal agent is deployed at the edge to perform embedding decisions, and the frequency agent is deployed to determine the embedding parameters. The parameters of the temporal agent policy network are initialized, the parameters of the frequency agent policy network are initialized, and the learning rate and discount factor are initialized. Step 5, in the vehicle The time-domain state vector of the time-domain agent and the frequency-domain state vector of the frequency-domain agent are constructed at the end. Step 6, vehicle The temporal agent selects whether to embed based on the temporal state vector; Step 7, vehicle The frequency domain agent is activated when the time domain agent selects embedding, and information is embedded according to the actions of the frequency domain agent. Step 8, vehicle Calculate the rewards for the time-domain agent and the frequency-domain agent separately; Step 9: After each time slot, the state, action, reward, and next state of the time-domain agent and the frequency-domain agent are combined into an experience tuple and stored in the experience pool. During training, a batch of experiences is randomly sampled, and the Q network parameters of the time-domain agent and the frequency-domain agent are updated using the temporal difference error, respectively. The Adam optimizer is used to update the parameters. Step 10: Repeat steps 6 to 9 until the time-domain agent and frequency-domain agent policies converge, the reward function stabilizes, and the vehicle-to-everything (V2X) system adaptively selects the optimal embedding strategy based on the audio content features and the vehicle communication network status.

2. The method according to claim 1, characterized in that, In step 2, let the first The audio signals collected by the vehicles are continuous analog signals. Each audio segment is [length missing] Seconds, corresponding number of sampling points , For the sampling rate, the discrete-time series Windowing is performed for frame division, using a Hamming window function, denoted as... , No. The windowing signal for the frame is: ； in, This indicates element-wise multiplication.

3. The method according to claim 1, characterized in that, Step 3 includes: Perform a Discrete Fourier Transform on each windowed signal frame: ； in, This represents the Discrete Fourier Transform operator. For the first The first vehicle Frame windowing signal, output Let be a complex frequency spectrum vector, corresponding to the frequency of... , Indicates the number of sampling points. For the length of the window, Sampling rate; The output complex spectrum vector Decomposed into amplitude spectrum and phase spectrum After taking the positive frequency portion, ,common There are points, with the shape of... , .

4. The method according to claim 1, characterized in that, In step 5, at the The first signal for adding windows to a vehicle Frame, calculate the energy of the current frame. Frame energy is defined as: ； in, For the length of the window, For the first The first vehicle Frame windowing signal; Calculate the average gradient of the current frame Defined as: ； Where n is the sampling index at discrete time points. For vehicles In the The first frame windowed signal The amplitude of each sampling point; Get the current frame index and embedding the start frame index And record the current embedding progress. Construct the state vector of the time-domain intelligent agent for: ； The frequency-domain agent is responsible for determining the embedding parameters and constructing the state vector at the frequency domain level. for: ； in, This is the amplitude spectrum.

5. The method according to claim 1, characterized in that, In step 6, the current time-domain state vector is... The input is fed into a pre-trained temporal agent policy network, and the actions of the temporal agent are... Defined as a discrete binary set ,in, Indicates "skip". Indicating "embed", the temporal agent policy network outputs the probability distribution of selecting "embed" in the current state: ； in, For feature extractors, For the parameters of the temporal intelligent agent policy network, The function is sigmoid; the probability distribution of the temporal agent policy network output choosing "skip" in the current state is: ； Sampling is performed based on the probability distribution to obtain the final action. This determines whether to execute the embedded decision.

6. The method according to claim 5, characterized in that, In step 7, the vehicle Frequency domain intelligent agents in When activated, information is embedded based on the actions of the frequency domain agent. The action space of the frequency domain agent is: ； in, These are the normalized start and stop frequencies of the embedding band. For embedding strength coefficient, For encoding schemes; First, the secret message is encoded into a frequency domain noise signal. The coding scheme is based on the action. Select, then modify the amplitude spectrum within the selected frequency band. Internally, according to the embedding strength coefficient The frequency domain noise signal of the secret message is superimposed onto the amplitude spectrum: ； in, For the first The amplitude spectrum of the original embedding at frame time, This is a normalization factor used to control the maximum intensity of the noise; Finally, time-domain reconstruction is performed, combined with the original phase spectrum. and the modified amplitude spectrum Perform inverse Fourier transform to reconstruct time-domain dense audio frames .

7. The method according to claim 1, characterized in that, In step 8, the temporal agent reward function is defined. for: ； in, The capacity reward is positively correlated with the number of embeddable bits; The robust reward for the temporal agent is positively correlated with signal similarity. As a delay penalty, it is applied when a frame is skipped; These are weight parameters; Frequency domain agent reward function for: ； in, The reward is concealed and is positively correlated with the signal-to-noise ratio; The robustness reward for the temporal agent is positively correlated with decoding similarity. To detect risk penalties, the energy distribution difference between the original and dense audio is measured based on JS divergence. These are the weight parameters.

Images