A cognitive interference integrated beamforming method and electronic device based on deep reinforcement learning

Abstract

The present invention discloses a cognitive interference-integrated beamforming method based on deep reinforcement learning, comprising: establishing an optimization problem with the goal of maximizing the sum of communication rates of legal receivers; mapping the optimization problem into a Markov decision process, using a deep reinforcement learning network model to predict the beamforming matrix of a legal transceiver, using training samples as input to train the deep reinforcement learning network model, and repeating the iterative training process until the deep reinforcement learning network model converges; collecting state information of the communication environment and inputting it into the trained deep reinforcement learning model to obtain an optimal decision strategy, wherein the decision strategy is an interference-integrated beamforming matrix; the present invention can enable the legal transceiver to communicate with L legal receivers and interfere with a non-cooperative adversarial network at the same time when the channel state information is unknown, thereby improving the reliability of communication, the accuracy of interference, and the cognitive level of interference-integrated through training.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a cognitive communication-interference integrated beamforming method. Background Art

[0002] Communication and interference are two important elements in the electromagnetic field and have been widely studied in academia and industry. On the one hand, communication is generally to ensure the effectiveness and reliability of information transmission, while interference is generally to reduce the quality of the opponent's information transmission, so the two are considered contradictory. On the other hand, due to the similarity of the two in signal propagation and the fact that they can be converted into each other under certain conditions, the two are also unified.

[0003] According to the above analysis, a new concept can be introduced, namely, the integration of communication and interference. In general, the integration of communication and interference can achieve simultaneous communication and interference in one system by sharing frequency, power, data, and software and hardware, with the advantages of high spectrum / energy / data efficiency and low hardware cost. However, there are still some shortcomings in the integrated communication and interference system, such as the need to obtain prior information of users / targets before making decisions, which can be achieved for communication partners, but it is almost impossible to achieve interference with non-cooperative targets. In addition, in a dynamic unknown environment, the user / target state changes quickly, and the decision cycle of the integrated communication and interference system is long, the delay is large, and the reliability is low. Therefore, how to achieve real-time, accurate, and reliable communication and interference in a dynamic unknown environment is an interesting and challenging problem. Summary of the invention

[0004] The present invention provides a cognitive communication and interference integrated beamforming method based on deep reinforcement learning, which has the characteristics of dynamic adjustment strategy, shorter decision-making time, and more accurate and reliable communication and interference.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The first aspect of the present invention provides a cognitive interference integrated beamforming method based on deep reinforcement learning, comprising:

[0007] Collecting state information of the communication environment and inputting it into the trained deep reinforcement learning model to obtain an optimal decision strategy, wherein the decision strategy is a beamforming matrix integrating communication and interference;

[0008] The construction and training process of the deep reinforcement learning model includes:

[0009] Build a communication and interference integrated model based on the communication simulation environment; add the interference constraints of the suspicious communication group in the non-cooperative adversarial network and the maximum transmission power constraints of the legitimate transceiver, and establish an optimization problem with the goal of maximizing the sum of the communication rates of the legitimate receivers;

[0010] The optimization problem is mapped into a Markov decision process, and a deep reinforcement learning network model is constructed with the legal transceiver as the intelligent agent. The deep reinforcement learning network model is used to predict the beamforming matrix of the legal transceiver. Training samples are obtained by interacting between the legal transceiver and the communication simulation environment. The training samples are used as input to train the deep reinforcement learning network model. The iterative training process is repeated until the deep reinforcement learning network model converges.

[0011] Furthermore, the communication and interference integrated model is configured with a single legal transceiver, L legal receivers and a non-cooperative adversarial network; the legal transceiver is equipped with N lt Antennas distributed in linear arrays; K radiation source nodes are set in the non-cooperative adversarial network; in a state of unknown channel state information, the legal transceiver communicates with L legal receivers and interferes with the non-cooperative adversarial network at the same time.

[0012] Furthermore, adding the interference constraint of the suspicious communication group in the non-cooperative adversarial network and the maximum transmission power constraint of the legitimate transceiver, the method of establishing the optimization problem with the goal of maximizing the sum of the communication rates of the legitimate receivers includes:

[0013]

[0014] St C1:

[0015] C2:

[0016] In the formula, R l,l is the communication rate of the lth legal receiver; W is the communication beamforming matrix transmitted by the legal transceiver; V is the interference beamforming matrix transmitted by the legal transceiver; γ a,k It is represented by the signal-to-noise ratio of the kth radiation source node in the non-cooperative adversarial network; φ represents the minimum signal-to-noise ratio at which the kth radiation source node can successfully demodulate the message in the non-cooperative adversarial network; P max is the maximum transmission power of a legal transmitter; represents the conjugate transpose of a vector or matrix; tr(·) represents the trace of a matrix.

[0017] Furthermore, the communication beamforming matrix transmitted by the legal transceiver is expressed as:

[0018]

[0019]

[0020]

[0021] In the formula, w l represents the communication beamforming vector transmitted by the legal transceiver; h represents the number of beams transmitted by the legal transceiver; It represents the transmission power allocated by the legal transceiver to the lth legal user; represents the beam direction from the legal transceiver to the lth legal user; d0 represents the distance between antennas in the legal transceiver; θ l It is represented as the pointing angle from the legal transceiver to the lth legal user; j′ is represented as the imaginary part.

[0022] Furthermore, the interference beamforming matrix transmitted by the legitimate transceiver is expressed as:

[0023]

[0024]

[0025]

[0026] In the formula, v k represents the interference beamforming vector transmitted by the legitimate transceiver; h represents the number of beams transmitted by the legitimate transceiver; It is represented by the transmission power allocated by the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; d0 is represented as the distance between antennas in the legitimate transceiver; θ k It is represented as the pointing angle from the legitimate transceiver to the kth radiating source node in the non-cooperative adversarial network; j′ is represented as the imaginary part.

[0027] Furthermore, the expression formula of the communication rate of the lth legal receiver is:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] In the formula, γ l,l Expressed as the signal-to-interference-noise ratio of the lth legal receiver; It represents the transmission power allocated by the legal transceiver to the nth legal user; h ll,l It is expressed as the channel gain between the legal transceiver and the lth legal receiver; It is represented as the beam direction from the legal transceiver to the nth legal user; d ll，l It is expressed as the distance between the legal transceiver and the lth legal receiver; It is represented by the transmission power allocated by the legitimate transceiver to the i-th radiator node in the non-cooperative adversarial network; represents the beam direction from the legitimate transceiver to the i-th radiation source node in the non-cooperative adversarial network; N0 represents the variance of the circularly symmetric complex Gaussian noise; α represents the path loss coefficient; K ll，l It is expressed as Rice factor; It is represented as the fast fading part between the legal transceiver and the lth legal receiver; Represented as the direct path component between the legal transceiver and the lth legal receiver; β ll，l It is expressed as the line-of-sight angle between the legal transceiver and the lth legal receiver.

[0034] Furthermore, the expression formula of the signal to noise ratio of the kth radiation source node in the non-cooperative adversarial network is:

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] In the formula, It represents the transmission power allocated by the legal transceiver to the mth legal user; h ls,k It is expressed as the channel gain between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legal transceiver to the mth legal user; g aa,k It is represented as the channel gain between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; p s It is represented by the transmission power of the radiation source node in the non-cooperative adversarial network; It is represented by the transmission power allocated by the legitimate transceiver to the i-th radiator node in the non-cooperative adversarial network; represents the beam direction from the legitimate transceiver to the i-th radiating source node in the non-cooperative adversarial network; N0 represents the variance of the circularly symmetric complex Gaussian noise; α represents the path loss coefficient; dls，k K is the distance between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; ls,k and K aa，k It is expressed as Rice factor; It is represented as the fast fading part between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the direct path component between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; β ls，k It is represented as the line-of-sight angle between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; d aa，k It is represented as the distance between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented as the fast fading part between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented as the direct path component between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network.

[0043] Further, methods for mapping the optimization problem into a Markov decision process include:

[0044] The Markov decision process includes a state set S, an action set A, a transfer function P and a reward function R;

[0045] The state set S contains the beamforming matrices of communication and interference, that is, S = {s:s t =[W t ,V t ]}, where W t It is represented as the communication beamforming matrix at time t, V t The interference beamforming matrix at time t is expressed as: and a set of pointing angles Θ = {0, 2π / C, ..., 2π-2π / C}; , where B and C represent the discrete numbers of transmit power and pointing angles, respectively;

[0046] The action set A is defined as the set of transmit power and the pointing angle set Θ = {0, 2π / C, ..., 2π-2π / C} are traversed element by element; the next state S is determined by the current action A through the transfer function P;

[0047] The reward function R is expressed as:

[0048]

[0049] Among them, a′ and b′ are the communication weight coefficient and interference weight coefficient respectively, and denote the communication reward and interference reward at time t respectively; r t Represented as the reward value at time t.

[0050] Furthermore, the method of using the training samples as input to the deep reinforcement learning network model for training, and repeating the iterative training process until the deep reinforcement learning network model converges includes:

[0051] The training samples are used as input to the deep reinforcement learning network model for training. The formula for calculating the training loss value is:

[0052]

[0053] In the formula, r t+1 Represented as the selected action a at time t t The reward value obtained after t Represented as the state of the deep reinforcement learning network model at time t; a t Represented as the deep reinforcement learning network model action at time t; θ - Represents the parameters of the target network in the deep reinforcement learning network model; is represented as the value function of the target network in the deep reinforcement learning network model; θ is represented as the parameter of the evaluation network in the deep reinforcement learning network model; Q(s t ,a t ; θ) represents the value function of the evaluation network in the deep reinforcement learning network model;

[0054] The parameters of the deep reinforcement learning network model are optimized based on the training loss value, and the iterative training process is repeated until the deep reinforcement learning network model converges.

[0055] A second aspect of the present invention provides an electronic device comprising a storage medium and a processor; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the method described in the first aspect of the present invention.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention collects state information of the communication environment and inputs it into a trained deep reinforcement learning model to obtain an optimal decision-making strategy, wherein the decision-making strategy is a beamforming matrix that integrates communication and interference. The present invention is applicable to the simultaneous communication and interference problems in dynamic confrontation scenarios and has strong adaptability.

[0058] The present invention establishes an optimization problem with the goal of maximizing the sum of the communication rates of legal receivers; maps the optimization problem into a Markov decision process, constructs a deep reinforcement learning network model with legal transceivers as intelligent agents, uses the deep reinforcement learning network model to predict the beamforming matrix of the legal transceiver, and uses the training samples as input to train the deep reinforcement learning network model. Compared with traditional separate communication or interference systems, the present invention can enable the legal transceiver to communicate with L legal receivers and simultaneously interfere with a non-cooperative adversarial network under the state of unknown channel state information, thereby improving the reliability of communication and the accuracy of interference through training. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flow chart of the cognitive interference integrated beamforming method based on deep reinforcement learning of the present invention;

[0060] Figure 2 A reward comparison diagram of the deep reinforcement learning network model training process in an embodiment of the present invention;

[0061] Figure 3 A comparison chart of the communication rates of legitimate receivers corresponding to different interference algorithms provided by the present invention;

[0062] Figure 4 A comparison chart of the communication rates of legal receivers corresponding to different numbers of antennas provided by the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0064] Example 1

[0065] like Figure 1 As shown, this implementation provides a cognitive interference integrated beamforming method based on deep reinforcement learning, including:

[0066] A communication and interference integrated model is constructed based on a communication simulation environment; the communication and interference integrated model is configured with a single legal transceiver, L legal receivers and a non-cooperative adversarial network; the legal transceiver is equipped with N lt Antennas distributed in linear arrays; K radiation source nodes are set in the non-cooperative adversarial network; in a state of unknown channel state information, the legal transceiver communicates with L legal receivers and interferes with the non-cooperative adversarial network at the same time.

[0067] The communication beamforming matrix transmitted by the legal transceiver is expressed as:

[0068]

[0069]

[0070]

[0071] In the formula, w l represents the communication beamforming vector transmitted by the legal transceiver; h represents the number of beams transmitted by the legal transceiver; It represents the transmission power allocated by the legal transceiver to the lth legal user; represents the beam direction from the legal transceiver to the lth legal user; d0 represents the distance between antennas in the legal transceiver; θ l It is represented as the pointing angle from the legal transceiver to the lth legal user; j′ is represented as the imaginary part.

[0072] The interference beamforming matrix transmitted by the legitimate transceiver is expressed as:

[0073]

[0074]

[0075]

[0076] In the formula, v k represents the interference beamforming vector transmitted by the legitimate transceiver; h represents the number of beams transmitted by the legitimate transceiver; It is represented by the transmission power allocated by the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; d0 is represented as the distance between antennas in the legitimate transceiver; θ k It is represented as the pointing angle from the legitimate transceiver to the kth radiating source node in the non-cooperative adversarial network; j′ is represented as the imaginary part.

[0077] The expression formula of the communication rate of the lth legal receiver is:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] In the formula, γ l,l Expressed as the signal-to-interference-noise ratio of the lth legal receiver; It represents the transmission power allocated by the legal transceiver to the nth legal user; h ll,l It is expressed as the channel gain between the legal transceiver and the lth legal receiver; It is represented as the beam direction from the legal transceiver to the nth legal user; d ll，l It is expressed as the distance between the legal transceiver and the lth legal receiver; It is represented by the transmission power allocated by the legitimate transceiver to the i-th radiator node in the non-cooperative adversarial network; represents the beam direction from the legitimate transceiver to the i-th radiation source node in the non-cooperative adversarial network; N0 represents the variance of the circularly symmetric complex Gaussian noise; α represents the path loss coefficient; K ll，l It is expressed as Rice factor; It is represented as the fast fading part between the legal transceiver and the lth legal receiver; Represented as the direct path component between the legal transceiver and the lth legal receiver; β ll，l It is expressed as the line-of-sight angle between the legal transceiver and the lth legal receiver.

[0084] The expression formula of the signal to noise ratio of the kth radiation source node in the non-cooperative adversarial network is:

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] In the formula, It is represented by the transmission power allocated by the legal transceiver to the mth legal user; hls,k is represented by the channel gain between the legal transceiver and the kth radiation source node in the non-cooperative adversarial network; It is represented as the beam direction from the legal transceiver to the mth legal user; g aa,k It is represented as the channel gain between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; p s It is represented by the transmission power of the radiation source node in the non-cooperative adversarial network; It is represented by the transmission power allocated by the legitimate transceiver to the i-th radiator node in the non-cooperative adversarial network; represents the beam direction from the legitimate transceiver to the i-th radiating source node in the non-cooperative adversarial network; N0 represents the variance of the circularly symmetric complex Gaussian noise; α represents the path loss coefficient; d ls，k K is the distance between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; ls,k and K aa，k It is expressed as Rice factor; It is represented as the fast fading part between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the direct path component between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; β ls，k It is represented as the line-of-sight angle between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; d aa，k It is represented as the distance between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented as the fast fading part between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented as the direct path component between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network.

[0093] Adding the interference constraint of the suspicious communication group in the non-cooperative adversarial network and the maximum transmission power constraint of the legitimate transceiver, the method of establishing the optimization problem with the goal of maximizing the sum of the communication rates of the legitimate receivers includes:

[0094]

[0095] st C1

[0096] C2:

[0097] In the formula, R l,l is the communication rate of the lth legal receiver; W is the communication beamforming matrix transmitted by the legal transceiver; V is the interference beamforming matrix transmitted by the legal transceiver; γ a,k It is represented by the signal-to-noise ratio of the kth radiation source node in the non-cooperative adversarial network; φ represents the minimum signal-to-noise ratio at which the kth radiation source node can successfully demodulate the message in the non-cooperative adversarial network; P max is the maximum transmission power of a legal transmitter; represents the conjugate transpose of a vector or matrix; tr(·) represents the trace of a matrix.

[0098] Methods for mapping optimization problems into Markov decision processes include:

[0099] The Markov decision process includes a state set S, an action set A, a transfer function P and a reward function R;

[0100] The state set S contains the beamforming matrices of communication and interference, that is, S = {s:s t =[W t ,V t ]}, where W t It is represented as the communication beamforming matrix at time t, V t The interference beamforming matrix at time t is expressed as: and a set of pointing angles Θ = {0, 2π / C, ..., 2π-2π / C}; , where B and C represent the discrete numbers of transmit power and pointing angles, respectively;

[0101] The action set A is defined as the set of transmit power and the pointing angle set Θ = {0, 2π / C, ..., 2π-2π / C} are traversed element by element; the next state S is determined by the current action A through the transfer function P;

[0102] The reward function R is expressed as:

[0103]

[0104] Among them, a′ and b′ are the communication weight coefficient and interference weight coefficient respectively, and denote the communication reward and interference reward at time t respectively; r t Represented as the reward value at time t.

[0105] A deep reinforcement learning network model is constructed with the legal transceiver as the intelligent agent. The beamforming matrix of the legal transceiver is predicted using the deep reinforcement learning network model. Training samples are obtained by interacting with the legal transceiver and the communication simulation environment.

[0106] The method of training the deep reinforcement learning network model by taking the training samples as input and repeating the iterative training process until the deep reinforcement learning network model converges includes:

[0107] The training samples are used as input to the deep reinforcement learning network model for training. The formula for calculating the training loss value is:

[0108]

[0109] In the formula, r t+1 Represented as the selected action a at time t t The reward value obtained after t Represented as the state of the deep reinforcement learning network model at time t; a t Represented as the deep reinforcement learning network model action at time t; θ- Represents the parameters of the target network in the deep reinforcement learning network model; is represented as the value function of the target network in the deep reinforcement learning network model; θ is represented as the parameter of the evaluation network in the deep reinforcement learning network model; Q(s t ,a t ; θ) represents the value function of the evaluation network in the deep reinforcement learning network model;

[0110] The parameters of the deep reinforcement learning network model are optimized based on the training loss value, and the iterative training process is repeated until the deep reinforcement learning network model converges.

[0111] The state information of the communication environment is collected and input into the trained deep reinforcement learning model to obtain the optimal decision-making strategy, which is a beamforming matrix integrating communication and interference. This embodiment first senses the surrounding dynamic radio environment, uses reasoning and evaluation methods enabled by artificial intelligence to learn from the radio environment, stores the sensed data and learned knowledge in various knowledge bases, and adapts to environmental changes or uncertainties through a real-time decision-making mechanism using multi-dimensional resources (such as transmit power, carrier frequency, and space resources).

[0112] This implementation can bring three types of gains, namely integration gains, synergy gains and cognitive gains.

[0113] First, the integration gain is the essential gain of the integrated system, which is mainly manifested in the following three aspects: ① Higher spectrum / energy efficiency: Compared with the traditional separate communication or interference system, this implementation can achieve flexible spectrum / energy management. In addition, the sensing and communication waveform reuse can reduce the cost of spectrum use, and the design of embedding interference signals in communication symbols can effectively utilize previously unused spectrum. The above results show that it can bring higher spectrum / energy efficiency. ② Higher data utilization efficiency: Perception is the basis of communication and interference systems, and effective data transmission is the prerequisite for rapid decision-making. It is worth noting that traditional data transmission faces the problems of information redundancy, inefficiency and high transmission risk, while this implementation, due to its integrated design and database sharing, can share data according to communication or interference requirements without data transmission, thereby obtaining higher data utilization efficiency and achieving rapid decision-making. ③ Lower hardware cost: Traditional communication or interference systems are either separated or directly combined, which leads to large hardware volume and high cost. In contrast, this implementation integrates multiple functions into one system through waveform design, software radio and OODA loop, which can not only reduce the volume and cost of hardware, but also improve the scalability of the system through the joint design of each part.

[0114] Second, this embodiment organically integrates the functions of perception, communication and interference into one system, giving full play to the advantages of the three, promoting each other, and thus obtaining the following synergistic gains. ① Reliable communication and precise interference assisted by perception: This embodiment emphasizes the key role of perception in decision-making. Specifically, through perception, information such as the location, frequency and channel status of users and targets can be obtained, providing prior information support for waveform design and resource allocation. Especially in a dynamic environment, real-time acquisition of user and target status can adjust strategies in real time to achieve reliable communication and precise interference. ② Communication-assisted collaborative perception and interference: A single device is suitable for scenarios with a small range and fewer targets. When there are multiple targets and a wide range, multiple devices need to work together to achieve better performance. Communication can connect multiple devices into a network to achieve network-level perception and interference. ③ Interference-assisted secure communication: When there are eavesdroppers in the environment, this embodiment can make communication and perception safer through the transmitted interference signals. On the one hand, beamforming technology is used to transmit data to the receiving end of the communication user to reduce the communication signal received by the eavesdropper; on the other hand, it can use the remaining power to send interference signals to the eavesdropper, thereby increasing the interference signal received by the eavesdropper.

[0115] Third, cognitive gain is mainly derived from cognitive, intelligence and learning theories, and its main features are as follows: ① Adaptability and controllability: Traditional integrated communication and jamming systems can make fast and accurate decisions for known users / targets. However, for unknown targets in dynamic environments, a longer decision cycle may be required with lower accuracy. This embodiment infers the current state of the user / target through learning and reasoning, and then quickly adjusts the strategy and reasonably controls the output to achieve optimal communication and jamming performance. ② Autonomous learning and growth: This embodiment can autonomously learn how to communicate with users and jam targets through interaction with the environment, as well as how to effectively store and apply past experience. It mainly relies on machine learning methods to stimulate the user / target system, observe the response of the user / target and learn automatically, and try to use different waveforms and styles to quickly find effective integrated communication and jamming signals. ③ Concealment and anti-destruction: Traditional integrated communication and jamming systems still need to use high power to suppress targets, which can easily expose their positions and lead to anti-radiation strikes. In addition, when there are potential eavesdroppers, there is a risk of data being eavesdropped. It is noted that this embodiment can greatly improve the concealment and anti-destruction capabilities of the system by sensing environmental conditions, reasonably utilizing environmental information, and selecting a more appropriate interference strategy (deception or induced interference).

[0116] This embodiment has five major functions and four closed loops.

[0117] (1) Five functions. The perception function is the basis and premise of the cognitive system. It mainly obtains information (user / target location, frequency, channel status, etc.) from the external environment and stores it in the database for subsequent extraction and utilization. In addition, the database is gradually enriched and updated iteratively over time. The reasoning function extracts features based on the data obtained by perception, and then identifies and classifies users / targets through machine learning, and further judges their behaviors (such as power control, frequency switching, etc.). The evaluation function evaluates the target's threat level, system situation, and perception, communication, and interference performance (minimum mean square error (MMSE), quality of service (QoS), interference signal power ratio (JSR), energy efficiency (EE), etc.) based on the reasoning results. The decision part first analyzes the reasoning and evaluation results, and then optimizes the multi-dimensional resources and integrated waveforms through intelligent algorithms. The action function reconstructs the policy parameters output by the decision function, generates new integrated sensing, communication, and interference signals, and transmits them to the physical environment.

[0118] (2) Four closed loops

[0119] Rational cognitive loop (perception-reasoning-evaluation-decision-action-perception): This is a complete forward loop that includes all five functions. Starting from perception, first obtain external environmental information, then reason and analyze the state and behavior of the user / target, evaluate the overall situation, and then make a response decision to the external environmental stimulus and adjust the parameters in time. The advantage of this forward loop is that all steps are based on rational judgment, its cause and effect relationship can be explained, and it has a low decision delay.

[0120] Memory cognitive loop (perception-reasoning-evaluation-case library-action-perception): Unlike the rational cognitive loop, it does not require a decision-making function, but directly compares the existing situation with the cases in the case library. If there is a similar situation, the system is reconstructed using the existing parameter settings, and then the integrated signal is transmitted to the environment. This loop is mainly based on experience and memory, which can achieve faster actions, but it relies on some accumulated experience.

[0121] Active perception loop (perception-reasoning-perception): The implicit guidance and control feedback of reasoning on perception is called task attention, which emphasizes that the perception of the environment requires subjective goals / tasks. Perceiving the environment according to the requirements of reasoning and evaluation can reduce information redundancy and improve perception efficiency.

[0122] Active cognitive loop (action-decision-evaluation-reasoning-perception-action): Compared with the rational cognitive loop, it emphasizes the initiative of action. Starting from the action, the integrated signal is first transmitted to the environment, and then the possible changes of the user / target are inferred and evaluated. In order to confirm whether the real change is consistent with the inference result, the perception function observes the environment and forms feedback. The action function adjusts and optimizes the parameters to achieve better communication and interference performance.

[0123] In this embodiment, the relevant parameters are set as follows: Considering that the small-scale fading between the cognitive communication and interference integrated device and the communication user / target follows Rician fading. The uncertainty of the target position is described by the uncertainty of the angle. The path fading index is 2.5, the noise spectrum density is -169dbm / Hz, the communication users and targets are randomly distributed on a circle with a radius of 100 meters, and the maximum transmission power of the cognitive communication and interference integrated device is 1w.

[0124] first, Figure 2 The convergence curve of the proposed algorithm 5-3 is shown, including the total reward and the interference-related reward. Figure 2 It can be seen that within 10,000 iterations, all curves have converged. In addition, an interesting finding is that the total reward in the last period coincides with the interference-related reward curve. According to the reward function expression, it can be concluded that the communication-related reward remains at 0, which indicates that the performance (objective function) of legal communication has reached the maximum critical value and cannot be further increased, and the algorithm tends to converge.

[0125] Secondly, Figure 3 The performance of different algorithms is compared, including the DQN-based joint optimization algorithm 5-3 for transmit power and beamforming proposed in this implementation, the traditional fractional programming (FP) algorithm, and the traditional random algorithm. Please note that the fractional programming method here assumes that the channel state information of all links is known under ideal conditions, while the random algorithm adopts a random action selection strategy that satisfies the power constraint. The sum of the communication rates of legitimate users is the objective function of the original problem and is also the performance indicator compared here. It can be seen that the sum of the communication rates of legitimate users obtained by this implementation gradually converges with the increase of training rounds, and the rate is increased by nearly 20bps compared with the random algorithm. It is worth noting that despite the assumption that the channel state information is unknown, the communication rate of the proposed algorithm 5-3 is still close to the ideal fractional programming algorithm.

[0126] also, Figure 4 The performance comparison of this implementation under different numbers of transmit antennas is shown. It can be seen that the proposed algorithm tends to converge under different settings of the number of transmit antennas, and the sum of the communication rates of legitimate users gradually increases with the increase of the number of antennas.

[0127] Example 2

[0128] This embodiment provides an electronic device including a storage medium and a processor; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the method described in Embodiment 1.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cognitive interference integrated beamforming method based on deep reinforcement learning, characterized in that: include: Collecting state information of the communication environment and inputting it into the trained deep reinforcement learning model to obtain an optimal decision strategy, wherein the decision strategy is a beamforming matrix integrating communication and interference; The construction and training process of the deep reinforcement learning model includes: A communication and interference integrated model is constructed based on a communication simulation environment; the communication and interference integrated model is configured with a single legal transceiver, L legal receivers and a non-cooperative adversarial network; the legal transceiver is equipped with antennas distributed in a linear array; K radiation source nodes are arranged in the non-cooperative adversarial network; in a state where the channel state information is unknown, the legal transceiver communicates with L legal receivers and interferes with the non-cooperative adversarial network at the same time; Add the interference constraints of the suspicious communication group in the non-cooperative adversarial network and the maximum transmission power constraints of the legitimate transceiver, and establish an optimization problem with the goal of maximizing the sum of the communication rates of the legitimate receivers. Specifically, it includes: ； In the formula, It is represented as the communication rate of the lth legal receiver; W is represented as the communication beamforming matrix transmitted by the legal transceiver; V is represented as the interference beamforming matrix transmitted by the legal transceiver; It is expressed as the signal-to-interference-noise ratio of the kth radiation source node in the non-cooperative adversarial network; It represents the minimum signal-to-interference-noise ratio (SINR) at which the kth radiation source node in the non-cooperative adversarial network can successfully demodulate the message; is the maximum transmission power of a legal transmitter; Represents the conjugate transpose of a vector or matrix; Expressed as the trace of a matrix; The communication beamforming matrix transmitted by the legal transceiver is expressed as: , , ； ； In the formula, represents the communication beamforming vector transmitted by the legitimate transceiver; It represents the number of beams transmitted by legal transceivers; It represents the transmission power allocated by the legal transceiver to the lth legal user; It is represented as the beam direction from the legal transceiver to the lth legal user; It is expressed as the distance between antennas in a legal transceiver; It is represented as the pointing angle from the legal transceiver to the lth legal user; It is expressed as the imaginary part; The interference beamforming matrix transmitted by the legitimate transceiver is expressed as: , ； ； ； In the formula, represents the interference beamforming vector transmitted by the legitimate transceiver; It represents the number of beams transmitted by legal transceivers; It is represented by the transmission power allocated by the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; It is expressed as the distance between antennas in a legal transceiver; It is represented as the pointing angle from the legitimate transceiver to the kth radiator node in the non-cooperative adversarial network; Map the optimization problem into a Markov decision process, which includes: The Markov decision process includes a state set S, an action set A, and a transition function And the reward function ; The state set S contains the beamforming matrices for communication and interference, namely ,in Expressed as the communication beamforming matrix at time t, The interference beamforming matrix at time t is expressed as: and the pointing angle set ,in, and The discrete numbers representing the transmission power and the pointing angle respectively; The action set A is defined as the set of transmit power and the pointing angle set Perform element-by-element traversal; through the transfer function The next state set S is determined by the current action set A; The reward function The expression formula is: ； in, and are the communication weight coefficient and the interference weight coefficient, and denote the communication reward and interference reward at time t respectively; Represented as the reward value at time t; A deep reinforcement learning network model is constructed with the legal transceiver as the intelligent agent. The beamforming matrix of the legal transceiver is predicted using the deep reinforcement learning network model. Training samples are obtained through the interaction between the legal transceiver and the communication simulation environment. The training samples are used as input to the deep reinforcement learning network model for training, and the iterative training process is repeated until the deep reinforcement learning network model converges, specifically including: The training samples are used as input to the deep reinforcement learning network model for training. The formula for calculating the training loss value is: ； In the formula, Represented as the selected action at time t The reward value obtained after Represented as the state of the deep reinforcement learning network model at time t; Represented as the deep reinforcement learning network model action at time t; Represents the parameters of the target network in the deep reinforcement learning network model; Represented as the value function of the target network in the deep reinforcement learning network model; Represented as the parameters of the evaluation network in the deep reinforcement learning network model; Represented as the value function of the evaluation network in the deep reinforcement learning network model; The parameters of the deep reinforcement learning network model are optimized based on the training loss value, and the iterative training process is repeated until the deep reinforcement learning network model converges.

2. The interference integrated beamforming method according to claim 1, characterized in that: The expression formula of the communication rate of the lth legal receiver is: ； ； ； ； ； In the formula, Expressed as the signal-to-interference-noise ratio of the lth legal receiver; represents the transmission power allocated by the legal transceiver to the nth legal user; It is expressed as the channel gain between the legal transceiver and the lth legal receiver; It is represented as the beam direction from the legal transceiver to the nth legal user; It is expressed as the distance between the legal transceiver and the lth legal receiver; It is represented by the transmission power allocated by the legitimate transceiver to the i-th radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legitimate transceiver to the i-th radiating source node in the non-cooperative adversarial network; Expressed as the variance of circularly symmetric complex Gaussian noise; represents the path loss coefficient; It is expressed as Rice factor; It is represented as the fast fading part between the legal transceiver and the lth legal receiver; It is represented as the direct path component between the legal transceiver and the lth legal receiver; It is expressed as the line-of-sight angle between the legal transceiver and the lth legal receiver.

3. The interference-integrated beamforming method according to claim 1, characterized in that: The expression formula of the signal to noise ratio of the kth radiation source node in the non-cooperative adversarial network is: ； ； ； ； ； ； ； In the formula, It represents the transmission power allocated by the legal transceiver to the mth legal user; It is expressed as the channel gain between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legal transceiver to the mth legal user; It is expressed as the channel gain between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented by the transmission power of the radiation source node in the non-cooperative adversarial network; It is represented by the transmission power allocated by the legitimate transceiver to the i-th radiator node in the non-cooperative adversarial network; It is represented as the beam direction from the legitimate transceiver to the i-th radiating source node in the non-cooperative adversarial network; Expressed as the variance of circularly symmetric complex Gaussian noise; represents the path loss coefficient; It is represented as the distance between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; and It is expressed as Rice factor; It is represented as the fast fading part between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the direct path component between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the line-of-sight angle between the legitimate transceiver and the kth radiator node in the non-cooperative adversarial network; It is represented as the distance between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented as the fast fading part between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network; It is represented as the direct path component between the kth radiation source node and its corresponding transmitter in the non-cooperative adversarial network.

4. An electronic device, comprising a storage medium and a processor; the storage medium is used to store instructions; characterized in that: The processor is configured to operate according to the instructions to execute the method according to any one of claims 1 to 3.

Images