A decoupled Q-learning intelligent codebook selection method for SCMA-V2X system

By decoupling Q-learning intelligent codebook selection method, the user access delay and interference problems in the SCMA-V2X system are solved, and the codebook exclusively is realized for each user, improving system performance and capacity.

CN115515101BInactive Publication Date: 2024-11-26NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211167893.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-23
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the existing SCMA-V2X systems, the traditional codebook selection method leads to increased user access delay and serious interference problems between V2I users and V2V users.

Method used

Adopting the decoupled Q-learning intelligent codebook selection method, through orthogonal channel resource division and adaptive user-codebook selection strategy, avoid interference between V2I users and V2V users, and ensure that through Q table update and ε-greedy strategy Each user finds a unique codebook.

Benefits of technology

It effectively reduces user access delay, improves system uplink access efficiency, avoids interference between users, and improves system capacity and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115515101B_ABST
    Figure CN115515101B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of mobile communication technology, and discloses a decoupled Q learning intelligent codebook selection method for SCMA-V2X system. In the initial stage of learning, all Q values ​​in the Q table of V2I users and V2V users are initialized to 0; V2I users and V2V users randomly select codebooks from the codebook pool, and update the Q values ​​in the Q table according to the received feedback; in each learning process, V2I users and V2V users select new actions according to the ε-greedy strategy and continue to update the Q value in the table according to the received feedback until convergence; and finally determine the optimal user-codebook selection strategy for V2I users and V2V users. Through the decoupled Q learning intelligent codebook selection method provided by the present invention, each V2I user and V2V user can find a unique codebook, thereby solving the codebook collision problem, greatly improving system performance, reducing user access delay, and effectively avoiding interference between users.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of mobile communication technology, and in particular to a decoupled Q-learning intelligent codebook selection method for a SCMA-V2X system. Background Art

[0002] At present, SCMA is a code domain non-orthogonal multiple access technology with high spectrum utilization efficiency, which can effectively enhance the system capacity of V2X. In the traditional SCMA-V2X system, the commonly used codebook selection method is that the base station allocates a codebook to the user, which will increase the transmission delay and generate certain signaling overhead. In addition, V2I (vehicle-to-infrastructure) users and V2V (vehicle-to-vehicle) users share the same channel and codebook, which causes V2I users and V2V users to interfere with each other.

[0003] In recent years, SCMA has attracted widespread attention as a non-orthogonal multiple access technology in the code domain. Due to its high spectrum utilization efficiency, it has great potential in enhancing V2X communication performance. SCMA introduces a sparse codebook to enable multiple users to reuse the same wireless communication resources (including time domain, frequency domain and other resources). The receiving end uses an advanced message passing decoding algorithm to achieve joint decoding of multiple users, thereby improving the utilization efficiency of wireless spectrum resources. In the SCMA-V2X system, the core issue is the allocation method of the codebook. The traditional method usually adopts the form of base station scheduling, that is, the user first needs to send an access request to the base station, and the base station allocates the corresponding codebook to the user after receiving the request. Although this method can avoid the problem of user codebook conflicts, it will greatly increase the user's access delay and reduce the access efficiency. At the same time, in the current SCMA-V2X system, V2I users and V2V users usually share a channel, which will cause serious interference between users. To this end, it is urgent to propose an effective codebook allocation method for the SCMA-V2X system.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) The existing codebook selection method will greatly increase the user's access delay and reduce the access efficiency.

[0006] (2) In the current SCMA-V2X system, V2I users and V2V users usually share a channel, which will cause serious interference between users. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a decoupled Q-learning intelligent codebook selection method for a SCMA-V2X system, and more particularly, relates to a decoupled Q-learning intelligent codebook selection method, system, medium, device and terminal.

[0008] The present invention is implemented as follows: first, in order to avoid mutual interference between V2I users and V2V users, V2I users and V2V users are separated by orthogonal channel resource division; then, in order to solve the code book collision problem between V2I users and V2V users during uplink transmission, decoupled Q learning is used to implement an adaptive user-code book selection strategy. Users continuously learn and adjust their own code book selection strategies, and ultimately each user finds a unique code book, thereby solving the code book collision problem during user uplink transmission and improving the uplink access efficiency of the system.

[0009] Furthermore, the decoupled Q-learning intelligent codebook selection method further includes:

[0010] In the initial stage of learning, all Q values ​​in the Q table of V2I users and V2V users are initialized to 0; V2I users and V2V users randomly select codebooks from the codebook pool and update the Q values ​​in the Q table according to the received rewards; in each learning process, V2I users and V2V users select new actions according to the ε-greedy strategy and continue to update the Q values ​​in the table according to the received rewards until convergence; finally, the optimal user-codebook selection strategy for V2I users and V2V users is obtained according to the following formula:

[0011]

[0012]

[0013] Further, the decoupled Q learning intelligent codebook selection method comprises the following steps:

[0014] Step 1: Construct the state space S and action space A;

[0015] Step 2: Set the reward function r m,1 and r n,2 ;

[0016] Step 3: Calculate the Q function Q m,1 and Q n,2 .

[0017] Furthermore, the construction of the state space S in step 1 includes: the state space is defined as a set of codebooks, and each codebook is regarded as a state. Since the V2I user and the V2V user always select codebooks from the same codebook pool, the codebook space of the V2I user and the V2V user is the same, and the state space is represented as S = {s1, s2, ..., s J}; For any V2I user u m,1 and V2V user u n,2 , the states at time t are expressed as and

[0018] The construction of the action space A includes: the judgment made by the agent is regarded as an action, all possible actions constitute the action space, and the action represents the agent to select the codebook; for V2I users and V2V users, the action space is the same, expressed as A = {a1, a2, ..., a J}; For any V2I user u m,1 and V2V user u n,2 , the actions at time t are expressed as and

[0019] Furthermore, the reward function r in step 2 is set m,1 and r n,2 include:

[0020] The reward is used to evaluate the quality of the action taken by the agent. m,1 If a unique codebook is selected and not selected by other V2I users, the reward value is r m,1 = +1. If there are multiple V2I users who select the same codebook, a specific user is designated to use the codebook, and the remaining users reselect the codebook; the designated user is determined to be the user with the maximum transmission signal-to-noise ratio. m,1 The current state and action are and At this time, V2I user u m,1 In the current state-action pair The reward function is expressed as:

[0021]

[0022] Among them, μ m,1 represents V2I user u m,1 The received SNR, Γ 1,j Represents all selected codebooks c j The SNR set of V2I users; similarly, V2V users and V2I users have the same reward and punishment mechanism, so V2V user u n,2 In the current state-action pair The reward function is expressed as:

[0023]

[0024] Among them, μ n,2 represents V2V user u n,2 The received SNR, Γ 2,j Represents all selected codebooks c j The SNR set of V2V users.

[0025] Furthermore, the calculation Q function Q in step 3 m,1 and Q n,2 include:

[0026] Each V2I user and V2V user has a Q table to store the Q value of the state-action pair; for V2I user u m,1 and V2V user u n,2 , the Q value is updated according to the following criteria:

[0027]

[0028]

[0029] Among them, α m,1 ,α n,2 ∈(0,1] represents V2I user u m,1 and V2V user u n,2 The learning rate, γ m,1 ,γ n,2 ∈[0,1] represents V2I user u m,1 and V2V user u n,2 The discount factor.

[0030] Another object of the present invention is to provide a decoupled Q learning intelligent codebook selection system using the decoupled Q learning intelligent codebook selection method, the decoupled Q learning intelligent codebook selection system comprising:

[0031] An initialization module, for initializing all Q values ​​in the Q tables of the V2I user and the V2V user to 0 in the initial stage of learning;

[0032] The codebook selection module is used for V2I users and V2V users to randomly select a codebook from the codebook pool and update the Q value in the Q table according to the received feedback;

[0033] The Q-value updating module is used for each learning process. Both the V2I user and the V2V user select new actions according to the ε-greedy strategy and continue to update the Q-value in the table according to the received rewards until convergence;

[0034] The optimal strategy determination module is used to determine the optimal user-codebook selection strategy for V2I users and V2V users.

[0035] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the decoupled Q learning intelligent codebook selection method.

[0036] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the decoupled Q learning intelligent codebook selection method.

[0037] Another object of the present invention is to provide an information data processing terminal, which is used to implement the decoupled Q learning intelligent codebook selection system.

[0038] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0039] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0040] In order to overcome the problems of the prior art, the present invention proposes a random codebook selection scheme, and solves the conflict problem caused by random codebook selection through a new decoupled Q learning intelligent codebook selection method for sparse code multiple access vehicle-to-everything (SCMA-V2X) system. Specifically, V2I users and V2V users transmit data using frequency division multiple access, which can effectively avoid interference between V2I users and V2V users; then, V2I users and V2V users randomly select codebooks from the shared codebook pool and transmit data according to the selected codebooks. Since random codebook selection is bound to lead to codebook conflict problems, the present invention proposes decoupled Q learning to solve the codebook conflict problem between V2I users and V2V users, greatly improving the system capacity.

[0041] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0042] SCMA technology is a key technology in 5G communication and still has a wide range of applications in V2X communication. The decoupled Q learning intelligent codebook selection method for SCMA-V2X network proposed in the present invention involves key technologies in the field of mobile communications. Through the decoupled Q learning intelligent codebook selection method provided by the present invention, each V2I user and V2V user can find a unique codebook, thereby solving the codebook collision problem and greatly improving system performance. The present invention reduces user access delay and effectively avoids interference between users, which has important guiding significance for the SCMA-V2X system to move from theory to application.

[0043] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0044] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0045] The main application background of the present invention is the V2X network, which is used to solve the uplink access problem of vehicle users in large-scale V2X networks, and aims to realize adaptive vehicle-to-base station and vehicle-to-vehicle communication in an intelligent way. The expected benefits after the transformation of the technical solution of the present invention are closely related to the maturity of the current V2X development. At present, the development of V2X is still in its rising stage, and the expected benefits will also increase year by year with the maturity of V2X development. After the transformation of the results, it can be applied to large-scale V2X communication to improve the efficiency and intelligence level of vehicle-to-base station and vehicle-to-vehicle communication.

[0046] (2) The technical solution of the present invention solves a technical problem that people have been eager to solve but have never been able to solve successfully:

[0047] At present, there are two main technical problems in the uplink transmission for SCMA-V2X network, namely: 1) the mutual interference problem between V2I users and V2V users; 2) the codebook collision problem during user uplink transmission. The present invention proposes a decoupled Q learning intelligent codebook selection method, which first solves the mutual interference problem when V2V users and V2I users share the spectrum. At the same time, in order to improve the communication efficiency and capacity for SCMA-V2X network, decoupled Q learning is used to solve the codebook collision problem during user uplink access in an intelligent way, greatly improving the capacity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is a flow chart of a decoupled Q learning intelligent codebook selection method provided by an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of a codebook mapping solution provided by an embodiment of the present invention;

[0051] Figure 3 Schematic diagram of convergence of the decoupled Q-learning algorithm provided in an embodiment of the present invention; (a) is a schematic diagram of the number of V2V users without codebook collision, and (b) is a schematic diagram of V2I users and rates;

[0052] Figure 4 Schematic diagram of the effect of the interruption probability on the V2I user and rate provided by an embodiment of the present invention, M=15, R min =2.06bps / Hz, v=60km / h;

[0053] Figure 5 is a schematic diagram comparing the performance of RMA-SCMA with codebook conflict resolution and without codebook conflict resolution provided by an embodiment of the present invention, P out =10 -2 , R min =2.06bps / Hz, v=60km / h. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] In view of the problems existing in the prior art, the present invention provides a decoupled Q-learning intelligent codebook selection method, system, medium, device and terminal. The present invention is described in detail below with reference to the accompanying drawings.

[0056] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands and describes the technical solution of the claims.

[0057] like Figure 1 As shown, the decoupled Q learning intelligent codebook selection method provided by the embodiment of the present invention includes the following steps:

[0058] S101, in the initial stage of learning, all Q values ​​in the Q tables of V2I users and V2V users are initialized to 0;

[0059] S102, the V2I user and the V2V user randomly select a codebook from the codebook pool, and update the Q value in the Q table according to the received feedback;

[0060] S103, in each learning process, the V2I user and the V2V user select new actions according to the ε-greedy strategy and continue to update the Q value in the table according to the received rewards until convergence;

[0061] S104, determining an optimal user-codebook selection strategy for the V2I user and the V2V user.

[0062] As a preferred embodiment, the decoupled Q learning intelligent codebook selection method provided in the embodiment of the present invention specifically includes the following steps:

[0063] 1. Codebook mapping scheme

[0064] In the current access opportunity, assuming that there are L time slot resources, each codebook only occupies K time slots out of L time slots, then the number of available codebooks is The codebook set can be expressed as where c j =[c 1,j ,...,c l,j ,...,c L,j ] and c l,j ∈{0,1},|c j |=K. In order to avoid mutual interference between V2I users and V2V users, the total bandwidth is divided into two parts, namely βB and (1-β)B, where β is the bandwidth allocation coefficient, which is considered a constant here. βB bandwidth is used for V2I user transmission, and (1-β)B bandwidth is used for V2V user transmission. V2I users and V2V users use the same codebook set. A codebook is randomly selected from the time slot and data is transmitted in the corresponding time slot according to the selected codebook. For convenience of representation, the embodiment of the present invention introduces two factors a m,j and b n,j denote the V2I user-codebook selection relationship and the V2V user-codebook selection relationship respectively, and if the V2I user u m,1 and V2V user u n,2 Select codebook c j , then there is a m,j =1,b n,j =1, otherwise a m,j =0,b n,j = 0. Due to the random codebook selection scheme, when two or more V2I users or V2V users select the same codebook, a codebook collision will occur, which will affect multi-user decoding and deteriorate system performance. When there is no codebook collision, multiple users can be successfully decoded using advanced message passing algorithms. Figure 2 A random codebook mapping scheme is shown. As can be seen from the figure, since V2I users and V2V users transmit in an orthogonal manner in the frequency domain, even if the same codebook is selected, there will be no codebook collision, which can effectively avoid mutual interference between users. On the contrary, if the same type of users select the same codebook, a codebook collision will occur, thus affecting system performance.

[0065] 2. SCMA signal transmission model

[0066] Assume that V2I user u m,1 and V2V user u n,2 Codebook c has been selected j , for convenience, it is rewritten as and and Based on the above definition, the received signal of the V2I user in time slot l can be expressed as:

[0067]

[0068] Where P is the transmit power of the user and it is assumed that the transmit power of all V2I users is the same. Represents user u m,1 The channel gain, represents the channel coefficient, G is the path loss constant, d m,1 is the transmission distance from the user to the base station, and α is the path loss index. m,l,1 Represents user u m,1 The transmission signal in time slot l and E[|x m,l,1 | 2 ] = 1. n l,1 The mean is 0 and the variance is σ 2 Additive White Gaussian Noise (AWGN). Similarly, the received signal of the V2V user in time slot l can be expressed as:

[0069]

[0070] Where P is the transmit power of the user and it is assumed that the transmit power of the V2V user is the same as that of the V2I user. Represents user u n,2 The channel gain, represents the channel coefficient, G is the path loss constant, d n,2 is the transmission distance from user to user, and α is the path loss exponent. n,l,2 Represents user u n,2 The transmission signal in time slot l and E[|x n,l,2 | 2 ] = 1. n l,2 The mean is 0 and the variance is σ 2 AWGN.

[0071] 3. Build and rate maximization issues

[0072] From equations (1) and (2), we can see that V2I user u m,1 The transmission rate can be expressed as:

[0073]

[0074] Among them, I m,l,1 Represents user u m,1 The interference from other users is worth noting that this interference only occurs between the same codebooks, that is, the interference is caused by the V2I user u m,1 The interference is generated by users who choose the same codebook. The interference can be expressed as:

[0075]

[0076] Similarly, V2V user u n,2 The transmission rate can be expressed as:

[0077]

[0078] Similarly, I n,l,2 Represents user u n,2 The interference from other users can be expressed as:

[0079]

[0080] Generally speaking, V2X communication usually requires high transmission data rate for V2I users and high reliability for V2V users. In order to ensure V2V reliability while maximizing the sum rate of V2I users, the embodiment of the present invention constructs the following user-codebook selection optimization problem, namely:

[0081]

[0082] Among them, constraint C1 in R min represents the minimum permissible transmission data rate of the V2V user, P out represents the interruption probability, and this constraint is used to ensure the reliability requirements of V2V users. Constraint C2 is used to ensure that each codebook is selected by at most one V2I user. Constraint C3 is used to ensure that each codebook is selected by only one V2V user. Constraints C4 and C5 are the value ranges of the optimization variables themselves. In order to ensure the reliability requirements of V2V users, that is, to meet constraint C1, the transmission bandwidth of V2V users should be large enough. This constraint can be achieved by selecting a suitable bandwidth allocation coefficient β. It can be seen from (7) that this optimization problem is a discrete optimization problem. The optimal solution can be obtained through the traditional exhaustive search method, but the implementation complexity is very high, which is obviously impractical. Intuitively, in order to maximize the sum rate of V2I users, the codebook collision selected by the user should be as small as possible. The ideal situation is that each V2I user or V2V user can select a unique codebook, but under the condition of random codebook selection, codebook collision is bound to occur. In order to enable each user to select a unique codebook, the embodiment of the present invention introduces an intelligent learning strategy, that is, the user gradually selects a unique codebook by continuously trying and adjusting his or her codebook selection strategy. The specific implementation method is shown in the next section.

[0083] 4. Decoupled Q-learning intelligent codebook selection method

[0084] In order to achieve the goal of maximizing V2I users and rates, an embodiment of the present invention proposes a decoupled Q-learning intelligent codebook selection method. This technology separates V2I users and V2V users, allowing V2I users and V2V users to use Q-learning to implement adaptive user-codebook selection strategies, and by continuously learning and adjusting their own codebook selection strategies, each user can eventually find a unique codebook, thereby avoiding the codebook collision problem. When codebook collisions are completely avoided, optimization problem (7) is maximized. The specific implementation steps are as follows:

[0085] (1) Constructing the state space S: The state space is defined as a set of codebooks, and each codebook can be regarded as a state. The codebook space of V2I users and V2V users is the same because they always select codebooks from the same codebook pool. The state space can be expressed as S = {s1, s2, ..., s J}. For any V2I user u m,1 and V2V user u n,2 , their states at time t can be expressed as and

[0086] (2) Constructing the action space A: The judgment made by the agent is regarded as an action, and all possible actions constitute the action space. The action here means that the agent selects the codebook. For V2I users and V2V users, the action space is the same and can be expressed as A = {a1, a2, ..., a J}. For any V2I user u m,1 and V2V user u n,2 , their actions at time t can be expressed as and

[0087] (3) Set the reward function r m,1 and r n,2 :The reward is used to evaluate the quality of the action taken by the agent. For V2I user u m,1 For example, if it selects a unique codebook (that is, not selected by other V2I users), its reward value is r m,1 = +1. If there are multiple V2I users who have selected the same codebook, a specific user needs to be designated to use the codebook, and the remaining users reselect the codebook. The designated user will be determined as the user with the maximum transmission signal-to-noise ratio (SNR). Assume that V2I user u m,1 The current state and action are and At this time, V2I user u m,1 In the current state-action pair The reward function can be expressed as:

[0088]

[0089] Among them, μ m,1 represents V2I user u m,1 The received SNR, Γ 1,j Represents all selected codebooks c j The SNR set of V2I users. Similarly, V2V users and V2I users have the same reward and punishment mechanism. Therefore, V2V user u n,2 In the current state-action pair The reward function can be expressed as:

[0090]

[0091] Among them, μ n,2 represents V2V user u n,2 The received SNR, Γ 2,j Represents all selected codebooks c j The SNR set of V2V users.

[0092] (4) Calculate the Q function Q m,1 and Q n,2 :Each V2I user and V2V user has a Q table to store the Q value of the state-action pair. m,1 and V2V user u n,2 , their Q values ​​can be updated according to the following criteria:

[0093]

[0094]

[0095] Among them, α m,1 ,α n,2 ∈(0,1] represents V2I user u m,1 and V2V user u n,2 The learning rate, γ m,1 ,γ n,2 ∈[0,1] represents V2I user u m,1 and V2V user u n,2 The discount factor.

[0096] The decoupled Q-learning intelligent codebook selection process is as follows. First, in the initial stage of learning, all Q values ​​in the Q table of V2I users and V2V users are initialized to 0. Then, V2I users and V2V users randomly select codebooks from the codebook pool and update the Q values ​​in the Q table according to formulas (10) and (11) based on the received rewards. In each subsequent learning process, V2I users and V2V users select new actions according to the ε-greedy strategy and continue to update the Q values ​​in the table according to the received rewards until convergence. Finally, the optimal user-codebook selection strategy for V2I users and V2V users can be obtained by the following formula:

[0097]

[0098]

[0099] It is worth noting that since the non-overload scenario is considered (that is, the number of users does not exceed the number of codebooks), the final learning process will converge. Through the above decoupled Q learning intelligent codebook selection method, each V2I user and V2V user can find a unique codebook, thus solving the codebook collision problem and greatly improving system performance.

[0100] The decoupled Q-learning intelligent codebook selection system provided by the embodiment of the present invention includes:

[0101] An initialization module, for initializing all Q values ​​in the Q tables of the V2I user and the V2V user to 0 in the initial stage of learning;

[0102] The codebook selection module is used for V2I users and V2V users to randomly select a codebook from the codebook pool and update the Q value in the Q table according to the received feedback;

[0103] The Q-value updating module is used for each learning process. Both the V2I user and the V2V user select new actions according to the ε-greedy strategy and continue to update the Q-value in the table according to the received rewards until convergence;

[0104] The optimal strategy determination module is used to determine the optimal user-codebook selection strategy for V2I users and V2V users.

[0105] 2. Application Examples: In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.

[0106] This application embodiment describes the application of the decoupled Q-learning intelligent codebook selection method proposed in the present invention to the SCMA-V2X network, which can be used to solve the mutual interference problem in vehicle-to-base station communication and vehicle-to-vehicle communication in application fields such as Internet of Vehicles and autonomous driving, as well as the codebook collision problem during user uplink access, thereby improving the system capacity of the V2X network.

[0107] 3. Evidence of the effects of the embodiments. The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art. The following content is described in conjunction with the data, charts, etc. of the test process.

[0108] Simulation verification: The embodiment of the present invention will present a series of simulation results to evaluate the performance of the proposed decoupled Q-learning intelligent codebook selection method. Considering the suburban V2X communication scenario, the average distance of V2V is set to 2.5s×v, where v (m / s) is the vehicle speed. The total number of time slots and the number of time slots occupied by the codebook are L=6 and K=2, respectively, so the number of codebooks in the codebook pool is J=15. The transmission channel model and other key parameters are determined according to 3GPP, and the detailed parameter values ​​are shown in Table 1.

[0109] Table 1 V2X system simulation parameters

[0110] parameter Value Transmission bandwidth, B 10MHz Noise Power Spectral Density -174dBm / Hz User transmit power, P 15dBm Minimum receiving SNR for V2V users 3-9dB Minimum data transmission rate for V2V users 1.58-3.16bps / Hz User path loss model 128.1+37.6log10(d)(km) Learning Rate 0.1 Discount Factor 0.95 Average vehicle speed 10-80km / h

[0111] Figure 3 (a) and (b) show the relationship between the number of V2V users without codebook collision and the number of decoupled Q learning times, and the relationship between the V2I users and the rate and the number of decoupled Q learning times, respectively. Figure 3 (a) It can be seen that when the number of V2V users N = 6, 9, the decoupled Q learning can converge after 2 iterations, and when N = 12, 15, the decoupled Q learning can converge after 3 iterations. It is easy to see that as the number of V2V users increases, the convergence speed of decoupled Q learning gradually slows down. Figure 3 (b) can be obtained with Figure 3 (a) Similar conclusion, as the number of V2I users increases, the convergence speed of decoupled Q learning gradually slows down, but Figure 3 As can be seen from (a) and (b), the decoupled Q-learning converges quickly and can converge after only a few iterations, which demonstrates the high learning efficiency of the proposed decoupled Q-learning.

[0112] Figure 4The effect of interruption probability on the sum rate of V2I users is shown. As can be seen from the figure, as the interruption probability decreases, the sum rate of V2I users also gradually decreases. This is because the smaller the interruption probability, the higher the reliability requirement for V2V users, which also requires a larger transmission bandwidth for V2V users. Therefore, the transmission bandwidth of V2I users will decrease, resulting in a decrease in the sum rate of V2I users. Therefore, in practical applications, it is necessary to determine the appropriate interruption probability based on the reliability requirements of V2V users.

[0113] Figure 5 The performance of RMA-SCMA with and without decoupled Q-learning to solve codebook conflicts is shown. As can be seen from the figure, when the number of V2I users is small, the performance of the two is close. As the number of V2I users increases, the use of decoupled Q-learning intelligent codebook selection can effectively avoid codebook conflicts, so the performance will increase linearly. When decoupled Q-learning is not used to solve codebook conflicts, codebook conflicts will occur, thus affecting system performance. Therefore, the performance difference between the two gradually increases.

[0114] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0115] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A decoupled Q-learning intelligent codebook selection method, characterized in that: The decoupled Q-learning intelligent codebook selection method comprises: separating the V2I user and the V2V user, so that the V2I user and the V2V user respectively use Q-learning to implement an adaptive user-codebook selection strategy, and finally achieve each user finding a unique codebook by continuously learning and adjusting their own codebook selection strategy; The decoupled Q learning intelligent codebook selection method comprises the following steps: Step 1: Construct the state space S and action space A; Step 2: Set the reward function r m,1 and r n,2 ; Step 3: Calculate the Q function Q m,1 and Q n,2 ; The decoupled Q learning intelligent codebook selection method also includes: In the initial stage of learning, all Q values ​​in the Q table of V2I users and V2V users are initialized to 0; V2I users and V2V users randomly select codebooks from the codebook pool and update the Q values ​​in the Q table according to the received rewards; in each learning process, V2I users and V2V users select new actions according to the ε-greedy strategy and continue to update the Q values ​​in the table according to the received rewards until convergence; finally, the optimal user-codebook selection strategy for V2I users and V2V users is obtained according to the following formula: The construction of the state space S in step 1 includes: the state space is defined as a set of codebooks, and each codebook is regarded as a state; since the V2I user and the V2V user always select codebooks from the same codebook pool, the codebook space of the V2I user and the V2V user is the same, and the state space is represented as S = {s1, s2, ..., s J }; For any V2I user u m,1 and V2V user u n,2 , the states at time t are expressed as and The construction of the action space A includes: the judgment made by the agent is regarded as an action, all possible actions constitute the action space, and the action represents the agent to select the codebook; for V2I users and V2V users, the action space is the same, expressed as A = {a1, a2, ..., a J }; For any V2I user u m,1 and V2V user u n,2 , the actions at time t are expressed as and The reward function r in step 2 is set m,1 and r n,2 include: The reward is used to evaluate the quality of the action taken by the agent; for V2I user u m,1 If a unique codebook is selected and not selected by other V2I users, the reward value is r m,1 = +1; if there are multiple V2I users who select the same codebook, a specific user is designated to use the codebook, and the remaining users reselect the codebook; the designated user is determined to be the user with the maximum transmission signal-to-noise ratio; when the V2I user u m,1 The current state and action are and At this time, V2I user u m,1 In the current state-action pair The reward function is expressed as: Among them, μ m,1 represents V2I user u m,1 The received SNR, Γ 1,j Represents all selected codebooks c j The SNR set of V2I users; similarly, V2V users and V2I users have the same reward and punishment mechanism, so V2V user u n,2 In the current state-action pair The reward function is expressed as: Among them, μ n,2 represents V2V user u n,2 The received SNR, Γ 2,j Represents all selected codebooks c j The SNR set of V2V users; The calculation Q function Q in step 3 m,1 and Q n,2 include: Each V2I user and V2V user has a Q table to store the Q value of the state-action pair; for V2I user u m,1 and V2V user u n,2 , the Q value is updated according to the following criteria: Among them, α m,1 ,α n,2 ∈(0,1] represents V2I user u m,1 and V2V user u n,2 The learning rate, γ m,1 ,γ n,2 ∈[0,1] represents V2I user u m,1 and V2V user u n,2 The discount factor.

2. A decoupled Q learning intelligent codebook selection system using the decoupled Q learning intelligent codebook selection method according to claim 1, characterized in that: The decoupled Q-learning intelligent codebook selection system comprises: An initialization module, for initializing all Q values ​​in the Q tables of the V2I user and the V2V user to 0 in the initial stage of learning; The codebook selection module is used for V2I users and V2V users to randomly select a codebook from the codebook pool and update the Q value in the Q table according to the received feedback; The Q-value updating module is used for each learning process. Both the V2I user and the V2V user select new actions according to the ε-greedy strategy and continue to update the Q-value in the table according to the received rewards until convergence; The optimal strategy determination module is used to determine the optimal user-codebook selection strategy for V2I users and V2V users.

3. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the decoupled Q learning intelligent codebook selection method according to claim 1.

4. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the decoupled Q-learning intelligent codebook selection method according to claim 1.

5. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the decoupled Q learning intelligent codebook selection system as claimed in claim 2.