A layered deep Q network-based offshore VHF spectrum resource cooperative allocation method
Patent Information
- Application Number
- CN202611061192.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-16
- Publication Date
- 2026-09-15
Smart Images
Figure CN122765740A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of maritime communication technology, and more particularly to a method for collaborative allocation of maritime VHF spectrum resources based on hierarchical deep Q networks. Background Technology
[0002] The VHF band at sea has long been responsible for critical services such as ship scheduling, navigation safety, collision avoidance communication, search and rescue communication, and VTS voice reporting. Traditional VHF RT analog voice communication still occupies an important position in port, waterway, and near-shore traffic organization. The maritime VHF band is divided into multiple basic channels, and the system operates in TDMA mode, with each time slot length consistent with the VDE-TER physical frame length. Analog voice services typically rely on manual channel selection, and service arrival is characterized by burstiness, randomness, and scene dependence. Some channels have significant idle time windows, resulting in low spectrum utilization. To meet the demands of broadband, digital, and intelligent communication at sea, VDES digital communication technology is gradually developing. VDE-TER defines multiple link configuration IDs (LCIDs), each corresponding to different bandwidth requirements, required number of basic channels, modulation order, coding rate, and minimum decoding signal-to-noise ratio threshold. Digital communication links need to achieve spectrum reuse without affecting the secure communication of existing analog voice services.
[0003] Existing methods for managing VHF spectrum resources at sea mainly include fixed-threshold modulation and coding scheme selection, adaptive link selection, and traditional dynamic spectrum access methods. These methods typically only focus on the current channel signal-to-noise ratio or immediate idle state. After analog voice services occupy the channel, the available spectrum resources of the VHF channel exhibit discretization and segmentation characteristics. While extending the VDE-TER LCID configuration can provide multiple modulation and coding schemes for digital services, LCID selection is directly constrained by the availability of consecutive channel blocks. The accessibility of digital links is not only determined by the current channel state but also by the probability of future analog voice services reappearing. A transmitting digital link may be forced to release spectrum and be rescheduled due to intrusion by analog voice services. There are mutual exclusion constraints between consecutive spectrum blocks among multiple digital links. If the digital occupancy vector is not updated in real time, spectrum conflicts between digital links can easily occur. Existing deep reinforcement learning spectrum access schemes can learn access strategies in dynamic environments, but when LCID and channel access are jointly enumerated as single actions, the action space expands rapidly. Solving them independently with greediness may lead to higher layers excessively pursuing throughput or lower layers being overly conservative, making it difficult to form a long-term optimal spectrum reuse strategy.
[0004] Therefore, existing technologies cannot simultaneously solve the following problems: balancing analog voice service priority protection with opportunistic digital spectrum access, coupled decision-making regarding LCID selection and access to consecutive spectrum blocks, pre-assessment of future spectrum idleness risks, dynamic migration of digital links after analog intrusion, and mutual exclusion coordination among multiple digital links. In particular, there is a lack of an intelligent resource management method that can explicitly characterize the coupling relationship between LCID selection and channel access, while simultaneously considering analog service priority protection, digital link mutual exclusion constraints, spectrum prediction risks, and dynamic migration attribution. Summary of the Invention
[0005] In view of this, the purpose of this invention is to propose a collaborative allocation method for maritime VHF spectrum resources based on hierarchical deep Q networks, so as to solve the technical problem of mutual exclusion constraints of continuous spectrum blocks between multiple digital links in the prior art.
[0006] The technical means employed in this invention are as follows: A collaborative allocation method for maritime VHF spectrum resources based on hierarchical deep Q-networks includes the following steps: S1. Initialize system and network parameters; S2. In each time slot, update the link status and construct a queue to be scheduled. The queue to be scheduled consists of migrating links, blocked waiting links and newly added links connected in series according to migration priority. S3. Take out digital links one by one from the queue to be scheduled, input the current link's high-layer instant reward into the high-layer DQN, and output the link configuration identifier LCID according to the greedy strategy. The output action of the high-layer DQN is constrained by the mask formed by the signal-to-noise ratio threshold. S4. Input the low-level state of the same link into the low-level DQN in the same time slot, and output the channel start or blocking action according to the low-level instant reward. The low-level state includes the LCID information output by S3, the predicted access confidence vector of all candidate start frequencies under the LCID, the actual occupancy status of the current time slot analog service in each channel, and the digital link occupancy status before scheduling the current link. The output action of the low-level DQN is subject to the mask constraint composed of four constraints: channel range, analog voice occupancy, digital link mutual exclusion, and predicted access confidence threshold. S5. Execute spectrum access or blocking based on the output of S4: If blocking, no spectrum is allocated in the current time slot; if channel start, data transmission is performed using continuous spectrum blocks configured according to the LCID output of S3; calculate the high-level instant reward and low-level instant reward based on the hierarchical reward function, weight the high-level instant reward and joint reward and store them as the high-level training reward in the high-level experience replay pool, and weight the low-level instant reward and joint reward and store them as the low-level training reward in the low-level experience replay pool. The joint reward is synthesized from the high-level instant reward and low-level instant reward; when the link enters the migration queue due to simulated voice intrusion, calculate the high-level migration penalty and low-level migration penalty respectively, attenuate the high-level migration penalty by delay discount and add it to the high-level reward of the corresponding historical experience sample, and attenuate the low-level migration penalty by delay discount and add it to the low-level reward of the corresponding historical experience sample; update the network parameters of the high-level DQN and low-level DQN, enter the next time slot until training is completed, and output the trained maritime VHF spectrum resource collaborative allocation model based on hierarchical deep Q network; S6. Input the set of links to be scheduled into the trained maritime VHF spectrum resource collaborative allocation model based on hierarchical deep Q network. The trained maritime VHF spectrum resource collaborative allocation model based on hierarchical deep Q network executes S1-S4 and outputs the optimal link configuration identifier LCID and the optimal access channel start point or blocking action.
[0007] Furthermore, in S1, the system parameters include: The maritime VHF band is divided into There are 1 basic channel, and the channel set is as follows: Each channel has a bandwidth of The system operates in TDMA mode, with a time slot set of [missing information]. Each time slot is [length] The maximum number of concurrent links is ;link The path loss of the maritime channel is defined as:
[0008] in, Represents frequency, The mean square root of ocean waves, For reference distance The free space loss value under the following conditions To conform to a mean of 0 and a standard deviation of A normally distributed random variable, The definition is , The definition is , Represents the signal wavelength; the LCID candidate set is defined as follows: For the first LCID configuration, defining physical layer parameters ,in To consume bandwidth, For the required number of basic channels, For signed numbers, The modulation order, For coding rate, This is the minimum decoding signal-to-noise ratio threshold. The number of bits that can be transmitted per time slot is expressed as .
[0009] Furthermore, in S2, the newly added link is generated through time-slotted service arrival and candidate vessel pair screening, specifically including the following steps: Digital service arrival adopts a time-slotted Poisson model, assuming the first... The average arrival intensity of each time slot is The number of newly arriving digital services is The size of each new service data packet follows a uniform distribution. Set time slots The collection of idle ships is Any candidate ship The distance is The effective signal-to-noise ratio of the ship pair is then... The set of candidate ship pairs is represented as:
[0010] when At that time, new links are selected uniformly and randomly from the candidate set, that is:
[0011] If the candidate set is empty, the arrival of this service is recorded as a pairing block and will not enter the spectrum access scheduling queue.
[0012] Furthermore, in S2, the update of the link state specifically includes the following steps: Link state It means that among them This indicates whether the link is complete or inactive. This indicates that the link is transmitting data. This indicates that the link is in a waiting-for-rescheduling state; the first Link in time slot The remaining amount of data to be transmitted is denoted as ; The set of links that were transmitting in the previous time slot is:
[0013] Assume analog voice service in channel and time slot The actual occupancy status is ,in This indicates that the channel is being used by a simulated service. Indicates the channel is idle; if the link is... The spectrum blocks occupied in the previous successful access operation were Then the simulated service intrusion indicator variable for the current time slot is:
[0014] when At that time, link Release the original spectrum block and enter the migration queue; when At that time, the link continues to use the previous time slot configuration and transmit. Data packets of a certain size.
[0015] Furthermore, in S4, the method for calculating the predicted access confidence is as follows: Assume the spectrum prediction module is in the decision time slot Output all channels in the future The idle probability of each time slot is represented by a three-dimensional idle probability vector:
[0016] in, Indicates in time slot When making decisions, the prediction channel Future Time Slot The probability of being idle; for links Using LCID And from the channel Start accessing continuous spectrum blocks Its prediction time domain length is defined as:
[0017] in, Representative link Using LCID The number of time slots required to transmit the remaining data; the spectrum accessibility confidence is calculated using geometric mean aggregation and expressed as:
[0018] in, To prevent positive numbers with singular logarithms; Predicted collision risk is defined as:
[0019] The access confidence level for blocking actions is defined as 0, and the predicted collision risk is defined as 1.
[0020] Furthermore, in S3 and S4, the method of constructing the mask specifically includes the following steps: Define digital link occupancy vector Its elements Channel It has been occupied by the digital link. This indicates that the channel is not occupied; the set of possible actions for higher-layer LCID is given by the signal-to-noise ratio threshold:
[0021] Given LCID Subsequently, the set of low-level channel access actions is as follows:
[0022] in, This is the minimum confidence threshold for accessibility. This represents a blocking action; blocking actions do not occupy a real channel and are always retained in the lower-level action set; let... and Let these represent the actual selection actions of the higher and lower layers, respectively. Then, the single-link two-layer decision-making process satisfies:
[0023] And regarding actions that cannot be taken: .
[0024] Furthermore, in S3, the higher-level state is constructed as follows: No. Link in time slot The high-level state is defined as:
[0025] in, Represents the normalized amount of data to be transmitted. One-hot encoding for linked queue type, Let be the normalized signal-to-noise ratio margin vector, where:
[0026] This represents the maximum predictive access confidence that can be obtained for different LCIDs under the current digital occupancy state, and its i.e., the The components are:
[0027] If there are no candidate starting points that satisfy the constraints, then let ; and These are the normalized codes for the LCID number and channel start point in the previous successful access action, respectively. This indicates whether a simulated service intrusion occurred in the previous time slot.
[0028] Furthermore, in S4, the lower-level state is constructed as follows: No. Link in time slot The lower-level state is defined as:
[0029] in, One-hot encoding for the selected LCID, For LCID The predicted access confidence vector of all channel starting points. This represents the current simulated actual service usage status. For scheduling links The occupancy status of the digital link in the previous time slot.
[0030] Furthermore, in S5, the calculation method for the high-level instant reward and the low-level instant reward is as follows: set up The maximum number of bits transmitted per time slot across all LCIDs; when the higher layer selects LCID. Then, the potential number of transmittable bits for this LCID is:
[0031] The actual normalized throughput is:
[0032] Among them, when blocked ; Define LCID An indicator variable indicating whether a real accessible channel block exists:
[0033] If it exists, the optimal access confidence level is:
[0034] If it does not exist, then take it. ; For the actual selection action at the lower level, the definition is:
[0035] make The variable represents the blocking indicator; the high-level immediate reward is defined as:
[0036] Low-level instant rewards are defined as:
[0037] in, , , , For reward and punishment parameters, The penalty item represents the punishment for choosing to block when there is a channel available for access.
[0038] Furthermore, the method for combining the joint reward and the high-level training reward and low-level training reward described in S5 is as follows: The high-level reward and the low-level reward are linearly combined into a joint reward, expressed as follows:
[0039] The actual training reward written into the experience replay pool is:
[0040] in, This is the combined reward coefficient.
[0041] Compared with the prior art, the present invention has the following advantages: This invention performs digital spectrum reuse under the priority protection constraint of analog VHF RT services, and can actively release channels and perform migration when analog voice services occur, reducing the impact on the secure communication of existing analog voice communication systems.
[0042] This invention decomposes LCID selection and channel access into two discrete decision layers (high and low) and learns the long-term coupling relationship between the two through joint reward learning, thereby avoiding an excessively large joint action space.
[0043] This invention constructs access confidence for continuous spectrum blocks by predicting idle probability, so that digital link access decisions not only consider the current idle state, but also the risk of analog services reappearing in the future.
[0044] This invention employs a migration-priority scheduling queue, which can prioritize the recovery of digital links that have been intruded by simulated services, thereby improving service continuity and reducing rescheduling waiting time.
[0045] This invention introduces a delayed migration penalty mechanism, which traces the migration cost back to the historical access action that led to the current intrusion, thereby improving the accuracy of reward attribution and learning stability.
[0046] This invention introduces a mature VDE-TER LCID configuration, which can adaptively select modulation and coding strategies based on link signal-to-noise ratio, traffic volume, and spectrum status, making it easy to implement in engineering. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a framework diagram of the method of the present invention. Detailed Implementation
[0049] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0051] like Figure 1 As shown, this invention provides a method for collaborative allocation of maritime VHF spectrum resources based on hierarchical deep Q-networks, comprising the following steps: S1. Initialize system and network parameters; Consider by A self-organizing network of digital ships for maritime cognitive purposes, comprising a collection of ships. The maritime VHF band is divided into There are 1 basic channel, and the channel set is as follows: Each channel has a bandwidth of The system operates in TDMA mode, with a time slot set of [missing information]. Each time slot is [length] The default timeslot length is 26.67ms, consistent with the VDE-TER physical frame length. Each ship can participate in a maximum of one digital communication link at any given time, therefore the maximum number of concurrent links is [missing information]. Each digital link By the sending ship and receiving ships Composition, the physical distance of the link is The maritime channel model is affected by the distance between transmitting and receiving nodes, sea conditions, and signal frequency, resulting in a link... The path loss of the maritime channel is defined as: (2) in, Represents frequency, measured in GHz; The root mean square of the ocean waves, in meters; For the reference distance is The free space loss value is as follows; To conform to a mean of 0 and a standard deviation of , A normally distributed random variable, where , The definition is as follows: Representing the signal wavelength, this invention sets... .
[0052] (3) (4) This patent references the VDE-TER LCID configuration in the IALA.G1139 standard as a modulation and coding scheme for digital links, as shown in Table 1. VDE-TER defines 9 link configuration IDs, from 11 to 19. The LCID set can be defined as... For the first LCID configuration, defining physical layer parameters ,in To consume bandwidth, For the required number of basic channels, For signed numbers, The modulation order, For coding rate, This is the minimum decoding signal-to-noise ratio threshold. The number of bits that can be transmitted per time slot can be expressed as If the link In the time slot From the channel Start accessing and adopting LCID Then the continuous spectrum block it occupies is ,in .link In LCID The received signal-to-noise ratio under the given conditions can be expressed as: (5) in For transmission power, For bandwidth The noise power below.
[0053] Table 1 VDE-TER Channel Configuration Table
[0054] Digital service arrivals are modeled using a time-slotted Poisson model. Let the... The average arrival intensity of each time slot is The number of newly arriving digital services is The size of each new service's data packet follows a uniform distribution. Upon arrival of new service, a random ship pair is selected from candidate ship pairs that meet the link reliability requirements to establish a communication link. Let time slots be defined. The collection of idle ships is Any candidate ship The distance is The effective signal-to-noise ratio of the ship pair is then... Then the set of candidate ship pairs can be represented as (6) Among them, default This aligns with the minimum LCID reliable transmission threshold. When At that time, new links are selected uniformly and randomly from the candidate set, that is... (7) If the candidate set is empty, the arrival of this service is recorded as a pairing block and will not enter the spectrum access scheduling queue.
[0055] S2. In each time slot, update the link status and construct a queue to be scheduled. The queue to be scheduled consists of migrating links, blocked waiting links and newly added links connected in series according to migration priority. Link state It means that among them This indicates whether the link is complete or inactive. This indicates that the link is transmitting data. This indicates that the link is in a waiting-for-rescheduling state. Link in time slot The remaining amount of data to be transmitted is denoted as The set of links that were transmitting in the previous time slot was... (8) Assume analog voice service in channel and time slot The actual occupancy status is ,in This indicates that the channel is being used by a simulated service. This indicates the channel is idle. If the link... The spectrum blocks occupied in the previous successful access operation were Then the simulated service intrusion indicator variable for the current time slot is (9) when At that time, link Release the original spectrum block and enter the migration queue; when At this time, the link continues to use the previous time slot LCID and channel block configuration, and transmits data packets of the following formula size: (10) If the amount of data remaining after transmission is zero, the link is complete. Therefore, time slots The links that need to be rescheduled consist of three types of queues: the migration link set, the blocked waiting link set, and the new link set, denoted as [list of queues]. (11) This invention employs a migration-first scheduling strategy, namely... (12) in This indicates a concatenation operation that maintains queue order. After processing each link, the system immediately updates the digital link occupancy vector to ensure that subsequent links do not conflict with already allocated digital spectrum blocks.
[0056] S3. Take out digital links one by one from the queue to be scheduled, input the current link's higher-layer status into the higher-layer DQN, and output the link configuration identifier LCID according to the greedy strategy. The output action of the higher-layer DQN is constrained by the mask formed by the signal-to-noise ratio threshold. High-level strategy: LCID selection No. Link in time slot The high-level state is defined as (13) in, Represents the normalized amount of data to be transmitted. One-hot encoding is used for link queue types, corresponding to newly added links, blocked waiting links, and migrated links. Let be the normalized signal-to-noise ratio margin vector, where (14) This represents the maximum predictive access confidence that can be obtained for different LCIDs under the current digital occupancy state, and its i.e., the Each component is (15) If there are no candidate starting points that satisfy the channel range and digital mutual exclusion constraints, then let . and These are the normalized codes for the LCID number and channel start point in the previous successful access action, respectively, when the link If no successful access record is found, the value is 0. This indicates whether a simulated service intrusion occurred in the previous time slot.
[0057] High-level actions are defined as (16) The LCID selection result based on high-level DQN is: (17) in, For high-level actions, value function These are the parameters for the higher-level main network. The corresponding LCID selection indicator variable is... (18) Define digital link occupancy vector This vector consists of two parts: first, the historical link occupancy in the current time slot that has not been intruded upon by simulated services and continues to transmit; and second, the link occupancy in the current time slot that has higher priority and has already been scheduled. Its elements... Channel It has been occupied by the digital link. This indicates that the channel is not occupied. The set of possible actions for higher-layer LCID is given by the signal-to-noise ratio threshold: (19) S4. Input the low-level state of the same link into the low-level DQN in the same time slot, and output the channel start or blocking action according to the greedy strategy. The low-level state includes the LCID information output by S3, the predicted access confidence vector of all candidate start frequencies under the LCID, the actual occupancy status of the current time slot analog service in each channel, and the digital link occupancy status before scheduling the current link. The output action of the low-level DQN is subject to the mask constraint composed of four constraints: channel range, analog voice occupancy, digital link mutual exclusion, and predicted access confidence threshold. In maritime hybrid analog-digital communication systems, before a digital link connects to a real channel, it is necessary to assess the likelihood that the selected spectrum block will remain idle for a period of time. Assume the spectrum prediction module is in the decision slot... Output all channels in the future The idle probability of each time slot is represented by a three-dimensional idle probability vector. (20) in Indicates in time slot When making decisions, the prediction channel Future Time Slot This represents the probability of being idle. For a link... Using LCID And from the channel Start accessing continuous spectrum blocks Its predicted time slot length is defined as (twenty one) in, Representative link Using LCID The number of time slots required to transmit the remaining data. The spectrum accessibility confidence is calculated using geometric mean aggregation and can be expressed as... (twenty two) in, To prevent logarithmically singular positive numbers, a geometric mean is used. This is because directly multiplying the prediction time domain and continuous channel width simultaneously would excessively suppress the confidence of higher-order modulation and coding schemes within a longer prediction window, hindering fair comparisons between different bandwidth LCID configurations. Accordingly, the prediction collision risk is defined as... (twenty three) The blocking action does not occupy the real spectrum, its access confidence is defined as 0, and the predicted collision risk is defined as 1.
[0058] Low-layer strategy: Spectrum access Low-level strategies determine LCID at higher levels Then select the real channel starting point or the blocking action. Link in time slot The lower-level state is defined as (twenty four) in, One-hot encoding for the selected LCID, For LCID The predicted access confidence vector of all channel starting points. This represents the current simulated actual service usage status. For scheduling links The occupancy status of the digital link in the previous time slot.
[0059] Low-level actions are defined as (25) During the utilization phase, the lower-layer DQN outputs the decision access channel. (26) like If the link does not obtain actual spectrum resources, it enters a waiting state; if Then the link access And send in the current time slot (27) The corresponding channel access indicator variable is (28) Given LCID Subsequently, the set of low-level channel access actions is as follows: (29) in This is the minimum confidence threshold for accessibility. This represents a blocking action. Blocking actions do not occupy a real channel and are always retained in the lower-level action set to ensure that the lower-level decision set is not empty. Let... and These represent the actual choices made by the higher and lower levels, respectively, with the binary decision variables for the higher and lower levels being respectively... (30) in, link In the time slot Select LCID Indicator variables, Representative link In the time slot Select Channel The indicator variable for initial access must satisfy the single-link two-layer decision-making process. (31) And have the right to act without action (31) The above constraints are applied directly to the Q value of the DQN output through high-level and low-level action masks, so that inactive actions are invalidated during the selection process.
[0060] S5. Execute spectrum access or blocking based on the output of S4: If blocking, no spectrum is allocated to the current time slot; if channel start, data transmission is performed using continuous spectrum blocks configured according to the LCID output of S3; calculate the high-level instant reward and low-level instant reward according to the hierarchical reward function, weight the high-level instant reward and joint reward and store them as the high-level training reward in the high-level experience replay pool, and weight the low-level instant reward and joint reward and store them as the low-level training reward in the low-level experience replay pool. The joint reward is synthesized from the high-level instant reward and low-level instant reward; when the link enters the migration queue due to simulated voice intrusion, calculate the high-level migration penalty and low-level migration penalty respectively, attenuate the high-level migration penalty by delay discount and add it to the high-level reward of the corresponding historical experience sample, and attenuate the low-level migration penalty by delay discount and add it to the low-level reward of the corresponding historical experience sample; update the network parameters of the high-level DQN and low-level DQN, and enter the next time slot until training is completed.
[0061] This invention designs high-level rewards, low-level rewards, and combined rewards. High-level rewards emphasize the potential throughput and accessibility brought by LCID selection, while low-level rewards emphasize the actual throughput after actual access, the confidence level of the selected channel, and the avoidance of blocking penalties. Let... This represents the maximum number of bits transmitted per time slot across all LCIDs. This is the threshold for the higher layer to select an LCID. Then, the potential number of transmittable bits for this LCID is (32) The actual normalized throughput is (33) When blocked Define LCID An indicator variable indicating whether a real accessible channel block exists. (34) If a truly accessible channel block exists, the optimal access confidence level that higher layers can perceive is: (35) If it does not exist, then take it. For the actual selection action at the lower level, define... (36) make This indicates a blocking indicator variable. The high-level instant reward is defined as... (37) Low-level instant rewards are defined as (38) in, , , , These are the reward and punishment parameters, respectively. The penalty term represents the penalty for choosing to block when an accessible channel exists. Equation (15) emphasizes the impact of higher-layer LCID selection on potential throughput and accessibility, and penalizes the absence of a real access opportunity and blocking; Equation (16) emphasizes whether the actual channel selection at the lower layer brings real transmission and high-confidence access, and only penalizes avoidable blocking. The higher-layer reward and the lower-layer reward are linearly combined into a joint reward, expressed as follows: (39) To inject joint objective information into each layer while maintaining the difference in rewards between upper and lower layers, the actual training reward written into the experience replay pool is... (40) in This represents the combined reward coefficient. It should be specifically noted that this paper designs the migration penalty as a delayed attribution mechanism. If the link... In the time slot If a link enters the migration queue due to an intrusion of analog voice services, this migration event does not penalize the current recovery action. Instead, it penalizes the historical action that previously enabled successful access to the intruded spectrum block. This ensures that the action that actually caused the collision event receives the appropriate penalty. Assume that the channel access action that caused the collision occurred in a time slot. The access confidence level at that time was The high-level and low-level delayed migration penalties are respectively (41) in and This is the migration penalty coefficient. and To penalize the scaling factor, This is the delay discount factor. (Function) This is assigned a risk weight. Accordingly, the historical action reward is updated to... (42) If the corresponding experience is still pending or has not yet been covered by the replay pool, the reward value in the experience replay pool is synchronously corrected. This design attributes migration consequences to historical spectrum access actions that lead to future intrusion risks, rather than penalizing migration actions performed in the current time slot to restore communication, thus making reward calculation more consistent with the dynamic spectrum access process. Based on the higher-layer LCID selection strategy and the lower-layer channel access strategy, the time slot... The combined action can be represented as (43) The overall layered strategy can be decomposed into (44) in, Given the set of parameters for both high-level and low-level networks, the joint reward of the system is defined as follows: (45) Therefore, the long-term optimization objective of the hierarchical deep Q-network is (46) Example In this embodiment, a hybrid VHF digital-analog communication scenario at sea is considered. Analog VHF voice service, as the highest priority primary service, must not conflict with the digital ship communication link when occupying a channel. The digital ship communication link can only opportunistically access continuous spectrum blocks that are currently not occupied by analog voice and are predicted to remain idle for a relatively long period. When the spectrum block occupied by the currently transmitting digital link is re-occupied by analog voice service, the digital link must immediately release its original channel resources and enter a migration queue, whereby the method proposed in this invention reselects the LCID and accesses the channel.
[0062] This embodiment employs a hierarchical deep Q network for spectrum access decisions. The higher-layer DQN is responsible for selecting the VDE-TER link configuration LCID based on link traffic volume, channel quality, historical migration status, and spectrum prediction results. The lower-layer DQN, after determining the LCID at the higher layer, further selects the actual channel start point or blocking action to complete spectrum access or link migration. Higher-layer and lower-layer decisions are executed sequentially within the same time slot, with the lower-layer access result inversely influencing the higher-layer reward, enabling the higher layer to learn the long-term coupling relationship between LCID selection and spectrum accessibility. Simultaneously, this embodiment includes a delayed migration penalty mechanism. Only when a simulated voice intrusion event occurs and triggers migration is the migration penalty retroactively attributed to a channel access action that historically caused the link to occupy a risky spectrum block, without penalizing the migration action used to restore communication in the current time slot.
[0063] In this embodiment, the maritime VHF band is divided into several basic channels, each with a bandwidth of [missing information]. The system operates in TDMA mode, with each time slot having a length of [length missing]. The LCID candidate set is as follows: Different LCIDs correspond to different channel bandwidths, modulation schemes, coding rates, number of bits that can be transmitted per time slot, and minimum decoding signal-to-noise ratio thresholds. The data packet size, service arrival process, ship position, link signal-to-noise ratio, and analog voice channel occupancy status of the digital ship link all change dynamically over time slots.
[0064] The specific implementation steps are as follows.
[0065] Step 1: Initialize environment and network parameters.
[0066] Initialize the location of digital ship nodes in the current sea area, simulate the actual occupancy status of voice channels, and the future idle probability matrix output by the spectrum prediction module.
[0067] Initialize the main Q network parameters during the high-level LCID selection phase. and target Q network parameters Initialize the main Q network parameters during the low-level channel access phase. and target Q network parameters The input to the higher-layer DQN is the higher-layer state vector, and the output is the Q value of each LCID action; the input to the lower-layer DQN is the lower-layer state vector, and the output is the Q value of each channel start action and blocking action.
[0068] Initialize the high-level experience replay pool and low-level experience replay pool Set the total number of training rounds. Maximum number of time slots per round Batch sampling size High-level discount factor Low-level discount factor Learning rate Initial value of exploration rate Exploration rate lower limit Target network synchronization period Minimum spectrum access confidence threshold Predicted time domain length High and low level reward mixing coefficient and , and set time slot count .
[0069] Step 2: Environment reset.
[0070] After resetting the environment, clear the rewards and experience pool. and .
[0071] Step 3: Update the link status and construct the queue to be scheduled.
[0072] In each time slot Update the link status and construct the queue to be scheduled.
[0073] (1) If no analog service intrusion occurred on the digital link that was transmitting in the previous time slot, then the link will continue to use the LCID and channel block configuration of the previous time slot. If the remaining data volume after transmission If the value is zero, the link completes the transmission and releases the corresponding digital spectrum resources; otherwise, the link continues to maintain the transmission state.
[0074] (2) For a digital link that was transmitting in the previous time slot, if at least one channel in its occupied spectrum block is reoccupied by analog voice service, it is determined that an analog service intrusion has occurred on the link, and it is added to the migration link set. And call the delayed migration penalty function delayed_migration_penalty().
[0075] (3) Read the blocked waiting link .
[0076] (4) The new link adopts the Poisson distribution model, assuming that in each time slot... If a digital service arrives and forms a digital link, then that link is included in the newly added link. .
[0077] (5) Subsequently, the system will form a scheduling queue of links that need to be rescheduled in this time slot according to the priority order of migrating links, blocked waiting links, and newly added links:
[0078] in, This indicates a concatenation operation that maintains the internal order of the queue.
[0079] Step 4: Perform hierarchical decision-making for each link in the scheduling queue.
[0080] For scheduling queues Each link in Perform the following operations: (1) Call the step_hl() function to obtain the optimal LCID configuration action. ; (2) Call the step_ll() function to obtain the optimal channel access action. ; (3) If For blocking actions, update the current link. The type is a blocking link, and the amount of data sent in the current time slot is 0; (4) If Access to real channels, according to Configure spectrum blocks and transmit data. If the remaining data packets are 0 after the current time slot transmission is completed, the transmission is complete; otherwise, update the current link. The type is a link in transit, and will , The decision is passed to the delayed migration penalty function delayed_migration_penalty() and saved.
[0081] (5) Update the high / low layer instant rewards, high / low layer training rewards and combined rewards respectively according to the formula.
[0082] Step 5: Update network parameters and proceed to the next time slot.
[0083] Once the number of samples in the high-level experience pool and the low-level experience pool reaches the training requirement, the system will then... and Randomly sample a small batch of samples, then minimize the TD error loss functions of the high-level and low-level layers respectively, and update... and Every time I pass by In each training step, the parameters of the high- and low-level main networks are synchronized to the high- and low-level target networks, respectively. If the current time slot has not reached the maximum number of time slots in a single round and there are still unfinished services in the system, return to step three; otherwise, end the current round. If the number of training rounds has not reached... If the environment is reset, the next round begins; when the number of training rounds reaches [number missing], the training environment is reset and the next round begins. Training ends when the time is right.
[0084] In this embodiment, the specific process of the high-level step function step_hl() is as follows: HL_S1: Read link The remaining traffic volume, link signal-to-noise ratio, historical access records, queue type, and spectrum prediction results.
[0085] HL_S2: Generates a high-level action mask based on the lowest decoding signal-to-noise ratio threshold of each LCID, and marks LCIDs that do not meet the signal-to-noise ratio condition as non-actionable.
[0086] HL_S3: For each feasible LCID, calculate the maximum predictive access confidence that can be obtained under the current digital link occupancy status.
[0087] HL_S4: Input the high-level state into the high-level DQN, according to... The -greedy strategy outputs the LCID to select an action.
[0088] HL_S5: Outputs the selected LCID and its bandwidth, the required number of consecutive channels, the number of bits that can be transmitted per timeslot, and the signal-to-noise ratio threshold.
[0089] In this embodiment, the specific process of the low-level step function step_ll() is as follows: LL_S1: Calculates the required number of consecutive basic channels based on the LCID output from the higher layer, and enumerates all possible channel starting points.
[0090] LL_S2: For each candidate channel starting point, check the channel range constraints, analog voice protection constraints, digital link mutual exclusion constraints, and minimum predicted access confidence constraints.
[0091] LL_S3: Generates a low-level action mask and adds blocking actions to the low-level action set.
[0092] LL_S4: Input the low-level state into the low-level DQN, according to... The -greedy strategy outputs the channel start point or blocking action.
[0093] In this embodiment, the specific process of the delayed migration penalty function delayed_migration_penalty() is as follows: DM_S1: When a link is detected When the currently occupied spectrum block is intruded by analog voice services, read the most recent successful access record of that link.
[0094] DM_S2: Determine whether the experience corresponding to the successful access record still exists in the high-level experience pool and the low-level experience pool; if it does not exist, only record the migration event and do not modify the experience pool.
[0095] DM_S3: If the corresponding experience still exists, calculate the higher-layer migration penalty and lower-layer migration penalty based on the historical access action occurrence time slot, the current intrusion time slot, the historical access confidence level, and the delay discount factor.
[0096] DM_S4: Write the calculated penalty back to the reward fields of the corresponding high-level experience and low-level experience, respectively.
[0097] DM_S5: Link Add to the migration queue and wait for subsequent time slots to be rescheduled according to the migration priority strategy.
[0098] Through the above embodiments, this invention enables adaptive LCID selection, opportunistic channel access, and dynamic spectrum migration for digital ship communication links while prioritizing analog voice service protection. Compared to the integrated decision-making approach of traditional modulation and coding scheme selection and channel access, hierarchical DQN decomposes the complex action space into two sub-problems: high-level LCID selection and low-level channel access, reducing the decision dimensionality. Simultaneously, through a joint reward and delay migration penalty mechanism, the algorithm can learn the trade-offs between high-throughput LCID, spectrum occupancy width, access confidence, blocking probability, and future migration risk, thereby improving spectrum utilization efficiency and digital service access stability in maritime VHF hybrid analog-digital communication scenarios.
[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for maritime VHF spectrum resource collaborative allocation based on hierarchical deep Q network, characterized in that, Includes the following steps: S1. Initialize system and network parameters; S2. In each time slot, update the link status and construct a queue to be scheduled. The queue to be scheduled consists of migrating links, blocked waiting links and newly added links connected in series according to migration priority. S3. Take out digital links one by one from the queue to be scheduled, input the current link's high-layer status into the high-layer DQN, and output the link configuration identifier LCID according to the high-layer instant reward. The output action of the high-layer DQN is constrained by the mask formed by the signal-to-noise ratio threshold. S4. Input the low-level state of the same link into the low-level DQN in the same time slot, and output the channel start or blocking action according to the low-level instant reward. The low-level state includes the LCID information output by S3, the predicted access confidence vector of all candidate start frequencies under the LCID, the actual occupancy status of the current time slot analog service in each channel, and the digital link occupancy status before scheduling the current link. The output action of the low-level DQN is subject to the mask constraint composed of four constraints: channel range, analog voice occupancy, digital link mutual exclusion, and predicted access confidence threshold. S5. Execute spectrum access or blocking according to the output of S4: If blocking, no spectrum is allocated to the current time slot; if channel start, data transmission is performed by configuring continuous spectrum blocks according to the LCID output of S3; calculate the high-level instant reward and low-level instant reward according to the hierarchical reward function, and store the high-level instant reward and joint reward as the high-level training reward in the high-level experience replay pool, and store the low-level instant reward and joint reward as the low-level training reward in the low-level experience replay pool. The joint reward is synthesized from the high-level instant reward and the low-level instant reward. When a link enters the migration queue due to simulated voice intrusion, the higher-level migration penalty and the lower-level migration penalty are calculated separately. The higher-level migration penalty is attenuated by delay discount and then added to the higher-level reward of the corresponding historical experience sample. The lower-level migration penalty is attenuated by delay discount and then added to the lower-level reward of the corresponding historical experience sample. Update the network parameters of the high-level DQN and low-level DQN, move to the next time slot until training is complete, and output the trained maritime VHF spectrum resource collaborative allocation model based on hierarchical deep Q network. S6. Input the set of links to be scheduled into the trained maritime VHF spectrum resource collaborative allocation model based on hierarchical deep Q network. The trained maritime VHF spectrum resource collaborative allocation model based on hierarchical deep Q network executes S1-S4 and outputs the optimal link configuration identifier LCID and the optimal access channel start point or blocking action.
2. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S1, the system parameters include: The maritime VHF band is divided into There are 1 basic channel, and the channel set is as follows: Each channel has a bandwidth of The system operates in TDMA mode, with a time slot set of [missing information]. Each time slot is [length] The maximum number of concurrent links is ;link The path loss of the maritime channel is defined as: in, Represents frequency, The mean square root of ocean waves, For reference distance The free space loss value under the following conditions To conform to a mean of 0 and a standard deviation of A normally distributed random variable, The definition is , The definition is , Represents the signal wavelength; the LCID candidate set is defined as follows: For the first LCID configuration, defining physical layer parameters ,in To consume bandwidth, For the required number of basic channels, For signed numbers, The modulation order, For coding rate, This is the minimum decoding signal-to-noise ratio threshold. The number of bits that can be transmitted per time slot is expressed as .
3. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S2, the newly added link is generated through time-slotted service arrival and candidate vessel pair screening, specifically including the following steps: Digital service arrival adopts a time-slotted Poisson model, assuming the first... The average arrival intensity of each time slot is The number of newly arriving digital services is The size of each new service data packet follows a uniform distribution. Set time slots The collection of idle ships is Any candidate ship The distance is The effective signal-to-noise ratio of the ship pair is then... The set of candidate ships is represented as: when At that time, new links are selected uniformly and randomly from the candidate set, that is: If the candidate set is empty, the arrival of this service is recorded as a pairing block and will not enter the spectrum access scheduling queue.
4. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S2, the update of the link state specifically includes the following steps: Link state It means that, among them This indicates whether the link is complete or inactive. This indicates that the link is transmitting data. This indicates that the link is in a waiting-for-rescheduling state; the first Link in time slot The remaining amount of data to be transmitted is denoted as ; The set of links that were transmitting in the previous time slot is: Assume analog voice service in channel and time slot The actual occupancy status is ,in This indicates that the channel is being used by a simulated service. Indicates the channel is idle; if the link is... The spectrum blocks occupied in the previous successful access operation were Then the simulated service intrusion indicator variable for the current time slot is: when At that time, link Release the original spectrum block and enter the migration queue; when At that time, the link continues to use the previous time slot configuration and transmit. Data packets of a certain size.
5. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S4, the method for calculating the predicted access confidence is as follows: Assume the spectrum prediction module is in the decision time slot Output all channels in the future The idle probability of each time slot is represented by a three-dimensional idle probability vector: in, Indicates in time slot When making decisions, the prediction channel Future Time Slot The probability of being idle; for links Using LCID And from the channel Start accessing continuous spectrum blocks Its prediction time domain length is defined as: in, Representative link Using LCID The number of time slots required to transmit the remaining data; the spectrum accessibility confidence is calculated using geometric mean aggregation and expressed as: in, To prevent positive numbers with singular logarithms; Predicted collision risk is defined as: The access confidence level for blocking actions is defined as 0, and the predicted collision risk is defined as 1.
6. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S3 and S4, the mask construction method specifically includes the following steps: Define digital link occupancy vector Its elements Channel It has been occupied by the digital link. This indicates that the channel is not occupied; the set of possible actions for higher-layer LCID is given by the signal-to-noise ratio threshold: Given LCID Subsequently, the set of low-level channel access actions is as follows: in, This is the minimum confidence threshold for accessibility. This represents a blocking action; blocking actions do not occupy a real channel and are always retained in the lower-level action set; let... and Let these represent the actual selection actions of the higher and lower layers, respectively. Then, the single-link two-layer decision-making process satisfies: And regarding actions that cannot be taken: 。 7. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S3, the high-level state is constructed as follows: No. Link in time slot The high-level state is defined as: in, Represents the normalized amount of data to be transmitted. One-hot encoding for linked queue type, Let be the normalized signal-to-noise ratio margin vector, where: This represents the maximum predictive access confidence that can be obtained for different LCIDs under the current digital occupancy state, and its i.e., the The components are: If there are no candidate starting points that satisfy the constraints, then let ; and These are the normalized codes for the LCID number and channel start point in the previous successful access action, respectively. This indicates whether a simulated service intrusion occurred in the previous time slot.
8. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S4, the lower-level state is constructed as follows: No. Link in time slot The lower-level state is defined as: in, One-hot encoding for the selected LCID, For LCID The predicted access confidence vector of all channel starting points. This represents the current simulated actual service usage status. For scheduling links The occupancy status of the digital link in the previous time slot.
9. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, In S5, the calculation methods for the high-level instant reward and the low-level instant reward are as follows: set up The maximum number of bits transmitted per time slot across all LCIDs; when the higher layer selects LCID. Then, the potential number of transmittable bits for this LCID is: The actual normalized throughput is: Among them, when blocked ; Define LCID An indicator variable indicating whether a real accessible channel block exists: If it exists, the optimal access confidence level is: If it does not exist, then take it. ; For the actual selection action at the lower level, the definition is: make The variable represents the blocking indicator; the high-level immediate reward is defined as: Low-level instant rewards are defined as: in, , , , For reward and punishment parameters, The penalty item represents the punishment for choosing to block when there is a channel available for access.
10. The method for collaborative allocation of marine VHF spectrum resources based on hierarchical deep Q-networks according to claim 1, characterized in that, The combined reward and the synthesis method of high-level training reward and low-level training reward described in S5 are as follows: The high-level reward and the low-level reward are linearly combined into a joint reward, expressed as follows: The actual training reward written into the experience replay pool is: in, This is the combined reward coefficient.