A reinforcement learning-based dynamic obstacle avoidance optimization method for wireless channels

Abstract

This application relates to the field of dynamic channel avoidance, specifically disclosing a wireless channel dynamic avoidance optimization method based on reinforcement learning. First, the instantaneous signal strength and sleep state of heterogeneous terminals are time-sensitively labeled, and a time-sensitive feature matrix is ​​constructed to accurately identify outdated data. Further, digital shadow inference technology is introduced to virtually complete the features of expired sleep nodes, and spatial dimension reorganization is performed with the real-time data of active nodes to generate a spatiotemporally aligned state tensor. Subsequently, a deep reinforcement learning network is used to evaluate this aligned tensor to extract the optimal channel action, and asynchronous distribution scheduling is performed based on the node sleep state in the state buffer block. Immediate and delayed instructions are sent to broadband and narrowband terminals respectively via real-time and suspended queues, effectively avoiding the disconnection and isolation effect caused by blind channel switching.

Description

Technical Field

[0001] This application relates to the field of dynamic channel avoidance, and more specifically, to a wireless channel dynamic avoidance optimization method based on reinforcement learning. Background Technology

[0002] With the rapid development of the Industrial Internet of Things (IIoT) and wireless communication technologies, massive numbers of heterogeneous radio frequency (RF) terminals are often densely deployed in complex application scenarios such as substations and smart factories. Wireless spectrum resources in these environments are becoming increasingly congested, and electromagnetic interference exhibits high dynamism and suddenness. Traditional channel allocation and switching methods based on fixed rules or static thresholds are ill-suited to rapidly changing network conditions, easily leading to a sharp decline in communication link quality. Reinforcement learning technology, with its powerful environmental interaction and dynamic decision-making capabilities, can autonomously learn the optimal frequency band switching strategy through a trial-and-error mechanism. Therefore, constructing a dynamic wireless channel avoidance optimization scheme based on reinforcement learning has become a key technological evolution direction for improving the anti-interference capability of heterogeneous network communication and ensuring data transmission stability.

[0003] However, existing reinforcement learning-based dynamic channel avoidance schemes for wireless networks suffer from serious inherent limitations when deployed in real-world heterogeneous networks. Existing reinforcement learning schemes are primarily based on Markov Decision Processes (MDPs), whose theoretical foundation is the instantaneous completeness of the input state. In real-world heterogeneous scenarios, broadband devices such as video surveillance are typically powered continuously and possess the ability to perceive channel interference in real time; while low-power narrowband devices, such as partial discharge devices and temperature and humidity sensors, rely on battery power and must adhere to a strict sleep-wake-communication-sleep cycle. When constructing environmental features, existing schemes often simply structure and package the data from all devices and input it into the network. Due to the lack of spatiotemporal alignment mechanisms, dormant narrowband nodes report outdated observation data, while broadband nodes report millisecond-level new data. This forced spatiotemporal misalignment of heterogeneous data leads the reinforcement learning network to perceive a phantom state that does not exist in the physical world, resulting in incorrect assessments and decisions. Furthermore, existing schemes typically employ a one-size-fits-all, instant command issuance mode when executing decisions; active broadband devices immediately switch channels, while dormant narrowband devices miss the command. When a narrowband device is woken up on an old channel, it will become an isolated node because the base station has already moved to a new channel, resulting in a complete break in the data link in the industrial field.

[0004] This contradiction between asynchronous perception and one-size-fits-all instruction execution urgently requires a completely new optimization mechanism to resolve. Summary of the Invention

[0005] To address the aforementioned technical challenges, this application is proposed. According to this application, a reinforcement learning-based dynamic obstacle avoidance optimization method for wireless channels includes: Step 1: Perform time-sensitivity marking and multi-dimensional structured encapsulation on the instantaneous signal strength and sleep / wake cycle state of each heterogeneous node in the collected heterogeneous radio frequency raw data to obtain the time-sensitivity feature matrix. Step 2: Perform digital shadow inference to complete the outdated feature dimensions in the time-sensitive feature matrix that have expired due to node dormancy to obtain a virtual complete feature set. Use the real-time observations of the peripheral active nodes as external observation correction inputs, and reorganize the virtual complete feature set and real-time features in spatial dimensions to obtain a spatiotemporally aligned state tensor. Step 3: Use a deep reinforcement learning network to perform reinforcement learning time-varying action evaluation on the spatiotemporal aligned state tensor to obtain the channel value vector representing the switching value of each candidate channel. At the same time, copy and store the current input state snapshot as a state cache block. Step 4: Extract the optimal channel action from the channel value vector and perform asynchronous distribution and scheduling based on the node sleep state in the state buffer block. In this step, the immediate instruction and the delayed instruction are sent to the broadband and narrowband terminals through the real-time queue and the suspended queue, respectively.

[0006] Compared with existing technologies, this application provides a reinforcement learning-based dynamic channel avoidance optimization method for wireless networks, addressing the issues of reinforcement learning state phantoms caused by asynchronous perception in heterogeneous communication networks and narrowband node isolation caused by one-size-fits-all command switching. First, the instantaneous signal strength and sleep state of heterogeneous terminals are time-sensitively labeled, constructing a time-sensitive feature matrix to accurately identify outdated data. To eliminate spatiotemporal misalignment, digital shadow inference technology is further introduced to virtually complete the features of expired sleep nodes, and spatially reorganize them with real-time data from active nodes to generate a spatiotemporally aligned state tensor. This provides a realistic and reliable complete environment input for the reinforcement learning network, completely resolving the state phantom problem. Subsequently, a deep reinforcement learning network is used to evaluate this aligned tensor to extract the optimal channel action, and asynchronous distribution scheduling is performed based on the node sleep states in the state buffer block. Immediate and delayed commands are sent to broadband and narrowband terminals respectively via real-time and suspended queues, effectively avoiding the disconnection and isolation effect caused by blind channel switching. Attached Figure Description

[0007] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0008] Figure 1 This is a flowchart of a reinforcement learning-based wireless channel dynamic avoidance optimization method according to an embodiment of this application.

[0009] Figure 2 This is a schematic diagram of data flow for a reinforcement learning-based wireless channel dynamic avoidance optimization method according to an embodiment of this application.

[0010] Figure 3 This is a flowchart of step 2 in the reinforcement learning-based wireless channel dynamic avoidance optimization method according to an embodiment of this application.

[0011] Figure 4 This is a flowchart of step 4 in the reinforcement learning-based wireless channel dynamic avoidance optimization method according to an embodiment of this application. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] To address the shortcomings in the aforementioned technical fields, a dynamic obstacle avoidance optimization method for wireless channels based on reinforcement learning is proposed. Figure 1 This is a flowchart of a reinforcement learning-based wireless channel dynamic avoidance optimization method according to an embodiment of this application. Figure 2 This is a schematic diagram of data flow for a reinforcement learning-based wireless channel dynamic obstacle avoidance optimization method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the wireless channel dynamic avoidance optimization method based on reinforcement learning includes: Step 1, performing time-sensitive marking and multi-dimensional structured encapsulation on the instantaneous signal strength and sleep / wake cycle state of each heterogeneous node in the collected heterogeneous radio frequency raw data to obtain a time-sensitive feature matrix; Step 2, performing digital shadow inference to complete the outdated feature dimensions in the time-sensitive feature matrix due to node sleep to obtain a virtual complete feature set, and using the real-time observations of peripheral active nodes as external observation correction inputs, and spatially reorganizing the virtual complete feature set and real-time features to obtain a spatiotemporal aligned state tensor; Step 3, using a deep reinforcement learning network to perform reinforcement learning time-varying action evaluation on the spatiotemporal aligned state tensor to obtain a channel value vector representing the switching value of each candidate channel, and simultaneously copying and storing the current input state snapshot as a state cache block; Step 4, extracting the optimal channel action from the channel value vector, and asynchronously distributing and scheduling it according to the node sleep state in the state cache block, wherein immediate instructions and delayed instructions are sent to broadband and narrowband terminals respectively through real-time queues and suspended queues. Step 1 involves timely marking and multi-dimensional structured encapsulation of the instantaneous signal strength and sleep / wake cycle status of each heterogeneous node in the collected heterogeneous RF raw data to obtain a time-sensitive feature matrix. Specifically, the heterogeneous RF raw data includes the amplitude of the in-phase component, the phase of the quadrature component, and the node's Media Access Control (MAC) wake-up timestamp. It is understandable that in a wireless communication environment where heterogeneous terminals coexist, different types of RF devices have completely different physical operating mechanisms. Broadband communication devices operate normally and continuously sense channel changes at high frequencies, while low-power narrowband devices strictly adhere to low duty cycle sleep / wake cycles, resulting in severe asynchronous discontinuities and spatial misalignments in the collected physical layer sensing data on the time axis. Directly inputting this unaligned, discrete, and chaotic physical sensing data into the subsequent evaluation network will severely interfere with the quality of channel feature extraction, leading to false environmental state perceptions. To completely eliminate the data fragmentation and perception misalignment caused by the differences in the operating mechanisms of heterogeneous nodes, this technical solution performs time-sensitive marking and multi-dimensional structured encapsulation of the instantaneous signal strength and sleep / wake cycle status of each heterogeneous node in the collected heterogeneous radio frequency raw data to obtain a time-sensitive feature matrix, thereby providing standardized and spatiotemporally aligned basic data support for subsequent environmental state inference.

[0014] Specifically, in an exemplary implementation, step 1 includes: synchronously acquiring the amplitude of the in-phase component and the phase of the quadrature component within each baseband sampling period, and performing instantaneous power calculation on the amplitude of the in-phase component and the phase of the quadrature component to obtain an instantaneous signal strength sequence; extracting the wake-up timestamp of the node's media access control layer as an absolute reference cursor, and performing truncation matching and sleep period invalid background noise removal on the instantaneous signal strength sequence in the time domain to obtain a time-bound tuple set; and performing dimension normalization filling and two-dimensional matrix template alignment on the time-bound tuple set through a matrix mapping engine to obtain a time-sensitive feature matrix.

[0015] The operation is detailed below: Heterogeneous RF raw data refers to the basic baseband signal data read directly from the registers of the underlying RF communication chips of each sensing node, without modification by the higher-level protocol stack. In actual working scenarios such as substations, this data specifically includes the amplitude of the in-phase component, the phase of the quadrature component, and the wake-up timestamp of the node's media access control layer. The physical layer analysis probe synchronously collects the amplitude of the in-phase component and the phase of the quadrature component in each baseband sampling cycle. Since the receiver at the underlying RF layer adopts a quadrature demodulation architecture, its output baseband signal is in complex form. Utilizing the mapping relationship between polar coordinates and rectangular coordinates in the complex domain, the two input components, the amplitude of the in-phase component and the phase of the quadrature component, are subjected to trigonometric transformation, summation of squares, and logarithmic transformation to accurately calculate the instantaneous power scalar of the baseband signal. To eliminate data glitches caused by instantaneous electromagnetic pulse interference, the multiple instantaneous power scalars calculated within the continuous sampling window are smoothed by moving average using a built-in digital low-pass filter, and the thermal noise compensation constant of the underlying RF hardware is superimposed to finally generate and output an instantaneous signal strength sequence characterizing the energy of the current electromagnetic environment. The mathematical expression for this process is as follows: in, This represents the final instantaneous signal strength scalar output at the k-th observation time after smoothing and compensation processing, expressed in decibels and milliwatts. This represents the size of the continuous sampling window during moving average processing. This value is a parameter that needs to be preset, and its setting is mainly based on the rate of change of the underlying electromagnetic environment of the target operating environment. In the scenario of high channel fluctuations in substations, in order to balance the ability to capture sudden electromagnetic interference with data smoothness, [the following parameter is used]. The default value is 64, which means that the average of 64 consecutive sampling points is used to represent the signal energy at the current moment. This is the time index for the sampling points within the sliding window. This represents the physical observation of the amplitude of the in-phase component acquired synchronously at time t. This represents the orthogonal component phase physical observations synchronously acquired at time t. In the formula... Essentially, the amplitude of the quadrature component is derived by inversely using the trigonometric function relationship between the in-phase component and the phase. Summing the squares of the two yields the absolute value of the instantaneous power of the complex signal at that sampling moment. This represents the inherent thermal noise compensation constant of the underlying receiving hardware. This constant is obtained through no-load calibration of the terminal in a microwave anechoic chamber and set to -105 dBmW. For example, considering the high dynamic range characteristics of signals in actual RF receiving links, after initial amplification by the underlying low-noise amplifier and analog-to-digital conversion matching, the baseband equivalent value of the in-phase component is approximately 50 mV. When the phase of the quadrature component is 45 degrees, the equivalent value of the quadrature component derived by mapping using the tangent function is also 50 mV. By performing a sum of squares on these two components, the integral representation of the current complex signal energy can be calculated as 5000 square millivolts. Substituting this into the first term of the physical measurement formula composed of a logarithmic function, and after conversion by a constant base of 10, a relative signal power of approximately 37 dB is obtained. Subsequently, the -105 dBmW thermal noise constant, pre-calibrated by the physical hardware in a microwave anechoic chamber, is rigorously superimposed for compensation bias. Finally, the instantaneous signal strength scalar at the current moment is calculated to be -68 dBmW, consistent with the strength of the real physical wireless environment. By continuously calculating over multiple sampling periods, a sequence of continuously fluctuating instantaneous signal strengths can be constructed.

[0016] After obtaining the continuously fluctuating instantaneous signal strength sequence, it is necessary to precisely truncate and align it in the time dimension. The time axis aligner, as the core logic unit for this step, first extracts the node's Media Access Control (MAC) wake-up timestamp reported by the low-power narrowband node during link-layer interactions. This wake-up timestamp is an absolute timestamp at the microsecond level automatically generated by the node's internal high-precision real-time clock module when the RF front-end is in receive or transmit mode, marking the precise starting point of the node's actual participation in electromagnetic environment interaction. The time axis aligner uses the extracted MAC wake-up timestamp as an absolute reference cursor to truncate and match the previously generated instantaneous signal strength sequence in the time domain, and removes invalid background noise during the sleep period. Low-power narrowband devices employ a deep sleep strategy; during their sleep period, the underlying RF chip is powered off. If signals are continuously acquired during this time, only irregular thermal background noise from the physical environment is obtained. This data is not only ineffective for channel feature extraction but also introduces serious numerical biases. Therefore, the timeline aligner reads the wake-up duration of a node based on its protocol stack configuration parameters. For example, if it knows that a local discharge sensor in a substation wakes up every 300 seconds, with each wake-up lasting 100 milliseconds, the timeline aligner uses this wake-up timestamp as the absolute starting point and strictly extracts a series of instantaneous signal strength sequences corresponding to these 100 milliseconds as valid sensing data. Simultaneously, it discards or masks the signal strength during the 299.9-second dormant period outside this wake-up interval. Through this truncation and matching mechanism, the actual electromagnetic environment sensing range of the device is accurately extracted. Subsequently, this extracted valid signal strength data is bound and concatenated with its corresponding physical timestamp using multi-dimensional key-value pairs, ultimately generating and outputting a structured, time-bound tuple set. For example, a bound tuple contains the node device's unique physical identifier, the absolute standard time when the wake-up occurs, and 20 valid instantaneous signal strength scalar values ​​of approximately -70 dB / mW extracted within these 100 milliseconds.

[0017] Subsequently, the matrix mapping engine performs dimension normalization and padding on the output time-bound tuple set, aligning it with the two-dimensional matrix template. The preset-dimensional two-dimensional matrix template is a fixed data structure pre-allocated in memory based on the maximum number of registered terminal nodes within the current control area and the historical observation time span required for environmental state deduction. Specifically, the template's row count is set to the maximum concurrent access limit in the network topology; for example, 50 rows correspond to 50 heterogeneous terminals within the area. Its column count is set to the number of historical time steps required to evaluate channel status; for example, 20 columns correspond to the most recent 20 decision periods. This two-dimensional matrix template is obtained during the control core initialization phase by reading the entire network's terminal registration device table and dynamically allocating memory space of the corresponding size based on control parameters. The matrix mapping engine's architecture includes an input parsing submodule, a spatial mapping submodule, and a time-alignment submodule.

[0018] The input parsing submodule of the matrix mapping engine first receives the time-series bound tuple set generated in the previous stage and extracts the physical time features and signal numerical features belonging to different nodes. Due to the significant differences in communication frequencies among different terminal devices—for example, broadband video surveillance nodes report dozens of continuous data per second, while low-power narrowband sensors only report valid data once every few minutes—a large number of missing data dimensions inevitably occur when mapping to a two-dimensional matrix template with the same uniform step size. To address this heterogeneity issue, the spatial mapping submodule uses the forward hold rule to fill in zeros or constants to achieve dimension normalization. Specifically, if a narrowband node is in a dormant state and no new bound tuples are input within the current processing step, the spatial mapping submodule extracts the valid instantaneous signal strength scalar value recorded by the node during its last wake-up and copies it to fill the missing matrix positions at the current moment; if the node has just joined the network and has never completed any valid reporting, it uses an extremely low noise floor constant for forced zero-value filling, thus ensuring the absolute integrity of the input data dimensions. After completing the spatial-level dimensional alignment, the temporal alignment submodule further adds time-related tags to each row of feature vectors in the reconstructed two-dimensional matrix template. The generation logic for these time-related tags is based on the difference between the current absolute time read by the control core and the original Media Access Control layer wake-up timestamp recorded in the time-bound tuple. The larger the difference, the older the currently filled data. This is achieved through an internally preset decay function, namely, introducing a standard time exponential decay model, mathematically expressed as: ,in To calculate the timeliness label weight of the output, This is the time difference (in seconds) between the current absolute time and the original data acquisition time. This is an empirical attenuation constant pre-calibrated based on the coherent time evolution law of the wireless channel in the target control domain. In this practical scenario, this constant has been precisely calibrated to 0.0095 through testing. This time difference is strictly mapped to a time-sensitivity label characterizing the attenuation weight of data freshness. For example, the timestamp of a broadband node differs from the current time by only 5 milliseconds, i.e. The value is 0.005 seconds. After substituting it into the above formula and applying a basic bit width precision limit, its timeliness label weight is assigned 0.99, indicating that the data is extremely fresh and highly reliable. However, the data filled using the forward hold rule by a certain deep sleep sensor has an original timestamp that differs from the current time by 200 seconds, meaning... The value is 200 seconds. Substituting this into the formula, we get... The timeliness label weight is reduced to 0.15, indicating that this outdated feature has extremely low reference value in subsequent analysis. Finally, the matrix mapping engine performs multi-dimensional joint encapsulation on the underlying data blocks carrying signal strength of a unified dimension and the freshness timeliness labels of each node, outputting a timeliness feature matrix that can be directly read and parsed by the downstream evaluation network.

[0019] Step 2 involves performing digital shadow inference to complete the outdated feature dimensions in the time-sensitive feature matrix due to node dormancy, resulting in expired time-sensitive labels, to obtain a virtual complete feature set. Real-time observations of active peripheral nodes are used as external observation correction inputs. The virtual complete feature set and real-time features are then spatially reorganized to obtain a spatiotemporally aligned state tensor. For example... Figure 3 As shown, Figure 3 This is a flowchart of step 2 in the reinforcement learning-based wireless channel dynamic avoidance optimization method according to an embodiment of this application. Correspondingly, after the initial data parsing and feature mapping, the constructed time-sensitive feature matrix, while achieving formal filling alignment in the matrix dimension, contains a large amount of outdated data filled using the forward hold rule due to nodes being in deep dormancy. In the rapidly changing and frequently disruptive industrial electromagnetic environment, this outdated data with severely expired time tags has long lost its ability to represent the current real-world interference situation. Reinforcement learning models based on Markov decision processes have a high theoretical dependence on the immediate completeness and authenticity of the input state. If this data with severe time discontinuities and failure characteristics is forcibly used as the environmental feature set input, the downstream evaluation network will perceive a distorted phantom state that does not exist in the physical world, leading to the blind issuance of subsequent channel switching actions and communication link breaks. To completely eliminate the asynchronous perception misalignment caused by the heterogeneous node dormancy mechanism and accurately restore the true global real-time picture of the physical environment, digital shadow inference is performed to complete the outdated feature dimensions in the time-sensitive feature matrix that have expired due to node dormancy, so as to obtain a virtual complete feature set. The real-time observations of the peripheral active nodes are used as external observation correction inputs, and the virtual complete feature set and real-time features are spatially reorganized to obtain a spatiotemporally aligned state tensor.

[0020] Specifically, in an exemplary implementation, step 2 includes: step 21, reading the time-sensitive labels of each sensing node in the time-sensitive feature matrix line by line using a state scanner, comparing the difference between the timestamp of each node and the current system absolute time with a preset effective time threshold, wherein real-time node data with a difference not exceeding the threshold is output as an active node feature set, and dormant node data with a difference exceeding the threshold and its last valid observation value are output as an old node state set; step 22, performing digital shadow Kalman inference on the old node state set and the active node feature set to obtain a virtual completion feature set; step 23, based on a global network topology dictionary, spatially merging, sorting, and splicing the active node feature set and the virtual completion feature set according to the unique physical address index of each node to obtain a spatiotemporally aligned state tensor.

[0021] The operation is detailed as follows: First, the state scanner performs row-by-row parsing and stream splitting of the input time-sensitive feature matrix. The state scanner sequentially reads the time-sensitive label and the latest data timestamp of each sensing node in the time-sensitive feature matrix output from the previous stage. The current absolute time is a nanosecond-level time reference directly provided by the high-precision hardware real-time clock module at the bottom layer of the control core, marking the exact physical time point that triggers the environmental state assessment decision. The state scanner calculates the difference between the timestamp of each node and the current absolute time to obtain the absolute lag time of the underlying sensing data of each node. Then, it performs a rigorous comparison operation with the preset effective time threshold. The preset effective time threshold is a key boundary parameter characterizing the maximum tolerance of the network control domain for the staleness of the node sensing data. The setting of this parameter is mainly based on the coherence time characteristics of the wireless channel in the target control area and the typical burst change frequency of electromagnetic interference. In scenarios with high-frequency pulse interference, such as substations, the channel state changes extremely rapidly. This threshold is preset to 100 milliseconds through historical channel fading statistical analysis. Based on the comparison calculation results, the state scanner performs a hard classification and splitting operation: For nodes whose calculated time lag difference does not exceed the 100-millisecond threshold, such as a broadband video surveillance node with constant power supply, the difference between its latest data timestamp and the current absolute time is only 5 milliseconds, indicating that the instantaneous signal strength of -68 dBmW recorded in its time-sensitivity feature matrix can fully reflect the current real channel interference situation. The state scanner directly extracts such high-frequency updated real-time node data and splits it into an active node feature set. Conversely, for nodes whose time lag difference exceeds the 100-millisecond threshold, such as a low-power narrowband partial discharge sensor that follows a deep sleep cycle, the timestamp of its last updated data is 200 seconds away from the current absolute time. The -70 dBmW value retained in the time-sensitivity feature matrix has been determined to be a severely outdated feature. The state scanner packages and peels off the data of such dormant nodes and their last valid observation value, and splits it into an outdated node state set.

[0022] Simply discarding the stale node state set from the outflow would severely deplete the input space dimension of the reinforcement learning network and impair the overall network state perspective. Therefore, it is necessary to obtain a virtual completion feature set by performing a digital shadow Kalman inference on the stale node state set and the active node feature set. In an exemplary implementation, step 22, performing a digital shadow Kalman inference on the stale node state set and the active node feature set to obtain a virtual completion feature set, includes: step 221, updating the last optimal state estimation vector in the stale node state set over time based on the state transition matrix to obtain a priori state estimates; step 222, using the real-time signal strength in the active node feature set as the external observation vector, calculating the dynamic Kalman gain through the prior error covariance matrix and the observation matrix, and performing residual weighted correction on the priori state estimates to obtain a virtual completion feature set.

[0023] The state transition matrix is ​​a discrete dynamic model matrix used to characterize the fading evolution of wireless channels within a local communication area. Its specific structure is an N×N square matrix, where N is the defined dimension of the feature state vectors. This matrix is ​​obtained before the initial deployment of the control network by deploying long-term spectrum monitoring probes within the target scenario to collect massive amounts of continuous time-series channel fading data. An autoregressive moving average model is used to fit the channel energy evolution trajectory, and the state transition evolution coefficients are extracted and pre-fixed. The inference engine extracts the last optimal state estimate vector of the node recorded in the old node state set at the last moment before entering sleep mode. It then uses the pre-constructed state transition matrix to perform iterative multiplication across time steps, thereby inferring the prior state estimate of the node at the current absolute time point. The core formula for this time update step is expressed as: .in, This represents the prior state estimation vector of the dormant node obtained purely from the evolutionary deduction of the theoretical model at the k-th decision time. The aforementioned predetermined state transition matrix contains elements that directly reflect the transition probability and natural attenuation ratio of the channel energy characteristics from the previous time step to the current time step. This represents the last optimal state estimate vector at time k-1, i.e., the last successful wake-up and state update record before the node entered hibernation. For example, if the hibernating node entered hibernation 200 seconds ago, its state vector would be... The recorded low-level signal noise floor is -70 dB / mW, after passing through the state transition matrix. Based on the extrapolation calculations of the channel's natural inertial fading model, it is possible to calculate its current prior state estimate. The drift decreased to -72 dBmW. However, this purely theoretical model lacks the ability to detect and correct sudden external electromagnetic interference, so it is necessary to introduce external real-time observation results for forced correction.

[0024] The inference engine extracts the real-time signal strength of active broadband nodes that are physically adjacent to the dormant node in the active node feature set, using this multi-dimensional data representing the latest electromagnetic fluctuations as the external observation vector of the current physical environment. The observation matrix is ​​a spatial mapping matrix used to accurately map the theoretical inference state space of the network model to the measurable external hardware observation space. The covariance matrix includes the prior error covariance matrix and the measurement noise covariance matrix; the former characterizes the uncertainty variance of the theoretical inference prior state, and the latter characterizes the measurement error variance of the underlying hardware of the active node sensors due to thermal noise. Both of these key matrices are empirical diagonal matrices pre-determined and solidified in a microwave anechoic chamber during the initialization phase of the control core through a standard Kalman filter parameter calibration process. The formula for calculating the dynamic Kalman gain and performing the final residual correction is expressed as: in, The dynamic Kalman gain matrix, calculated at the k-th decision time, is essentially a dynamic data trust weight allocation regulator that determines the operational logic's tendency to trust theoretical prior deductions versus trusting external real-time physical observations. This represents the prior error covariance matrix that was synchronously calculated and iterated in the previous time update phase; Representing the aforementioned pre-defined observation matrix, This is the algebraic transpose of the observation matrix; The measurement noise covariance matrix representing the hardware calibration; This represents the virtual disturbed state vector of the dormant node, which is generated after the entire process of simulation. This represents the external observation vector provided by the surrounding active nodes; This represents a theoretical prediction of the expected observed value based on a mapping of purely theoretical prior states to the observation space. In the specific computational logic, This constitutes a crucial physical residual term: the difference between the actual measurement results of the surrounding active nodes and the theoretically predicted results of the dormant nodes. This is achieved using dynamic Kalman gain. The residual term is weighted and amplified or reduced, and then strictly superimposed as a compensation term onto the prior state estimate. For example, if the external observation vectors of active peripheral nodes indicate that the current channel has suffered a sudden, high-power pulse interference, i.e., a surge in observation energy, the Kalman gain will superimpose this abrupt residual onto the prior estimated value of -72 dB / mW with a very high weighting, potentially increasing the corrected virtual disturbed state to -60 dB / mW. Finally, all corrected multidimensional state vectors of dormant nodes are packaged and output as a virtual complete feature set.

[0025] The global network topology dictionary is a statically mapped basic database deployed in the persistent memory area of ​​the control core. Its specific architecture includes key mapping fields such as the underlying physical address of nodes, the three-dimensional physical coordinate positioning information of nodes, the classification attributes of node hardware devices, and the corresponding preset matrix row number index. This dictionary is automatically collected and persistently constructed by the gateway central node through the flooding routing discovery protocol and ranging location calibration process during the initial stage of heterogeneous network overall network construction. First, the global network topology dictionary is read, and using the unique physical address as the primary key index, the real-time state vector of the active node feature set is matched with the inferred virtual state vector of the virtual completion feature set to accurately map and fill it into the corresponding planned row position of the global network environment state matrix, thereby completing the merging and strict sorting splicing at the physical space level to obtain a flattened two-dimensional basic matrix. Subsequently, a whitening algorithm was applied to the spliced ​​and recombined full-node hybrid data matrix for zero-mean denoising. By subtracting the statistical mean of the overall matrix features, environmental common-mode static background noise was filtered out, further highlighting the relative sudden interference peak characteristics of each local topological region. Finally, a tensor reshaping function was used, i.e., a multidimensional array transformation mapping operator based on data feature decoupling was executed. , among which input items The flattened two-dimensional fundamental matrix generated by the preceding processing. For example, if the maximum number of rows of heterogeneous sensing nodes in the global control topology is set to 50. If the number of historical observation time steps required for environmental simulation is set to 20, This represents the number of independent physical feature channels generated after spatial stripping. This function automatically extracts three types of heterogeneous features that were originally mixed in the data bits of the two-dimensional basic matrix: the real-time sensing / Kalman compensation signal strength, the calculated dynamic time-sensitive label weight, and the node's native sleep cycle identifier. These features are then stacked in parallel to each other as three independent depth data channels. In this way, the denoised two-dimensional splicing matrix is ​​expanded along the data channel dimension using high-dimensional tensor expansion, and a three-dimensional feature array with a dimension specification of 50×20×3 is reconstructed as a spatiotemporally aligned state tensor.

[0026] Step 3 involves using a deep reinforcement learning network to perform time-varying action evaluation on the spatiotemporally aligned state tensor to obtain a channel value vector representing the switching value of each candidate channel. Simultaneously, a snapshot of the input state at the current moment is copied and stored as a state cache block. It should be understood that after completing the spatiotemporal alignment deduction of the underlying data and obtaining a spatiotemporally aligned state tensor that highly approximates the real physical environment, although the perceptual asynchronous misalignment and state phantom caused by the heterogeneous node sleep mechanism has been completely eliminated from the dimension of the underlying data, traditional channel allocation mechanisms based on static rules, greedy algorithms, or fixed thresholds cannot accurately extract deep-level anti-interference switching strategies in the face of high-dimensional, complex, and nonlinearly evolving industrial radio electromagnetic environments. Conventional methods are prone to getting trapped in local optima and completely fail to consider the risk of timing discrepancies caused by the asynchronous execution of instructions by subsequent heterogeneous devices. In order to autonomously learn and extract the optimal anti-interference frequency band dynamic avoidance strategy from high-dimensional aligned data, and to provide a real and reliable state backtracking basis for the asynchronous instruction distribution and scheduling of subsequent narrowband hibernation nodes, a deep reinforcement learning network is used to perform reinforcement learning time-varying action evaluation on the spatiotemporal aligned state tensor to obtain the channel value vector representing the switching value of each candidate channel. At the same time, the current input state snapshot is copied and stored as a state cache block.

[0027] Specifically, in one exemplary implementation, step 3 includes: performing a shallow copy of the spatiotemporally aligned state tensor through a memory manager to obtain a state cache block; synchronously feeding the spatiotemporally aligned state tensor into a deep reinforcement learning network to obtain hidden layer feature vectors; feeding the hidden layer feature vectors into the state value evaluation branch and the action advantage evaluation branch respectively for parallel forward inference through the routing nodes of the competing network architecture, wherein the state value evaluation branch outputs a state value scalar representing the current environmental benchmark value, and the action advantage evaluation branch outputs an action advantage vector with the same dimension as the action space; and performing identifiability compensation and Q-value aggregation on the action advantage vector and the state value scalar to obtain a channel value vector.

[0028] The operation is detailed as follows: First, the input spatiotemporally aligned state tensor is shallowly copied using the memory manager to obtain a state cache block. Simultaneously, this spatiotemporally aligned state tensor is fed into the deep reinforcement learning network to obtain hidden feature vectors. The memory manager is a resident logical unit at the bottom layer of the control center, responsible for data flow scheduling and storage space allocation. When a high-dimensional spatiotemporally aligned state tensor is generated and ready to enter the network for forward inference, the memory manager intercepts the data flow and performs a shallow copy operation in an independent storage area using pointer indexes and memory block copy instructions. The shallow copy only copies the contents of the data matrix without changing its deep logical links in main memory, thus quickly capturing the complete environmental state image at the decision moment with extremely low computational overhead and encapsulating it as a state cache block. This state cache block is crucial; it will be persistently resided in memory, serving not only as benchmark data for offline training updates in the subsequent experience replay pool but also as the sole basis for backtracking the physical environment context of the current decision when asynchronously distributing and scheduling based on the metadata of each dormant node in subsequent stages.

[0029] While completing the caching, the memory manager formally feeds the spatiotemporally aligned state tensor into the main input port of the deep reinforcement learning network. This deep reinforcement learning network is a high-level artificial neural network architecture that includes a shared feature extraction layer, mainly composed of multiple cascaded two-dimensional convolutional layers, max pooling layers, and fully connected mapping layers. For the complex high-dimensional spatiotemporally aligned state tensor output from the previous stage, such as one consisting of 50 nodes, 20 historical observation steps, and multiple data channels, the front-end convolutional layers of the network use convolutional kernels of different sizes to perform sliding window convolution calculations in the temporal span dimension and the spatial node topology dimension, extracting the temporal correlation of local channel fading and the interference propagation features of spatially adjacent nodes. Subsequently, a modified linear unit is used as a nonlinear activation function to filter out redundant thermal noise at the bottom layer and introduce nonlinear mapping capabilities. After dimensionality reduction and condensation through multiple layers of convolution and pooling, the output multi-channel feature map is reshaped into a one-dimensional vector by a flattening operator and fed into the shared fully connected feature extraction layer. The fully connected layer performs linear combination and activation operations on its internal weight matrix and bias vector, ultimately transforming it into a highly abstract and compact hidden feature vector, such as a 256-dimensional floating-point vector. The weight matrices and bias parameters in this deep reinforcement learning network were accumulated in a simulation environment simulating real interference or a microwave anechoic chamber using Markov decision processes before formal deployment. The accumulated data, including state, action, and reward interactions, was stored in an experience replay pool. Subsequently, the parameters were solidified through tens of thousands of iterations of pre-training using an adaptive moment estimation optimization algorithm and backpropagation mechanism to minimize the temporal difference objective error.

[0030] Subsequently, the 256-dimensional hidden feature vectors are fed into the state value evaluation branch and the action advantage evaluation branch for parallel forward inference via the routing nodes of the competitive network architecture. The competitive network architecture is a two-stream neural network topology designed to decouple the inherent value of the environment from the additional benefits brought by specific actions. In traditional networks, regardless of the environment, the absolute value of each candidate channel needs to be evaluated. However, in extreme conditions such as sudden strong interference causing full-band congestion, the benefit of any channel switching action will be extremely low. If the network continues to perform redundant negative value updates for all channels, it will severely slow down the training convergence speed. The competitive network architecture receives the hidden feature vectors output from the front-end shared layer through the routing nodes and uses tensor partitioning and copying techniques to distribute them equivalently to two completely parallel independent network branches for specialized computation. The state value evaluation branch mainly consists of two cascaded fully connected networks. Its core function is to directly reduce the dimensionality of the input hidden feature vectors and aggregate them, implicitly fitting the inherent expected throughput and disturbance resistance properties of the current physical environment. This branch outputs a single-dimensional continuous real value at its end, representing a state value scalar that characterizes the current environmental baseline value. For example, a scalar value of 80 represents the basic communication quality score of the overall electromagnetic environment for the 50 heterogeneous nodes without any specific avoidance actions. Simultaneously, the action advantage assessment branch performs parallel computation on the same hidden layer feature vector. This branch also consists of multiple fully connected layers, but the number of neurons in its output layer strictly corresponds to the total number of candidate channels supported by the wireless communication protocol within the controlled area. For example, if the industrial field low-power wireless communication protocol specifies 16 orthogonal available physical channels within the 2.4 GHz band, this branch will output an action advantage vector of length 16. Each element within this vector precisely evaluates the expected relative gain or the attenuation of switching costs that a device could gain compared to the average environmental level if it actively switches to the corresponding frequency channel in the current state.

[0031] Finally, the computational logic performs discriminability compensation and formulaic value aggregation on the action advantage vector and state value scalar to obtain the final channel value vector. This step addresses the indiscernibility problem that arises when competing networks directly add state value and action advantage. If fusion is performed using only simple addition, the network cannot determine whether the increase in final reward is due to an improvement in the environment state itself or an exceptionally good channel switching action during backpropagation parameter updates. This leads to drastic oscillations in weight updates within the network, preventing convergence to a unique optimal parameter solution. To eliminate this overlapping coupling, forced de-meaning discriminability compensation is necessary.

[0032] The specific calculation operation is as follows: First, extract the values ​​of all 16 elements in the action advantage vector and calculate their arithmetic mean. Then, strictly subtract this mean from each original relative advantage value in the advantage vector. This de-meaning operation ensures that the sum of all relative advantages is strictly zero, thus establishing a stable numerical anchor point. Next, using the summation operator, add each de-meaned relative advantage value to the global baseline state value scalar output by the state value evaluation branch one by one to calculate the final absolute expected value of executing each candidate channel handover action under the current spatiotemporal alignment state. The encapsulator combines and concatenates these 16 absolute expected values ​​according to the physical frequency order, outputting a channel value vector representing the comprehensive handover value of each candidate channel. This complete forward inference and aggregation process is precisely expressed by the following core mathematical formula: The left side of the equation Represents the spatiotemporally aligned state tensor with respect to the current input. Evaluate and execute the i-th candidate channel handover action. The final comprehensive channel value obtained is a set of available orthogonal frequency points that are pre-divided and fixed in the system's physical channel frequency table within a specific operating frequency band, based on the underlying wireless communication protocol standards followed by heterogeneous terminals within the control area. This represents the set of fixed weight parameters for the shared convolutional and feature extraction layers at the front end of a deep reinforcement learning network. The weight matrix and bias parameters unique to the internal branch of the state value assessment; This represents the weight matrix and bias parameters specific to the action advantage evaluation branch. The first term on the right-hand side of the equation... This refers to the representative environmental baseline state value scalar calculated forward from the state value assessment branch. The entire portion within parentheses represents the action advantage term after identifiability compensation. The original relative advantage value output by the action advantage evaluation branch for the i-th candidate channel. This represents the total number of all feasible avoidance actions in the action space, such as the 16 channels mentioned above. This involves calculating the arithmetic mean of the relative advantage values ​​of all 16 candidate channels. Subtracting this mean from the original relative advantage values ​​completes the de-meaning operation.

[0033] Such as the state value scalar output from the front end The original advantage values ​​for channels 1, 2, and 3, output by the action advantage assessment branch, are 10, 5, and -3, respectively, with the arithmetic mean of all 16 advantage values ​​being 2. After identifiability compensation, the true relative advantage of channel 1 is corrected to 10 minus 2, equaling 8; the correction value for channel 2 is 5 minus 2, equaling 3; and the correction value for channel 3 is -3 minus 2, equaling -5. Ultimately, the aggregated comprehensive value of channel 1... Channel 2 is 80 plus 8 equals 88, channel 3 is 80 plus 3 equals 83, and channel 4 is 80 plus -5 equals 75. After a series of rigorous tensor extraction, branch splitting, and demeaning compensation aggregation operations, the final 16-dimensional channel value vector accurately characterizes the long-term expected gain of the current heterogeneous network shifting to various frequency points when facing complex interference.

[0034] In particular, the use of equal-weighted arithmetic mean for de-meaning in conventional identifiability compensation processing implies a premise severely out of touch with reality: that all candidate channels are assumed to be independent and completely equal-weighted in the advantage evaluation space. However, in the real electromagnetic environment of a substation in the 2.4 GHz industrial, scientific, and medical frequency band, this assumption deviates significantly from the objective physical laws of radio frequency signal propagation. Specifically, the single-channel bandwidth of on-site broadband video surveillance equipment can reach 20 MHz or even 40 MHz, while the single-channel bandwidth of a low-power narrowband sensing system is only 2 MHz, resulting in a high probability of overlap in their physical spectra. When a certain frequency band becomes the dominant strong interference source, its transmitted energy does not abruptly cut off at the preset frequency band boundary. Instead, limited by the limited suppression capability of the RF front-end physical filter, the sideband energy of its transmitter spectrum mask continuously diffuses and leaks into adjacent frequency bands in a non-linear decay manner with respect to the distance from the center frequency. In this context, conventional equal-weighted calculations can lead to serious decision-making biases on two levels: First, there is the baseline offset problem. Candidate channels near interference sources exhibit statistically inflated advantage values ​​due to instantaneous spectrum leakage. These values ​​are included in the calculation with equal weights, unreasonably raising the global evaluation baseline. This results in the relative advantage of truly high-quality, safe channels far from interference sources being systematically compressed after de-averaging. Second, there is the risk blind zone problem. Even if neighboring channels have a high instantaneous false advantage, the ongoing adjacent channel spectrum leakage risk at the physical level is not reflected in the final evaluation. This can easily induce the control network to select the dangerous frequency, causing terminal equipment to immediately fall into a secondary interference zone after switching. To completely overcome these difficulties, a cross-layer security bridge is established at the data level between the physical layer spectrum topology and the decision layer value assessment. By utilizing a distinguishable compensation and evaluation value aggregation mechanism based on spectrum adjacency coupling and explicit adjacent channel leakage penalty injection, the purely statistical expected indicators are upgraded to a security evaluation benchmark with embedded physical topology constraints.

[0035] Specifically, in one exemplary implementation, discriminability compensation and Q-value aggregation are performed on the action advantage vector and state value scalar to obtain the channel value vector, including: The interference source locator parses the center frequency of the dominant interference source within the current frequency band from the state buffer block and calculates the absolute spectral distance between the physical center frequency of each candidate channel and the center frequency of the dominant interference source. It should be understood that the interference sweep intensity within the physical frequency band directly depends on the absolute physical spacing between channels. This spatial topological distance needs to be used as a mandatory prior input for subsequent coupling metrics to accurately quantify the theoretical sweep range of each candidate frequency point in the entire network affected by the current core interference radiation. In specific implementation, the interference source locator deeply analyzes the state buffer block residing in the independent memory area to extract the parameters of the dominant interference frequency point currently causing severe channel congestion. Subsequently, based on the preset physical frequency dictionary of all 16 candidate channels in the action space, the absolute distance calculation is performed one by one. The core formula is: In this formula, The absolute spectral distance between the i-th candidate channel and the dominant interference source is represented by megahertz; The center frequency scalar represents the i-th candidate channel fixed in the physical frequency dictionary; This represents the center frequency scalar of the dominant interference source, as parsed from the state buffer block. For example, when a broadband interference source is identified as occupying channel 6 and its center frequency... At 2.437 GHz, for the first channel, its center frequency is... The absolute spectral distance is 2.412 GHz. Substituting this into the formula, we can obtain its absolute spectral distance. Up to 25 MHz; and for the 5th channel, its center frequency... The frequency is 2.432 GHz, and its absolute distance is calculated. It is only 5 megahertz.

[0036] The absolute spectral distance is mapped to the original coupling coefficients and then globally normalized to obtain the spectral coupling coefficient vector. Correspondingly, a purely linear numerical distance cannot accurately reflect the nonlinear energy leakage characteristics of the RF transmitter; therefore, a specific nonlinear kernel function is needed to transform it to conform to hardware characteristics, eliminating the dimensional differences in the frequency domain and providing a directly usable and safe weight allocation matrix that conforms to the sideband roll-off law for the reinforcement learning evaluation layer. The coupling modeler performs the key nonlinear mapping based on a Gaussian kernel attenuation function. The bell-shaped feature of the Gaussian kernel perfectly matches the attenuation characteristics of the physical transmitter's spectral mask. The mapping formula is expressed as follows: in is the original coupling coefficient of the i-th candidate channel after Gaussian kernel attenuation mapping. Its value is strictly limited to 0 and 1. The larger the value, the more fatal the corresponding channel is to the main interference adjacent channel leakage. This is the absolute spectral distance calculated from the preceding sequence; This is a Gaussian kernel bandwidth control parameter, corresponding to the sideband roll-off rate characteristic width of the interfering source transmitter. Its physical meaning corresponds to the sideband roll-off rate characteristic width of the spectral mask of the dominant interfering source transmitter, and is pre-calibrated by the characteristics of the RF front-end filter of the WAPI or narrowband device. A smaller value indicates that the spectral leakage of the interference source is more concentrated in the adjacent channel and the attenuation is steeper. The value of this parameter is preset and fixed by performing frequency sweep calibration on a typical broadband RF front-end in a microwave anechoic chamber, for example, preset to 5 MHz. Subsequently, a formula is applied for global overall processing. in This represents the normalized spectral coupling coefficient of the i-th channel in the final output spectral coupling coefficient vector. Let be the cardinality of the total number of 16 candidate channels, and j be the iteration index for traversing all candidate channels. By calculating the proportion of all elements, it is ensured that the sum of all vector elements is strictly equal to 1. Continuing the example above, channel 5, due to its close proximity, has a surge in its original coupling coefficient, resulting in a very large weight ratio after normalization; while channel 1, due to its distant proximity and being at the end of the attenuation range, has a normalized spectral coupling coefficient... It approaches a safe value of zero infinitely.

[0037] Using each element in the spectral coupling coefficient vector as a weight, the action advantage vector is weighted and mean-reduced with a coupling-aware penalty injection to obtain a coupled-corrected advantage vector. It is understandable that two major hidden dangers—baseline offset and risk blind spots—need to be eradicated simultaneously. This process uses weighting to restore the true baseline assessment level of the environment and uses an explicit penalty mechanism to mathematically forcibly deprive dangerous channels of their competitive advantage. First, the weighted mean-reducing calculation is performed by substituting the values ​​into the formula. here This represents the newly generated coupled-aware weighted mean baseline. The normalized spectral coupling coefficient; This represents the original relative dominance value output by the i-th candidate channel when the environment is in state S. This process ensures that channels more severely affected by adjacent channel leakage have a higher dominance in baseline determination. Explicit penalty reduction is then immediately applied, with the corresponding formula being: Here This represents the final relative advantage value of the i-th candidate channel in the coupled correction advantage vector after double correction; This is a pre-set spectral coupling penalty strength factor. This non-negative hyperparameter represents the severity of the control system's resistance to adjacent channel leakage risk, and is pre-set to a value of 20 based on the field's tolerance limit for link interruption. Taking channel 5 as an example, even if its original advantage instantly surges to 10, after removing the weighted average, the penalty factor of 20 and its maximum weight must be additionally deducted. The product of these factors causes its relative advantage to plummet into negative territory; conversely, the first channel, due to its coupling coefficient... Extremely low penalty, with almost zero deduction of penalty items, thus perfectly preserving its true and excellent avoidance attributes.

[0038] The channel value vector is obtained by summing the corrected advantage value of each channel in the coupling correction advantage vector with the state value scalar one by one. In other words, the handover decision execution module of the underlying device needs to intuitively read the absolute expected transfer value of each physical frequency point. It must organically superimpose the static assessment representing the overall network's anti-interference foundation with the dynamic relative advantage after removing any negative information. This outputs a comprehensive set of guiding indicators to the underlying device that considers both the overall environmental resilience and completely avoids hidden physical spectrum radiation traps. This aggregation process is performed by the summation operator according to the formula... In this expression, The absolute expected value scalar of the evaluation of the i-th physical channel in the channel value vector; This represents the current global electromagnetic interference immunity benchmark value scalar calculated by the front branch of the deep model; This is the final relative advantage value immediately after the penalty injection. For example, the base value of the global environment measured at the current moment. The first channel, rated at a value of 80, has a positive relative advantage of 8 after proper correction, resulting in a final comprehensive evaluation value of 88. The fifth channel, severely penalized due to leakage, has a relative advantage of -15, resulting in a final value of only 65 after aggregation. All 16 channels are synthesized and encapsulated using this logic, and the output channel value vector completely eliminates the misleading influence of false signals, ensuring that the optimal channel action extracted by the subsequent retrieval operator effectively guides all network terminals to migrate to truly clear wireless frequency bands.

[0039] Step 4: Extract the optimal channel action from the channel value vector and asynchronously distribute and schedule it based on the node's dormant state in the state buffer block. In this step, immediate and delayed instructions are sent to broadband and narrowband terminals via real-time and suspended queues, respectively. In other words, although the theoretical expected value of each candidate frequency band has been accurately assessed through preliminary deep feature deduction, the operating rhythms of devices differ drastically in real heterogeneous communication scenarios. Broadband nodes powered by constant power have the ability to respond to scheduling at any time, while narrowband nodes powered by batteries are in a deep dormant state with their RF modules turned off most of the time. If this timing discrepancy is ignored and the optimal switching action is immediately sent out in a one-size-fits-all manner, active nodes will instantly escape to the new frequency band, while dormant nodes will completely miss the synchronization beacon when they wake up in the old frequency band, becoming isolated nodes without network access. To completely overcome the technical bottleneck of severe disconnect between perception assessment and execution timing, this step is used to extract the optimal action from the channel value vector and perform precise asynchronous distribution scheduling for the dormant state, classifying and pushing instructions into different queues, thereby achieving smooth and seamless anti-interference dynamic avoidance of heterogeneous nodes across the entire network.

[0040] Figure 4 This is a flowchart of step 4 in the reinforcement learning-based wireless channel dynamic obstacle avoidance optimization method according to an embodiment of this application. Figure 4 As shown, specifically, in an exemplary implementation, step 4 includes: step 41, performing a maximum value index retrieval and matching with the physical channel frequency table on the channel value vector through the decision engine to decode and generate the optimal channel action; step 42, binding the optimal channel action with the state buffer block as key-value pairs and outputting them as state action experience pairs, synchronously parsing the sleep cycle metadata of each node in the state buffer block to deduce the theoretical wake-up time and calculate the delay suspension time to obtain the asynchronous timing scheduling tag; step 43, encapsulating the physical frequency parameters in the optimal channel action and the asynchronous timing scheduling tag according to the node address by the instruction assembler into the control frame, wherein the broadband protocol adaptation submodule compiles the immediately executable instructions into kernel-level wireless control instructions, and the narrowband protocol adaptation submodule converts the delay countdown instructions into radio frequency baseband register operation signaling, which are respectively pushed into the real-time processing queue and the sleep wake-up suspension queue.

[0041] The operation is detailed below: Following the previous step, a 16-dimensional channel value vector is output by the deep reinforcement learning network. This vector accurately depicts the long-term expected gain of transitioning to 16 candidate channels when facing complex interference; for example, the comprehensive value of channel 1 is 88, channel 2 is 83, and channel 3 is 75. Simultaneously, the memory manager has persistently stored the complete state cache block of the current decision section. The decision engine, as the logical hub within the control core responsible for generating policy decoding and execution mechanisms, first receives the channel value vector representing the value of each channel from the front-end network. The maximum value index retrieval operator configured within the decision engine employs a purely greedy strategy, performing indiscriminate linear sorting and extreme value retrieval on all consecutive floating-point numerical elements in the 16-dimensional vector. This pure greedy strategy means that the underlying logic at this stage does not involve any random exploration or probabilistic selection; it completely trusts and adopts the optimal expected value output by the deep reinforcement learning network, thereby quickly extracting the absolute array index sequence corresponding to the maximum value. For example, after extreme value retrieval and comparison, the maximum value of 88 is found to be in the first element position of the target vector. The maximum value index retrieval operator then outputs the corresponding array index value of 0. Subsequently, the decision engine performs a precise lookup and matching operation between the retrieved extreme value index and the physical channel frequency table. The physical channel frequency table is a static mapping table pre-embedded in the device's underlying read-only memory. This table is obtained during the initialization of the underlying network architecture and the device manufacturing stage, based on national radio management regulations and on-site industrial IoT communication network protocol specifications. The center frequency parameters, bandwidth occupancy range, and transmit power limit standards of all allowed orthogonal frequency points are burned into the memory chip. For example, when the input retrieval index value is 0, by consulting the physical channel frequency table, the center physical frequency parameter of 2.412 GHz can be precisely matched and extracted. Through this lookup and matching and mapping translation operation, the decision engine successfully strips away the abstract mathematical dimension of the reinforcement learning network output, perfectly decoding it to generate the optimal channel action with clear physical frequency meaning and underlying hardware control direction.

[0042] After successfully decoding the optimal channel action with explicit physical parameters, this optimal channel action is immediately bound to a high-dimensional key-value pair with a state cache block stored in an independent memory area, representing the current panoramic reality of the environment. This key-value pair combination is achieved by adding a timestamp-based anti-tampering stamp and network evaluation latency parameters, and then encapsulating them into a complete causal closed-loop data packet. This generates and outputs state-action experience pairs for subsequent offline experience playback and iterative updates by the deep learning model.

[0043] Meanwhile, because the low-power narrowband nodes within the controlled area follow extremely strict and varied sleep-wake rhythms, the scheduling and allocation platform must synchronously extract and parse the sleep cycle metadata of each sensing node recorded in the state cache block. This sleep cycle metadata contains the underlying operating rhythm configuration messages of all heterogeneous nodes across the network. The scheduling logic specifically filters and extracts the historical synchronization timestamps and inherent sleep cycle configuration information of narrowband terminals currently in a sleep-disconnected state. For each narrowband node determined to be in a sleep-locked state, the computing core directly adds its underlying hardware's inherent physical sleep waiting period to the high-precision synchronization timestamp of its last successful data interaction with the main control gateway. This accurately deduces the absolute time point at which the node will theoretically next activate its RF preamplifier and enter the wake-up receiving state. Then, the built-in scheduling arithmetic unit executes the following core mathematical calculation formulas to accurately calculate and determine the latency suspension time of each sleep node: .in, This represents the calculated final delay scalar for the nth micro-power node in a dormant state. This positive real value determines the exact number of milliseconds that the command needs to wait silently in the suspend buffer queue. This represents the high-precision synchronization timestamp recorded by the underlying clock when the nth node last reported environmental perception data before entering the current deep sleep lockout state; This represents the fixed physical sleep period duration preset by the media access control layer for this node; the sum of these two values ​​is... This is the theoretically expected absolute reference time for the next power-on wake-up of the RF front-end of this underlying node. The current absolute runtime scalar represents the time when the core decision-making trigger action matching calculation is completed. This represents the estimated baseband firmware instruction queuing and processing time based on multiple benchmark tests. This preset time is set to 5 milliseconds during the initialization of the control center, based on the processing speed of the underlying control chip and the degree of congestion on the fieldbus. The outermost nested formula... The function is used to execute the safety and fault tolerance protection logic at the exception boundary.

[0044] The following calculation is illustrated using a set of actual values ​​from within a substation: For example, if a deeply dormant narrowband sensor node responsible for monitoring partial discharge exists in the node array, reading the metadata in its status cache block reveals its last synchronization timestamp. This occurred 200,000 milliseconds ago, during the node's inherent physical sleep cycle. It is configured to run for 300,000 milliseconds. Adding these two values ​​together, the theoretical absolute wake-up time corresponds to the 300,000th millisecond since the last synchronization. If the trigger policy is issued, the current absolute running time... If the node is exactly at the 200,000th millisecond mark after synchronization, there is still a 100,000 millisecond gap before the theoretical wake-up time. Subtracting the estimated 5 milliseconds for baseband firmware instruction processing from this 100,000 milliseconds, the node's latency suspension time can be calculated. This equals 99995 milliseconds. If a rare instance of computing power congestion occurs during a complex environmental assessment, causing the subtraction difference (within parentheses) to be less than or equal to 0, it indicates that the sensor node has already autonomously woken up prematurely or is on the verge of fully waking up. The function will force it to be zero to prevent injecting illegal negative wait times into the underlying clock timer, which could cause a kernel crash.

[0045] Based on the set of delay suspension time scalars calculated one by one from the network-wide node state cache blocks, the timing control module assigns distinctly different action execution attributes to RF terminals currently in different operating states. For active broadband nodes, such as high-definition video surveillance cameras in industrial sites, which routinely have their RF modules enabled and have no sleep waiting periods, an absolute highest priority attribute of immediate execution is directly assigned. For low-power narrowband nodes that are in sleep mode and have a valid positive delay suspension time calculated using the above formula, a delay countdown execution attribute accurate to the millisecond level is strictly assigned. These two distinct execution attribute instruction tags are structured and merged for output, forming a timing allocation blueprint covering all heterogeneous terminals across the entire domain, namely, asynchronous timing scheduling tags.

[0046] Subsequently, the underlying interactive control is handed over to the instruction assembler to perform the final underlying control signaling encapsulation and distribution operations. The instruction assembler is a key protocol conversion bridge located between the core control platform and the physical radio frequency hardware driver layer. The payload encapsulation module of the instruction assembler first takes the explicit physical frequency switching parameters such as 2.412 GHz from the optimal channel action generated by the front-end decision, along with the asynchronous timing scheduling tag that has just been merged, and compactly writes them into the payload segment of the media access control frame according to their respective unique physical media access control layer node network addresses for formatted encapsulation. In the crucial distribution flow stage, the instruction assembler performs hard traffic splitting and routing processing based on its internally built dual-track protocol adaptation architecture. For control frames that carry and are parsed to have immediate execution attributes, the broadband protocol adaptation submodule performs high-speed, unimpeded translation, directly compiling them into kernel-level wireless control commands that can instantly drive the underlying radio frequency programmable oscillator to change its oscillation frequency, and directly pushing them into the real-time processing queue without any blocking. Once the underlying broadband hardware driver detects a new signaling instruction in the real-time processing queue, it will forcibly initiate a hardware interrupt within a very short microsecond, suspending any current regular transmission tasks and immediately forcing all broadband nodes to complete the physical frequency hopping avoidance action to the new 2.412 GHz channel.

[0047] For control frames carrying delay countdown attributes, such as the aforementioned micro-power partial discharge sensor switching command that requires a precise calculation of 99,995 milliseconds of silent waiting, a specially designed narrowband protocol adaptation submodule performs specialized processing. This submodule securely converts such upper-layer commands containing delay countdowns into suspend operation signaling directly targeting the RF baseband frequency register, and pushes it into a sleep / wake-up suspend queue with dedicated memory resources. This sleep / wake-up suspend queue is maintained asynchronously by an independent high-precision programmable hardware timer. When the hardware timer counts down strictly to 0, and coincides with the extremely brief moment on the physical timeline when the narrowband node releases deep sleep as scheduled and opens the RF front-end receiving window for listening, the sleep / wake-up suspend queue will forcefully inject the register operation signaling into the baseband processing chip of the narrowband node in the first sequence. This instructs it to complete the wake-up handshake confirmation on the currently interfered old frequency band channel, and instead of attempting to retransmit failed data, immediately and synchronously switch to a new, secure 2.412 GHz channel for subsequent services. This sophisticated dual-track asynchronous queue scheduling and spatiotemporal adaptation architecture not only ensures the rapid response requirements of active broadband devices to avoid sudden interference, but also perfectly fills the time vacuum blind spot caused by the deep sleep of low-power narrowband devices, thus fundamentally eliminating the occurrence of heterogeneous node channel disconnection and network topology island phenomenon.

[0048] In summary, the reinforcement learning-based dynamic channel avoidance optimization method based on the embodiments of this application is explained, addressing the issues of reinforcement learning state phantom caused by asynchronous perception in heterogeneous communication networks and narrowband node isolation caused by one-size-fits-all command switching. First, the instantaneous signal strength and sleep state of heterogeneous terminals are time-sensitively marked, and a time-sensitive feature matrix is ​​constructed to accurately identify outdated data. To eliminate spatiotemporal misalignment, digital shadow inference technology is further introduced to virtually complete the features of expired sleep nodes, and spatial dimension reorganization is performed with the real-time data of active nodes to generate a spatiotemporally aligned state tensor. This provides a realistic and reliable complete environment input for the reinforcement learning network, completely solving the state phantom problem. Subsequently, a deep reinforcement learning network is used to evaluate this aligned tensor to extract the optimal channel action, and asynchronous distribution scheduling is performed based on the node sleep state in the state buffer block. Immediate and delayed commands are sent to broadband and narrowband terminals respectively via real-time queues and suspended queues, effectively avoiding the disconnection and isolation effect caused by blind channel switching.

Claims

1. A wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning, characterized in that, include: Step 1: Perform time-sensitivity marking and multi-dimensional structured encapsulation on the instantaneous signal strength and sleep / wake cycle state of each heterogeneous node in the collected heterogeneous radio frequency raw data to obtain the time-sensitivity feature matrix. Step 2: Perform digital shadow inference to complete the outdated feature dimensions in the time-sensitive feature matrix that have expired due to node dormancy to obtain a virtual complete feature set. Use the real-time observations of the peripheral active nodes as external observation correction inputs, and reorganize the virtual complete feature set and real-time features in spatial dimensions to obtain a spatiotemporally aligned state tensor. Step 3: Use a deep reinforcement learning network to perform reinforcement learning time-varying action evaluation on the spatiotemporal aligned state tensor to obtain the channel value vector representing the switching value of each candidate channel. At the same time, copy and store the current input state snapshot as a state cache block. Step 4: Extract the optimal channel action from the channel value vector and perform asynchronous distribution and scheduling based on the node sleep state in the state buffer block. In this step, the immediate instruction and the delayed instruction are sent to the broadband and narrowband terminals through the real-time queue and the suspended queue, respectively.

2. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 1, characterized in that, The heterogeneous RF raw data includes the amplitude of the in-phase component, the phase of the quadrature component, and the wake-up timestamp of the node's media access control layer.

3. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 2, characterized in that, Step 1 includes: The amplitude of the in-phase component and the phase of the quadrature component are synchronously acquired in each baseband sampling period, and the instantaneous power of the amplitude of the in-phase component and the phase of the quadrature component is calculated to obtain the instantaneous signal strength sequence. The wake-up timestamp of the node's media access control layer is extracted as the absolute reference cursor. The instantaneous signal strength sequence is truncated and matched in the time domain, and the invalid background noise during the sleep period is removed to obtain a set of time-bound tuples. The time-series bound tuple set is dimension-normalized and filled using a matrix mapping engine and aligned with a two-dimensional matrix template to obtain the time-series feature matrix.

4. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 1, characterized in that, Step 2 includes: The timeliness label of each sensing node in the timeliness feature matrix is ​​read line by line by the state scanner. The difference between the timestamp of each node and the current system absolute time is compared with the preset effective time threshold. The real-time node data with the difference not exceeding the threshold is output as the active node feature set, and the dormant node data with the difference exceeding the threshold and its last effective observation value are output as the old node state set. Digital shadow Kalman calculus is performed on the state set of old nodes and the feature set of active nodes to obtain a virtual completion feature set; Based on the global network topology dictionary, the active node feature set and the virtual completion feature set are spatially merged, sorted and concatenated according to the unique physical address index of each node to obtain the spatiotemporally aligned state tensor.

5. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 4, characterized in that, Digital shadow Kalman inference is performed on the state set of old nodes and the feature set of active nodes to obtain a virtual completion feature set, including: The last optimal state estimation vector in the old node state set is updated over time based on the state transition matrix to obtain the prior state estimate. Using the real-time signal strength in the active node feature set as the external observation vector, the dynamic Kalman gain is calculated through the prior error covariance matrix and the observation matrix, and the prior state estimate is weighted and corrected by residuals to obtain the virtual complete feature set.

6. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 1, characterized in that, Step 3 includes: The spatiotemporally aligned state tensor is shallowly copied by the memory manager to obtain the state cache block, and the spatiotemporally aligned state tensor is simultaneously fed into the deep reinforcement learning network to obtain the hidden layer feature vector. The hidden feature vectors are fed into the state value evaluation branch and the action advantage evaluation branch respectively through the routing nodes of the competitive network architecture for parallel forward inference. The state value evaluation branch outputs a state value scalar that represents the current environmental benchmark value, and the action advantage evaluation branch outputs an action advantage vector with the same dimension as the action space. The channel value vector is obtained by performing identifiability compensation and Q-value aggregation on the action advantage vector and state value scalar.

7. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 1, characterized in that, Step 4 includes: The decision engine performs a maximum value index retrieval on the channel value vector and matches it with the physical channel frequency table to decode and generate the optimal channel action. The optimal channel action is bound to the state buffer block as a key-value pair and output as a state action experience pair. The sleep cycle metadata of each node in the state buffer block is parsed synchronously to deduce the theoretical wake-up time and calculate the delay suspension time to obtain the asynchronous timing scheduling tag. The instruction assembler encapsulates the physical frequency parameters and asynchronous timing scheduling tags in the optimal channel action into the control frame according to the node address. The broadband protocol adaptation submodule compiles the immediately executable instructions into kernel-level wireless control instructions, and the narrowband protocol adaptation submodule converts the delayed countdown instructions into radio frequency baseband register operation signaling, which are then pushed into the real-time processing queue and the sleep / wake-up pending queue, respectively.

8. The wireless channel dynamic obstacle avoidance optimization method based on reinforcement learning according to claim 6, characterized in that, The channel value vector is obtained by performing identifiability compensation and Q-value aggregation on the action advantage vector and state value scalar, including: The interference source locator resolves the center frequency of the dominant interference source in the current frequency band from the state buffer block, and calculates the absolute spectral distance between the physical center frequency of each candidate channel and the center frequency of the dominant interference source. The absolute spectral distance is mapped to the original coupling coefficients and then subjected to global normalization to obtain the spectral coupling coefficient vector; Using each element in the spectral coupling coefficient vector as a weight, the action advantage vector is subjected to coupling-aware weighted demeaning and penalty injection to obtain the coupling-corrected advantage vector; The channel value vector is obtained by adding the corrected advantage value of each channel in the coupling corrected advantage vector to the state value scalar one by one.

Images