Bluetooth networking topology self-optimization method based on deep reinforcement learning

Abstract

The application provides a Bluetooth networking topology self-optimization method based on deep reinforcement learning, a master device performs physical contact with multiple Bluetooth terminals by means of a near field communication induction module, acquires an identification information set of the Bluetooth terminals, and generates an initial networking topology structure according to the identification information set; then, connection stability features and signal coverage intensity features of the Bluetooth terminals are extracted, a pre-trained deep reinforcement learning strategy model is input, a dynamic adjustment strategy is generated, the initial networking topology structure is updated according to the dynamic adjustment strategy, an optimized networking topology structure is generated, and the optimized networking topology structure is synchronized to the master device and all the Bluetooth terminals to trigger a network configuration operation, thereby solving the problems of difficult information entry and difficult optimization of a networking topology structure of a traditional Bluetooth networking method in a harsh environment, and effectively improving the efficiency and reliability of Bluetooth networking.

Description

Technical Field

[0001] This invention relates to the field of Bluetooth networking technology, and more specifically, to a Bluetooth networking topology self-optimization method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of wireless communication technology, Bluetooth networking, as a key technology for enabling short-range wireless communication between devices, is widely used in many fields such as smart homes and industrial IoT.

[0003] Currently, the mainstream wireless Bluetooth networking method typically involves manually inputting the Bluetooth terminal's identification information (such as MAC address) through a human-machine interface, or obtaining this information by scanning a QR code on the Bluetooth terminal using a mobile app. After obtaining this identification information, the master device establishes a Bluetooth wireless network with the corresponding terminal based on the MAC address table.

[0004] However, in practical applications, especially in harsh environments such as underground coal mines, numerous problems arise. The high dust concentration in coal mines severely impairs visibility, and the dim lighting and confined spaces make manually entering MAC addresses extremely difficult, inefficient, and prone to errors. Furthermore, QR codes are easily covered, scratched, or damaged by dust in such environments, leading to scanning failures and the inability to properly obtain the Bluetooth terminal's identification information. Traditional Bluetooth pairing methods are not only complex to operate, but their applicability in these harsh environments is also significantly reduced, severely hindering the application and development of Bluetooth networking technology in these special scenarios. Summary of the Invention

[0005] In view of the aforementioned problems, embodiments of the present invention provide a Bluetooth networking topology self-optimization method based on deep reinforcement learning, the method comprising: The main device acquires a set of identification information of multiple Bluetooth terminals through a near-field communication sensing module. Each identification information in the set is generated through a physical contact operation between the near-field communication sensing module and the corresponding Bluetooth terminal. An initial network topology is generated based on the identification information set, and the connection stability characteristics and signal coverage strength characteristics of each Bluetooth terminal are extracted from the initial network topology. The connection stability features and the signal coverage strength features are input into a pre-trained deep reinforcement learning policy model to generate a dynamically adjusted policy. The topological connection relationship of the initial network topology is updated according to the dynamic adjustment strategy to generate an optimized network topology. The optimized network topology is synchronized to the master device and all Bluetooth terminals to trigger network configuration operations.

[0006] Based on the above embodiments, the main device obtains identification information by physically contacting the Bluetooth terminal using a near-field communication sensing module. This eliminates the need for traditional manual input and QR code scanning, effectively overcoming the difficulty of information input in harsh environments. This greatly improves the accuracy and convenience of Bluetooth networking information input, enabling Bluetooth networking to proceed smoothly in various complex environments. Furthermore, the analysis of connection stability characteristics and signal coverage strength characteristics based on a deep reinforcement learning strategy model can dynamically generate adjustment strategies and optimize the network topology. This fully considers various factors in the actual network environment, such as device distribution and signal interference, allowing the generated optimized network topology to better adapt to complex and ever-changing network environments, significantly improving the stability and communication quality of the Bluetooth network. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the execution flow of the Bluetooth networking topology self-optimization method based on deep reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0008] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a Bluetooth networking topology self-optimization method based on deep reinforcement learning, provided in one embodiment of the present invention. The following is a detailed description of this Bluetooth networking topology self-optimization method based on deep reinforcement learning.

[0009] Step S110: The master device obtains a set of identification information of multiple Bluetooth terminals through the near field communication sensing module. Each identification information in the set is generated through the physical contact operation between the near field communication sensing module and the corresponding Bluetooth terminal.

[0010] In real-world applications, such as complex industrial plant environments with issues like interference from electronic devices, insufficient lighting, and complex spatial layouts, traditional Bluetooth networking methods are difficult to implement effectively. In these situations, the Near Field Communication (NFC) sensing module on the main device plays a crucial role. The NFC sensing module on the main device possesses specific hardware circuitry and signal processing units, enabling communication via electromagnetic induction when in physical contact with a Bluetooth terminal. When the NFC sensing module approaches the NFC tag on the Bluetooth terminal, it generates an alternating magnetic field. This field induces a current in the NFC tag, activating it and causing it to send its stored identification information. This identification information can be something like the Bluetooth terminal's unique MAC address. The physical contact between the NFC sensing module and the Bluetooth terminal enables the generation and acquisition of this identification information. The NFC sensing module on the main device then performs preliminary verification and encoding of the acquired identification information to ensure its accuracy and completeness, gradually collecting and forming an identification information set.

[0011] Step S111: When the near-field communication sensing module detects an external sensing trigger operation, it obtains the duration and number of triggers corresponding to the sensing trigger operation.

[0012] In an industrial environment, when a worker holds the main device and brings the NFC sensing module close to a Bluetooth terminal, the trigger detection circuit inside the NFC sensing module immediately detects the external sensing trigger operation. The trigger detection circuit is based on sensor technology and can sense the distance and magnetic field changes between the NFC sensing module and the Bluetooth terminal. Once a trigger operation is detected, the timing and counting units inside the main device begin to operate. The timing unit accurately records the duration 't' from the start to the end of the sensing trigger operation, using a high-precision clock signal to ensure accurate time measurement. The counting unit counts the number of trigger signals 'n' during this sensing trigger operation. For example, when the NFC sensing module comes into contact with the Bluetooth terminal, the change in magnetic field generates a series of pulse signals; the counting unit counts these pulse signals to obtain the number of triggers 'n'.

[0013] Step S112: Determine the identification information collection mode according to the duration, wherein the first duration range corresponds to the identification information addition mode and the second duration range corresponds to the identification information deletion mode.

[0014] The master device has two pre-defined duration ranges: a first duration range T1 and a second duration range T2. These ranges are set based on actual application requirements and the characteristics of Bluetooth networking. When the acquired duration t falls within the first duration range T1, the master device determines that it is currently in the identifier information addition mode. This is because in this case, the duration of the sensor-triggered operation is relatively short, which may indicate that the user wants to add a new Bluetooth terminal to the network. When the duration t falls within the second duration range T2, the master device determines that it is currently in the identifier information deletion mode. The longer duration may mean that the user wants to remove a Bluetooth terminal from the network. The master device will automatically switch the identifier information acquisition mode based on the comparison between the duration t and the preset range to meet different networking needs.

[0015] Step S113: In the identification information addition mode, generate the identification information collection count of the target Bluetooth terminal according to the number of triggers, and collect the identification information of the target Bluetooth terminal according to the number of identification information collections through the near field communication sensing module and add it to the identification information set.

[0016] In the identification information addition mode, the master device generates the number of identification information collection attempts (m) of the target Bluetooth terminal based on the number of trigger attempts (n). This generation process can be implemented using a mapping function f(n), for example, f(n) = k. The function `n+b` maps the trigger count `n` to the identification information collection count `m`. Then, the NFC sensing module of the master device collects the identification information of the target Bluetooth terminal according to the collection count `m`. During each collection, the NFC sensing module communicates with the NFC tag of the Bluetooth terminal to obtain the identification information and stores it in the master device's temporary storage area. After collection, the master device integrates and verifies this identification information to ensure its accuracy and consistency before adding it to the identification information set.

[0017] Step S114: In the identification information deletion mode, delete historical identification information that matches the number of triggers from the identification information set according to the number of triggers.

[0018] In the identifier deletion mode, the master device deletes historical identifiers that match the trigger count n from the identifier information set. The master device internally maintains an identifier information index table, which records the index number corresponding to each identifier. The trigger count n is mapped to one or more index numbers in this index table. The master device uses these index numbers to search for the corresponding historical identifier in the identifier information set and deletes it from the set. During the deletion process, the master device updates and adjusts the identifier information set to ensure its integrity and consistency. Simultaneously, the master device records relevant information about the deletion operation for subsequent querying and auditing.

[0019] Step S120: Generate an initial network topology based on the identification information set, and extract the connection stability features and signal coverage strength features of each Bluetooth terminal in the initial network topology.

[0020] After acquiring the identification information set, the master device generates an initial network topology based on this information. The master device parses each identification information in the set to obtain relevant attribute information of the Bluetooth terminals, such as device type and communication protocol version. Based on this attribute information, the master device determines the position and connection relationship of each Bluetooth terminal in the initial network topology. During the generation of the initial network topology, the master device considers factors such as communication distance and signal strength between Bluetooth terminals to ensure network stability and reliability. After generating the initial network topology, the master device monitors each Bluetooth terminal, extracting its connection stability characteristics and signal coverage strength characteristics.

[0021] Step S121: Parse each identifier in the identifier information set to obtain the device type and communication protocol version of the corresponding Bluetooth terminal.

[0022] The master device performs detailed parsing of each identifier in the identifier information set. Identifier information is typically encoded, and the master device decodes it according to pre-defined encoding rules. Through this decoding process, the master device can extract the device type and communication protocol version information of the corresponding Bluetooth terminal. The device type may include core devices, extension devices, etc., while the communication protocol version determines the communication capabilities and compatibility between Bluetooth terminals. The master device stores the parsed device type and communication protocol version information in an internal database for subsequent network setup and management.

[0023] Step S122: Determine the hierarchical position of the Bluetooth terminal in the initial network topology based on the device type, wherein the first device type corresponds to the core layer and the second device type corresponds to the extended layer.

[0024] Based on the device type obtained from the analysis, the master device determines the hierarchical position of each Bluetooth terminal in the initial network topology. Assuming the first device type is the core device, these devices typically have strong processing and communication capabilities and occupy the core layer of the network. The second device type is the extension device, with relatively weaker processing and communication capabilities, and is located in the extension layer. The master device assigns Bluetooth terminals to different layers according to their device type. Bluetooth terminals in the core layer connect directly to the master device, while Bluetooth terminals in the extension layer connect indirectly to the master device through Bluetooth terminals in the core layer. This hierarchical division improves the stability and scalability of the network.

[0025] Step S123: Assign initial connection parameters to the Bluetooth terminal according to the communication protocol version. The initial connection parameters include the broadcast interval and the channel frequency hopping sequence.

[0026] The master device assigns initial connection parameters to the Bluetooth terminal based on its communication protocol version. Different communication protocol versions may support different broadcast intervals and channel hopping sequences. The master device selects an appropriate broadcast interval and channel hopping sequence from a preset parameter set and assigns them to the Bluetooth terminal according to the requirements of the communication protocol version. The broadcast interval determines the time interval at which the Bluetooth terminal sends broadcast signals, while the channel hopping sequence is used to switch between different channels to avoid interference and improve communication quality. The master device then sends the assigned initial connection parameters to the corresponding Bluetooth terminal, which performs initialization settings based on these parameters.

[0027] Step S124: Generate the initial network topology based on the hierarchical position and initial connection parameters of all Bluetooth terminals, wherein the Bluetooth terminals in the core layer are directly connected to the master device, and the Bluetooth terminals in the extended layer are indirectly connected to the master device through the Bluetooth terminals in the core layer.

[0028] The master device comprehensively considers the hierarchical position and initial connection parameters of all Bluetooth terminals to generate an initial network topology. Core-level Bluetooth terminals establish direct connections with the master device, transmitting data via the Bluetooth communication protocol. Extended-level Bluetooth terminals indirectly connect to the master device through core-level terminals, forming a hierarchical network structure. The master device stores the initial network topology in its internal database and sends relevant information to each Bluetooth terminal to enable them to network and communicate.

[0029] Step S125: Collect historical data transmission records of the Bluetooth terminal in the initial network topology within a preset monitoring period. The historical data transmission records include the number of successful transmissions, the number of failed transmissions, and signal strength fluctuation data for each transmission.

[0030] The master device monitors and records the data transmission of Bluetooth terminals within the initial network topology within a preset monitoring period. The monitoring period is set based on actual application requirements and network characteristics; for example, it can be set to a time interval. Within the monitoring period, the master device records the data transmission of each Bluetooth terminal, including the number of successful transmissions (s), the number of failed transmissions (f), and signal strength fluctuation data for each transmission. Signal strength fluctuation data can be collected by the signal strength sensor of the Bluetooth terminal. The master device stores and analyzes this data for subsequent feature extraction.

[0031] Step S126: Generate the connection stability feature based on the ratio of the number of successful transmissions to the number of failed transmissions, wherein a ratio greater than a preset threshold corresponds to a high stability feature, and a ratio less than or equal to the preset threshold corresponds to a low stability feature.

[0032] The master device calculates the ratio r = s / f based on the number of successful transmissions (s) and the number of failed transmissions (f). Then, it compares this ratio r with a preset threshold R. If the ratio r is greater than the preset threshold R, the master device determines that the Bluetooth terminal has high stability characteristics; if the ratio r is less than or equal to the preset threshold R, the master device determines that the Bluetooth terminal has low stability characteristics. The preset threshold R is set based on actual application requirements and networking experience, and it can serve as a standard for judging the connection stability of Bluetooth terminals. The master device stores the connection stability characteristics in an internal database for subsequent analysis and processing.

[0033] Step S127: Perform sliding window mean processing on the signal strength fluctuation data to obtain a signal strength trend sequence, and generate the signal coverage strength feature based on the difference between the maximum and minimum values ​​of the signal strength trend sequence.

[0034] The master device performs sliding window averaging on the collected signal strength fluctuation data. Sliding window averaging is a common signal processing method that smooths signal fluctuations and extracts trend information. The master device sets a sliding window size *w*, slides the window across the signal strength fluctuation data, and calculates the average value of the data within the window to obtain a signal strength trend sequence. Then, the master device identifies the maximum value *Max* and the minimum value *Min* in the signal strength trend sequence and calculates their difference *d* = *Max* - *Min*. This difference *d* is the signal coverage strength feature, reflecting the fluctuation of the Bluetooth terminal's signal coverage range. The master device stores this signal coverage strength feature in an internal database for subsequent analysis and decision-making.

[0035] Step S130: Input the connection stability features and the signal coverage strength features into the pre-trained deep reinforcement learning policy model to generate a dynamically adjusted policy.

[0036] The host device inputs the extracted connectivity stability features and signal coverage strength features into a pre-trained deep reinforcement learning policy model. This deep reinforcement learning policy model is trained on a large amount of historical data and can generate appropriate dynamically adjusted policies based on the input features. Before inputting the features, the host device standardizes the connectivity stability features and signal coverage strength features to ensure they have the same scale and range, allowing the model to learn and process them more effectively.

[0037] Step S131: Standardize and merge the connection stability feature and the signal coverage strength feature into a state vector, and input the state vector into the deep reinforcement learning policy model.

[0038] The master device standardizes the connection stability and signal coverage strength features. Common methods such as z-score standardization can be used. For both features, the master device calculates their mean and standard deviation, then subtracts the mean from each feature value and divides by the standard deviation to obtain the standardized feature value. The standardized features are then combined to form a state vector. This state vector contains comprehensive information about the Bluetooth terminal's connection stability and signal coverage strength. The master device inputs this state vector into a deep reinforcement learning policy model for analysis and decision-making.

[0039] Step S132: Invoke the action space mapping of the state vector in the deep reinforcement learning policy model to obtain the probability distribution of multiple candidate adjustment actions.

[0040] After receiving the state vector, the deep reinforcement learning policy model maps it to the action space. The model contains multiple layers and modules. First, the state vector is input to a multi-head self-attention mechanism layer. This layer models the cross-correlation of multi-dimensional features in the state vector, capturing global dependencies between features and generating an enhanced feature vector containing these global dependencies. Then, a residual connection operation is performed between the enhanced feature vector and the original state vector to obtain a residual fusion vector. This residual connection operation avoids the vanishing gradient problem and improves the model's training performance. The residual fusion vector is then input into a multilayer perceptron for nonlinear transformation to obtain a high-dimensional hidden state representation. Feature decoupling encoding is performed on the high-dimensional hidden state representation to separate independent feature components that match the dimension of the action space. These independent feature components are mapped to the initial action probability space using a Gaussian error linear unit activation function. The initial action probability space is then sparsified based on the topological dynamic constraint matrix, eliminating candidate adjustment actions that conflict with the physical connection constraints of the current network topology. Finally, the probability distribution of multiple candidate adjustment actions is obtained.

[0041] Step S1321: Input the state vector into the multi-head self-attention mechanism layer of the deep reinforcement learning policy model, perform cross-correlation modeling on the multi-dimensional features in the state vector, and generate an enhanced feature vector containing global dependencies.

[0042] The multi-head self-attention mechanism layer is a crucial component of deep reinforcement learning policy models, consisting of multiple attention heads. Each attention head focuses on and processes different dimensions of the state vector. After the state vector is input into the multi-head self-attention mechanism layer, each attention head calculates the correlation weights between different elements in the state vector. These weights are then used to weight and sum the state vectors to obtain the output of each attention head. The outputs of multiple attention heads are concatenated to obtain an enhanced feature vector containing global dependencies. This enhanced feature vector captures the complex relationships between multi-dimensional features in the state vector, providing richer information for subsequent processing.

[0043] Step S1322: Perform a residual connection operation on the enhanced feature vector and the original state vector to obtain a residual fusion vector, and input the residual fusion vector into a multilayer perceptron for nonlinear transformation to obtain a high-dimensional hidden state representation.

[0044] After obtaining the enhanced feature vector, a residual connection operation is performed between it and the original state vector. The residual connection operation adds the enhanced feature vector to the original state vector, resulting in a residual fusion vector. This is done to avoid the vanishing gradient problem during model training, allowing the model to better learn the differences and changes between features. The residual fusion vector is then input into a multilayer perceptron, which consists of multiple hidden layers, each containing multiple neurons. The residual fusion vector undergoes a nonlinear transformation within the multilayer perceptron, converting the input signal into an output signal through the activation function of the neurons, ultimately yielding a high-dimensional hidden state representation. This high-dimensional hidden state representation can more comprehensively express the feature information of the state vector.

[0045] Step S1323: Perform feature decoupling encoding on the high-dimensional hidden state representation to separate independent feature components that match the dimension of the action space. The dimension of the independent feature components is consistent with the dimension of the preset topological dynamic constraint matrix.

[0046] The purpose of feature decoupling encoding on high-dimensional hidden state representations is to separate the features within these representations, obtaining independent feature components that match the dimension of the action space. Feature decoupling encoding can employ various methods, such as autoencoders. Through this process, different features in the high-dimensional hidden state representation are separated into independent feature components. The dimension of these independent feature components matches the dimension of the pre-defined topological dynamic constraint matrix, allowing for subsequent processing of the feature components based on this matrix.

[0047] Step S1324: The independent feature components are mapped to the initial action probability space through the Gaussian error linear unit activation function, and the initial action probability space is sparsified based on the topology dynamic constraint matrix to eliminate candidate adjustment actions that conflict with the physical connection constraints of the current network topology, and generate the probability distribution of multiple candidate adjustment actions.

[0048] Independent feature components are input into a Gaussian error linear unit activation function, which maps these components to a probability space, resulting in an initial action probability space. This initial action probability space contains the probability distributions of all candidate adjustment actions. Then, the initial action probability space is sparsified based on a topology dynamic constraint matrix. This matrix is ​​dynamically constructed based on the hierarchical connection relationships of the initial network topology and the communication protocol version of the Bluetooth terminal, recording the physical connection constraints of the current network topology. By operating the topology dynamic constraint matrix on the initial action probability space, candidate adjustment actions that conflict with the physical connection constraints of the current network topology are eliminated, ultimately yielding the probability distributions of multiple candidate adjustment actions.

[0049] Step S13241: Generate a hierarchical transition permission vector according to the device type and current hierarchical position of the Bluetooth terminal. Each element in the hierarchical transition permission vector represents the target hierarchical identifier that is allowed to migrate.

[0050] The master device generates a tier transition permission vector based on the Bluetooth terminal's device type and current tier position. Different device types and tier positions may have different tier transition rules. For example, core-tier devices may have more tier transition options, while extended-tier devices may be subject to certain restrictions. The master device generates a tier transition permission vector for each Bluetooth terminal according to preset rules, where each element represents the target tier identifier for permitted migration. The tier transition permission vector can be used as a constraint for subsequent action selection.

[0051] Step S13242: Perform a Hadamard product operation on the hierarchical transition permission vector and the independent feature components, and filter the feature values ​​in the independent feature components that correspond to the unreachable levels, wherein the hierarchical transition permission vector and the independent feature components have the same dimension.

[0052] The hierarchical transition permission vector is multiplied by the independent feature components using the Hadamard product. The Hadamard product is an element-wise multiplication operation, which filters out eigenvalues ​​in the independent feature components that correspond to unreachable levels. Since the hierarchical transition permission vector records the identifier of the target level to which migration is permitted, and the element corresponding to the unreachable level has a value of 0, the Hadamard product operation sets the eigenvalues ​​in the independent feature components corresponding to the unreachable level to 0, thus filtering out these eigenvalues.

[0053] Step S13243: Perform directional probability weighting on the filtered feature components, and dynamically scale the probability value of each candidate adjustment action through the directional weighting coefficient matrix. The directional weighting coefficient matrix is ​​generated based on the changing trend and gradient direction of the historical connection stability characteristics of the Bluetooth terminal.

[0054] The filtered feature components are then subjected to directional probability weighting. The directional weighting coefficient matrix is ​​generated based on the changing trends and gradient directions of the Bluetooth terminal's historical connection stability characteristics. The master device records the historical connection stability characteristics of the Bluetooth terminal. By analyzing the changing trends and gradient directions of these characteristics over time, the dynamic changes in the Bluetooth terminal's connection stability can be understood. For example, if the historical connection stability characteristics show an upward trend, it indicates that the Bluetooth terminal's connection stability is gradually improving. In this case, higher weights can be assigned to certain candidate adjustment actions that are conducive to further improving stability.

[0055] Each element of the orientation weighting coefficient matrix corresponds to a candidate adjustment action. By multiplying the filtered feature components element-wise with the orientation weighting coefficient matrix, the probability value of each candidate adjustment action is dynamically scaled. The purpose of this is to make targeted adjustments to different candidate adjustment actions based on the historical performance of the Bluetooth terminal, so that adjustment actions that are more in line with the current state and development trend of the Bluetooth terminal have a higher probability of being selected.

[0056] Step S13244: Normalize the scaled probability values ​​and generate the final probability distribution that satisfies the probability distribution constraints through a soft maximization function controlled by the temperature parameter.

[0057] After directional probability weighting and scaling the probability values ​​of candidate adjustment actions, these scaled probability values ​​need to be normalized to ensure they satisfy the probability distribution constraint, i.e., the sum of the probability values ​​of all candidate adjustment actions is 1. Here, a soft maximization function controlled by the temperature parameter is used to achieve this normalization.

[0058] The temperature parameter is adjustable and controls the sharpness of the final probability distribution. When the temperature parameter is large, the probability distribution is relatively smooth, and the probability values ​​of each candidate adjustment action are less different, meaning the model is more inclined to explore different adjustment actions. When the temperature parameter is small, the probability distribution becomes sharper, the advantage of candidate adjustment actions with higher probability values ​​becomes more obvious, and the model is more inclined to select the adjustment action with the highest probability.

[0059] The master device will adaptively adjust the temperature parameters based on the network diameter of the optimized network topology. The network diameter reflects the longest and shortest path length between any two nodes in the network topology. When the network diameter increases, it indicates that the network topology has become more complex and the communication distance between nodes is greater. In this case, lowering the temperature parameters enhances the sharpness of the probability distribution, making the model more inclined to select adjustment actions that can effectively optimize the network topology and shorten the communication distance.

[0060] Step S133: Select the candidate adjustment action with the highest probability value as the target adjustment action according to the probability distribution, and generate the dynamic adjustment strategy according to the parameters of the target adjustment action.

[0061] After obtaining the final probability distribution of multiple candidate adjustment actions, the master device selects the candidate adjustment action with the highest probability value as the target adjustment action. The target adjustment action may include at least one of the following: adjusting the hierarchical position, updating the broadcast interval, and switching the channel frequency hopping sequence.

[0062] If the target adjustment action is to adjust the layer position, the master device will determine the specific migration plan based on the current layer of the Bluetooth terminal and the allowed target layer. For example, the master device will analyze factors such as the load and communication capabilities of other Bluetooth terminals in the target layer to select the most suitable target location for migration.

[0063] If the target adjustment action is to update the broadcast interval, the master device will adjust the Bluetooth terminal's broadcast interval according to the interval increment value in the dynamic adjustment strategy. The interval increment value is determined based on the current network topology and the Bluetooth terminal's status, with the aim of enabling the Bluetooth terminal's broadcast interval to better match the master device's scanning cycle and improve communication efficiency.

[0064] If the target adjustment action is to switch the channel frequency hopping sequence, the master device will select the target channel with the lowest interference value from the preset channel set and update the frequency hopping sequence of the Bluetooth terminal. The master device will monitor the interference of each channel in real time, and select the channel with the least interference as the target channel by comparing the interference values ​​of different channels.

[0065] The master device will adjust the specific parameters of the action according to the target and generate a complete dynamic adjustment strategy, which contains detailed information on adjusting the Bluetooth terminal.

[0066] Step S140: Update the topology connection relationship of the initial network topology according to the dynamic adjustment strategy to generate an optimized network topology.

[0067] The master device updates the initial network topology based on the generated dynamic adjustment strategy. Depending on the specific adjustment action, it takes corresponding actions to change the connection methods and parameter settings between Bluetooth terminals, thereby generating an optimized network topology.

[0068] Step S141: When the target adjustment action is to adjust the level position, the Bluetooth terminal is moved from the current level to the target level, and the original connection is disconnected before establishing a connection relationship with other Bluetooth terminals in the target level.

[0069] If the target adjustment action is to change the tier position, the master device will first send a migration command to the Bluetooth terminal that needs to be migrated. After receiving the command, the Bluetooth terminal will stop its current communication with other Bluetooth terminals in the original tier and disconnect the corresponding connection. Then, the Bluetooth terminal will search for broadcast signals from other Bluetooth terminals in the target tier, and select a suitable target Bluetooth terminal to establish a connection based on factors such as signal strength and communication quality.

[0070] During the establishment of a new connection, the Bluetooth terminal authenticates and negotiates parameters with the target Bluetooth terminal to ensure normal communication between them. The master device monitors the migration process in real time to ensure that the Bluetooth terminal successfully migrates to the target layer and establishes stable connections with other Bluetooth terminals in the target layer. After the migration is complete, the master device updates the layer position and connection relationship information of the Bluetooth terminal in the initial network topology.

[0071] Step S142: When the target adjustment action is to update the broadcast interval, the broadcast interval of the Bluetooth terminal is adjusted according to the interval increment value in the dynamic adjustment strategy to adapt to the scanning cycle of the master device.

[0072] When the target adjustment action is to update the broadcast interval, the master device sends the interval increment value in the dynamic adjustment strategy to the corresponding Bluetooth terminal. After receiving the interval increment value, the Bluetooth terminal calculates the new broadcast interval based on the current broadcast interval and the interval increment value.

[0073] For example, a Bluetooth terminal's original broadcast interval might be a specific duration. Adding an interval increment yields a new broadcast interval. This new interval needs to better match the master device's scanning cycle to improve the efficiency of discovery and communication between the Bluetooth terminal and the master device. The Bluetooth terminal will update its broadcast parameter settings and send broadcast signals according to the new interval. The master device will then scan according to the new broadcast interval during subsequent scans to ensure timely discovery and communication with the Bluetooth terminal.

[0074] Step S143: When the target adjustment action is to switch the channel frequency hopping sequence, select the target channel with the smallest interference value with the current channel from the preset channel set and update the frequency hopping sequence of the Bluetooth terminal.

[0075] If the target adjustment action is a channel hopping sequence, the master device will monitor the interference of each channel in the preset channel set in real time. The master device will collect information such as signal strength and noise level of each channel, and calculate the interference value of each channel by analyzing this information.

[0076] Then, the master device compares the interference value of the channel currently used by the Bluetooth terminal with the interference values ​​of other channels in the preset channel set, and selects the channel with the lowest interference value as the target channel. The master device then sends a command to the Bluetooth terminal to switch the channel frequency hopping sequence, which includes relevant information about the target channel.

[0077] Upon receiving the command, the Bluetooth terminal will stop using the current channel frequency hopping sequence and switch to the target channel. During the switchover process, the Bluetooth terminal will perform channel synchronization and parameter configuration to ensure normal communication on the new channel. The master device will continuously monitor the Bluetooth terminal's communication status on the new channel to ensure a successful switchover and stable operation.

[0078] Step S144: Use the network topology after performing all target adjustment actions as the optimized network topology.

[0079] After completing all target adjustments, the master device will consider the current network topology as the optimized network topology. The master device will then perform a comprehensive check and verification of the optimized network topology to ensure that the connections between all Bluetooth terminals are correct, the parameter settings are reasonable, and that stable communication is possible.

[0080] The master device updates its internal topology database, storing information related to the optimized network topology for future management and maintenance. Simultaneously, the master device sends this optimized topology information to each Bluetooth terminal, informing them of the current network status.

[0081] Step S150: Synchronize the optimized network topology to the master device and all Bluetooth terminals to trigger network configuration operation.

[0082] The master device will synchronize the optimized network topology information to itself and all Bluetooth terminals to trigger network configuration operations, enabling the entire Bluetooth network to work according to the optimized topology.

[0083] Step S151: Encode the optimized network topology into a set of configuration instructions, which includes connection parameter update instructions and topology update instructions for each Bluetooth terminal.

[0084] The master device encodes the optimized network topology and converts it into a set of configuration instructions. This set of instructions includes connection parameter update instructions and topology update instructions for each Bluetooth terminal.

[0085] For connection parameter update commands, corresponding commands are generated based on the new settings of each Bluetooth terminal in the optimized network topology, such as the updated broadcast interval and channel frequency hopping sequence. These commands will clearly inform the Bluetooth terminal of the parameter values ​​that need to be updated and the specific operation methods.

[0086] For topology update commands, corresponding commands are generated based on the Bluetooth terminal's new hierarchical position and connection relationships in the optimized network topology. These commands instruct the Bluetooth terminal on which other Bluetooth terminals to establish or disconnect connections with, and how to adjust its role and communication methods within the network.

[0087] Step S152: Send the set of configuration instructions to all Bluetooth terminals via the broadcast channel, and receive the configuration confirmation response returned by each Bluetooth terminal.

[0088] The master device sends a set of configuration commands to all Bluetooth devices via a broadcast channel. A broadcast channel is a public communication channel that can send information to multiple Bluetooth devices simultaneously.

[0089] After receiving the configuration command set, each Bluetooth terminal parses and processes the commands. Based on the requirements of the commands, the Bluetooth terminal updates its own connection parameters and topology, and performs the corresponding configuration operations. After completing the configuration operation, the Bluetooth terminal returns a configuration confirmation response to the master device, informing the master device whether the configuration was successful.

[0090] Step S153: When all Bluetooth terminals respond with configuration confirmations indicating success, the master device switches to the optimized network topology and enables the new connection parameters.

[0091] If the master device receives configuration confirmation responses from all Bluetooth terminals, and these responses all indicate successful configuration, the master device will switch to the optimized network topology and enable the new connection parameters. The master device will update its own network configuration and communicate with each Bluetooth terminal according to the optimized topology.

[0092] The master device will re-establish the connection with the Bluetooth terminal, using new broadcast intervals, channel frequency hopping sequences, and other parameters for communication. Simultaneously, the master device will monitor the entire network operation to ensure the optimized network topology runs stably and that data transmission between the various Bluetooth terminals is normal.

[0093] Step S154: If a Bluetooth terminal does not return a configuration confirmation response, calculate the retry interval based on its historical connection stability characteristics, and resend the corresponding configuration command according to the retry interval until the maximum number of retries is reached or a successful response is received. If the maximum number of retries is reached and the task is still unsuccessful, mark the Bluetooth terminal as an abnormal node and update the identification information set.

[0094] If a Bluetooth terminal fails to return a configuration confirmation response, the master device will calculate the retry interval based on the Bluetooth terminal's historical connection stability characteristics. Historical connection stability characteristics reflect the stability of the Bluetooth terminal in past communication processes, such as the ratio of successful to failed transmissions and signal strength fluctuations.

[0095] The master device calculates the retry interval based on historical connection stability characteristics using preset calculation rules. The calculation of the retry interval can take into account the stability level of the Bluetooth terminal; Bluetooth terminals with poor stability may require a longer retry interval to ensure sufficient time for configuration operations.

[0096] The master device will resend the corresponding configuration command to the Bluetooth terminal that has not returned a configuration confirmation response according to the calculated retry interval. The master device will record the number of retries and increment the retry count after each retry. If a successful response is not received from the Bluetooth terminal after reaching the maximum number of retries, the master device will mark the Bluetooth terminal as an abnormal node.

[0097] The master device will update the identification information set, removing or specially marking information related to abnormal nodes to facilitate subsequent troubleshooting and handling. Simultaneously, the master device will continue to monitor the operation of other normal Bluetooth terminals to ensure the stability of the entire Bluetooth network.

[0098] Step S160: Determine the topology optimization direction and connection parameter update instructions in the dynamic adjustment strategy. In addition to target adjustment actions, the dynamic adjustment strategy also includes topology optimization direction and connection parameter update instructions. Determining the topology optimization direction helps to further improve the performance of the network topology, while the connection parameter update instructions ensure that Bluetooth terminals can communicate according to the optimized requirements.

[0099] Step S161: Generate a topology connection diagram based on the hierarchical position of each Bluetooth terminal in the initial network topology, and calculate the network diameter and the number of redundant paths in the topology connection diagram.

[0100] The master device generates a topology diagram based on the hierarchical position of each Bluetooth terminal in the initial network topology. A topology diagram is a graphical representation that uses nodes to represent Bluetooth terminals and edges to represent the connections between them.

[0101] The master device analyzes the topology graph, calculating its network diameter and the number of redundant paths. The network diameter refers to the length of the longest and shortest path between any two nodes in the topology graph, reflecting the communication distance between nodes in the network topology. The number of redundant paths refers to the number of paths in the topology graph other than the shortest path, reflecting the reliability and fault tolerance of the network topology.

[0102] Step S162: When the network diameter exceeds a preset diameter threshold, generate an optimization direction to increase the core level.

[0103] The master device will compare the calculated network diameter with a preset diameter threshold. If the network diameter exceeds the preset diameter threshold, it indicates that the communication distance between nodes in the network topology is too long, which may lead to problems such as increased communication latency and decreased communication quality.

[0104] At this point, the master device will generate optimization directions for adding a core layer. Adding a core layer can shorten the communication distance between nodes and improve communication efficiency. The master device will select appropriate Bluetooth terminals to be promoted to the core layer based on factors such as the device type and performance of the Bluetooth terminals, in order to optimize the network topology.

[0105] Step S163: When the number of redundant paths is lower than a preset path threshold, generate an optimized direction for adding backup connection relationships.

[0106] The master device compares the calculated number of redundant paths with a preset path threshold. If the number of redundant paths is lower than the preset path threshold, it indicates that the reliability and fault tolerance of the network topology are low, and a failure of a node or connection may lead to network communication interruption.

[0107] At this point, the master device generates optimized directions for adding backup connections. The master device analyzes the topology graph, identifies node pairs where backup connections can be added, and establishes backup connections between these nodes. Backup connections provide a backup communication path when the primary connection fails, improving network reliability.

[0108] Step S164: Combine the optimization direction with the target adjustment action output by the deep reinforcement learning policy model to generate the dynamic adjustment policy.

[0109] The master device combines the determined topology optimization direction with the target adjustment actions output by the deep reinforcement learning strategy model to generate a complete dynamic adjustment strategy. The dynamic adjustment strategy not only includes the target adjustment actions for specific adjustments to Bluetooth terminals, but also includes the direction for overall optimization of the network topology.

[0110] This dynamically adjusted strategy can more comprehensively consider the performance and requirements of the network topology, enabling the Bluetooth network to maintain stable and efficient operation in different scenarios.

[0111] Step S170: After receiving the identification information of the new Bluetooth terminal, the master device triggers a real-time optimization operation. When the master device receives the identification information of a new Bluetooth terminal, it will trigger a real-time optimization operation to ensure that the new Bluetooth terminal can be smoothly integrated into the existing network topology, while maintaining the stability and performance of the entire network.

[0112] Step S171: Add the identification information of the new Bluetooth terminal to the identification information set, and add the new Bluetooth terminal to the temporary extension layer of the current network topology according to the device type of the new Bluetooth terminal.

[0113] The master device adds the identification information of the new Bluetooth terminal to the identification information set for management and identification. Depending on the device type of the new Bluetooth terminal, the master device adds it to a temporary extension layer of the current network topology.

[0114] The Temporary Extension Layer is a temporary layer used to temporarily accommodate newly added Bluetooth devices for observation and evaluation. Within the Temporary Extension Layer, new Bluetooth devices can communicate with other Bluetooth devices, while the master device can collect relevant performance data.

[0115] Step S172: Periodically collect the connection stability characteristics and signal coverage strength characteristics of the new Bluetooth terminal during the preset observation period, and synchronously update the topology dynamic constraint matrix according to the device type of the new Bluetooth terminal.

[0116] During the preset observation period, the master device will periodically collect connection stability characteristics and signal coverage strength characteristics of new Bluetooth terminals. Connection stability characteristics can be calculated by monitoring information such as the number of successful and failed data transmissions of new Bluetooth terminals, while signal coverage strength characteristics can be determined by monitoring their signal strength fluctuations.

[0117] Simultaneously, the master device will synchronously update the topology dynamic constraint matrix based on the device type of the new Bluetooth terminal. The topology dynamic constraint matrix records the physical connection restrictions and hierarchical transition rules of each Bluetooth terminal in the network topology. The addition of a new Bluetooth terminal may change these rules, thus requiring an update to the topology dynamic constraint matrix.

[0118] Step S173: When the connection stability feature is lower than the average stability of the current network topology and the standard deviation of the signal coverage strength feature is less than a preset threshold, the deep reinforcement learning strategy model is invoked to generate a real-time adjustment strategy for the new Bluetooth terminal.

[0119] The master device will compare the connection stability characteristics of the new Bluetooth terminal with the average stability of the current network topology, and at the same time calculate the standard deviation of the fluctuation of the signal coverage strength characteristics and compare it with a preset threshold.

[0120] If the connection stability characteristics are lower than the average stability of the current network topology, and the standard deviation of the signal coverage strength characteristics is less than the preset threshold, it indicates that the connection stability of the new Bluetooth terminal at the current location is poor, and the signal coverage is relatively stable but not ideal.

[0121] At this point, the master device invokes a deep reinforcement learning strategy model, inputting the connection stability characteristics and signal coverage strength characteristics of the new Bluetooth terminal into the model to generate a real-time adjustment strategy for that Bluetooth terminal. The real-time adjustment strategy may include operations such as adjusting the hierarchical position, updating the broadcast interval, or switching the channel frequency hopping sequence to improve the connection performance of the new Bluetooth terminal.

[0122] Step S174: According to the real-time adjustment strategy, migrate the new Bluetooth terminal to the target level or adjust its connection parameters to integrate into the optimized network topology.

[0123] The master device will adjust the new Bluetooth terminal according to the generated real-time adjustment policy. If the real-time adjustment policy recommends migrating the new Bluetooth terminal to the target layer, the master device will send a migration command to the Bluetooth terminal, causing it to migrate from the temporary extended layer to the target layer and establish a new connection.

[0124] If the real-time adjustment strategy suggests adjusting the connection parameters of a new Bluetooth terminal, the master device will send a parameter update command to that Bluetooth terminal, causing it to update connection parameters such as broadcast interval and channel frequency hopping sequence. Through these adjustments, the new Bluetooth terminal can better integrate into the optimized network topology, improving the performance and stability of the entire Bluetooth network.

[0125] For example, in order for a deep reinforcement learning policy model to accurately generate appropriate dynamically adjusted policies based on the input features, it needs to be trained. The training process includes steps such as obtaining a training sample set, building an initial model, performing feature processing, and calculating loss.

[0126] Step S211: Obtain a training sample set, which includes training sample state vectors of historical network topologies and corresponding real adjustment action sequences.

[0127] Historical network topology data is collected, including connection stability characteristics and signal coverage strength characteristics of each Bluetooth terminal, and this information is combined into a training sample state vector. Simultaneously, the actual adjustment action sequences taken for different states during the historical network setup process are recorded.

[0128] The training sample state vector contains multi-dimensional feature information, comprehensively reflecting the state of the network topology. The real adjustment action sequences record the actual adjustment actions taken under different network conditions, including adjusting hierarchical positions, updating broadcast intervals, and switching channel frequency hopping sequences. The training sample state vectors and corresponding real adjustment action sequences are combined to form a training sample set for subsequent model training. To ensure the quality and diversity of the training sample set, the collected data needs to be cleaned and preprocessed. Noise and outliers are removed to ensure data accuracy and consistency. Simultaneously, features of different dimensions are standardized to ensure they have the same scale and range, avoiding negative impacts on model training performance due to excessive differences in feature scale.

[0129] Step S212: Construct an initial deep reinforcement learning policy model, which includes a multi-head self-attention mechanism layer, a multilayer perceptron, and a Gaussian error linear unit activation function.

[0130] When initializing a deep reinforcement learning policy model, each component module is constructed according to a specific architecture and parameters. The multi-head self-attention mechanism layer is a crucial part of the model, consisting of multiple attention heads. Each attention head can independently process the input training sample state vector, capturing the correlations between features of different dimensions. Through parallel processing of multiple attention heads, the global dependencies between features can be more comprehensively explored.

[0131] A multilayer perceptron (MLP) is a neural network structure with multiple hidden layers, each containing multiple neurons. MLPs can perform non-linear transformations on input features, learning more complex feature representations. When constructing an MLP, parameters such as the number of hidden layers, the number of neurons in each hidden layer, and the type of activation function need to be determined.

[0132] The Gaussian error linear unit activation function is used to map the model's output to a suitable range. It features smooth gradients and good non-linear characteristics, which can improve the model's training efficiency and generalization ability. During model initialization, the model's weights and biases also need to be randomly initialized to provide an initial starting point for subsequent training.

[0133] Step S213: Input the training sample state vector into the multi-head self-attention mechanism layer, perform global dependency modeling on the multi-dimensional features in the training sample state vector during the training phase, and generate a training feature encoding vector.

[0134] When the training sample state vector is input into the multi-head self-attention mechanism layer, each attention head processes the input multi-dimensional features independently. First, the attention head calculates the correlation weights between different elements in the training sample state vector. These correlation weights reflect the degree of association between different features. The training sample state vector is then weighted and summed using these weights to obtain the output of each attention head.

[0135] Concatenating the outputs of multiple attention heads generates a training feature encoding vector that contains global dependencies. This vector more comprehensively expresses the complex relationships between multidimensional features in the training sample state vector, providing richer information for subsequent processing. When calculating relevance weights, the dimensionality and value range of the features must be considered to ensure the weight calculation is reasonable and effective.

[0136] Step S214: Perform a residual connection operation on the training feature encoding vector and the training sample state vector to obtain a training residual fusion vector, and input the training residual fusion vector into the multilayer perceptron for nonlinear transformation to generate a training hidden state representation.

[0137] After obtaining the training feature encoding vector, a residual concatenation operation is performed between it and the training sample state vector. The residual concatenation adds the training feature encoding vector to the training sample state vector, resulting in a training residual fusion vector. This is done to avoid the vanishing gradient problem during model training, enabling the model to better learn the differences and changes between features.

[0138] The training residual fusion vector is input into a multilayer perceptron for nonlinear transformation. Each hidden layer of the multilayer perceptron processes the input training residual fusion vector, converting the input signal into an output signal through the activation function of the neuron. After multiple layers of nonlinear transformation, a training hidden state representation is generated. The training hidden state representation can more comprehensively express the feature information of the training sample state vector, providing a more accurate basis for subsequent action prediction.

[0139] Step S215: Decouple the training phase features of the training hidden state representation, separate the independent training feature components corresponding to the action space dimension, and map them to the initial action probability space through the Gaussian error linear unit activation function.

[0140] The purpose of decoupling training-stage features from the training hidden state representation is to separate the features within the training hidden state representation and obtain independent training feature components corresponding to the action space dimension. Feature decoupling can be achieved using various methods, such as autoencoders. Through feature decoupling, different features in the training hidden state representation are separated into independent feature components.

[0141] The dimensions of these independent training feature components are consistent with the dimensions of the action space, enabling accurate probability prediction for each candidate adjustment action. These independent training feature components are input into a Gaussian error linear unit activation function, which maps them to a probability space, resulting in the initial training action probability space. This initial training action probability space contains the initial probability distribution of all candidate adjustment actions.

[0142] Step S216: Generate a historical topology dynamic constraint matrix based on the hierarchical connection relationship of the historical network topology, perform a training phase Hadamard product operation on the historical topology dynamic constraint matrix and the initial training action probability space, and filter out candidate adjustment action probability values ​​in the initial training action probability space that do not meet the historical physical connection constraints.

[0143] Based on the hierarchical connection relationships of the historical network topology and the communication protocol versions of the Bluetooth terminals, a historical topology dynamic constraint matrix is ​​generated. This matrix records the physical connection constraints and hierarchical transition rules of each Bluetooth terminal during the historical network setup. The historical topology dynamic constraint matrix is ​​then subjected to a Hadamard product operation with the initial action probability space during the training phase.

[0144] The Hadamard product operation is an element-wise multiplication operation. This operation can filter out candidate adjustment action probabilities from the initial training action probability space that do not conform to the historical physical connection constraints. Since the historical topology dynamic constraint matrix records allowed connections and actions, the elements corresponding to candidate adjustment actions that do not conform to the historical physical connection constraints have a value of 0. The Hadamard product operation can set the probability values ​​of these actions in the initial training action probability space to 0, thereby filtering out the probability values.

[0145] Step S217: Calculate the policy gradient loss based on the filtered training action probability distribution and the real adjusted action sequence, and jointly optimize the network weights of the deep reinforcement learning policy model based on the policy gradient loss.

[0146] After obtaining the filtered training action probability distribution, it is compared with the actual adjusted action sequence to calculate the policy gradient loss. The policy gradient loss reflects the difference between the action probability distribution predicted by the model and the actual adjusted action sequence taken. Various methods can be used to calculate the policy gradient loss, such as the cross-entropy loss function.

[0147] Based on the calculated policy gradient loss, an optimization algorithm is used to jointly optimize the network weights of the deep reinforcement learning policy model. The optimization algorithm can employ stochastic gradient descent and its variants, such as the Adam algorithm. By continuously adjusting the network weights, the predicted action probability distribution gradually approaches the actual action sequence, thereby improving the model's prediction accuracy and performance. During the optimization process, appropriate parameters such as the learning rate need to be set to ensure that the model can stably converge to the optimal solution.

[0148] Throughout the training process of the deep reinforcement learning policy model, monitoring and evaluation are necessary. A validation set can be used to evaluate the model's performance, and training parameters and model structure can be adjusted based on the evaluation results to improve the generalization ability and stability of the deep reinforcement learning policy model. Simultaneously, to protect data privacy and security, encryption algorithms are used to encrypt sensitive data during data collection and processing to prevent data leakage. Appropriate security measures are also taken for the storage and transmission of the training sample set to ensure data integrity and confidentiality. Through these detailed training steps and processing methods, an accurate and stable deep reinforcement learning policy model can be trained for self-optimization of Bluetooth network topologies.

[0149] Furthermore, in practical applications, as Bluetooth networks continuously operate and change, it is necessary to regularly update and optimize the deep reinforcement learning policy model. New network topology data can be continuously collected and added to the training sample set to retrain the model, adapting it to new network environments and requirements. Simultaneously, the model's performance should be monitored in real time. When a decline in prediction accuracy is detected, timely adjustments and optimizations should be made to ensure the model consistently generates appropriate dynamic adjustment strategies, achieving efficient self-optimization of the Bluetooth network topology.

[0150] In terms of data acquisition, to ensure the accuracy and comprehensiveness of the data, various sensors and monitoring devices are used to monitor various parameters of the Bluetooth terminal in real time. For example, a signal strength sensor is used to monitor the signal strength fluctuations of the Bluetooth terminal in real time, and a data transmission recorder is used to record information such as the number of successful and failed data transmissions of the Bluetooth terminal. These monitoring devices transmit the collected data to the main device in real time, and the main device integrates and processes the data to generate training sample state vectors.

[0151] When processing the state vectors of training samples, in addition to standardization, feature selection and feature extraction operations can also be performed. Feature selection chooses the most representative and relevant features from the original multidimensional features, reducing the dimensionality of the features and improving the training efficiency and accuracy of the model. Feature extraction uses mathematical transformation methods to convert the original features into new features that better reflect the essential characteristics of the data.

[0152] When building a multilayer perceptron, different network structures and parameter settings can be tried. Experiments can be conducted to compare the model's performance under different settings, and the optimal network structure and parameters can be selected. For example, different numbers of hidden layers, neurons, and activation function types can be tried to observe the model's training effect and generalization ability.

[0153] During model training, batch training can be used, dividing the training sample set into multiple mini-batches for training. This reduces memory usage and improves training efficiency. Simultaneously, to prevent overfitting, regularization methods such as L1 and L2 regularization can be employed to constrain the model's weights, making the model more robust.

[0154] In practical applications, the model's real-time performance and response speed also need to be considered. When the Bluetooth network changes, the model needs to be able to generate appropriate dynamic adjustment strategies in a timely manner. To improve the model's real-time performance, it can be optimized to reduce its computational complexity. For example, pruning techniques can be used to simplify the model's network structure, removing unnecessary neurons and connections, thereby improving the model's inference speed.

[0155] When synchronizing the optimized network topology to the Bluetooth terminal, a retransmission and acknowledgment mechanism can be used to ensure communication reliability. After the master device sends a set of configuration commands, if it does not receive a configuration acknowledgment response from the Bluetooth terminal within a set time, the master device will resend the configuration commands until it receives an acknowledgment response or reaches the maximum number of retries. Simultaneously, after receiving the configuration commands, the Bluetooth terminal will perform verification and validation to ensure the accuracy and completeness of the commands, and then return a configuration acknowledgment response to inform the master device whether the configuration was successful.

[0156] When determining the direction of topology optimization, in addition to considering network diameter and the number of redundant paths, other factors can also be considered, such as network load balancing and energy consumption. For example, when the Bluetooth terminals at a certain level are overloaded, an optimization direction can be generated to adjust the position of the level, migrating some Bluetooth terminals to other levels to achieve network load balancing.

[0157] When handling newly added Bluetooth terminals, in addition to observing their connection stability and signal coverage characteristics, their device performance and functional features should also be considered. Based on the device performance and functional characteristics of the new Bluetooth terminal, assign it an appropriate hierarchical position and connection parameters to better integrate it into the existing network topology.

[0158] When training deep reinforcement learning policy models, a distributed training approach can be adopted, distributing the training task across multiple computing nodes for parallel computation. This can significantly improve training efficiency and shorten training time. However, during distributed training, the communication and synchronization between nodes need to be considered to ensure that the training results from each node are correctly integrated and used to update the model's network weights.

[0159] In practical applications, Bluetooth networks also require security management. For example, authentication and encryption technologies should be used to protect communication between Bluetooth terminals, preventing data leaks and malicious attacks. Simultaneously, regular security audits and vulnerability scans of the Bluetooth network should be conducted to promptly identify and address security risks, ensuring the secure and stable operation of the Bluetooth network.

[0160] During data collection, differential privacy technology is used to protect privacy of sensitive data, such as the location information and usage habits of Bluetooth terminals. Differential privacy technology adds noise to the data, allowing it to maintain certain statistical characteristics without revealing individual privacy, thus meeting the needs of model training. Simultaneously, strict access control is implemented for data access and use; only authorized personnel can access and process this sensitive data.

[0161] During model training, an early stopping strategy can be employed. Training should be stopped prematurely when the model's performance on the validation set no longer improves, thus preventing overfitting. Simultaneously, the loss function values ​​and evaluation metrics during training can be visualized to facilitate monitoring and analysis of the model's training effectiveness.

[0162] In practical applications, other technologies, such as genetic algorithms and ant colony optimization from artificial intelligence, can be combined to further optimize Bluetooth network topologies. For example, genetic algorithms can be used to optimize the parameters of deep reinforcement learning policy models, improving the model's performance and adaptability.

[0163] When updating the configuration of a Bluetooth terminal, an incremental update method can be used, updating only the changed configuration parameters in the Bluetooth terminal, reducing data transmission volume and update time. Simultaneously, a backup and restore mechanism should be implemented during the update process to prevent Bluetooth terminal malfunction due to update failure.

[0164] When constructing a topological connectivity graph, graph embedding techniques can be used to convert it into a low-dimensional vector representation, facilitating model processing and analysis. Graph embedding techniques can capture structural information and relationships between nodes in the topological connectivity graph, providing the model with richer feature information.

[0165] When determining the direction of topology optimization, a multi-objective optimization method can be used, considering multiple optimization objectives simultaneously, such as network performance, reliability, and energy consumption. By setting different weights, the various optimization objectives are weighted and summed to obtain a comprehensive optimization objective, and then the topology optimization direction is generated based on this comprehensive objective.

[0166] In handling abnormal nodes, in addition to marking abnormal nodes and updating the identification information set, fault diagnosis and repair can also be performed on abnormal nodes. For example, by analyzing the historical data and status information of abnormal nodes, the cause of the fault can be identified, and corresponding repair measures can be taken, such as restarting the node or updating the software.

[0167] In practical applications, real-time monitoring and evaluation of Bluetooth network performance are also necessary. Performance metrics such as data transmission rate, packet loss rate, and latency can be set and monitored and analyzed periodically. Based on the monitoring results, the parameters of the deep reinforcement learning strategy model and the direction of topology optimization can be adjusted in a timely manner to improve the performance and stability of the Bluetooth network.

[0168] Furthermore, embodiments of the present invention also provide a readable storage medium containing computer-executable instructions. When the processor executes the computer-executable instructions, the above-described Bluetooth networking topology self-optimization method based on deep reinforcement learning is implemented.

[0169] It should be noted that, in order to simplify the description of this invention and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of this invention sometimes combines multiple features into a single embodiment, drawing, or description thereof. Similarly, it should be noted that, in order to simplify the description of this invention and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of this invention sometimes combines multiple features into a single embodiment, drawing, or description thereof.

Claims

1. A Bluetooth networking topology self-optimization method based on deep reinforcement learning, characterized in that, The method includes: The main device acquires a set of identification information of multiple Bluetooth terminals through a near-field communication sensing module. Each identification information in the set is generated through a physical contact operation between the near-field communication sensing module and the corresponding Bluetooth terminal. An initial network topology is generated based on the identification information set, and the connection stability characteristics and signal coverage strength characteristics of each Bluetooth terminal are extracted from the initial network topology. The connection stability features and the signal coverage strength features are input into a pre-trained deep reinforcement learning policy model to generate a dynamically adjusted policy. The topological connection relationship of the initial network topology is updated according to the dynamic adjustment strategy to generate an optimized network topology. The optimized network topology is synchronized to the master device and all Bluetooth terminals to trigger network configuration operations; The step of inputting the connection stability features and the signal coverage strength features into a pre-trained deep reinforcement learning policy model to generate a dynamically adjusted policy includes: The connection stability feature and the standard ring of the signal coverage strength feature are merged into a state vector, and the state vector is input into the deep reinforcement learning policy model; The deep reinforcement learning policy model is invoked to perform action space mapping on the state vector to obtain the probability distribution of multiple candidate adjustment actions; The candidate adjustment action with the highest probability value is selected as the target adjustment action according to the probability distribution, and the dynamic adjustment strategy is generated according to the parameters of the target adjustment action. The candidate adjustment action includes at least one of adjusting the hierarchical position, updating the broadcast interval, and switching the channel frequency hopping sequence.

2. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 1, characterized in that, The master device acquires a set of identification information for multiple Bluetooth terminals through a near-field communication sensing module, including: When the near-field communication sensing module detects an external sensing trigger operation, it obtains the duration and number of triggers corresponding to the sensing trigger operation. The identification information collection mode is determined based on the duration, wherein the first duration range corresponds to the identification information addition mode, and the second duration range corresponds to the identification information deletion mode. In the identification information addition mode, the identification information collection count of the target Bluetooth terminal is generated according to the number of triggers, and the identification information of the target Bluetooth terminal is collected by the near-field communication sensing module according to the identification information collection count and added to the identification information set; In the identification information deletion mode, historical identification information matching the number of triggers is deleted from the identification information set according to the number of triggers.

3. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 2, characterized in that, The process of generating the initial network topology based on the set of identifier information includes: The device type and communication protocol version of the corresponding Bluetooth terminal are obtained by parsing each identifier in the identifier information set. The hierarchical position of the Bluetooth terminal in the initial network topology is determined based on the device type, wherein the first device type corresponds to the core layer and the second device type corresponds to the extended layer; The Bluetooth terminal is assigned initial connection parameters according to the communication protocol version, and the initial connection parameters include the broadcast interval and the channel frequency hopping sequence. The initial network topology is generated based on the hierarchical position and initial connection parameters of all Bluetooth terminals. The Bluetooth terminals in the core layer are directly connected to the master device, and the Bluetooth terminals in the extended layer are indirectly connected to the master device through the Bluetooth terminals in the core layer.

4. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of extracting the connection stability features and signal coverage strength features of each Bluetooth terminal from the initial network topology includes: Within a preset monitoring period, historical data transmission records of the Bluetooth terminal in the initial network topology are collected. The historical data transmission records include the number of successful transmissions, the number of failed transmissions, and signal strength fluctuation data for each transmission. The connection stability feature is generated based on the ratio of the number of successful transmissions to the number of failed transmissions, wherein a ratio greater than a preset threshold corresponds to a high stability feature, and a ratio less than or equal to the preset threshold corresponds to a low stability feature. The signal strength fluctuation data is processed by a sliding window mean to obtain a signal strength trend sequence. The signal coverage strength feature is generated based on the difference between the maximum and minimum values ​​of the signal strength trend sequence.

5. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 1, characterized in that, The step of invoking the deep reinforcement learning policy model to perform action space mapping on the state vector to obtain the probability distribution of multiple candidate adjustment actions includes: The state vector is input into the multi-head self-attention mechanism layer of the deep reinforcement learning policy model to perform cross-correlation modeling on the multi-dimensional features in the state vector, generating an enhanced feature vector containing global dependencies. A residual connection operation is performed on the enhanced feature vector and the original state vector to obtain a residual fusion vector. The residual fusion vector is then input into a multilayer perceptron for nonlinear transformation to obtain a high-dimensional hidden state representation. The high-dimensional hidden state representation is decoupled and encoded to separate independent feature components that match the dimension of the action space. The dimension of the independent feature components is consistent with the dimension of the preset topological dynamic constraint matrix. The independent feature components are mapped to the initial action probability space by the Gaussian error linear unit activation function, and the initial action probability space is sparsified based on the topology dynamic constraint matrix to eliminate candidate adjustment actions that conflict with the physical connection constraints of the current network topology, thereby generating a probability distribution of multiple candidate adjustment actions. The topology dynamic constraint matrix is ​​dynamically constructed based on the hierarchical connection relationship of the initial network topology and the communication protocol version of the Bluetooth terminal, and is used to limit the physical feasibility of the candidate adjustment actions.

6. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 5, characterized in that, The process of sparsifying the initial action probability space based on the topology dynamic constraint matrix, eliminating candidate adjustment actions that conflict with the physical connection constraints of the current network topology, and generating a probability distribution of multiple candidate adjustment actions includes: A hierarchical transition permission vector is generated based on the device type and current hierarchical position of the Bluetooth terminal. Each element in the hierarchical transition permission vector represents the target hierarchical identifier that is allowed to migrate. Perform a Hadamard product operation between the hierarchical transition permission vector and the independent feature components to filter out the feature values ​​in the independent feature components that correspond to unreachable levels, wherein the hierarchical transition permission vector and the independent feature components have the same dimension; The filtered feature components are subjected to directional probability weighting, and the probability value of each candidate adjustment action is dynamically scaled by a directional weighting coefficient matrix. The directional weighting coefficient matrix is ​​generated based on the changing trend and gradient direction of the historical connection stability characteristics of the Bluetooth terminal. The scaled probability values ​​are normalized, and a final probability distribution that satisfies the probability distribution constraints is generated by a soft maximization function controlled by the temperature parameter. The temperature parameter is adaptively adjusted according to the network diameter of the optimized network topology. When the network diameter increases, the temperature parameter is reduced to enhance the sharpness of the probability distribution.

7. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 1, characterized in that, The step of updating the topology connection relationship of the initial network topology according to the dynamic adjustment strategy to generate an optimized network topology includes: When the target adjustment action is to adjust the level position, the Bluetooth terminal is moved from the current level to the target level, and the original connection is disconnected before establishing a connection with other Bluetooth terminals in the target level. When the target adjustment action is to update the broadcast interval, the broadcast interval of the Bluetooth terminal is adjusted according to the interval increment value in the dynamic adjustment strategy to adapt to the scanning cycle of the master device. When the target adjustment action is to switch the channel frequency hopping sequence, the target channel with the smallest interference value with the current channel is selected from the preset channel set and the frequency hopping sequence of the Bluetooth terminal is updated. The network topology after performing all target adjustment actions is taken as the optimized network topology.

8. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 1, characterized in that, The step of synchronizing the optimized network topology to the master device and all Bluetooth terminals to trigger network configuration operations includes: The optimized network topology is encoded into a set of configuration instructions, which includes connection parameter update instructions and topology update instructions for each Bluetooth terminal. The configuration instruction set is sent to all Bluetooth terminals via a broadcast channel, and a configuration confirmation response is received from each Bluetooth terminal. When all Bluetooth terminals respond with configuration confirmations indicating success, the master device switches to the optimized network topology and enables the new connection parameters. If a Bluetooth terminal fails to return a configuration confirmation response, the retry interval is calculated based on its historical connection stability characteristics, and the corresponding configuration command is resent according to the retry interval until the maximum number of retries is reached or a successful response is received. If the maximum number of retries is reached and the connection still fails, the Bluetooth terminal is marked as an abnormal node and the identification information set is updated.

9. The Bluetooth networking topology self-optimization method based on deep reinforcement learning according to claim 8, characterized in that, The dynamic adjustment strategy also includes topology optimization direction and connection parameter update instructions, wherein the topology optimization direction is determined in the following way: A topology connection diagram is generated based on the hierarchical position of each Bluetooth terminal in the initial network topology, and the network diameter and number of redundant paths in the topology connection diagram are calculated. When the network diameter exceeds a preset diameter threshold, an optimization direction for increasing the core level is generated; When the number of redundant paths is lower than a preset path threshold, an optimization direction for adding backup connection relationships is generated; The optimization direction is combined with the target adjustment action output by the deep reinforcement learning strategy model to generate the dynamic adjustment strategy; Upon receiving the identification information of a new Bluetooth terminal, the master device triggers a real-time optimization operation, which includes: Add the identification information of the new Bluetooth terminal to the identification information set, and add the new Bluetooth terminal to the temporary extension layer of the current network topology according to the device type of the new Bluetooth terminal; During the preset observation period, the connection stability characteristics and signal coverage strength characteristics of the new Bluetooth terminal are periodically collected, and the topology dynamic constraint matrix is ​​updated synchronously according to the device type of the new Bluetooth terminal. When the connection stability feature is lower than the average stability of the current network topology and the standard deviation of the signal coverage strength feature is less than a preset threshold, the deep reinforcement learning strategy model is invoked to generate a real-time adjustment strategy for the new Bluetooth terminal. According to the real-time adjustment strategy, the new Bluetooth terminal is migrated to the target level or its connection parameters are adjusted to integrate into the optimized network topology.

Images