A communication method for marine Internet of Things buoy networking

Abstract

The present invention relates to the field of buoy networking communication technology, and provides a communication method for marine Internet of Things buoy networking, the method comprising: determining the current state of the marine buoy networking system; inputting the current state into a preset marine group target monitoring network to determine the output action; the output action is used to decide the state of the buoy, so that the buoy performs full coverage monitoring of the marine group target; the marine group target monitoring network and the noise depth Q network constitute a double Q network framework; the noise depth Q network includes noise parameters, the network parameters of the marine group target monitoring network are the same as the network parameters of the noise depth Q network, and the noise depth Q network is trained with Markov decision making through a target monitoring training sample set. The present invention can reasonably control the state of the buoy, the buoy networking target monitoring scheduling speed is fast, it is helpful to reduce the buoy energy consumption in the buoy networking, and improve the overall target monitoring probability of the networking.

Description

Technical Field

[0001] The present invention relates to the field of buoy networking communication technology, and in particular to a communication method for marine Internet of Things buoy networking. Background Art

[0002] The continuous development of marine resources by humans has made the monitoring of the marine environment a key issue. In order to better manage and protect these precious resources, marine monitoring buoy systems that integrate monitoring sensors, data collection and processing, and communication technologies have been widely deployed. The marine monitoring buoy networking system deploys multiple buoys in the monitoring area to monitor marine group targets. Through the integration of IoT technology, these buoys can send the collected data to the central server in real time through wireless communication, allowing users to access and analyze this information in real time through mobile devices or online platforms. In addition, IoT technology can also realize the interconnection between buoys to form a multi-hop self-organizing communication network, realize automatic data collection and transmission, and improve work efficiency. The selection and deployment of marine buoys have a decisive impact on the accuracy and practicality of data. How to optimize the deployment and control of buoys has become a research focus in the field of marine IoT. Summary of the invention

[0003] In order to solve the technical problems existing in the prior art, the present invention provides a communication method for marine Internet of Things buoy networking, which can reasonably control the status of the buoy, and the buoy networking target monitoring and scheduling speed is fast, which helps to reduce the buoy energy consumption in the buoy networking and improve the overall target monitoring probability of the network.

[0004] The present invention provides a communication method for marine Internet of Things buoy networking, comprising: determining the current state of a marine buoy networking system; the marine buoy networking system comprises a plurality of buoys and a plurality of marine group targets; the current state comprises the states of the plurality of buoys and the monitoring state of the buoys on the marine group targets; inputting the current state into a preset marine group target monitoring network to determine an output action; the output action is used to decide the state of the buoy so that the buoy performs full coverage monitoring on the marine group targets; the marine group target monitoring network and a noise depth Q network form a double Q network framework; wherein the noise depth Q network comprises noise parameters, and the network parameters of the marine group target monitoring network are the same as the network parameters of the noise depth Q network, performing Markov decision training on the noise depth Q network through a target monitoring training sample set, and assigning the network parameters obtained by the noise depth Q network training to the marine group target monitoring network to obtain the final marine group target monitoring network.

[0005] According to a communication method for marine Internet of Things buoy networking provided by the present invention, the Euclidean distance between the buoy and the marine group target in the marine buoy networking system is:

[0006] ;

[0007] The probability that the buoy detects the ocean group target is:

[0008] ;

[0009] in, For buoy With the Ocean Group Target The Euclidean distance between Targeting the marine community The horizontal axis of For buoy The horizontal axis of Targeting the marine community The vertical coordinate of For buoy The vertical coordinate of For buoy Marine group targets detected The probability of For buoy energy radius.

[0010] According to a communication method for marine Internet of Things buoy networking provided by the present invention, the operating time of each buoy in the marine buoy networking system is:

[0011] ;

[0012] The constraints of the ocean buoy networking system are:

[0013] ;

[0014] in, time is the duration of a feasible solution, is a feasible solution for the buoy network target coverage. The feasible solution is defined as the activation state of each buoy under the condition that the buoy is in full communication state. =1 is a buoy You can send buoys Transmit information, i.e. buoy and buoys The communication link between them is a one-way communication link. For buoy Communication coverage radius.

[0015] According to a communication method for marine Internet of Things buoy networking provided by the present invention, the state space of the Markov decision is:

[0016] ;

[0017] The action space of the Markov decision is:

[0018] ;

[0019] The reward of the Markov decision is:

[0020] ;

[0021] in, for The state space at time, For buoy The activation state of For buoy The remaining energy, Targeting the marine community The number of times the buoy detected the is the action space, For the n Actions, For The action of the moment, argmax To obtain the maximum function, it is defined as choosing a simple The action of the function, For the noisy deep Q network, For The action of the moment, is the noise parameter of the noise deep Q network, for The rewards of the moment, For buoy The collection of monitored marine group targets, For buoy of total energy.

[0022] According to a communication method for marine Internet of Things buoy networking provided by the present invention, the loss function of the Markov decision training of the noise deep Q network is:

[0023] ;

[0024] The action value prediction is:

[0025] ;

[0026] in, loss is the loss function, For the noisy deep Q network, for The state of the moment, are the network parameters of the noisy deep Q network, is the action value prediction, is the discount factor, Network parameters of the network are monitored for the marine swarm target.

[0027] According to a communication method for marine Internet of Things buoy networking provided by the present invention, the noisy deep Q network includes an input layer, a first hidden layer, a second hidden layer and an output layer; the number of neural nodes in the input layer is equal to the dimension of the state data, the number of neural nodes in the first hidden layer is greater than the number of neural nodes in the second hidden layer, and the number of neural nodes in the output layer is equal to the number of buoys in the marine buoy networking system.

[0028] According to a communication method for marine Internet of Things buoy networking provided by the present invention, the Markov decision training of the noisy deep Q network through the target monitoring training sample set includes: initializing the network parameters and the noise parameters of the noisy deep Q network; initializing the cumulative reward; constructing the state space of the Markov decision according to the target monitoring training sample set; selecting the output action according to the state space and the noise parameter execution strategy; updating the reward and the state space; putting the experience data into a memory pool; the experience data includes the output action, the updated reward and the state space; updating the cumulative reward according to the memory pool, and constructing the action value according to the updated cumulative reward. Value prediction; randomly selecting training samples from the experience data of the memory pool to train the noisy deep Q network, and updating the network parameters of the noisy deep Q network according to the loss function, and updating the network parameters of the ocean group target monitoring network after a fixed number of steps of training, so that the network parameters of the ocean group target monitoring network are the same as the network parameters of the noisy deep Q network; outputting the cumulative reward obtained in this round and the running time of the ocean buoy networking system; repeatedly and iteratively executing the step of initializing the cumulative reward to perform the next round of training until the cumulative reward reaches a preset reward value and the running time of the ocean buoy networking system reaches a preset time, thereby completing the training of the noisy deep Q network.

[0029] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the communication method for marine Internet of Things buoy networking as described in any one of the above is implemented.

[0030] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the communication method for marine Internet of Things buoy networking as described in any of the above-mentioned methods is implemented.

[0031] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the communication method for marine Internet of Things buoy networking as described in any of the above.

[0032] The present invention provides a communication method for marine Internet of Things buoy networking, the method comprising: determining the current state of the marine buoy networking system; the marine buoy networking system comprises a plurality of buoys and a plurality of marine group targets; inputting the current state into a preset marine group target monitoring network to determine the output action; the output action is used to determine the state of the buoy, so that the buoy performs full coverage monitoring of the marine group target; the marine group target monitoring network and the noise depth Q network constitute a double Q network framework; the noise depth Q network comprises noise parameters, the network parameters of the marine group target monitoring network are the same as the network parameters of the noise depth Q network, and the noise depth Q network is trained with Markov decision making through a target monitoring training sample set. The present invention can reasonably control the state of the buoy, the buoy networking target monitoring scheduling speed is fast, it is helpful to reduce the buoy energy consumption in the buoy networking, and improve the overall target monitoring probability of the networking. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of a communication method for marine Internet of Things buoy networking provided by the present invention.

[0035] Figure 2 It is a schematic diagram of the communication scenario of the marine Internet of Things buoy networking provided by the present invention.

[0036] Figure 3 It is a structural schematic diagram of a communication device for marine Internet of Things buoy networking provided by the present invention.

[0037] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] With the continuous development of marine resources by humans, marine environmental monitoring has become increasingly important. In order to more effectively protect and manage marine resources, marine environmental monitoring buoy systems and Internet of Things transmission technologies have emerged. These technologies provide more efficient data support for marine environmental monitoring.

[0040] The marine environment monitoring buoy network system is a comprehensive system that integrates sensors, data collection and processing, and communication technology. By deploying buoys at different locations in the ocean, it is possible to monitor seawater temperature, salinity, turbidity, dissolved oxygen and other parameters, as well as environmental factors such as wind speed, wave height, and humidity in real time. These data are of great significance for studying marine ecosystems, climate change, and fishery resource management.

[0041] The buoy system combined with the Internet of Things technology can play a more effective monitoring role. Through the Internet of Things technology, the buoy data can be transmitted to the remote server in real time, and users can view and analyze the data at any time through the mobile application or the web page. In addition, the Internet of Things technology can also realize the interconnection between buoys, realize the automatic collection and transmission of data, and improve work efficiency. The development of the buoy system has gone through several stages. From the initial manual operation to the current automated monitoring, the accuracy and stability of the buoy system have been continuously improved. With the application of new materials and new technologies, the design and manufacture of buoys are also constantly improving to meet different monitoring needs. The Internet of Things transmission technology is the key to realizing efficient monitoring of the buoy system. Through wireless communication, satellite communication and other technologies, the buoy data can be quickly transmitted to the remote data center to realize real-time analysis and processing of data. In addition, the Internet of Things technology can also realize the secure transmission and storage of data, ensuring the security and reliability of data.

[0042] In short, the combination of the marine environment monitoring buoy networking system and the Internet of Things transmission technology provides strong support for marine environmental protection and management. With the continuous advancement and development of technology, we have reason to believe that this comprehensive monitoring system will play a greater role in the future and provide a basis for decision-making for humans to better develop and protect marine resources.

[0043] There are two main existing methods for monitoring and scheduling buoy network targets: one is a heuristic optimization algorithm based on iterative optimization, and the other is a fast optimization algorithm based on reinforcement learning. Among them, the heuristic task allocation method has a long operation time, and it is difficult to meet the real-time requirements when facing the monitoring and scheduling problems with a large number of buoys and targets. The scheduling algorithm based on reinforcement learning often requires continuous debugging of the training environment and network parameters to improve the algorithm scheduling effect during algorithm training, and the overall solution effect of the model is less robust.

[0044] In order to solve the technical problems existing in the prior art, the present invention considers the multi-target monitoring task scheduling scenario under the buoy communication networking scenario, takes maximizing the life cycle of the ocean buoy networking system as the optimization goal, and considers the communication constraints between buoys and the monitoring quantity requirements of ocean group targets. A marine Internet of Things buoy target monitoring scheduling (networking communication) method based on noisy deep Q network reinforcement learning is proposed to formulate activation strategies for multiple buoys.

[0045] Please refer to Figure 1 , Figure 1 A flow chart of a communication method for marine Internet of Things buoy networking provided by the present invention.

[0046] The present invention provides a communication method for marine Internet of Things buoy networking, comprising:

[0047] 101: Determine the current state of the ocean buoy networking system; the ocean buoy networking system includes a plurality of buoys and a plurality of ocean group targets; the current state includes the states of the plurality of buoys and the monitoring state of the buoys on the ocean group targets;

[0048] 102: Input the current state into the preset ocean group target monitoring network to determine the output action; the output action is used to decide the state of the buoy so that the buoy can perform full coverage monitoring of the ocean group targets; the ocean group target monitoring network and the noise depth Q network constitute a double Q network framework; wherein the noise depth Q network includes noise parameters, and the network parameters of the ocean group target monitoring network are the same as the network parameters of the noise depth Q network, and the noisy depth Q network is trained with a target monitoring training sample set for Markov decision training, and the network parameters obtained by the noise depth Q network training are assigned to the ocean group target monitoring network to obtain the final ocean group target monitoring network.

[0049] In the field of marine Internet of Things, buoy networking is an important technical means for monitoring the marine environment and marine group targets. In order to achieve full coverage monitoring of marine group targets by buoys and improve monitoring efficiency and accuracy, the present invention proposes a communication method for marine Internet of Things buoy networking, which first needs to determine the current state of the marine buoy networking system. The marine buoy networking system includes a number of buoys and a number of marine group targets. The state of each buoy includes parameters such as whether it is activated, remaining energy, position, and communication capability, while the monitoring state of the buoy to the marine group target includes parameters such as monitoring range and monitoring accuracy. The determined current state is then input into the preset marine group target monitoring network. The network is a deep learning network for processing the state information and monitoring state information of the buoy. The marine group target monitoring network will determine the output action based on the input current state. The output action is a series of decision instructions used to adjust the state of the buoy to achieve full coverage monitoring of the marine group target.

[0050] The present invention adopts a dual Q network framework, including an ocean group target monitoring network and a noise depth Q network. The ocean group target monitoring network is responsible for processing monitoring tasks, while the noise depth Q network is used to simulate environmental noise and improve the robustness of the monitoring network. The noise depth Q network includes noise parameters, which are the same as the network parameters of the ocean group target monitoring network. The noise depth Q network is trained with a target monitoring training sample set to simulate different environmental noise conditions. The network parameters obtained by the noise depth Q network training are assigned to the ocean group target monitoring network. In this way, the ocean group target monitoring network can better adapt to environmental noise in practical applications and improve the accuracy and stability of monitoring. The finally obtained ocean group target monitoring network is used to perform full coverage monitoring of ocean group targets. By adjusting the state of the buoy, it is ensured that each target is within the monitoring range of at least one buoy to achieve efficient monitoring.

[0051] The present invention can reasonably control the state of the buoy, and the buoy network target monitoring scheduling speed is fast, which helps to reduce the energy consumption of the buoy in the buoy network, improve the overall target monitoring probability of the network, and realize efficient and accurate full coverage monitoring of marine group targets. The application of the dual Q network framework improves the robustness of the monitoring system, enabling it to work stably in a complex marine environment. In addition, through the training of simulating environmental noise, the adaptability of the monitoring network to the actual environment is improved, thereby improving the accuracy of monitoring.

[0052] As a preferred embodiment, the Euclidean distance between the buoy and the ocean group target in the ocean buoy networking system is:

[0053] ;

[0054] The probability of the buoy detecting the ocean group target is:

[0055] ;

[0056] in, For buoy With the Ocean Group Target The Euclidean distance between Targeting the marine community The horizontal axis of For buoy The horizontal axis of Targeting the marine community The vertical coordinate of For buoy The vertical coordinate of For buoy Marine group targets detected The probability of For buoy energy radius.

[0057] In this embodiment, it is assumed that the target coverage problem model for the marine IoT buoy defines the existence of n buoys and m The state of the ocean group target can be stationary. The sensors carried by the buoys need to cover all the ocean group targets, and the buoys need to communicate smoothly. The set of buoys is , the set of stationary ocean group targets is The range of the sensor carried by the buoy is set to be centered on the buoy, the buoy energy is set to 1-3, and the radius is r The coverage relationship between the buoy and the ocean group target is only related to the distance between them. The target coverage problem model for the marine IoT buoy defines the Euclidean distance between the buoy and the ocean group target in the ocean buoy networking system and the probability of the buoy detecting the ocean group target.

[0058] Please refer to Figure 2 , Figure 2 A schematic diagram of the communication scenario of the marine Internet of Things buoy network provided by the present invention.

[0059] The ocean buoy networking system includes 5 buoys and 4 ocean group targets. It is determined that buoy 1, buoy 4 and buoy 5 are in the activated state, buoy 1 and buoy 4 have less remaining energy, and buoy 2 and buoy 3 are in the inactivated state; the current state is input into the preset ocean group target monitoring network, and the output action is determined to activate buoy 2 and buoy 3.

[0060] Among them, buoy 1 can monitor target 1 and target 4, buoy 2 can monitor target 1, target 2 and target 3, buoy 3 can monitor target 2 and target 3, buoy 4 can monitor target 3, and buoy 5 can monitor target 4. If buoy 1 and buoy 4 have no energy, buoy 2 can continue to monitor target 1, and buoy 3 can continue to monitor target 3, so that the buoys can fully monitor the marine group targets, maximize the life cycle of the marine buoy networking system, and take into account the communication constraints between buoys and the monitoring quantity requirements of the marine group targets.

[0061] As a preferred embodiment, the operating time of each buoy in the ocean buoy networking system is:

[0062] ;

[0063] The constraints of the ocean buoy networking system are:

[0064] ;

[0065] in, time is the duration of a feasible solution, is a feasible solution for the buoy network target coverage. The feasible solution is defined as the activation state of each buoy under the condition that the buoy is in full communication state. =1 is a buoy You can send buoy Transmit information, i.e. buoy and buoys The communication link between them is a one-way communication link. For buoy Communication coverage radius.

[0066] In this embodiment, the operation time of the entire buoy networking system is set as the objective function of the target coverage problem. The network connectivity between buoys in the buoy networking system is a constraint condition for the target coverage problem. The definition of whether two buoys are connected is related to the straight-line distance between the two and their corresponding communication radius. The formation of a buoy networking system requires two-way connectivity between all buoys. When the buoy is activated, 1 unit of energy is consumed per unit time to keep the communication link and sensor running.

[0067] As a preferred embodiment, the state space of Markov decision making is:

[0068] ;

[0069] The action space of Markov decision is:

[0070] ;

[0071] The reward for Markov decision is:

[0072] ;

[0073] in, for The state space at time, For buoy The activation state of For buoy The remaining energy, Targeting the marine community The number of times the buoy detected the is the action space, For the n Actions, For The action of the moment, argmax To obtain the maximum function, it is defined as choosing a simple The action of the function, is a noisy deep Q network, For The action of the moment, is the noise parameter of the noise deep Q network, for The rewards of the moment, For buoy The collection of monitored marine group targets, For buoy of total energy.

[0074] In this embodiment, a Markov decision process in reinforcement learning is defined.

[0075] Among them, the state space of reinforcement learning consists of the state of the buoy (whether it is activated, the remaining energy) and the coverage of the target. =1 means float activation, =0 means float Not activated. In the action space of reinforcement learning, the agent selects a buoy node to activate, and the agent makes decisions based on the noise behavior selection strategy. The noise parameter is used to characterize the noise action.

[0076] In the buoy networking system, a reward function is constructed based on the number of ocean group targets covered by the buoy and the energy consumption of the buoy networking system. The specific construction principle is: the reward is positively correlated with the number of ocean group targets monitored by a single buoy; under the premise of the same monitoring quantity, the reward is positively correlated with the residual energy of the buoy networking system.

[0077] It should be noted that Markov property means that in a random process, the state at the next moment is only related to the current state and has nothing to do with the previous state. If in a random process, any two states satisfy the Markov property, then the random process is a Markov random process. A Markov random process can be described by a binary (S, P), where S is the finite state set of the process and P is the transition probability matrix between these states. The Markov decision process (MDP) is based on the Markov random process.

[0078] As a preferred embodiment, the loss function for the Markov decision training of the noisy deep Q network is:

[0079] ;

[0080] The action value prediction is:

[0081] ;

[0082] in, loss is the loss function, is a noisy deep Q network, for The state of the moment, are the network parameters of the noise deep Q network, is the action value prediction, is the discount factor, the value is 0.8, Network parameters of the network for marine swarm targeting monitoring.

[0083] As a preferred embodiment, the noisy deep Q network includes an input layer, a first hidden layer, a second hidden layer and an output layer; the number of neural nodes in the input layer is equal to the dimension of the state data, the number of neural nodes in the first hidden layer is greater than the number of neural nodes in the second hidden layer, and the number of neural nodes in the output layer is equal to the number of buoys in the ocean buoy networking system.

[0084] In this embodiment, a double Q network framework is used to construct a noise deep Q network. The marine group target monitoring network and the noise deep Q network have the same network structure but different parameters. The noise deep Q network is constructed using 4 fully connected layers, where the number of neural nodes in the input layer is equal to the dimension of the state data, the number of neural nodes in the first hidden layer is 256, the number of neural nodes in the second hidden layer is 128, the number of neural nodes in the output layer is equal to the number of buoys in the buoy networking system, and the noise deep Q network uses Relu as the activation function.

[0085] A noisy network adds noise to the parameter space. A noisy network means that at the beginning of each round, when the agent interacts with the environment, the agent uses the Q function to take action. The Q function is a network. Adding a Gaussian noise to each parameter of the network turns the original Q function into a noise Q-function. By adding a Gaussian noise to each parameter, we get a new network .

[0086] As a preferred embodiment, the Markov decision training of the noisy deep Q network is performed through the target monitoring training sample set, including: initializing the network parameters and noise parameters of the noisy deep Q network; initializing the cumulative reward; constructing the state space of the Markov decision according to the target monitoring training sample set; selecting the output action according to the state space and noise parameter execution strategy; updating the reward and state space; putting the experience data into the memory pool; the experience data includes the output action, the updated reward and the state space; updating the cumulative reward according to the memory pool, and constructing the action value prediction according to the updated cumulative reward; randomly selecting training samples from the experience data in the memory pool to train the noisy deep Q network, and updating the network parameters of the noisy deep Q network according to the loss function, and updating the network parameters of the marine group target monitoring network after a fixed number of steps of training, so that the network parameters of the marine group target monitoring network are the same as the network parameters of the noisy deep Q network; outputting the cumulative reward obtained in this round and the running time of the marine buoy networking system; repeating the step of initializing the cumulative reward for the next round of training until the cumulative reward reaches the preset reward value and the running time of the marine buoy networking system reaches the preset time, and the training of the noisy deep Q network is completed.

[0087] In this embodiment, a target monitoring training sample set of 500,000 groups is first generated. In the randomly generated sample set, the buoy coordinate position range is , the number of buoys is , buoy sensing radius , buoy sensing radius , the coordinate position range of the ocean group target node is , the number of target nodes in the ocean group is .

[0088] The Markov decision training process of the noise deep Q network using the target monitoring training sample set is as follows:

[0089] (1) Initialize the network parameters of the noise Q network and noise parameters The parameters of the ocean group target monitoring network and the noise Q network use the same settings;

[0090] (2) Start training and initialize the accumulated rewards;

[0091] (3) Construct state space based on training data;

[0092] (4) The agent selects actions based on the state space and the noise action execution strategy;

[0093] (5) Update rewards and state space;

[0094] (6) Put the experience data into the memory pool; the experience data includes the output action, the updated reward and the state space;

[0095] (7) Update the accumulated reward and construct the predicted action value;

[0096] (8) Randomly select training samples from the experience data in the experience pool for training, and update the parameters of the noise Q network. After a fixed number of steps of training, update the parameters of the ocean group target monitoring network to keep them consistent with the parameters of the noise Q network.

[0097] (9) Output the cumulative reward of this training and the running time of the ocean buoy networking system;

[0098] (10) Go to step (2) for the next round of training until the training convergence condition is met (the cumulative reward reaches the preset reward value and the running time of the ocean buoy networking system reaches the preset time), completing the training of the noisy deep Q network.

[0099] The present invention uses a scheduling method based on noise Q learning to learn the scheduling strategy of buoy networking according to the energy of buoys in the buoy network, whether they are activated and the monitoring of targets, so as to control the activation of buoys in the network and improve the number of monitored targets under the constraint of saving the energy of the buoy network.

[0100] Since the present invention uses a noisy Q network to implement a node scheduling strategy for buoy networking, it overcomes the problem in the prior art that a large amount of training is required to improve the algorithm exploration performance when dealing with buoy networking scheduling problems. The present invention has the characteristics of good algorithm solving performance and fast scheduling speed, which helps to reduce the energy consumption of buoys in the buoy network and improve the overall target monitoring probability of the network.

[0101] The communication device for marine Internet of Things buoy networking provided by the present invention is described below. The communication device for marine Internet of Things buoy networking described below and the communication method for marine Internet of Things buoy networking described above can be referenced to each other.

[0102] Please refer to Figure 3 , Figure 3 A schematic structural diagram of a communication device for marine Internet of Things buoy networking provided by the present invention.

[0103] The present invention also provides a communication device for marine Internet of Things buoy networking, including: a state determination module 301, used to determine the current state of the marine buoy networking system; the marine buoy networking system includes a plurality of buoys and a plurality of marine group targets; the current state includes the states of the plurality of buoys and the monitoring state of the buoys on the marine group targets; an action determination module 302, used to input the current state into a preset marine group target monitoring network to determine the output action; the output action is used to decide the state of the buoy so that the buoy performs full coverage monitoring of the marine group targets; the marine group target monitoring network and the noise depth Q network constitute a double Q network framework; wherein the noise depth Q network includes noise parameters, and the network parameters of the marine group target monitoring network are the same as the network parameters of the noise depth Q network, the noise depth Q network is Markov decision trained through the target monitoring training sample set, and the network parameters obtained by the noise depth Q network training are assigned to the marine group target monitoring network to obtain the final marine group target monitoring network.

[0104] Figure 4 An example of a structural diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute a communication method for marine Internet of Things buoy networking, the method comprising: determining the current state of the marine buoy networking system; the marine buoy networking system comprises a plurality of buoys and a plurality of marine group targets; the current state comprises the state of the plurality of buoys and the monitoring state of the buoys on the marine group targets; the current state is input into a preset marine group target monitoring network to determine the output action; the output action is used to decide the state of the buoy so that the buoy can perform full coverage monitoring of the marine group targets; the marine group target monitoring network and the noise depth Q network constitute a dual Q network framework; wherein the noise depth Q network comprises noise parameters, and the network parameters of the marine group target monitoring network are the same as the network parameters of the noise depth Q network, the noise depth Q network is Markov decision trained through the target monitoring training sample set, and the network parameters obtained by the noise depth Q network training are assigned to the marine group target monitoring network to obtain the final marine group target monitoring network.

[0105] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the communication method for marine Internet of Things buoy networking provided by the above methods, the method including: determining the current state of the marine buoy networking system; the marine buoy networking system includes a number of buoys and a number of marine group targets; the current state includes the state of the several buoys and the monitoring state of the buoys on the marine group targets; the current state is input into a preset marine group target monitoring network to determine the output action; the output action is used to decide the state of the buoy so that the buoy performs full coverage monitoring of the marine group targets; the marine group target monitoring network and the noisy deep Q network constitute a dual Q network framework; wherein the noisy deep Q network includes noise parameters, and the network parameters of the marine group target monitoring network are the same as the network parameters of the noisy deep Q network, the noisy deep Q network is Markov decision trained through the target monitoring training sample set, and the network parameters obtained by the training of the noisy deep Q network are assigned to the marine group target monitoring network to obtain the final marine group target monitoring network.

[0107] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a communication method for marine Internet of Things buoy networking provided by the above-mentioned methods, the method comprising: determining the current state of the marine buoy networking system; the marine buoy networking system comprises a plurality of buoys and a plurality of marine group targets; the current state comprises the state of the plurality of buoys and the monitoring state of the buoys on the marine group targets; the current state is input into a preset marine group target monitoring network to determine an output action; the output action is used to decide the state of the buoy so that the buoy performs full coverage monitoring of the marine group targets; the marine group target monitoring network and the noisy deep Q network constitute a dual Q network framework; wherein the noisy deep Q network comprises noise parameters, and the network parameters of the marine group target monitoring network are the same as the network parameters of the noisy deep Q network, the noisy deep Q network is Markov decision trained through a target monitoring training sample set, and the network parameters obtained by the training of the noisy deep Q network are assigned to the marine group target monitoring network to obtain the final marine group target monitoring network.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication method for marine Internet of Things buoy networking, characterized in that: include: Determine the current state of the ocean buoy networking system; the ocean buoy networking system includes a plurality of buoys and a plurality of ocean group targets; the current state includes the states of the plurality of buoys and the monitoring state of the buoys on the ocean group targets; Inputting the current state into a preset ocean group target monitoring network to determine an output action; The output action is used to determine the state of the buoy so that the buoy can perform full coverage monitoring of the ocean group target; The ocean group target monitoring network and the noisy deep Q network constitute a dual Q network framework; wherein, the noisy deep Q network includes noise parameters, and the network parameters of the ocean group target monitoring network are the same as the network parameters of the noisy deep Q network. The noisy deep Q network is subjected to Markov decision training through a target monitoring training sample set, and the network parameters obtained by the noisy deep Q network training are assigned to the ocean group target monitoring network to obtain the final ocean group target monitoring network.

2. The communication method for marine Internet of Things buoy networking according to claim 1 is characterized in that: The Euclidean distance between the buoy and the ocean group target in the ocean buoy networking system is: ； The probability that the buoy detects the ocean group target is: ； in, For buoy b n With the Ocean Group Target t m The Euclidean distance between x m Targeting the marine community t m The horizontal axis of x n For buoy b n The horizontal axis of y m Targeting the marine community t m The vertical coordinate of y n For buoy b n The vertical coordinate of p ( b n , t m ) is a buoy b n Marine group targets detected t m The probability of r For buoy b n energy radius.

3. The communication method for marine Internet of Things buoy networking according to claim 1 is characterized in that: The operating time of each buoy in the ocean buoy networking system is: ； The constraints of the ocean buoy networking system are: ； in, time is the duration of a feasible solution, is a feasible solution for the buoy network target coverage. The feasible solution is defined as the activation state of each buoy under the condition that the buoy is in full communication state. For buoy b i You can send buoy b j Transmit information, i.e. buoy b i and buoys b j The communication link between them is a one-way communication link. r c ( b i ) is a buoy b i Communication coverage radius.

4. The communication method for marine Internet of Things buoy networking according to claim 1 is characterized in that: The state space of the Markov decision is: ； The action space of the Markov decision is: ； The reward of the Markov decision is: ； in, S t for t The state space at time, s n For buoy b n The activation state of For buoy b n The remaining energy, Targeting the marine community t m The number of times the buoy detected the is the action space, a n For the n Actions, For t The action of the moment, argmax To obtain the maximum function, it is defined as choosing a simple The action of the function, For the noisy deep Q network, a t-1 For t -1 moment of action, is the noise parameter of the noise deep Q network, r t for t The rewards of the moment, Targeting the marine community t m The number of times the buoy detected the For buoy b n The collection of monitored marine group targets, For buoy b n of total energy.

5. The communication method for marine Internet of Things buoy networking according to claim 4 is characterized in that: The loss function for the Markov decision training of the noisy deep Q network is: ； The action value prediction is: ； in, loss is the loss function, For the noisy deep Q network, S t+1 for t +1 moment status, are the network parameters of the noisy deep Q network, is the action value prediction, is the discount factor, Network parameters of the network are monitored for the marine swarm target.

6. The communication method for marine Internet of Things buoy networking according to claim 1 is characterized in that: The noisy deep Q network includes an input layer, a first hidden layer, a second hidden layer and an output layer; the number of neural nodes in the input layer is equal to the dimension of the state data, the number of neural nodes in the first hidden layer is greater than the number of neural nodes in the second hidden layer, and the number of neural nodes in the output layer is equal to the number of buoys in the ocean buoy networking system.

7. The communication method for marine Internet of Things buoy networking according to claim 5 is characterized in that: The Markov decision training of the noise deep Q network using the target monitoring training sample set includes: Initialize the network parameters of the noise deep Q network and the noise parameters; initialize the accumulated reward; Constructing the state space of the Markov decision according to the target monitoring training sample set; An execution strategy selects an output action according to the state space and the noise parameter; updating the reward and the state space; Putting experience data into a memory pool; the experience data includes the output action, the updated reward and the state space; updating the accumulated reward according to the memory pool, and constructing the action value prediction according to the updated accumulated reward; Randomly select training samples from the experience data of the memory pool to train the noisy deep Q network, and update the network parameters of the noisy deep Q network according to the loss function, and update the network parameters of the marine group target monitoring network after training for a fixed number of steps, so that the network parameters of the marine group target monitoring network are the same as the network parameters of the noisy deep Q network; Output the cumulative reward obtained in this round and the running time of the ocean buoy networking system; Return to the step of initializing the cumulative reward to perform the next round of training and repeat the iteration until the cumulative reward reaches the preset reward value and the running time of the ocean buoy networking system reaches the preset time, completing the training of the noisy deep Q network.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the communication method for marine Internet of Things buoy networking as described in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the communication method for marine Internet of Things buoy networking as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the communication method for marine Internet of Things buoy networking as described in any one of claims 1 to 7 is implemented.

Images