Method for accessing multiple heterogeneous devices to time-sensitive network based on deep reinforcement learning

Abstract

The invention provides a method for accessing multiple heterogeneous devices to a time-sensitive network based on deep reinforcement learning, and particularly relates to the technical field of network communication, and the scheme comprises the steps: obtaining a current state of a heterogeneous network environment, inputting the current state to a trained deep reinforcement learning network model, and generating an optimal access strategy; and based on the optimal access strategy, executing the operation of accessing the corresponding equipment in the time-sensitive network. According to the scheme, deep reinforcement learning is applied to access management of nuclear power site multi-heterogeneous equipment, a time-sensitive network and a deep neural network are combined, the environment perception capability is improved through the deep neural network, optimization of access decisions is realized by utilizing reinforcement learning, a complex network environment and diversified equipment requirements are adapted, and the access management of nuclear power site multi-heterogeneous equipment is realized. And the high-reliability requirement of accessing various nuclear power field heterogeneous equipment to the time-sensitive network can be met.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning. Background Art

[0002] As a key infrastructure in the industrial field, nuclear power plants are accelerating their digitalization and intelligentization process. During the operation and maintenance of nuclear power field-level equipment, they need to face a variety of heterogeneous equipment access requirements, which puts extremely high demands on the real-time and reliability of the network.

[0003] The existing nuclear power site equipment access technology mainly relies on the traditional network architecture, which limits the type and number of access devices and cannot meet the high-reliability access requirements of various heterogeneous equipment on the nuclear power site. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning, aiming to solve the problem that the prior art cannot meet the high-reliability network access requirements of various heterogeneous devices on nuclear power sites.

[0005] In order to achieve the above object, the first aspect of the present invention provides a method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning, comprising:

[0006] Obtaining the current state of the heterogeneous network environment, where the current state is used to reflect the current network environment state and the operating state and service requirements of each device in the heterogeneous network;

[0007] Inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy;

[0008] Based on the optimal access strategy, an operation of accessing a corresponding device in the time-sensitive network is performed.

[0009] Optionally, obtaining the current state of the heterogeneous network environment includes:

[0010] Collecting status information of the heterogeneous network environment, wherein the status information includes at least one of location information, operation status information, and service demand information of each device in the heterogeneous network;

[0011] Based on at least one of the state information, a current state of the heterogeneous network environment is generated.

[0012] Optionally, inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy includes:

[0013] In response to the trained deep reinforcement learning network model receiving the current state, outputting a target action for each of the devices, the target action being used to represent a mapping relationship between the device and the target access location;

[0014] Based on all said target actions, an optimal access strategy is generated.

[0015] Optionally, the performing an operation of accessing a corresponding device in a time-sensitive network based on the optimal access strategy includes:

[0016] Based on the optimal access strategy, generating a network configuration instruction;

[0017] Determine the access location and configuration information of each of the devices according to the network configuration instructions;

[0018] An operation of accessing a corresponding device in a time-sensitive network is performed according to the access location and the configuration information.

[0019] Optionally, the training process of the deep reinforcement learning network model includes:

[0020] Acquire the historical status of the heterogeneous network environment, where the historical status is used to reflect the environmental status of the heterogeneous network and the operating status and service requirements of each device within a preset time period;

[0021] Inputting the historical state into a preset policy function and outputting an action, wherein the action is used to represent a mapping relationship between the device and the predicted access location;

[0022] In response to the next state of the action, a reward value is obtained;

[0023] Based on the reward value, the initial deep reinforcement learning network model is iteratively trained to obtain a trained deep reinforcement learning network model.

[0024] Optionally, obtaining a reward value in response to a next state of the action includes:

[0025] In response to the next state of the action, determining business requirement satisfaction and network connection quality;

[0026] A reward value is obtained based on the business demand satisfaction and the network connection quality.

[0027] Optionally, the iterative training of the initial deep reinforcement learning network model based on the reward value to obtain a trained deep reinforcement learning network model includes:

[0028] Based on each of the reward values, a corresponding experience sample is obtained, wherein the experience sample includes a state, an action, and a reward value of a heterogeneous network environment;

[0029] Using a number of the experience samples to train a Q-value estimation network to obtain a reward value;

[0030] Based on the reward value and the preset expected reward value, the initial deep reinforcement learning network model is iteratively trained to obtain a trained deep reinforcement learning network model.

[0031] A second aspect of the present invention provides a system for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning, the system comprising:

[0032] An information acquisition module is used to acquire the current state of the heterogeneous network environment, where the current state is used to reflect the current network environment state and the operating state and service requirements of each device in the heterogeneous network;

[0033] A decision module, used for inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy;

[0034] The access module is used to execute the operation of accessing the corresponding device in the time-sensitive network based on the optimal access strategy.

[0035] The third aspect of the present invention provides an intelligent terminal, which includes a memory for storing executable instructions; and a processor for calling and running the executable instructions in the memory to execute any step of the above-mentioned method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning.

[0036] A fourth aspect of the present invention provides a computer-readable storage medium, which stores program instructions. When the program instructions are executed by a processor, any step of the above-mentioned method of connecting multiple heterogeneous devices to a time-sensitive network based on deep reinforcement learning is implemented.

[0037] Compared with the prior art, the beneficial effects of this solution are as follows:

[0038] This application generates an optimal access strategy by inputting the current state of the heterogeneous network environment into a trained deep reinforcement learning network model, and based on the optimal access strategy, executes the operation of accessing the corresponding device in the time-sensitive network. It can be seen that the present invention applies deep reinforcement learning to the access management of multiple heterogeneous devices at the nuclear power site, combines the time-sensitive network with the deep neural network, improves the environmental perception ability through the deep neural network, and optimizes the access decision by using reinforcement learning, adapting to complex network environments and diversified equipment requirements, and can meet the high reliability requirements of heterogeneous devices at various nuclear power sites to access the time-sensitive network. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of the architecture of multiple heterogeneous devices accessing a time-sensitive network according to the present invention;

[0041] Figure 2 A flow chart of a method for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning according to the present invention;

[0042] Figure 3 A flow chart of the training process of the deep reinforcement learning network model of the present invention;

[0043] Figure 4 A schematic diagram of a system module for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning according to the present invention;

[0044] Figure 5 It is a schematic diagram of the structure of the intelligent terminal of the present invention. DETAILED DESCRIPTION

[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0046] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0047] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0048] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0050] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] In the existing technology, Time-Sensitive Networking (TSN), as a new network architecture, provides high-precision time synchronization and low-latency data transmission guarantee, which is suitable for scenarios with high real-time requirements such as industrial control. However, the existing TSN gateway technology is limited in the type and number of access devices and cannot meet the access requirements of many heterogeneous devices at nuclear power sites. Due to the different communication protocols and data formats used by different devices, it is difficult for existing technologies to achieve highly standardized and unified protocol conversion.

[0053] Reinforcement Learning (RL) is a machine learning technology that continuously adjusts its strategy through the interaction between the agent and the environment to maximize the cumulative reward. In terms of network access selection, the RL algorithm can automatically select the optimal access node based on historical data and current network status. Traditional RL algorithms have low learning efficiency when faced with complex network environments and diverse devices, and it is difficult to form an effective access strategy in a short period of time. RL algorithms mainly rely on historical experience and immediate feedback, and lack the ability to fully perceive the environmental state, resulting in limited decision-making reliability in a dynamically changing network environment.

[0054] Deep reinforcement learning (DRL) combines the advantages of deep learning and reinforcement learning, and improves environmental perception and decision-making efficiency through deep neural networks. Although DRL has achieved remarkable results in many fields, it has been less used in nuclear power plant equipment access.

[0055] Therefore, a new technical solution is needed to combine the high real-time performance of TSN and the intelligent decision-making ability of DRL, design an efficient network gateway architecture that is suitable for the access of multiple heterogeneous devices at nuclear power sites, and propose a method for accessing time-sensitive networks of multiple heterogeneous devices based on this architecture to solve the problem that the existing technology of multiple heterogeneous devices at nuclear power sites accessing time-sensitive networks cannot meet the diversity and high reliability requirements.

[0056] Based on the above analysis, the embodiment of the present invention proposes an architecture for multiple heterogeneous devices to access a time-sensitive network, such as Figure 1As shown, the architecture mainly includes three modules: heterogeneous network environment, terminal environment agent module, and decision module based on deep reinforcement learning (DRL). Among them, the heterogeneous network environment includes a variety of heterogeneous devices and network nodes at the nuclear power site, wherein the network node itself represents the location set in the network for accessing the nuclear power site-level equipment. The network where the network node is located can be a simple TSN network or a fusion network between 5G, 6G and other wireless networks and TSN networks. The present invention does not make specific restrictions on this. All methods for accessing the corresponding network through network nodes based on the concept of the present invention are within the protection scope of the present invention. The terminal environment agent module is responsible for collecting the status information of the network and field-level equipment (such as nuclear power equipment) through the information acquisition module. The status information includes at least one of the location information, operation status information and business demand information of each device in the heterogeneous network, such as signal strength, equipment type, load condition, etc., and interacts with the decision module generated based on deep reinforcement learning. The decision module is used to input the current state into the trained deep reinforcement learning network model and generate the best access strategy according to the current state. The trained deep reinforcement learning network model includes a data collection unit, a state integration unit, a strategy generation unit, an action execution unit, a reward calculation unit, a network update unit and a system optimization unit. The data collection unit is used to collect the status information of the heterogeneous network environment, including the signal strength, delay, packet loss rate and jitter of each device, and collects the location information, device operation status information and business requirements of each device; the state integration unit is used to generate the historical status of the heterogeneous network environment by one or more information collected by the data collection unit; the strategy generation unit is used to output actions using the historical status information through the strategy function, and the actions are used to represent the mapping relationship between the device and the predicted access location; the action execution unit is used to perform the access operation at the access location corresponding to the action output by the strategy generation unit, and generate the execution result in response to the access operation, and determine the business demand satisfaction and network connection quality according to the execution result; the reward calculation unit is used to calculate the reward value according to the business demand and the network state corresponding to the execution result; the network update unit is used to update the deep reinforcement learning network model according to the reward value; the system optimization unit is used to use the deep reinforcement learning network model after the current round of update, reuse the status information of the heterogeneous network environment collected by the data collection unit for iterative update, and obtain the trained deep reinforcement learning network model. The access module is used to perform the operation of accessing the corresponding device in the time-sensitive network through a number of reserved access points according to the optimal access strategy.

[0057] Based on the above architecture, the embodiment of the present invention provides a method for accessing multiple heterogeneous devices to a time-sensitive network based on deep reinforcement learning, which is deployed on electronic devices such as computers and servers and is applied to the scenario of accessing various types of nuclear power field-level heterogeneous devices to an existing time-sensitive network. The present invention does not limit the type of nuclear power field-level devices, which can be various electrical devices such as motors, electrical appliances, cables, batteries, chargers, control protection devices, sensors, etc. Specifically, Figure 2 As shown, the steps of the method in this embodiment include:

[0058] Step S100: Acquire the current state of the heterogeneous network environment, where the current state is used to reflect the current network environment state and the operating state and service requirements of each device in the heterogeneous network;

[0059] Specifically, the state information of the heterogeneous network environment is collected in real time through the terminal environment agent module, and the state information includes at least one of the location information, operation state information and business demand information of each device in the heterogeneous network, wherein the operation state information includes the signal strength, delay, packet loss rate and jitter of each device. Then, based on at least one of the state information, the current state of the heterogeneous network environment is generated, and the current state is used to describe the business needs of the current network environment and the device. For example, for a sensor that requires low latency and a controller that requires high bandwidth, the state can be expressed as a vector containing these device types and network demand parameters.

[0060] Step S200: inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy;

[0061] Specifically, in response to the trained deep reinforcement learning network model receiving the current state, a target action for each of the devices is output, and the target action is used to represent the mapping relationship between the device and the target access location (i.e., the access point); based on all the target actions, an optimal access strategy is generated. This embodiment realizes intelligent access management of heterogeneous devices at nuclear power sites by combining a trained deep reinforcement learning network with the decision-making capabilities of deep neural networks and reinforcement learning, and can perceive complex network environments, dynamically adjust access strategies, and ensure real-time and efficient device access.

[0062] Step S300: Based on the optimal access strategy, an operation of accessing a corresponding device in a time-sensitive network is performed.

[0063] Specifically, based on the optimal access strategy constructed by the trained deep reinforcement learning network, a network configuration instruction is generated; according to the network configuration instruction, the access location and configuration information of each of the devices are determined; according to the access location and the configuration information, an operation of accessing the corresponding device in the time-sensitive network is performed. For example, if the action selected by the model is [sensor: access location A; controller: access location B], the system will configure the network accordingly to enable this action to be implemented. It can be seen that in this embodiment, by using deep reinforcement learning technology, the system can dynamically select the most suitable network access location (i.e., access location) for each device.

[0064] This embodiment combines TSN with DRL, fully utilizing the performance of TSN technology to provide high-precision time synchronization and low-latency data transmission guarantee, supports the access of multiple heterogeneous devices, improves the compatibility and scalability of the system, and ensures that the overall performance and service quality of the network are not affected when multiple heterogeneous devices are accessed concurrently, thereby meeting the high requirements of nuclear power sites for real-time and reliability.

[0065] In a preferred embodiment, since the deep reinforcement learning network has the ability to self-learn, the system can quickly adapt and make the best decision when facing new devices and new network states, while taking into account the diversity, heterogeneity and diversity of business needs of various types of heterogeneous devices. This embodiment uses the deep reinforcement learning network model to continuously perform online learning, so that the system can continuously adjust the access strategy according to the changes in the on-site equipment and network environment, forming an intelligent access location selection mechanism, improving its adaptive ability in abnormal network environments, reducing the failure rate that may occur during device access, and ensuring that the system maintains the best operating state to optimize the utilization rate of network resources and improve the overall service quality.

[0066] Furthermore, redundant backup mechanisms can be used according to actual application requirements. For example, in terms of hardware, multiple servers can be deployed as computing nodes, and the continuous operation of the system can be ensured through load balancing and failover mechanisms. In terms of software, in the deep reinforcement learning framework, multiple processes or threads can be designed to perform the same task, and / or, in the process of data transmission and storage, a multi-copy strategy can be adopted to ensure data redundancy. In terms of network construction, in network communication, multi-path transmission technology can be used to ensure reliable data transmission, and / or, network reliability and stability can be improved through redundant configuration of network equipment. That is, by optimizing algorithms and redundant mechanisms, the reliability of the system in a complex network environment can be improved, and the failure rate of device access can be reduced.

[0067] It is worth stating that the embodiment of the present invention takes the nuclear power dedicated network as an example, and introduces in detail the design of a time-sensitive network architecture for accessing multiple heterogeneous devices, as well as the specific implementation process of a method for accessing a time-sensitive network for multiple heterogeneous devices based on deep reinforcement learning based on the architecture design. However, the application scenarios of the present invention are not limited to nuclear power dedicated networks. Any network architecture that can be implemented based on the design concept of the present invention and the method for accessing multiple heterogeneous devices to the network are within the protection scope of the present invention.

[0068] like Figure 3 As shown, in one embodiment, the training process of the deep reinforcement learning network model includes:

[0069] Step M100: Acquire the historical status of the heterogeneous network environment, where the historical status is used to reflect the environmental status of the heterogeneous network and the operating status and service requirements of each device within a preset time period;

[0070] Specifically, the terminal environment agent module collects the status information of the heterogeneous network environment, including the signal strength, delay, packet loss rate and jitter of each device, and collects the location information, device operation status information and service requirements of each device. Using at least one of the status information, the historical status of the heterogeneous network environment is generated. For example, using the network status N t , Equipment operating status τ t and business needs t Composition history state S t ,Right now:

[0071] S t = {N t ,τ t ,U t} (1)

[0072] Among them, S t is the state of the heterogeneous network environment at time t, N t is the network environment state, indicating the network nodes and signal information at time t; τ t U is the equipment operation status, indicating the location information and status information at time t; t is the business demand, which indicates the bandwidth demand, delay demand, etc. at time t.

[0073] Step M200: inputting the historical state into a preset policy function and outputting an action, wherein the action is used to represent a mapping relationship between the device and the predicted access location;

[0074] Specifically, the historical state is input into a preset policy function, and an action is output, wherein the action is used to represent the mapping relationship between the device and the predicted access location. Then, the predicted access location of the corresponding device can be determined through the action according to the mapping relationship, and each action represents a mapping from a specific device to a specific access location. For example, the action output at time t is a t , a t ∈{a1,a2,…,a m}, represents the access position of the corresponding device selected at time t, where the subscript represents the number of access positions. Since the ∈-greedy strategy (epsilon-greedy strategy) is a commonly used strategy in machine learning and reinforcement learning, it is mainly used to find a balance between exploration and exploitation. This strategy is implemented by introducing a parameter ∈ (epsilon) between 0 and 1, where ∈ represents the probability of exploration and 1-∈ represents the probability of exploitation. Therefore, in a preferred embodiment, the ∈-greedy strategy can be selected as the strategy function. It is easy to understand that other functions that can implement policy decisions can also be selected as strategy functions based on actual requirements such as calculation accuracy, calculation amount and efficiency, and the present invention does not make specific restrictions on this.

[0075] Step M300: obtaining a reward value in response to the next state of the action;

[0076] Specifically, based on the deep reinforcement learning network, the system uses a policy function to select an access action a from the possible action space. t , the decision process depends on the state S t Takes as input and outputs an action a t , the action is the selected access location, and then responds to the action a t The next state of the terminal environment agent module determines the business demand satisfaction and network connection quality based on the execution results. The terminal environment agent module uses a deep reinforcement learning network to evaluate the potential effect of each mapping to calculate the reward value R t The reward calculation is based on evaluating the actual network performance of the system, including the quality of the device's network connection and the satisfaction of business needs. It can be seen that the reward includes both rewards based on the successful connection of the device and additional rewards or penalties based on the satisfaction of the device's business needs (such as latency and bandwidth).

[0077] Step M400: Based on the reward value, iteratively train the initial deep reinforcement learning network model to obtain a trained deep reinforcement learning network model.

[0078] Specifically, based on each of the reward values, a corresponding experience sample (St ,a t ,R t ,S t+1 ), where R t represents the reward value obtained at time t, S t The invention discloses a method for obtaining a state of a heterogeneous network environment at time t+1, wherein the experience sample includes the state, action and reward value of the heterogeneous network environment, and stores the experience sample in an experience pool, and uses the experience pool to record the state, action, reward and next state (i.e., transfer result) experienced by the heterogeneous device in the process of interacting with the environment. As time goes by, the number of experience samples in the experience pool increases, so that it is convenient to select experience samples from the experience pool to train and update the deep reinforcement learning network model in the future; a Q-value estimation network in the initial deep reinforcement learning network model is trained by regularly sampling a number of the experience samples from the experience pool to obtain a reward value; based on the reward value and a preset expected reward value, the initial deep reinforcement learning network model is iteratively trained to obtain a trained deep reinforcement learning network model. Specifically, use several of the experience samples to train the Q-value estimation network to obtain the Q-value, which is used to represent the expected reward of selecting the next action in the current state; obtain the real Q-value of the real Q-value network at the time corresponding to the Q-value, and calculate the Q-value deviation between the Q-value and the real Q-value; based on the Q-value deviation, iteratively train the deep reinforcement learning network model to update the parameters of the Q-value estimation network and the real Q-value network to obtain a trained deep reinforcement learning network model. In this way, the deep reinforcement learning network model is continuously optimized to adapt to changes in network status and dynamic needs of the device.

[0079] For example, suppose that in a certain round of training, the Q value output by the Q-value estimation network is Q(S t ,a t ),Right now:

[0080]

[0081] Among them, Q(S t ,a t ) is in state S t Next select action a t The expected return R t is the reward value, indicating that action a is selected t γ is the discount factor, which indicates the decay value of future rewards.

[0082] Then calculate the reward value R based on business needs and network status t ,Right now:

[0083] R t (S t ,a t)=α(a t )q(S t ,a t ) (3)

[0084] Among them, R t Indicates that in state S t Next select action a t The reward value of t ) indicates action a t Function of q(S t ,a t ) represents the satisfaction of business demand (SBD) and the quality of network connection (QoC), where the satisfaction of business demand includes bandwidth demand, latency demand, etc., and the quality of connection includes indicators such as signal strength, latency, packet loss rate, and jitter. It is worth noting that if the various parts of the reward function have different dimensions or ranges, normalization or standardization may be required to ensure that the contribution of each part to the total reward is set as needed. Furthermore, as the network environment and business needs change, the parameters and logic of the reward function may need to be dynamically adjusted to ensure that the agent can continue to adapt to the new environment.

[0085] Suppose a scenario where multiple types of heterogeneous devices are connected to a nuclear power network, and the network needs to be dynamically connected at different times according to the needs of different types of devices. Then, the system built using the deep reinforcement learning network model trained above can update the state based on the data collected in real time and calculate the reward value for each possible combination of device and access location. For example, for a surveillance camera that requires instant data transmission, the system may recommend an access location because it provides the lowest latency. If the access is successful and the data transmission quality meets the requirements, the action receives a high reward, otherwise it is punished. Such feedback can be used to further train and optimize the model for more accurate decision-making.

[0086] It can be seen that the training mechanism of the deep reinforcement learning network model provided in this embodiment provides rich training samples for the training model by continuously collecting state transition data of various types of heterogeneous devices; by combining the deep reinforcement learning algorithm to design a reward function that takes into account both business demand satisfaction and network connection quality, an intelligent system that can automatically optimize the selection of network access locations is constructed, thereby significantly improving the overall service quality of network resource utilization, which in turn is beneficial to improving network performance and user experience.

[0087] In summary, the present invention applies deep reinforcement learning (DRL) to the access management of multiple heterogeneous devices at nuclear power sites, improves environmental perception capabilities through deep neural networks, and uses reinforcement learning to optimize access decisions to adapt to complex network environments and diverse equipment requirements. The DRL algorithm can perceive network status and equipment requirements in real time, dynamically adjust access strategies, and ensure the real-time and high efficiency of equipment access. By introducing the DRL algorithm, the combination of deep neural networks and reinforcement learning is achieved to realize intelligent access management of heterogeneous devices at nuclear power sites. Further utilizing the self-learning ability of the DRL algorithm, through continuous online learning and optimization, the system can dynamically adjust the access strategy according to changes in on-site equipment and network environment, maintain the best operating state, and improve the system's adaptability in abnormal network environments, thereby effectively improving the overall performance of the system and user experience.

[0088] like Figure 4 As shown, corresponding to the method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning, an embodiment of the present invention further provides a system for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning. The system for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning includes:

[0089] The information acquisition module 410 is used to acquire the current state of the heterogeneous network environment, where the current state is used to reflect the current network environment state and the operating state and service requirements of each device in the heterogeneous network;

[0090] A decision module 420, configured to input the current state into a trained deep reinforcement learning network model to generate an optimal access strategy;

[0091] The access module 430 is used to execute the operation of accessing the corresponding device in the time-sensitive network based on the optimal access strategy.

[0092] Specifically, in this embodiment, the specific functions of the system for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning can also refer to the corresponding description in the method for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning, and will not be repeated here.

[0093] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The intelligent terminal can be used to execute the method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning provided in the above embodiment. For the sake of brevity, it will not be repeated here. The intelligent terminal includes: a processor, the processor is coupled to a memory, the memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions stored in the memory, so that the method in the above method embodiment is executed.

[0094] The present invention also provides a computer-readable storage medium on which computer instructions for implementing the method in the above method embodiment are stored.

[0095] For example, when the computer program is executed by a computer, the computer can implement the method in the above method embodiment.

[0096] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to implement the method in the above method embodiment.

[0097] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A method for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning, characterized in that: The following steps are involved: Obtaining the current state of the heterogeneous network environment, where the current state is used to reflect the current network environment state and the operating state and service requirements of each device in the heterogeneous network; Inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy; Based on the optimal access strategy, an operation of accessing a corresponding device in the time-sensitive network is performed.

2. The method for accessing a time-sensitive network of multiple heterogeneous devices based on deep reinforcement learning according to claim 1 is characterized in that: The obtaining of the current state of the heterogeneous network environment includes: Collecting status information of the heterogeneous network environment, wherein the status information includes at least one of location information, operation status information, and service demand information of each device in the heterogeneous network; Based on at least one of the state information, a current state of the heterogeneous network environment is generated.

3. The method for accessing a time-sensitive network of multiple heterogeneous devices based on deep reinforcement learning according to claim 1 is characterized in that: The inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy includes: In response to the trained deep reinforcement learning network model receiving the current state, outputting a target action for each of the devices, the target action being used to represent a mapping relationship between the device and the target access location; Based on all said target actions, an optimal access strategy is generated.

4. The method for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning according to claim 1, characterized in that: The performing of accessing a corresponding device in a time-sensitive network based on the optimal access strategy includes: Based on the optimal access strategy, generating a network configuration instruction; Determine the access location and configuration information of each of the devices according to the network configuration instructions; An operation of accessing a corresponding device in a time-sensitive network is performed according to the access location and the configuration information.

5. The method for accessing a time-sensitive network of multiple heterogeneous devices based on deep reinforcement learning according to any one of claims 1 to 4, characterized in that: The training process of the deep reinforcement learning network model includes: Acquire the historical status of the heterogeneous network environment, where the historical status is used to reflect the environmental status of the heterogeneous network and the operating status and service requirements of each device within a preset time period; Inputting the historical state into a preset policy function and outputting an action, wherein the action is used to represent a mapping relationship between the device and the predicted access location; In response to the next state of the action, a reward value is obtained; Based on the reward value, the initial deep reinforcement learning network model is iteratively trained to obtain a trained deep reinforcement learning network model.

6. The method for accessing a time-sensitive network of multiple heterogeneous devices based on deep reinforcement learning according to claim 5 is characterized in that: The step of obtaining a reward value in response to a next state of the action comprises: In response to the next state of the action, determining business requirement satisfaction and network connection quality; A reward value is obtained based on the business demand satisfaction and the network connection quality.

7. The method for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning according to claim 5, characterized in that: The iterative training of the initial deep reinforcement learning network model based on the reward value to obtain a trained deep reinforcement learning network model includes: Based on each of the reward values, a corresponding experience sample is obtained, wherein the experience sample includes a state, an action, and a reward value of a heterogeneous network environment; Using a number of the experience samples to train a Q-value estimation network to obtain a reward value; Based on the reward value and the preset expected reward value, the initial deep reinforcement learning network model is iteratively trained to obtain a trained deep reinforcement learning network model.

8. A system for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning, characterized in that: The system comprises: An information acquisition module is used to acquire the current state of the heterogeneous network environment, where the current state is used to reflect the current network environment state and the operating state and service requirements of each device in the heterogeneous network; A decision module, used for inputting the current state into a trained deep reinforcement learning network model to generate an optimal access strategy; The access module is used to execute the operation of accessing the corresponding device in the time-sensitive network based on the optimal access strategy.

9. An intelligent terminal, characterized in that: include: A memory for storing executable instructions; A processor is used to call and run the executable instructions in the memory to execute the steps of the method for multiple heterogeneous devices to access a time-sensitive network based on deep reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions. When the program instructions are executed by the processor, the method for accessing a time-sensitive network with multiple heterogeneous devices based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images