Underground cable safety assessment method and system based on cloud edge cooperation and multi-agent deep reinforcement learning

Abstract

The invention discloses an underground cable safety assessment method and system based on cloud edge collaboration and multi-agent deep reinforcement learning. The method comprises the following steps: acquiring temperature, vibration, deformation and image data of multi-end acquisition nodes of an underground cable; uploading to an edge service node, and extracting spatio-temporal features of video image data, temperature, vibration and deformation data at the edge service node by adopting a bidirectional long-short-term memory network; and uploading the feature data preprocessed by the edge service node to a cloud computing service center for multi-source feature data fusion, constructing a multi-agent double-adversarial deep Q network, taking the fused feature data as the state input of the multi-agent double-adversarial deep Q network, and training to obtain a cable safety state assessment model. Therefore, the safety state of the underground cable can be identified and evaluated. Effective feature extraction and fusion of underground cable multi-source data are realized, and rapid and accurate evaluation of the safety state of the underground cable is realized based on fusion features.

Description

Technical Field

[0001] The present invention relates to the technical field of cable safety assessment, and in particular to an underground cable safety assessment method and system based on cloud-edge collaboration and multi-agent deep reinforcement learning. Background Art

[0002] With the acceleration of my country's urbanization process, the coverage rate and asset scale of power cables are increasing at an unprecedented rate. In recent years, the proportion of underground cable power supply in the power distribution system has become increasingly larger, but the cables are often damaged by external forces during use, and the advantages of safe and reliable cable power supply are seriously affected. The contradiction between social infrastructure construction and underground cable power supply is becoming more and more prominent. How to effectively protect the safety of underground cables in the process of infrastructure construction has become an urgent problem to be solved.

[0003] For the anti-damage monitoring of power pipelines such as underground cable tunnels, it is difficult to solve the problem of external force damage through daily line inspections. For random external force damage, only by strengthening the real-time monitoring of the line can the line's ability to resist external forces be improved. Some power departments have introduced some scientific and technological means to monitor the status of transmission lines to determine whether the transmission lines are damaged, such as leakage current detection method, power line carrier detection method, and point strain sensor detection method.

[0004] However, the above methods all have obvious limitations: the leakage current detection method can only reflect the results after the transmission line is damaged, and it cannot work during a power outage; the power line carrier detection method faces problems such as high noise, large impedance changes, and large signal transmission loss during the monitoring process; the point strain sensor detection method can monitor the strain state of the transmission line in real time. When an external force damage event occurs, the transmission line will produce strain due to the tension, and the tension is transmitted to the location of the sensor through the line, causing it to also produce strain, so that it can be judged whether the cable is damaged by external force. However, the point strain sensor detection method also has shortcomings. It cannot locate the fault point and can only be arranged at some special locations on the transmission line. The monitoring range is not comprehensive enough, and the strain detection value has a certain deviation compared with the actual value, which cannot fully reflect the real situation.

[0005] Therefore, how to avoid large-scale power outages and accurately locate the operating status of underground cables in real time and know it in advance has become an urgent problem and research hotspot that needs to be solved for urban power supply safety. Summary of the invention

[0006] Purpose of the invention: In order to solve the problems arising from the above-mentioned prior art, the present invention provides an underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning, which solves the problem that the operating status of underground cables cannot be located in real time and accurately when a large-scale power outage occurs. The present invention also provides an underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning.

[0007] Technical solution: According to the first aspect of the present invention, a method for underground cable safety assessment based on cloud-edge collaboration and multi-agent deep reinforcement learning is provided, the method comprising:

[0008] Build a cloud-edge collaborative service architecture for underground cable operation data transmission and processing, including edge service nodes and cloud computing service centers;

[0009] Acquire temperature data, vibration data, deformation data and video image data of multiple monitoring nodes of underground cables, where the number of monitoring nodes is recorded as M, use a sensor with Raman optical time domain reflectometry technology to collect cable temperature data at M, use a sensor with phase sensitive time domain reflectometry technology to collect cable vibration data at M, use a sensor with Brillouin optical time domain analysis technology to collect cable deformation data at M, and use a high-definition full-time monitoring camera to collect key ground image data of the cable at M;

[0010] The multi-source data obtained by each monitoring node is uploaded to the edge service node. Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data; at the same time, for the underground cable operation data in the form of video images, the key features of the image data are extracted by constructing a convolutional neural network and a bidirectional long short-term memory network;

[0011] Each edge service node uploads the multi-source data of underground cable operation after key feature extraction to the cloud computing service center. Based on the fully connected neural network, the multi-source data of underground cable operation after key feature extraction is mapped to a unified dimension to realize data fusion processing;

[0012] Construct a multi-agent dual-adversarial deep Q network cable safety status assessment model with feedback learning mechanism: design an agent for each data monitoring node of the underground cable, and use the multi-source fusion feature data of each data monitoring node as the state input of each agent's dual-adversarial deep Q network. Through a large amount of interactive training between the multi-agent and the data environment, the underground cable safety status assessment model is obtained. The output of each agent's dual-adversarial deep Q network cable safety status assessment model is the assessment of the safety status of the underground cable.

[0013] Furthermore, the method also includes:

[0014] The multi-agent dual-adversarial deep Q network cable safety status assessment model obtained after training is systematically deployed. The specific steps of the deployment are: saving the network parameters of the dual-adversarial deep Q network in the intelligent agent corresponding to each trained monitoring node in the cloud computing service center, reading the multi-source data fusion features transmitted by each edge service node in real time, and evaluating the safety status of each cable operation monitoring point.

[0015] Further, including:

[0016] The cloud-edge collaborative service architecture for underground cable operation data transmission and processing includes:

[0017] Constructing an edge service node, wherein the edge service node includes an edge microserver and a macro base station, wherein the edge microserver is used to collect and pre-process the original underground cable operation data from each data collection node, and the macro base station is used to communicate and forward the processed underground cable operation multi-source data features to the cloud computing service center;

[0018] The cloud computing service center is used to uniformly manage multi-source data features from various edge service nodes and provide a platform for the training and application of underground cable safety status assessment algorithms.

[0019] Further, including:

[0020] Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data, specifically including:

[0021] For the underground cable operation timing data, the temperature, vibration, and deformation formatted data collected by each edge service node are extracted and normalized to obtain normalized data.

[0022] A two-layer long-short-term neural network model is constructed to extract time series features of temperature, vibration and deformation data, and the normalized data is used as the state input of the long-short-term neural network. Each layer of the long-short-term neural network controls the flow of information through the forget gate, input gate and output gate. The second layer of the long-short-term neural network updates its internal state according to the information of the first layer and finally obtains the final temperature, vibration and deformation feature data.

[0023] Further, including:

[0024] The method of mapping the underground cable operation multi-source data after key feature extraction to a unified dimension based on a fully connected neural network to realize data fusion processing specifically includes:

[0025] The input data corresponding to the fully connected neural network is the multi-source data features after key feature extraction, and the output data is the data features mapped to a unified dimension. The data characteristics They respectively represent the processed multi-source feature fusion data, namely, feature data related to cable temperature, feature data related to cable vibration, feature data related to cable deformation, and feature data related to cable ground image.

[0026] Further, including:

[0027] The construction of a multi-agent dual-adversarial deep Q network cable safety status assessment model with a feedback learning mechanism specifically includes:

[0028] The construction of a multi-agent dual-adversarial deep Q network cable safety status assessment model with feedback learning mechanism is transformed into a problem of minimizing the cable safety status assessment error at M, including decision variables and objective functions;

[0029] The cable safety status assessment error minimization problem at the M locations is remodeled as a Markov decision process, and M cable safety status assessment agents based on a multi-agent dual adversarial deep Q network are constructed to solve the Markov decision process;

[0030] An intelligent agent model for underground cable safety status assessment is obtained through extensive interactive training of multiple agents and data environments.

[0031] Further, including:

[0032] The solution to the cable safety status assessment error minimization problem at location M includes decision variables and objective functions;

[0033] The decision variables are: the safety status evaluation a of the cable data monitoring node at M i ={e low ,…,e high},1≤i≤M, where: i represents the i-th cable data collection point, the safety status assessment is represented by multiple discrete values, e low ,e high Respectively represent the minimum safety factor and the maximum safety factor of the current cable;

[0034] The objective function is:

[0035]

[0036] Where: T represents the total number of time slots for safety status assessment of each cable, a′ i,t Indicates the true safety status of each cable.

[0037] Further, including:

[0038] The problem of minimizing the error of the cable safety status assessment at the M locations is remodeled as a Markov decision process, which is expressed as:

[0039]

[0040] a i,t ={e low ,...,e high},1≤i≤M,1≤t≤T

[0041] r i,t =-(a′ i,t -a i,t ) 2

[0042] Where: s i,t , a i,t , r i,t They represent the state, behavior, and reward function of the Markov decision process related to the safety status assessment of each underground cable; They respectively represent the processed multi-source feature fusion data, namely, feature data related to cable temperature, feature data related to cable vibration, feature data related to cable deformation, and feature data related to cable ground image.

[0043] Further, including:

[0044] The construction of M cable safety status assessment agents based on a multi-agent double adversarial deep Q network for solving the Markov decision process specifically includes:

[0045] The constructed M cable safety status assessment agents include value function networks with the same structure and the target value function network The value function network and the target value function network are composed of an input layer, multiple hidden layers and two output layers. The two output layers are: behavior value function output layer And advantage function output layer

[0046] The output value of the value function network Output layer of behavior value function And advantage function output layer Jointly decide that: Among them: U and They respectively represent the number of discrete action dimensions that the agent can choose and the evaluation levels of other safety states except the current evaluation output a.

[0047] Further, including:

[0048] The underground cable safety status assessment agent model is obtained through a large amount of interactive training between multiple agents and data environment, and the training steps include:

[0049] Initialize the data environment and repeat the iteration steps until the preset number of training segments K is reached. The length of each training segment is set to N, and the update frequency of the agent is set to K. update ;

[0050] The iterative steps include:

[0051] Input the processed multi-source feature fusion data into The intelligent agent i makes a decision for each cable safety status assessment, which is the current cable safety status e i,t ;

[0052] The reward function r that the agent gets feedback from i,t Run multi-source fusion feature data with the next cable Evaluation output a i,t And the current multi-source fusion feature data The four-tuple (s i,t ,a i,t ,s i,t+1 ,r i,t ) is stored in the shared experience pool of the agent;

[0053] Determine if the agent update condition is met, that is, the number of fragments K that meet the update frequency is reached i %K update =0, where: K i is the current number of fragments. At this time, a combination of the priority experience replay mechanism and the agent update mechanism is proposed, that is, each agent extracts a small batch of four-tuples (s, a, s, r) from the shared experience pool as training samples, and uses the multi-agent dual adversarial deep Q network algorithm combined with the joint loss function of the priority experience replay mechanism to train the agent, and the value function network is updated. to update.

[0054] Further, including:

[0055] The specific principle of the priority experience replay mechanism implemented in the agent update process is:

[0056] Calculate the sampling weight of each experience tuple in the experience pool, and perform small batch quadruple sampling based on the sampling weight. The sampling weight calculation formula is as follows:

[0057]

[0058] Where: κ represents the sampling weight of the priority experience replay mechanism; Represents the total number of experience tuples stored for each agent in the shared experience pool; Indicates the priority of a certain experience tuple among all currently stored experience tuples; p indicates the absolute value of the TD error between a certain experience tuple and all other experience tuples; represents the absolute value set of TD errors between a certain experience tuple and all other experience tuples, sum(·) is a summation function; φ represents a sampling weight coefficient; and represents the value function and target value function of agent i, s′ and a′ represent the cable operation characteristic data and target action value of the next time slot of a certain experience tuple respectively; Represents the target value of agent i.

[0059] Further, including:

[0060] The joint loss function of the multi-agent dual adversarial deep Q network algorithm combined with the priority experience replay mechanism is expressed as:

[0061]

[0062] In each update round, the agent is trained by minimizing the joint loss function constructed above.

[0063] On the other hand, the present invention also provides an underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning, the system comprising:

[0064] A service architecture building module for building a cloud-edge collaborative service architecture for underground cable operation data transmission and processing, including edge service nodes and cloud computing service centers;

[0065] A data acquisition module is used to obtain temperature data, vibration data, deformation data and video image data of multiple monitoring nodes of underground cables, wherein the number of monitoring nodes is recorded as M, and a sensor with Raman optical time domain reflectometry technology is used to collect cable temperature data at M, a sensor with phase sensitive time domain reflectometry technology is used to collect cable vibration data at M, a sensor with Brillouin optical time domain analysis technology is used to collect cable deformation data at M, and a high-definition full-time monitoring camera is used to collect key ground image data of the cable at M;

[0066] The preprocessing module is used to upload the multi-source data obtained by each monitoring node to the edge service node. Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data; at the same time, for the underground cable operation data in the form of video images, the key features of the image data are extracted by constructing a convolutional neural network and a bidirectional long short-term memory network;

[0067] A fusion processing module is used to uniformly upload the multi-source data of underground cable operation after key features are extracted by each edge service node to the cloud computing service center, and map the multi-source data of underground cable operation after key features are extracted to a unified dimension based on a fully connected neural network to realize data fusion processing;

[0068] The safety status assessment module is used to build a multi-agent dual-adversarial deep Q network cable safety status assessment model with a feedback learning mechanism: an agent is designed for each data monitoring node of the underground cable, and the multi-source fusion feature data of each data monitoring node is used as the state input of each agent's dual-adversarial deep Q network. The underground cable safety status assessment model is obtained through a large amount of interactive training between the multi-agent and the data environment. The output of each agent's dual-adversarial deep Q network cable safety status assessment model is an assessment of the safety status of the underground cable.

[0069] Furthermore, the system also includes:

[0070] The system deployment module is used to deploy the multi-agent dual-adversarial deep Q network cable safety status assessment model obtained after training. The specific steps of deployment are: saving the network parameters of the dual-adversarial deep Q network in the intelligent agent corresponding to each trained monitoring node in the cloud computing service center, reading the multi-source data fusion features transmitted by each edge service node in real time, and evaluating the safety status of each cable operation monitoring point.

[0071] Finally, the present invention also provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the above-mentioned underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning is executed.

[0072] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0073] The present invention provides a method and system for assessing the safety status of underground cables based on a multi-agent dual-adversarial deep Q network under a cloud-edge collaborative framework that considers multi-source fusion data feature input. Compared with the centralized data collection and processing method, the proposed cloud-edge collaborative architecture for underground cable operation data transmission and processing can effectively reduce problems such as computing burden and communication overload, and is suitable for scenarios such as real-time assessment of the safety status of underground cables.

[0074] Compared with the method based on single-type data input, the method and system of this application can achieve a more comprehensive safety status assessment. With the development of Internet of Things technology, the efficiency of data collection has been greatly improved. By using high-precision and highly intelligent sensors to collect massive data, it is not necessary to know the accurate dynamic model of underground cable operation. The underground cable safety status assessment model can be efficiently constructed through data-driven methods.

[0075] Compared with other deep learning-based methods that consider multi-source data feature input, the proposed method transforms the data classification problem into a Markov decision process, fully explores the potential of time series data application, and improves data utilization efficiency; in summary, the method of the present invention can achieve high-accuracy underground cable safety status assessment.

[0076] With the rapid development of Internet of Things technology, the widespread application of highly sensitive and high-precision sensors has made it possible to collect real-time and accurate information on the operation status of underground cables. Therefore, comprehensive analysis of the characteristics of multi-source perception data and intelligent assessment of the safety status of underground cables based on artificial intelligence methods will provide great support for improving the safety of urban power cables. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a flow chart of an underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning as described in Example 1 of the present invention;

[0078] Figure 2 It is a framework diagram of the multi-agent dual adversarial deep Q network algorithm considering multi-source data feature processing and input as described in Example 1 of the present invention;

[0079] Figure 3 It is a cloud-edge collaborative framework diagram for transmitting and processing multi-source fusion data features of underground cables as described in Example 1 of the present invention;

[0080] Figure 4 It is a flowchart of constructing a multi-agent dual-adversarial deep Q network cable safety status assessment model with feedback learning mechanism as described in Example 1 of the present invention;

[0081] Figure 5 This is a structural schematic diagram of an underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning as described in Example 3 of the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Example 1

[0084] like Figure 1 As shown, the present invention proposes an underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning. In this embodiment, the temperature, deformation, vibration and other data in the underground cable operation data used are obtained by modeling the underground cable operation process through COMSOL and ANSYS simulation platforms, and the ground operation status of the underground cable is obtained by shooting with a real camera. Specifically, the following steps are included:

[0085] S1. Build a cloud-edge collaborative service architecture for underground cable operation data transmission and processing, including edge service nodes and cloud computing service centers.

[0086] S1.1. Construct an edge service node, including an edge microserver and a macro base station. The edge microserver is used to collect and preprocess the original underground cable operation data from each data collection node. The macro base station is used to communicate and forward the processed underground cable operation multi-source data features to the cloud computing service center.

[0087] S1.2. Build a cloud computing service center to uniformly manage multi-source data features from various edge service nodes, and provide a platform for the training and application of underground cable safety status assessment algorithms.

[0088] like Figure 3 As shown, sensors are used to collect data, and the collection results are sent to the edge micro server, which processes the data and then sends it to the macro base station.

[0089] S2. Obtain temperature data, vibration data, deformation data and video image data of multiple monitoring nodes of underground cables. Different types of data are affected by the acquisition device and communication link, and there are problems such as data noise and data loss. Among them, a sensor with Raman optical time domain reflection technology is used to collect data on the temperature status of the cable at M, a sensor with phase-sensitive time domain reflection technology is used to collect the vibration data of the cable at M, a sensor with Brillouin optical time domain analysis technology is used to collect the deformation data of the cable at M, and a high-definition full-time monitoring camera is used to collect key ground image data of the cable at M.

[0090] S3, upload the multi-source data obtained by each acquisition node to the edge service node. Each edge service node builds a long short-term memory neural network for underground cable operation timing data such as temperature, vibration, deformation, etc. to achieve noise filtering, redundant data elimination and key feature extraction; at the same time, for underground cable operation data in the form of video images, the key features of the image data are extracted by constructing a convolutional neural network and a bidirectional long short-term memory network, such as Figure 2 shown.

[0091] S3.1. First, the formatted data such as temperature, vibration, deformation, etc. collected by each edge service node will be normalized for time series feature data extraction:

[0092]

[0093] Among them, D w , D z and D x are the original data of temperature, vibration and deformation respectively; D w,norm , D z,norm and D x,norm are the normalized data of temperature, vibration and deformation respectively; D w,max , D z,max , D x,max and D w,min , D z,min , D x,min They are the maximum and minimum values ​​corresponding to the original data of temperature, vibration and deformation respectively. A two-layer long-short term neural network model is constructed to extract the time series features of temperature, vibration and deformation data, and the normalized data is used as the state input of the long-short term neural network. Among them, each layer of the long-short term neural network controls the flow of information through the forget gate, input gate and output gate. The second layer of the long-short term neural network updates its internal state according to the information of the first layer and finally obtains the final temperature, vibration and deformation feature data.

[0094] The specific process can be expressed as follows:

[0095] D t ={D w,norm ,D z,norm ,D x,norm}

[0096] i t =δ(W i ·[H t-1 ,D t ]+V i )

[0097] f t =δ(W f ·[H t-1,D t ]+V f )

[0098]

[0099] O t =δ(W O ·[H t-1 ,D t ]+V O )

[0100] H t =O t *tanh(C t )

[0101] Where: i t , f t , O t Respectively represent the forget gate, input gate and output gate; C t Indicates the state of the memory unit; represents the value of the candidate state of the memory unit; δ and tanh represent the Sigmod and tanh activation functions respectively; W and V represent the weight matrix and bias matrix respectively; D t To input underground cable operation data; H t Run data features for extracted underground cables.

[0102] S3.2. The temporal features of the cable ground operation status data in the form of video images are preliminarily extracted through a convolutional neural network containing multiple convolutional layers, and input into a bidirectional long short-term memory neural network to further extract key features.

[0103] S4. Each edge service node uploads the pre-processed multi-source data of underground cable operation to the cloud computing service center, and maps the pre-processed multi-source data to a unified dimension based on a fully connected neural network to realize data fusion processing.

[0104] In this embodiment, the above steps are mainly completed by a deep neural network, whose input data is the preprocessed multi-source data features, and the output data is the data features mapped to a unified dimension. They respectively represent the processed multi-source feature fusion data, namely, feature data related to cable temperature, feature data related to cable vibration, feature data related to cable deformation, and feature data related to cable ground image.

[0105] S5. Construct a multi-agent dual-adversarial deep Q network cable safety status assessment model with a feedback learning mechanism, design an agent for each data collection point of the underground cable, use the multi-source fusion feature data of each data collection point as the state input of each agent's dual-adversarial deep Q network, and obtain the underground cable safety status assessment model through a large amount of interactive training between the multi-agent and the data environment. The output of each agent's dual-adversarial deep Q network model is the assessment of the safety status of the underground cable. Figure 4 As shown, the specific implementation steps are as follows:

[0106] S5.1. First, the construction of a multi-agent dual-adversarial deep Q network cable safety status assessment model with feedback learning mechanism is transformed into a problem of solving the cable safety status assessment error minimization problem at M, including decision variables and objective functions;

[0107] The decision variables are: safety status evaluation a for the cable data collection point at M i ={e low ,...,e high},1≤i≤M;

[0108] Where: i represents the i-th cable data collection point, the safety status assessment is represented by multiple discrete values, e low ,e high Respectively represent the minimum safety factor and the maximum safety factor of the current cable;

[0109] The objective function is: Where: T represents the total number of time slots for safety status assessment of each cable, a i ' ,t Indicates the true safety status of each cable.

[0110] S5.2. Secondly, the problem of minimizing the error of the cable safety status assessment at the M locations is remodeled as the following Markov decision process:

[0111]

[0112] a i,t ={e low ,...,e high},1≤i≤M,1≤t≤T;

[0113] r i,t =-(a′ i,t -a i,t ) 2 ;

[0114] Where: s i,t , a i,t , r i,tThey represent the state, behavior, and reward function of the Markov decision process related to the safety status assessment of each underground cable; They respectively represent the processed multi-source feature fusion data, namely, feature data related to cable temperature, feature data related to cable vibration, feature data related to cable deformation, and feature data related to cable ground image.

[0115] S5.3. Then, construct M cable safety status assessment agents based on a multi-agent double adversarial deep Q network to solve the above Markov decision process.

[0116] The M agents constructed are mainly composed of value function networks with the same structure and the target value function network constitute.

[0117] Among them: the value function network and the target value function network are composed of an input layer, multiple hidden layers and two output layers. The two output layers are the behavior value function output layer And advantage function output layer

[0118] Specifically, in order to avoid the overfitting problem of the value function network output in the existing deep reinforcement learning technology, a confrontation network framework is proposed, that is, the output value of the value function network Output layer of behavior value function And advantage function output layer Jointly decide that:

[0119]

[0120] Among them: U and They respectively represent the number of discrete action dimensions that the agent can choose and the evaluation levels of other safety states except the current evaluation output a.

[0121] S5.4. Finally, the intelligent agent model for underground cable safety status assessment is obtained through a large amount of interactive training between multiple agents and the data environment, and the trained intelligent agent is deployed in the underground cable safety status assessment system. The training steps mainly include:

[0122] Initialize the data environment and repeat the iteration steps until the preset number of training segments K is reached. The length of each training segment is set to N, and the update frequency of the agent is set to K. update ;

[0123] The iterative steps include:

[0124] According to the processed multi-source feature fusion data input The intelligent agent i makes a decision for each cable safety status assessment, which is the current cable safety status ei,t ;

[0125] The reward function r for the agent to get feedback i,t Run multi-source fusion feature data with the next cable Evaluation output a i,t And the current multi-source fusion feature data The four-tuple (s i,t ,a i,t ,s i,t+1 ,r i,t ) is stored in the shared experience pool of the agent;

[0126] Determine if the agent update condition is met, that is, the number of fragments K that meet the update frequency is reached i %K update =0, where: K i The priority experience replay mechanism is combined with the agent update mechanism. Each agent extracts a small batch of quadruple (s, a, s, r) from the experience pool as a training sample, and uses the joint loss function of the multi-agent dual adversarial deep Q network algorithm combined with the priority experience replay mechanism to train the agent, and the value function network is updated. to update.

[0127] Among them: The specific principle of the priority experience replay mechanism implemented in the agent update process is: calculate the sampling weight of each experience tuple in the experience pool, and perform small batch quadruple sampling according to the sampling weight. The sampling weight calculation formula is as follows:

[0128]

[0129] Where: κ represents the sampling weight of the priority experience replay mechanism; Represents the total number of experience tuples stored for each agent in the shared experience pool; Indicates the priority of a certain experience tuple among all currently stored experience tuples; p indicates the absolute value of the TD error between a certain experience tuple and all other experience tuples; represents the absolute value set of TD errors between a certain experience tuple and all other experience tuples, sum(·) is a summation function; φ represents a sampling weight coefficient; and represents the value function and target value function of agent i, s′ and a′ represent the cable operation characteristic data and target action value of the next time slot of a certain experience tuple respectively; Represents the target value of agent i.

[0130] like Figure 2 As shown, the joint loss function of the multi-agent dual-adversarial deep Q network algorithm proposed in the agent update stage during the implementation of this embodiment is as follows:

[0131]

[0132] In each update round, the agent is trained by minimizing the joint loss function constructed above.

[0133] At the same time, the update of the objective value function network adopts a soft update mechanism, namely: Where: σ is the soft update mechanism coefficient, which is used to constrain the update amplitude of the objective value function network.

[0134] Example 2

[0135] On the basis of Example 1, the method of the present invention further comprises:

[0136] S6. Systematically deploy the obtained underground cable safety status assessment model.

[0137] The specific steps of the deployment are: save the network parameters of the dual-adversarial deep Q network in the trained intelligent agents of each underground cable data monitoring point in the cloud computing service center, read the multi-source data fusion features transmitted by each edge service node in real time, and evaluate the safety status of each cable operation monitoring point.

[0138] Example 3

[0139] like Figure 5 As shown, the present invention also provides an underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning, the system comprising:

[0140] A service architecture building module for building a cloud-edge collaborative service architecture for underground cable operation data transmission and processing, including edge service nodes and cloud computing service centers;

[0141] A data acquisition module is used to obtain temperature data, vibration data, deformation data and video image data of multiple monitoring nodes of underground cables, wherein the number of monitoring nodes is recorded as M, and a sensor with Raman optical time domain reflectometry technology is used to collect cable temperature data at M, a sensor with phase sensitive time domain reflectometry technology is used to collect cable vibration data at M, a sensor with Brillouin optical time domain analysis technology is used to collect cable deformation data at M, and a high-definition full-time monitoring camera is used to collect key ground image data of the cable at M;

[0142] The preprocessing module is used to upload the multi-source data obtained by each monitoring node to the edge service node. Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data; at the same time, for the underground cable operation data in the form of video images, the key features of the image data are extracted by constructing a convolutional neural network and a bidirectional long short-term memory network;

[0143] A fusion processing module is used to uniformly upload the multi-source data of underground cable operation after key features are extracted by each edge service node to the cloud computing service center, and map the multi-source data of underground cable operation after key features are extracted to a unified dimension based on a fully connected neural network to realize data fusion processing;

[0144] The safety status assessment module is used to build a multi-agent dual-adversarial deep Q network cable safety status assessment model with a feedback learning mechanism: an agent is designed for each data monitoring node of the underground cable, and the multi-source fusion feature data of each data monitoring node is used as the state input of each agent's dual-adversarial deep Q network. The underground cable safety status assessment model is obtained through a large amount of interactive training between the multi-agent and the data environment. The output of each agent's dual-adversarial deep Q network cable safety status assessment model is an assessment of the safety status of the underground cable.

[0145] Example 4

[0146] Based on Example 3, the system further includes:

[0147] The system deployment module is used to deploy the multi-agent dual-adversarial deep Q network cable safety status assessment model obtained after training. The specific steps of deployment are: saving the network parameters of the dual-adversarial deep Q network in the intelligent agent corresponding to each trained monitoring node in the cloud computing service center, reading the multi-source data fusion features transmitted by each edge service node in real time, and evaluating the safety status of each cable operation monitoring point.

[0148] Finally, the present invention also provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the above-mentioned underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning is executed.

[0149] The other technical features of the underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning described in the present invention are similar to the corresponding underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning, and will not be repeated here.

[0150] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0152] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for underground cable safety assessment based on cloud-edge collaboration and multi-agent deep reinforcement learning, characterized in that: The method includes: Build a cloud-edge collaborative service architecture for underground cable operation data transmission and processing, including edge service nodes and cloud computing service centers; Acquire temperature data, vibration data, deformation data and video image data of multiple monitoring nodes of underground cables, where the number of monitoring nodes is recorded as M, use a sensor with Raman optical time domain reflectometry technology to collect cable temperature data at M, use a sensor with phase sensitive time domain reflectometry technology to collect cable vibration data at M, use a sensor with Brillouin optical time domain analysis technology to collect cable deformation data at M, and use a high-definition full-time monitoring camera to collect key ground image data of the cable at M; The multi-source data obtained by each monitoring node is uploaded to the edge service node. Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data; at the same time, for the underground cable operation data in the form of video images, the key features of the image data are extracted by constructing a convolutional neural network and a bidirectional long short-term memory network; Each edge service node uploads the multi-source data of underground cable operation after key feature extraction to the cloud computing service center. Based on the fully connected neural network, the multi-source data of underground cable operation after key feature extraction is mapped to a unified dimension to realize data fusion processing; Construct a multi-agent dual-adversarial deep Q network cable safety status assessment model with feedback learning mechanism: design an agent for each data monitoring node of the underground cable, and use the multi-source fusion feature data of each data monitoring node as the state input of each agent's dual-adversarial deep Q network. Through a large amount of interactive training between the multi-agent and the data environment, the underground cable safety status assessment model is obtained. The output of each agent's dual-adversarial deep Q network cable safety status assessment model is the assessment of the safety status of the underground cable.

2. The underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning as described in claim 1 is characterized in that: The method further includes: The multi-agent dual-adversarial deep Q network cable safety status assessment model obtained after training is systematically deployed. The specific steps of the deployment are: saving the network parameters of the dual-adversarial deep Q network in the intelligent agent corresponding to each trained monitoring node in the cloud computing service center, reading the multi-source data fusion features transmitted by each edge service node in real time, and evaluating the safety status of each cable operation monitoring point.

3. The safety assessment method according to claim 1 or 2, characterized in that: The cloud-edge collaborative service architecture for underground cable operation data transmission and processing includes: Constructing an edge service node, wherein the edge service node includes an edge microserver and a macro base station, wherein the edge microserver is used to collect and pre-process the original underground cable operation data from each data collection node, and the macro base station is used to communicate and forward the processed underground cable operation multi-source data features to the cloud computing service center; The cloud computing service center is used to uniformly manage multi-source data features from various edge service nodes and provide a platform for the training and application of underground cable safety status assessment algorithms.

4. The safety assessment method according to claim 1 or 2, characterized in that: Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data, specifically including: For the underground cable operation timing data, the temperature, vibration, and deformation formatted data collected by each edge service node are extracted and normalized to obtain normalized data. A two-layer long-short-term neural network model is constructed to extract time series features of temperature, vibration and deformation data, and the normalized data is used as the state input of the long-short-term neural network. Each layer of the long-short-term neural network controls the flow of information through the forget gate, input gate and output gate. The second layer of the long-short-term neural network updates its internal state according to the information of the first layer and finally obtains the final temperature, vibration and deformation feature data.

5. The safety assessment method according to claim 1 or 2, characterized in that: The method of mapping the underground cable operation multi-source data after key feature extraction to a unified dimension based on a fully connected neural network to realize data fusion processing specifically includes: The input data corresponding to the fully connected neural network is the multi-source data features after key feature extraction, and the output data is the data features mapped to a unified dimension. The data characteristics They respectively represent the processed multi-source feature fusion data, namely, feature data related to cable temperature, feature data related to cable vibration, feature data related to cable deformation, and feature data related to cable ground image.

6. The safety assessment method according to claim 5, characterized in that: The construction of a multi-agent dual-adversarial deep Q network cable safety status assessment model with a feedback learning mechanism specifically includes: The construction of a multi-agent dual-adversarial deep Q network cable safety status assessment model with feedback learning mechanism is transformed into a problem of solving the cable safety status assessment error minimization problem at M, including decision variables and objective functions; The cable safety status assessment error minimization problem at the M locations is remodeled as a Markov decision process, and M cable safety status assessment agents based on a multi-agent dual adversarial deep Q network are constructed to solve the Markov decision process; An intelligent agent model for underground cable safety status assessment is obtained through extensive interactive training of multiple agents and data environments.

7. The safety assessment method according to claim 6, characterized in that: The solution to the cable safety status assessment error minimization problem at location M includes decision variables and objective functions; The decision variables are: the safety status evaluation a of the cable data monitoring node at M i ={e low ,…,e high },1≤i≤M, where: i represents the i-th cable data collection point, the safety status assessment is represented by multiple discrete values, e low ,e high Respectively represent the minimum safety factor and the maximum safety factor of the current cable; The objective function is: Where: T represents the total number of time slots for safety status assessment of each cable, a i ' ,t Indicates the true safety status of each cable.

8. The safety assessment method according to claim 7, characterized in that: The problem of minimizing the error of the cable safety status assessment at the M locations is remodeled as a Markov decision process, which is expressed as: to i,t ={and low ,...,And high },1≤i≤M,1≤t≤T r i,t =-(a′ i,t -a i,t ) 2 Where: s i,t , a i,t , r i,t They represent the state, behavior, and reward function of the Markov decision process related to the safety status assessment of each underground cable; They respectively represent the processed multi-source feature fusion data, namely, feature data related to cable temperature, feature data related to cable vibration, feature data related to cable deformation, and feature data related to cable ground image.

9. The safety assessment method according to claim 8, characterized in that: The construction of M cable safety status assessment agents based on a multi-agent double adversarial deep Q network for solving the Markov decision process specifically includes: The constructed M cable safety status assessment agents include value function networks with the same structure and the target value function network The value function network and the target value function network are composed of an input layer, multiple hidden layers and two output layers. The two output layers are: behavior value function output layer And advantage function output layer The output value of the value function network Output layer of behavior value function And advantage function output layer Jointly decide that: Among them: U and They respectively represent the number of discrete action dimensions that the agent can choose and the evaluation levels of other safety states except the current evaluation output a.

10. The security assessment method according to claim 9, characterized in that: The underground cable safety status assessment agent model is obtained through a large amount of interactive training between multiple agents and data environment, and the training steps include: Initialize the data environment and repeat the iteration steps until the preset number of training segments K is reached. The length of each training segment is set to N, and the update frequency of the agent is set to K. update ; The iterative steps include: Input the processed multi-source feature fusion data into The intelligent agent i makes a decision for each cable safety status assessment, which is the current cable safety status e i,t ; The reward function r that the agent gets feedback from i,t Run multi-source fusion feature data with the next cable Evaluation output a i,t And the current multi-source fusion feature data The four-tuple (s i,t ,a i,t ,s i,t+1 ,r i,t ) is stored in the shared experience pool of the agent; Determine if the agent update condition is met, that is, the number of fragments K that meet the update frequency is reached i %K update =0, where: K i is the current number of fragments. At this time, a combination of the priority experience replay mechanism and the agent update mechanism is proposed, that is, each agent extracts a small batch of four-tuples (s, a, s, r) from the shared experience pool as training samples, and uses the multi-agent dual adversarial deep Q network algorithm combined with the joint loss function of the priority experience replay mechanism to train the agent, and the value function network is updated. to update.

11. The safety assessment method according to claim 10, characterized in that: The specific principle of the priority experience replay mechanism implemented in the agent update process is: Calculate the sampling weight of each experience tuple in the experience pool, and perform small batch quadruple sampling based on the sampling weight. The sampling weight calculation formula is as follows: Where: κ represents the sampling weight of the priority experience replay mechanism; Represents the total number of experience tuples stored for each agent in the shared experience pool; Indicates the priority of a certain experience tuple among all currently stored experience tuples; p indicates the absolute value of the TD error between a certain experience tuple and all other experience tuples; represents the absolute value set of TD errors between a certain experience tuple and all other experience tuples, sum(·) is a summation function; φ represents a sampling weight coefficient; and represents the value function and target value function of agent i, s′ and a′ represent the cable operation characteristic data and target action value of the next time slot of a certain experience tuple respectively; Represents the target value of agent i.

12. The security assessment method according to claim 11, characterized in that: The joint loss function of the multi-agent dual adversarial deep Q network algorithm combined with the priority experience replay mechanism is expressed as: In each update round, the agent is trained by minimizing the joint loss function constructed above.

13. An underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning, characterized in that: The system includes: A service architecture building module for building a cloud-edge collaborative service architecture for underground cable operation data transmission and processing, including edge service nodes and cloud computing service centers; A data acquisition module is used to obtain temperature data, vibration data, deformation data and video image data of multiple monitoring nodes of underground cables, wherein the number of monitoring nodes is recorded as M, and a sensor with Raman optical time domain reflectometry technology is used to collect cable temperature data at M, a sensor with phase sensitive time domain reflectometry technology is used to collect cable vibration data at M, a sensor with Brillouin optical time domain analysis technology is used to collect cable deformation data at M, and a high-definition full-time monitoring camera is used to collect key ground image data of the cable at M; The preprocessing module is used to upload the multi-source data obtained by each monitoring node to the edge service node. Each edge service node constructs a long short-term memory neural network for the underground cable operation timing data formed by temperature, vibration, and deformation, thereby realizing noise filtering, redundant data elimination, and key feature extraction in the underground cable operation timing data; at the same time, for the underground cable operation data in the form of video images, the key features of the image data are extracted by constructing a convolutional neural network and a bidirectional long short-term memory network; A fusion processing module is used to uniformly upload the multi-source data of underground cable operation after key features are extracted by each edge service node to the cloud computing service center, and map the multi-source data of underground cable operation after key features are extracted to a unified dimension based on a fully connected neural network to realize data fusion processing; The safety status assessment module is used to build a multi-agent dual-adversarial deep Q network cable safety status assessment model with a feedback learning mechanism: an agent is designed for each data monitoring node of the underground cable, and the multi-source fusion feature data of each data monitoring node is used as the state input of each agent's dual-adversarial deep Q network. The underground cable safety status assessment model is obtained through a large amount of interactive training between the multi-agent and the data environment. The output of each agent's dual-adversarial deep Q network cable safety status assessment model is an assessment of the safety status of the underground cable.

14. The underground cable safety assessment system based on cloud-edge collaboration and multi-agent deep reinforcement learning as claimed in claim 13, characterized in that: The system also includes: The system deployment module is used to deploy the multi-agent dual-adversarial deep Q network cable safety status assessment model obtained after training. The specific steps of deployment are: saving the network parameters of the dual-adversarial deep Q network in the intelligent agent corresponding to each trained monitoring node in the cloud computing service center, reading the multi-source data fusion features transmitted by each edge service node in real time, and evaluating the safety status of each cable operation monitoring point.

15. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, an underground cable safety assessment method based on cloud-edge collaboration and multi-agent deep reinforcement learning as described in any one of claims 1-12 is executed.

Images