Large-scale equipment updating and reconstruction decision-making system and method for grey target reinforcement learning

Abstract

The invention relates to a large-scale equipment updating and reconstruction decision-making system and method for grey target reinforcement learning, and belongs to the technical field of equipment operation and maintenance and artificial intelligence. The system is composed of a data acquisition module, a preprocessing module, a grey feature extraction module, a series reinforcement learning decision module and a grey target evaluation feedback module. The method comprises the following steps: collecting and preprocessing multi-source data of equipment, fusing a text vector and dynamic trend characteristics extracted by a grey model, and constructing a comprehensive state vector; a reinforcement learning model in which event classification and a strategy generation network are connected in series is adopted to simulate expert logic of'first diagnosis and second decision 'for hierarchical decision making; a grey target model is innovatively introduced to calculate a multi-target comprehensive reward, and network parameters are jointly optimized and decided in combination with cross entropy loss of supervision signals. According to the method, depth perception and prospective prediction of the equipment state are realized, the problems of insufficient data utilization, decision black box, single optimization target and the like are solved, and scientificity, accuracy and comprehensive benefits of equipment updating and reconstruction decision are remarkably improved.

Description

Technical Field

[0001] This invention relates to a large-scale equipment upgrade and renovation decision-making system and method based on gray target reinforcement learning, belonging to the interdisciplinary fields of power system asset management, equipment operation and maintenance and artificial intelligence, and is particularly applicable to a large-scale equipment upgrade and renovation decision-making system based on gray target reinforcement learning. Background Technology

[0002] Large-scale power equipment upgrades and renovations are core measures for power grid companies to ensure power supply security, improve operational efficiency, and achieve energy transition. With the deepening of the construction of new power systems, the power grid structure is becoming increasingly complex, the number of equipment is surging, and the aging problem is becoming more prominent. Against this backdrop, scientifically, accurately, and efficiently formulating equipment upgrade strategies is of crucial practical significance for optimizing the input-output ratio of limited capital expenditures, preventing major operational risks, and improving the level of full life-cycle management of power grid assets. This not only relates to the physical security of the power grid but also directly affects the stable operation of the social economy and the sustainable development of enterprises.

[0003] Currently, there is considerable research and practice in the industry regarding decision-making methods for power equipment upgrades and renovations. These methods can be broadly categorized as follows: The first category is rule- and standard-based methods, such as rigid replacement based strictly on test data and lifespan specified in the "Maintenance Regulations for Transmission and Transformation Equipment." While logically clear and easy to implement, these methods fail to adequately consider the individualized operational differences of equipment, potentially leading to either over-maintenance or under-maintenance. The second category is risk assessment matrix-based methods. These methods construct a two-dimensional model of "risk level - importance" to classify equipment into different risk levels and formulate differentiated strategies accordingly. While this method can identify critical equipment, its assessment indicators largely rely on static or recent structured data (such as failure rate and load rate), lacking a dynamic depiction of long-term performance degradation trends and making it difficult to quantify the overall effect of decisions. The third category is operations research-based optimization models, such as Data Envelopment Analysis (DEA), which allocate resources with input-output efficiency as the core objective. While these methods have clear optimization objectives, they typically rely on complete historical data. They are not well-suited for new equipment with small sample sizes and limited information, or for older equipment with incomplete data records. Furthermore, they cannot effectively handle the large amounts of unstructured textual information present in equipment inspection reports and defect records. In addition, although some studies have attempted to introduce machine learning for state prediction, they generally suffer from severe "black box" decision-making processes, an inability to simulate the hierarchical reasoning logic of human experts who "diagnose first, then implement measures," and a singular optimization objective (often focusing only on cost or reliability), making it difficult to balance multiple, potentially conflicting, objectives such as safety, economy, and efficiency.

[0004] In summary, existing technologies generally suffer from three major bottlenecks: First, at the data level, there is insufficient capability for fusing and extracting deep features from multi-source heterogeneous data (especially unstructured text and time-series data), resulting in incomplete and unforeseen decision-making basis. Second, at the model level, the decision-making logic differs from the complex cognitive processes of human experts, failing to effectively simulate the progressive decision-making structure of "state assessment - solution generation," leading to poor accuracy and interpretability of strategies. Third, at the optimization level, there is a lack of mechanisms to integrate multi-dimensional and comprehensive human experience evaluation standards into the model's self-learning process, causing decision results to often excel in a single objective but fail to maximize overall benefits. Therefore, there is an urgent need for an innovative method that can connect the entire chain of data perception, intelligent decision-making, and comprehensive optimization to improve the overall intelligence level of large-scale equipment upgrade and transformation decisions. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a decision system and method for equipment upgrading and renovation that can make full use of multi-source data, have dynamic prediction capabilities, simulate hierarchical decision-making logic, and perform multi-objective comprehensive optimization, namely, a large-scale equipment upgrading and renovation decision system and method based on gray target reinforcement learning.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A large-scale equipment upgrade and renovation decision-making system based on gray target reinforcement learning is presented. This system is a closed-loop intelligent agent integrating data perception, feature extraction, intelligent decision-making, and feedback optimization. Combined with... Figure 1 The feature is that it consists of a data acquisition module (1), a data preprocessing module (2), a gray model feature extraction module (3), a cascaded reinforcement learning decision module (4), and a gray target evaluation feedback optimization module (5).

[0008] The data acquisition module (1) serves as the perception layer of the system and is responsible for collecting multi-source heterogeneous raw data related to equipment status and decision-making in a comprehensive and automated manner; it is connected to the data preprocessing module (2) and the gray target evaluation feedback optimization module (5) respectively.

[0009] The data preprocessing module (2) cleans, standardizes and fuses the collected raw data to obtain equipment static features and time series data; it is connected to the gray model feature extraction module (3) and the cascaded reinforcement learning decision module (4) respectively.

[0010] The gray model feature extraction module (3) extracts dynamic features that reflect the trend of equipment performance degradation using the gray model for the preprocessed time series data; and is connected to the cascaded reinforcement learning decision module (4).

[0011] The static features from the data preprocessing module (2) are fused with the dynamic features from the gray model feature extraction module (3) to obtain the comprehensive state features.

[0012] The cascaded reinforcement learning decision module (4) is implemented by building a reinforcement learning framework using two cascaded deep neural networks: an event classification network and a policy generation network. Its output is the final decision output of the system. The event classification network takes as input comprehensive state features and outputs as a probability distribution of an event type. The policy generation network takes as input a concatenated vector of the output of the event classification network and comprehensive state features and outputs as a probability distribution of a decision action.

[0013] The gray target evaluation feedback optimization module (5) uses the gray target model to conduct a multi-objective comprehensive evaluation of the decision-making effect of the cascaded reinforcement learning decision module (4), generates a reward and feeds it back to the cascaded reinforcement learning decision module (4) to drive the continuous optimization of its parameters; it is connected to the cascaded reinforcement learning decision module (4).

[0014] A large-scale equipment upgrade and renovation decision-making method based on gray target reinforcement learning, combined with Figure 2 It includes the following steps:

[0015] S1: Multi-source data acquisition and preprocessing: Collect equipment text, time series, static and historical decision data, and clean, vectorize and fuse them to form standardized equipment static features and time series data;

[0016] S2: Gray dynamic feature extraction: Based on the gray model GM(1,1), dynamic features are extracted from the preprocessed time series data and fused with static features to generate a comprehensive state vector;

[0017] S3: Construction of the cascaded reinforcement learning decision model: Following the principle of diagnosing event types first and then generating specific decision actions, simulating the expert logic of "diagnosis before treatment", a reinforcement learning framework is constructed using two cascaded deep neural networks, an event classification network and a policy generation network, to build the cascaded reinforcement learning decision model.

[0018] S4: Gray target evaluation and feedback optimization: Calculate the multi-dimensional effect reward of the decision using the gray target model, and optimize the parameters of the reinforcement learning decision model after integrating environmental rewards;

[0019] S5: System Deployment and Continuous Learning: Deploy the trained cascaded reinforcement learning decision model online to provide decision support for large-scale equipment upgrades and renovations, while collecting new data for continuous training to achieve system self-evolution.

[0020] Furthermore, step S1 specifically includes:

[0021] S101: Using the data acquisition module (1), the following four types of data are collected in parallel from the Production Management System (PMS), the Data Acquisition and Monitoring Control System (SCADA), the Online Monitoring System, the Asset Management System (AMS), and the Historical Archives:

[0022] a) Device text data, denoted as T: unstructured text automatically captured from the device;

[0023] b) Equipment time-series data, denoted as X: periodically acquire structured data of the equipment arranged in a time sequence;

[0024] c) Static data of equipment: static attributes of the equipment;

[0025] d) Historical decision sample data: Collect historical records of decisions that have been executed, manually label the event types and decision types, and analyze the decisions...

[0026] The evaluation is based on multiple dimensions of performance.

[0027] S102: The data preprocessing module (2) cleans the collected device text data and inputs it into the pre-trained language model to convert it into V. T ;

[0028] S103: The data preprocessing module (2) fills in missing values, detects and processes outliers in the device time series data, and aligns data from different sources under a unified timestamp to obtain preprocessed time series data.

[0029] S104: Data preprocessing module (2) will process the text feature vector V T By concatenating or weighting the static data of the equipment, a comprehensive static feature vector V of the equipment is formed. S .

[0030] Furthermore, step S2 specifically includes:

[0031] S201: For the preprocessed time series data [x] (0) (1), ..., x (0) (n)] is processed by first-order accumulation to generate (1-AGO), resulting in a new sequence [x] (1) (1), ..., x (1) n)];wherein, n is a positive integer;

[0032] S202: Establish the whitening equation for the GM(1,1) model: dx (1) (f) / dt+a·x (1) (t) = u, and the least squares method is used to solve for the model parameters a and u;

[0033] S203: Based on the model parameters a and u, predict the value at the next time step to obtain the predicted value. Simultaneously, the fitting accuracy index C of the model is calculated, and these parameters, predicted values, and accuracy index are combined to construct a dynamic feature vector V that reflects the trend of equipment performance degradation. G ,Right now in, S2 is the variance of the predicted values, and S1 is the variance of the preprocessed time series data;

[0034] S204: Integrate the static feature vector V S With dynamic feature vector V G The vectors are concatenated to generate a comprehensive state feature vector S, i.e., S = Concat(V S V G ).

[0035] Furthermore, the deep neural network described in step S3 is composed of a CNN network connected in series with a fully connected layer and a Softmax layer.

[0036] Furthermore, step S3 specifically includes:

[0037] S301: The event classification network receiving device integrates the state feature vector S, and performs event type e classification diagnosis through the fully connected layer and the Softmax output layer to obtain the event type e;

[0038] S302: After concatenating the event type e with the comprehensive state feature S, input the concatenation into the policy generation network to obtain the decision action A;

[0039] S303: Record a single interaction of the model as a quintuple (S, e, A, r). e , S′) and store them in the experience replay pool; where r e S is the environmental reward, and S′ is the new comprehensive state feature after performing action A;

[0040] S304: Calculate the cross-entropy loss for event classification and the cross-entropy loss for decision actions, and train the cascaded reinforcement learning decision model.

[0041] Furthermore, step S4 specifically includes:

[0042] S401: Extract the multi-dimensional scoring data corresponding to each historical decision sample i from the historical decision sample library of the data acquisition module (1), and construct the effect vector U. i ;

[0043] S402: Construct the bullseye vector U0 using the optimal values ​​of the scores for each dimension of the indicators;

[0044] S403: For each decision sample in the experience replay pool, calculate its effect vector U. i The distance γ from the target center to the target center vector U0 i And convert the target center distance into a gray target reward r g =-γ i ;

[0045] S404: Reward gray target r g Environmental rewards e Weighted fusion is performed to form a reward r = α·r e +β·r g Where α and β are hyperparameters;

[0046] S405: Employs the Proximal Policy Optimization (PPO) algorithm to maximize the expected total future reward. It samples batch data from the experience replay pool, calculates the policy gradient, and backpropagates to simultaneously update the parameters of the event classification network and the policy generation network.

[0047] An electronic device includes at least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0048] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to execute the aforementioned gray target reinforcement learning-based large-scale equipment upgrade and modification decision-making method.

[0049] Finally, the present invention also discloses a computer-readable storage medium storing computer instructions for causing a processor to execute and implement the above-described gray target reinforcement learning method for large-scale equipment upgrade and modification decision-making.

[0050] The beneficial effects of this invention are as follows: It provides a large-scale equipment upgrade and renovation decision-making system and method based on gray target reinforcement learning. By integrating textual semantics and gray dynamic prediction, it achieves holographic perception of equipment status. It utilizes a cascaded reinforcement learning architecture to simulate the expert decision-making logic of "diagnosis before treatment," significantly improving the accuracy of status assessment and the interpretability of the decision-making process. Simultaneously, it innovatively introduces a gray target model and classification cross-entropy loss to construct a multi-objective reward function, enabling the decision to automatically balance multiple objectives such as safety, economy, and reliability, achieving optimal comprehensive benefits. Finally, it constructs a complete closed loop from data perception to decision optimization. The system has the ability to continuously evolve from historical and real-time data, providing scientific, accurate, and adaptive intelligent decision support for large-scale equipment upgrades and renovations. Attached Figure Description

[0051] To make the objectives and technical solutions of this invention clearer, the following figures are provided for illustration:

[0052] Figure 1 This is an architecture diagram of a large-scale equipment upgrade and transformation decision system based on gray target reinforcement learning in this invention; wherein, the arrows indicate the data transmission direction, and the dashed arrows indicate the training feedback direction.

[0053] Figure 2 This is a flowchart of a large-scale equipment upgrade and renovation decision-making method based on gray target reinforcement learning in this invention.

[0054] Figure 3 This is a diagram of the cascaded reinforcement learning decision model architecture in Embodiment 1 of the present invention; wherein, the arrows indicate the data transmission direction, and the dashed arrows indicate the training feedback direction;

[0055] Figure 4 This is a schematic diagram of the electronic device in Embodiment 3 of the present invention. Detailed Implementation

[0056] To make the objectives and technical solutions of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0057] Example 1: Taking the large-scale upgrade and renovation of substation circuit breakers as an example, historical data includes inspection reports (text data), monthly load rate, number of operations, and fault records (time-series data) for a certain type of circuit breaker over the past three years, as well as decision records and post-event evaluations (changes in safety, cost, and outage time) for each maintenance or replacement in history. To achieve precise and intelligent operation and maintenance of substation circuit breakers, extend their service life, ensure power grid safety, and optimize asset investment efficiency, this invention proposes a "large-scale equipment upgrade and renovation decision-making system based on gray target reinforcement learning."

[0058] Combination Figure 1 The system consists of a data acquisition module (1), a data preprocessing module (2), a gray model feature extraction module (3), a cascaded reinforcement learning decision module (4), and a gray target evaluation feedback optimization module (5), forming a closed-loop intelligent decision-making body.

[0059] The data acquisition module (1) is connected to the data preprocessing module (2) and the gray target evaluation feedback optimization module (5) respectively; the data preprocessing module (2) is connected to the gray model feature extraction module (3) and the cascaded reinforcement learning decision module (4) respectively; the gray model feature extraction module (3) is connected to the cascaded reinforcement learning decision module (4); the gray target evaluation feedback optimization module (5) is connected to the cascaded reinforcement learning decision module (4).

[0060] The workflow of each module is as follows:

[0061] The data acquisition module (1), as the perception layer of the system, is responsible for collecting multi-source heterogeneous data of this type of circuit breaker from sources such as the production management system and SCADA system over the past three years. This includes: inspection report text, time-series data such as monthly load rate and number of operations, as well as historical maintenance / replacement decision records and their multi-dimensional post-event evaluation data (safety changes, costs, power outage time). The output of this module is provided to the data preprocessing module (2) and the gray target evaluation feedback optimization module (5), respectively.

[0062] The data preprocessing module (2) cleans and standardizes the received raw data. It uses a pre-trained language model to convert the text report into a text feature vector and normalizes the time series data. This module merges the text features with the static attributes of the device into static device features and outputs the normalized time series data to the gray model feature extraction module (3), while outputting the static device features to the cascaded reinforcement learning decision module (4).

[0063] The gray model feature extraction module (3) constructs a GM(1,1) gray prediction model on the normalized time series data, extracts key parameters such as the development coefficient a and the next prediction value, and forms a dynamic feature vector representing the performance degradation trend. This vector is sent to the cascaded reinforcement learning decision module (4).

[0064] The cascaded reinforcement learning decision module (4) is the intelligent core of this system. It receives static features from the data preprocessing module (2) and dynamic features from the gray model feature extraction module (3), and merges them into a comprehensive state feature vector S.

[0065] The module employs a two-level serial network architecture: the event classification network takes vector S as input and outputs the probability distribution of event types (such as "normal", "mechanical jam", "insulation degradation"); the policy generation network takes the vector [S, e] concatenated with the event type code and the state vector S as input and outputs the probability distribution of the final decision action (such as "continue to observe", "planned maintenance", "immediate replacement").

[0066] During the training phase, this module employs a hybrid loss function for optimization: in addition to the reinforcement learning objective based on gray target rewards, it also introduces human-labeled real event types and decision types. By calculating cross-entropy loss, it directly guides the outputs of the two networks to align with expert knowledge, thereby accelerating convergence and improving the accuracy and interpretability of decisions.

[0067] The gray target evaluation feedback optimization module (5) obtains multi-dimensional post-evaluation data of historical decisions from the data acquisition module (1), constructs an effect vector containing indicators such as safety and economy, and calculates the target distance between it and the optimal target. This target distance is converted into gray target reward and fused with environmental reward to form a total reward, which is fed back to the cascaded reinforcement learning decision module (4) to drive the iterative update of its parameters and ensure that the decision strategy tends to the multi-objective comprehensive optimum.

[0068] This system enables proactive assessment of the health status of circuit breakers and generates scientific, economical, and safe upgrade and renovation decisions, achieving an intelligent transformation and upgrade from "post-event maintenance" to "precise prediction and proactive intervention".

[0069] Example 2: In response to the decision-making scenario of large-scale upgrading and renovation of substation circuit breakers described in Example 1, this invention also proposes "a large-scale equipment upgrading and renovation decision-making method based on gray target reinforcement learning".

[0070] Combination Figure 2 The method includes the following steps:

[0071] S1: Multi-source data acquisition and preprocessing: Collecting textual, time-series, static, and historical decision data from equipment, which is then cleaned, vectorized, and fused to form standardized static features and time-series data of the equipment. Specifically,

[0072] S101: Using the data acquisition module (1), the following four types of data are collected in parallel from the Production Management System (PMS), the Data Acquisition and Monitoring Control System (SCADA), the Online Monitoring System, the Asset Management System (AMS), and the Historical Archives:

[0073] a) Equipment text data, denoted as T: Automatically captures unstructured text such as equipment inspection records, defect descriptions, and maintenance reports;

[0074] b) Equipment time-series data, denoted as X: periodically acquire structured data of equipment such as load, temperature, vibration, and number of operations arranged in time series;

[0075] c) Static data of equipment: Static attributes such as equipment model, commissioning date, and rated parameters are synchronized once or periodically;

[0076] d) Historical decision sample data: Collect historical records of maintenance, replacement and other decisions, manually label the event type and decision type, and score the multi-dimensional effects of the decision, such as cost, reliability and security.

[0077] S102: The data preprocessing module (2) cleans (denoises and segments) the collected device text data and inputs it into a pre-trained language model (such as BERT) to convert it into a text feature vector V.T ;

[0078] S103: The data preprocessing module (2) fills in missing values, detects and processes outliers in the device time series data, and aligns data from different sources under a unified timestamp to obtain preprocessed time series data.

[0079] S104: Data preprocessing module (2) will process the text feature vector V T By concatenating or weighting the static data of the equipment, a comprehensive static feature vector V of the equipment is formed. S .

[0080] S2: Gray Dynamic Feature Extraction: Based on the gray model GM(1,1), dynamic features are extracted from the preprocessed time-series data and fused with static features to generate a comprehensive state vector. Specifically,

[0081] S201: For the preprocessed time series data [x] (0) (1), ..., x (0) (n)] is processed by first-order accumulation to generate (1-AGO), resulting in a new sequence [x] (1) (1), ..., x (1) (n)]; where, n is a positive integer;

[0082] S202: Establish the whitening equation for the GM(1,1) model: dx (1) (t) / dt+a·x (1) (t) = u, and the least squares method is used to solve for the model parameters a and u;

[0083] S203: Based on the model parameters a and u, predict the value at the next time step to obtain the predicted value. Simultaneously, the fitting accuracy index C of the model is calculated, and these parameters, predicted values, and accuracy index are combined to construct a dynamic feature vector V that reflects the trend of equipment performance degradation. G ,Right now in, S2 is the variance of the predicted values, and S1 is the variance of the preprocessed time series data;

[0084] S204: Integrate the static feature vector V S With dynamic feature vector V G The vectors are concatenated to generate a comprehensive state feature vector S, i.e., S = Concat(V S V G ).

[0085] S3: Construction of the Serial Reinforcement Learning Decision Model: Following the principle of diagnosing event types first and then generating specific decision actions, simulating the expert logic of "diagnosis before treatment," a reinforcement learning framework is constructed using two Serial deep neural networks: an event classification network and a policy generation network, to build the Serial Reinforcement Learning Decision Model.

[0086] The deep neural network described is composed of a CNN network, a fully connected layer, and a Softmax layer connected in series.

[0087] Specifically,

[0088] S301: The event classification network receiving device integrates the state feature vector S, and performs event type e classification diagnosis through the fully connected layer and the Softmax output layer to obtain the event type e;

[0089] S302: After concatenating the event type e with the comprehensive state feature S, input the concatenation into the policy generation network to obtain the decision action A;

[0090] S303: Record a single interaction of the model as a quintuple (S, e, A, r). e , S′) and store them in the experience replay pool; where r e S is the environmental reward, and S′ is the new comprehensive state feature after performing action A;

[0091] S304: Calculate the cross-entropy loss L_event = CrossEntropy(P(e|S), e_true) for event classification and the cross-entropy loss L_action = CrossEntropy(P(A|S, e), a_true) for decision action, and train the cascaded reinforcement learning decision model. Here, e_true and a_true are the human-annotated true labels for event type and decision type.

[0092] S4: Gray Target Evaluation and Feedback Optimization: Utilizing a gray target model, the multi-dimensional effect reward of the decision is calculated. Environmental rewards are then incorporated, and the feedback optimization optimizes the parameters of the cascaded reinforcement learning decision model. Specifically,

[0093] S401: Extract the multi-dimensional scoring data corresponding to each historical decision sample i from the historical decision sample library of the data acquisition module (1), and construct the effect vector U. i ;

[0094] S402: Construct the bullseye vector U0 using the optimal values ​​of the scores for each dimension of the indicators;

[0095] S403: For each decision sample in the experience replay pool, calculate its effect vector U. i The distance γ from the target center to the target center vector U0 i And convert the target center distance into a gray target reward rg =-γ i ;

[0096] S404: Reward gray target r g Environmental rewards e Weighted fusion is performed to form a reward r = α·r e +β·r g Where α and β are hyperparameters;

[0097] S405: Employs the Proximal Policy Optimization (PPO) algorithm to maximize the expected total future reward. It samples batch data from the experience replay pool, calculates the policy gradient, and backpropagates to simultaneously update the parameters of the event classification network and the policy generation network.

[0098] S5: Deploy a well-trained cascaded reinforcement learning decision model online to provide real-time decision support for large-scale equipment upgrades and renovations. At the same time, collect newly generated data for continuous training to achieve system self-evolution and performance improvement.

[0099] Example 3: For the scenario described in Example 1, Figure 4 A schematic diagram of an electronic device 90 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0100] Electronic devices can also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0101] like Figure 4 As shown, the electronic device 90 includes at least one processor 91 and a memory, such as a read-only memory (ROM) 92 or a random access memory (RAM) 93, communicatively connected to the at least one processor 91. The memory stores computer programs executable by the at least one processor. The processor 91 can perform various appropriate actions and processes based on the computer program stored in the ROM 92 or loaded into the RAM 93 from storage unit 98. The RAM 93 can also store various programs and data required for the operation of the electronic device 90. The processor 91, ROM 92, and RAM 93 are interconnected via a bus 94. An input / output (I / O) interface 95 is also connected to the bus 94.

[0102] Multiple components in electronic device 90 are connected to I / O interface 95, including: input unit 96, such as keyboard, mouse, etc.; output unit 97, such as various types of displays, speakers, etc.; storage unit 98, such as disk, optical disk, etc.; and communication unit 99, such as network card, modem, wireless transceiver, etc. Communication unit 99 allows electronic device 90 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 91 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 91 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 91 performs the various methods and processes described above, such as a large-scale equipment upgrade and modification decision-making method using gray target reinforcement learning.

[0104] In some embodiments, a large-scale equipment upgrade and renovation decision-making method based on gray target reinforcement learning can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 98. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 90 via ROM 92 and / or communication unit 99. When the computer program is loaded into RAM 93 and executed by processor 91, one or more steps of the large-scale equipment upgrade and renovation decision-making method based on gray target reinforcement learning described above can be performed. Alternatively, in other embodiments, processor 91 can be configured to perform a large-scale equipment upgrade and renovation decision-making method based on gray target reinforcement learning by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0111] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A gray-box target reinforcement learning-based large-scale equipment revamping decision system, characterized in that, The data acquisition module (1), the data preprocessing module (2), the grey model feature extraction module (3), the series reinforcement learning decision module (4) and the grey target evaluation feedback optimization module (5) are connected. The data acquisition module (1) is connected with the data preprocessing module (2) and the grey target evaluation feedback optimization module (5); the data preprocessing module (2) is connected with the grey model feature extraction module (3) and the series reinforcement learning decision module (4); the grey model feature extraction module (3) is connected with the series reinforcement learning decision module (4); and the grey target evaluation feedback optimization module (5) is connected with the series reinforcement learning decision module (4).

2. A gray target reinforcement learning large-scale equipment revamping decision method, characterized in that, The method comprises the following steps: S1: multi-source data acquisition and preprocessing: collecting equipment text, time series, static and historical decision data, and forming standardized equipment static features and time series data through cleaning, vectorization and fusion; S2: grey dynamic feature extraction: extracting dynamic features from the preprocessed time series data based on the grey model GM(1,1), and fusing the dynamic features with the static features to generate a comprehensive state vector; S3: construction of a series reinforcement learning decision model: according to the expert logic of "diagnosis first and treatment second", the event classification network and the policy generation network are used to construct a deep neural network to build a reinforcement learning framework, and a series reinforcement learning decision model is constructed; S4: grey target evaluation and feedback optimization: the multi-dimensional effect reward of the decision is calculated by using the grey target model, and the parameters of the series reinforcement learning decision model are fed back and optimized after fusing the environmental reward; S5: system deployment and continuous learning: the trained series reinforcement learning decision model is deployed online to provide decision support for large-scale equipment updating and reconstruction, and new data is collected for continuous training to realize self-evolution of the system.

3. The grey-box reinforcement learning mass equipment revamping decision-making method according to claim 2, characterized in that, The step S1 specifically comprises: S101: collecting equipment text data, equipment time series data, equipment static data and historical decision sample data from a production management system, a data acquisition and monitoring control system, an online monitoring system, an asset management system and a historical archive library in parallel; S102: The collected device text data is cleaned and input into a pre-trained language model to convert it into V T ; S103: filling in missing values, detecting and processing abnormal values of the equipment time series data, and aligning the data of different sources under a unified timestamp to obtain preprocessed time series data; S104: Concatenate or weightedly fuse the text feature vector V T with the device static data to form a comprehensive static feature vector V S of the device. 4.The grey target reinforcement learning-based large-scale equipment updating and modification decision method according to claim 2, characterized in that, The step S2 specifically comprises: S201: For the preprocessed time series data [x] (0) (1), ..., x (0) (n)] is generated by first-order accumulation to obtain a new sequence [x (1) (1), ..., x (1) (n)]; where, n is a positive integer; S202: Establish the whitening equation of GM(1,1) model: dx (1) (t) / dt+a·x (1) (t) = u, and solve the model parameters a and u by least square method; S203: based on the model parameters a and u, the prediction of the next time value is carried out to obtain the predicted value At the same time, the fitting accuracy index C of the calculation model is calculated, and the model parameters, the predicted value and the accuracy index are combined to construct a dynamic feature vector V reflecting the performance degradation trend of the device G That is Wherein, S2 is the variance of the predicted value, and S1 is the variance of the preprocessed time series data; S204: Concatenate the static feature vector V S with the dynamic feature vector V G to generate a comprehensive state feature vector S, i.e., S = Concat(V S , V G ).

5. The grey-box reinforcement learning mass equipment upgrade retrofit decision method of claim 2, wherein, The deep neural network of the step S3 is composed of a CNN network, a fully connected layer and a Softmax layer in series.

6. The grey-box reinforcement learning mass equipment revamping decision-making method according to claim 2, characterized in that, The step S3 specifically comprises: S301: the event classification network receives the equipment comprehensive state feature vector S, classifies and diagnoses the event type e through a fully connected layer and a Softmax output layer, and obtains the event type e; S302: after splicing the event type e and the comprehensive state feature S, the spliced result is input into the policy generation network to obtain the decision action A; S303: record a five-tuple (S, e, A, r e , S') of one interaction of the model as a record and store it in the experience replay pool; wherein r e is the environment reward, and S' is the new comprehensive state feature after performing the action A. S304: the cross-entropy loss of event classification and the cross-entropy loss of decision action are calculated, and the series reinforcement learning decision model is trained.

7. The grey-box reinforcement learning mass equipment revamping decision-making method according to claim 2, characterized in that, The step S4 specifically comprises: S401: Extract multi-dimensional score data corresponding to each historical decision sample i from historical decision sample data, and construct an effect vector U; S402: Use the optimal value of each dimension index score to form a target vector U0; S403: For each decision sample in the experience replay pool, calculate its effect vector U1 and the target distance γ of the target vector U0 t , and convert the target distance into the gray target reward r g = - γ t ; S404: Reward gray target r g Environmental rewards e Weighted fusion is performed to form a reward r = α·r e +β·r g Where α and β are hyperparameters; S405: Using the proximal policy optimization (PPO) algorithm, sampling batch data from the experience replay pool, calculating the policy gradient, and updating the parameters of the event classification network and the policy generation network simultaneously by back propagation, with the goal of maximizing the expected value of future cumulative total rewards.

8. An electronic device, comprising: The electronic device comprises at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the grey target reinforcement learning large-scale equipment updating and transformation decision method of any one of claims 2-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to implement the grey target reinforcement learning large-scale equipment updating and transformation decision method of any one of claims 2-7 when executed.

Images