A sensor configuration optimization method and system based on deep reinforcement learning

Abstract

The application discloses a kind of sensor configuration optimization method and system based on deep reinforcement learning, it is related to machine learning technical field, method includes: obtaining the historical full-amount sensor dataset for target equipment, training fault diagnosis teacher model and constructing deep reinforcement learning environment;The state vector formed by current selected sensor channel and its monitoring data is used as the environment state, the joint selection of candidate sensor channel and global sampling rate is used as action, the deep reinforcement learning intelligent agent is iteratively trained by reward signal, and the target sensor configuration optimization strategy is generated;Monitoring data is collected according to configuration optimization strategy.Through the technical scheme of the application, the collaborative optimization of the number of sensors and the sampling frequency is realized, the data acquisition, transmission and storage costs are reduced under the premise of ensuring the fault diagnosis performance, and the automation and deployability of sensor configuration design are improved.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a sensor configuration optimization method and a sensor configuration optimization system based on deep reinforcement learning. Background Technology

[0002] Currently, the condition monitoring and fault diagnosis of complex electromechanical equipment such as railway switch machines typically involve the design and optimization of sensor configuration schemes. Taking railway switch machines as an example, as key track equipment ensuring train route switching and traffic safety, existing systems usually deploy various types of sensors, such as vibration and current sensors, in multiple key parts such as the drive mechanism, locking mechanism, and switching components to achieve accurate monitoring of their operating status, and continuously collect signals at a high sampling rate. While this "multi-point measurement + high-frequency sampling" configuration is beneficial for obtaining more comprehensive status information, it also introduces problems such as a large number of sensors, complex deployment, high hardware and wiring costs, and a huge amount of data, putting significant pressure on communication bandwidth, storage resources, and backend computing resources. At the same time, there are often strong correlations and significant redundancies between different measurement points, and not all channels and sampled data contribute substantially to fault diagnosis.

[0003] Existing technologies for sensor configuration optimization mainly employ three approaches: The first is based on statistical criteria such as information entropy and variance contribution rate, which are not directly related to specific diagnostic tasks. This involves selecting a subset of sensors using methods like principal component analysis and chi-square tests. However, this approach lacks direct consistency between the optimization objective and the final fault diagnosis accuracy, easily resulting in suboptimal configurations. The second approach treats the configuration problem as a combinatorial optimization problem, using heuristic algorithms such as evolutionary algorithms and particle swarm optimization to search a vast solution space. However, evaluating each candidate configuration typically requires complete training and validation of the downstream diagnostic model, leading to excessively high computational costs and hindering large-scale application in engineering practice. The third approach involves domain experts designing surrogate evaluation criteria, combined with heuristic search for optimization. However, the correlation between these surrogate criteria and actual diagnostic performance cannot be guaranteed, and the upper limit of the optimization results is strongly constrained by human experience.

[0004] Against this backdrop, for complex electromechanical equipment such as railway switch machines, there is still a lack of a systematic method that is oriented towards specific fault diagnosis tasks and can automatically balance the acquisition and maintenance costs brought about by the number of sensors and sampling frequency, given a set of candidate sensor channels and sampling rates, in order to obtain a near-optimal sensor configuration scheme. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a sensor configuration optimization method and system based on deep reinforcement learning. By training a fault diagnosis teacher model and constructing a deep reinforcement learning environment on this basis, it achieves automatic search and evaluation of candidate sensor channels and global sampling rate combinations in an offline simulation environment. By iteratively executing a Markov decision process of state construction, action selection, and reward feedback in the deep reinforcement learning environment, it can optimize sensor configuration strategies that balance diagnostic effectiveness and data cost. Through the strategy generation stage and the online deployment stage, it achieves coordinated compression of the number of sensors and sampling frequency while maintaining fault diagnosis capabilities, thereby improving the economy and deployability of the sensor configuration scheme.

[0006] To achieve the above objectives, this invention provides a sensor configuration optimization method based on deep reinforcement learning, comprising:

[0007] Obtain a historical full sensor dataset collected for the target device, the historical full sensor dataset including monitoring signals collected by candidate sensor channels at a preset highest sampling frequency;

[0008] A fault diagnosis teacher model is trained based on the historical full sensor dataset, and a deep reinforcement learning environment is constructed based on the historical full sensor dataset and the trained fault diagnosis teacher model.

[0009] In the deep reinforcement learning environment, a sensor configuration optimization process is constructed. The state vector formed by the currently selected sensor channels and their monitoring data is taken as the environment state. The decisions on the selection of candidate sensor channels and the selection of the global sampling rate are taken as actions. The deep reinforcement learning agent performs sequential decisions in the deep reinforcement learning environment. The deep reinforcement learning environment evaluates the current sensor configuration based on the actions and outputs the updated environment state and the reward signal used to train the deep reinforcement learning agent.

[0010] Based on the reward signal, state construction, action selection and reward feedback are iteratively performed in the deep reinforcement learning environment to optimize the sensor configuration until convergence, thereby obtaining a sensor configuration optimization strategy.

[0011] During the policy generation phase, the converged deep reinforcement learning agent is invoked to output a target sensor configuration policy, including the set of selected sensor channels and the target global sampling rate, under preset constraints.

[0012] During the online deployment phase, the sensor acquisition system is controlled according to the target sensor configuration strategy to collect monitoring data from only the sensor channels specified in the target sensor configuration strategy at the target global sampling rate.

[0013] In the above technical solution, preferably, the process by which the deep reinforcement learning environment evaluates the current sensor configuration and outputs the reward signal specifically includes:

[0014] Based on the historical full sensor dataset, the monitoring data corresponding to the currently selected sensor channel is extracted to construct a data subset. The data subset is then downsampled according to the global sampling rate obtained from the current decision. The downsampled data is then input into the fault diagnosis teacher model to obtain diagnostic performance indicators used to characterize the diagnostic effect.

[0015] Based on the data volume relationship between the data subset and the historical full sensor dataset, a cost-saving score index is determined to characterize data costs. The diagnostic performance index and the cost-saving score index are then weighted and fused to generate the reward signal used to train the deep reinforcement learning agent.

[0016] In the above technical solution, preferably, in the step of constructing the sensor configuration optimization process in the deep reinforcement learning environment, the state vector includes: a global statistical feature vector z calculated based on the monitoring data of the currently selected sensor channels, and a binary mask vector m used to indicate the selection history of candidate sensor channels. t ;

[0017] Wherein, the global statistical feature vector z is constructed based on the monitoring signals of the currently selected sensor channels, and includes at least the mean and variance of the monitoring signals of each of the selected sensor channels; or it is obtained by feature extraction of the monitoring signals of the currently selected sensor channels by a pre-trained autoencoder.

[0018] The state vector is composed of the content feature vector z and the binary mask vector m. t The data is spliced ​​together to simultaneously characterize the current sensor data features and channel selection.

[0019] In the above technical solution, preferably, the process by which the deep reinforcement learning agent performs the sequential decision-making in the deep reinforcement learning environment includes:

[0020] In the preset number of decision rounds M+1, the first M decisions sequentially select a sensor channel from the set of currently unselected candidate sensor channels, and the last decision selects one from the preset set of global sampling rates as the global sampling rate corresponding to the current decision.

[0021] In the above technical solution, preferably, the process of generating the reward signal by weighted fusion based on the diagnostic performance index and the cost-saving score index includes: after the deep reinforcement learning agent completes the selection of sensor channels and global sampling rate for the current sensor configuration, calculating the terminal reward R.terminal ,in,

[0022] R terminal =w acc ·R acc +w comp ·R comp ;

[0023] Among them, R acc R is the diagnostic performance reward output by the fault diagnosis teacher model. comp The cost-saving reward is determined based on the data compression rate of the subset of data and the historical full sensor dataset, w acc and w comp These are preset weighting coefficients used to control the trade-off between diagnostic performance and data cost.

[0024] In the above technical solution, preferably, the deep reinforcement learning agent adopts a neural network structure based on the Actor-Critic architecture, the neural network structure including: a function for processing the content feature vector z and the mask vector m in the state vector respectively. t A dual-stream input layer;

[0025] A shared feature extraction backbone composed of a one-dimensional convolutional network and gated recurrent units is used to extract local temporal patterns and temporal dependencies in the decision-making process;

[0026] And a multi-head output layer connected to the shared feature extraction backbone, the multi-head output layer including a first action output head for outputting the sensor channel selection probability distribution, a second action output head for outputting the global sampling rate selection probability distribution, and a value output head for outputting the current state value assessment.

[0027] In the above technical solution, preferably, the fault diagnosis teacher model is a fault diagnosis model built on a convolutional neural network. It is trained and its parameters are fixed in the offline stage using the historical full sensor dataset. In the deep reinforcement learning environment, it acts as a performance evaluator and outputs a diagnostic performance score for the subset of data determined by the global sampling rate obtained from the currently selected sensor channels and the current decision, in order to support the generation of the reward signal.

[0028] In the above technical solution, preferably, the target device is a railway switch machine, and the specific process of controlling the sensor acquisition system according to the target sensor configuration strategy includes:

[0029] Based on the selected sensor channel set, the sensor installation positions on different structural components of the railway switch machine are determined, and the data acquisition equipment is configured to collect monitoring data of the selected sensor channels only at the target global sampling rate. The collected monitoring data is then input into the downstream fault diagnosis system for real-time health status assessment.

[0030] In the above technical solution, preferably, in the policy generation stage, a complete deterministic decision-making process is performed on the converged deep reinforcement learning agent. At each decision step, the action with the highest probability is selected based on the action probability distribution of the policy output. The sensor channel selection and global sampling rate selection are completed in sequence, thereby outputting a unique target sensor configuration policy under the condition of the given historical full sensor dataset.

[0031] This invention also proposes a sensor configuration optimization system based on deep reinforcement learning, which applies the sensor configuration optimization method based on deep reinforcement learning disclosed in any of the above technical solutions, including:

[0032] The data acquisition module is used to acquire the historical full sensor dataset collected for the target device. The historical full sensor dataset includes monitoring signals collected by candidate sensor channels at a preset highest sampling frequency.

[0033] The teacher model training module is used to train a fault diagnosis teacher model based on the historical full sensor dataset.

[0034] The learning environment construction module is used to construct a deep reinforcement learning environment based on the historical full sensor dataset and the trained fault diagnosis teacher model;

[0035] The configuration strategy optimization module is used to construct a sensor configuration optimization process in the deep reinforcement learning environment. It takes the state vector formed by the currently selected sensor channel and its monitoring data as the environment state, and takes the decision on the selection of candidate sensor channels and the selection of global sampling rate as actions. It enables the deep reinforcement learning agent to perform sequential decision-making in the deep reinforcement learning environment, and iteratively updates the deep reinforcement learning agent based on the updated environment state and reward signal of the environment output until convergence, so as to obtain the sensor configuration optimization strategy.

[0036] The strategy generation module is used to call the converged deep reinforcement learning agent during the strategy generation stage and output a target sensor configuration strategy including the selected sensor channel set and the target global sampling rate under preset constraints.

[0037] An online deployment control module is used to control the sensor acquisition system according to the target sensor configuration strategy during the online deployment phase, and to collect only the monitoring data of the sensor channels specified in the target sensor configuration strategy at the target global sampling rate.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] (1) By training a fault diagnosis teacher model based on the historical full sensor dataset and building a deep reinforcement learning environment on this basis, the automatic search and evaluation of candidate sensor channels and global sampling rate combinations in the offline simulation environment is realized, avoiding repeated changes of sensor configuration and adjustment of sampling rate on actual equipment, and reducing the on-site test cost and debugging risk in the configuration optimization process.

[0040] (2) By iteratively executing the Markov decision process of state construction, action selection and reward feedback in a deep reinforcement learning environment, the joint effect of sensor channel selection and sampling frequency configuration is modeled and globally optimized. Compared with configuration methods based on empirical rules or greedy search, it is more conducive to obtaining an optimized sensor configuration strategy that balances diagnostic effect and data cost under preset constraints.

[0041] (3) By calling the output of the converged deep reinforcement learning agent in the policy generation stage, including the selected sensor channel set and the target global sampling rate, and collecting monitoring data of the sensor channels specified in the target sensor configuration strategy at the target global sampling rate in the online deployment stage, the collaborative compression of the number of sensors and sampling frequency is achieved while maintaining the fault diagnosis capability. This reduces the investment in sensor hardware and the back-end data acquisition, transmission and storage overhead, and improves the economy and deployability of the sensor configuration scheme. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a sensor configuration optimization method based on deep reinforcement learning disclosed in one embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the sensor configuration network architecture of the PSRONet agent disclosed in one embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The present invention will now be described in further detail with reference to the accompanying drawings:

[0046] like Figure 1 and Figure 2 As shown, the sensor configuration optimization method based on deep reinforcement learning provided by the present invention is executed in three stages: "offline training and optimization - policy generation - online deployment and application".

[0047] First, a historical full-volume sensor dataset is constructed based on the monitoring data collected during the long-term operation of the target equipment (such as a complex electromechanical device). The dataset includes the raw signals collected by all candidate sensor channels at the highest sampling frequency (such as 512 Hz) and covers a variety of operating conditions and fault modes.

[0048] During the offline training and optimization phase, a high-precision fault diagnosis teacher model is trained using the historical full sensor dataset. The preferred teacher CNN is a convolutional neural network structure. After training, the model parameters are frozen so that it can be used as a performance evaluator.

[0049] Then, based on the teacher model and the historical full sensor dataset, a deep reinforcement learning environment is constructed, and the sensor configuration problem is formalized as a Markov decision process. The deep reinforcement learning agent performs sequential decision-making in this environment: in each round of decision-making, a new sensor channel and the final global sampling rate are selected according to the current environment state. The environment updates the sensor configuration, extracts the corresponding data subset, evaluates the diagnostic performance through the teacher model, and generates a reward signal based on the degree of data compression, which is then fed back to the agent.

[0050] In multiple rounds of interaction, the agent iteratively executes state construction, action selection, and reward feedback, continuously optimizing its policy parameters until the configuration policy converges, thus obtaining the sensor configuration optimization policy.

[0051] During the policy generation phase, the converged deep reinforcement learning agent is loaded and allowed to perform a complete decision-making process in the deep reinforcement learning environment, outputting a specific and clear target sensor configuration policy, including the set of selected sensor channels and the target global sampling rate.

[0052] During the online deployment phase, the target configuration strategy is sent to the field sensor acquisition system. Only the sensor channels specified by the target strategy are enabled in the data acquisition equipment, and the sampling frequency of these channels is uniformly set to the target global sampling rate. "Few but high-quality" monitoring data are collected online and sent to the downstream fault diagnosis system for real-time health status assessment.

[0053] In this implementation, the sensor configuration problem is modeled as a deep reinforcement learning problem. An offline training environment is built using a teacher model, which enables automatic searching and optimization of sensor channel and sampling rate combinations in the simulation environment. This reduces the need for repeated trial installations and parameter adjustments on real devices, lowers on-site testing costs and debugging risks, and provides a unified data flow and decision framework for subsequent detailed schemes in subordinate weights.

[0054] In the above implementation, preferably, during the process of the deep reinforcement learning environment evaluating the current sensor configuration and outputting reward signals, the following steps are specifically performed: after each step of the sequential decision is completed, based on the set of sensor channels currently selected by the agent, the monitoring signals of the corresponding channels are extracted from the historical full sensor dataset to construct a data subset, and the data subset is downsampled according to the global sampling rate obtained from the current decision to form a sensor data subset under a low sampling rate.

[0055] The downsampled subset of data is then input into the teacher CNN, which has been trained offline and has its parameters frozen. The teacher CNN outputs a diagnostic performance metric (such as prediction accuracy R) to characterize the diagnostic effectiveness of the current configuration. acc Simultaneously, based on the compression ratio of the data subset to the historical full sensor dataset in terms of data volume (e.g., the proportion of currently used data to the original data), a cost-saving score R, used to characterize data costs, is calculated. comp The cost-saving score increases with the increase of data compression rate.

[0056] Finally, the diagnostic performance index and the cost-saving score index are weighted and fused according to preset weights to construct a reward signal for training the deep reinforcement learning agent. The reward signal is fed back to the agent all at once when the configuration is completed, so as to guide the agent to make trade-offs between diagnostic performance and data cost.

[0057] In this implementation, a multi-objective reward signal is constructed by using the diagnostic performance index output by the teacher's CNN and the cost-saving index based on the data compression rate. This enables a quantitative comprehensive evaluation of the two key objectives of "diagnostic accuracy" and "data cost" under a unified framework, guiding the agent to learn to compress the amount of data as much as possible while ensuring diagnostic performance, thereby improving the cost-effectiveness of the sensor configuration optimization results.

[0058] In the above embodiments, preferably, in the step of constructing the sensor configuration optimization process in a deep reinforcement learning environment, the environmental state s is... t The design consists of a content feature vector z and a binary mask vector m. t The concatenated state vector, where m t A mask vector for recording decision history, used to indicate which candidate sensor channels have been selected before step t, and unselected channels in step m. t The corresponding position in the middle is 0, and the corresponding position of the selected channel is 1.

[0059] In a preferred embodiment, the content feature vector z is a global statistical feature vector constructed based on the monitoring signals of the currently selected sensor channels. It includes at least the mean and variance of the monitoring signals for each selected channel, and is used to summarize the intensity level and fluctuation characteristics of the current sensor data in a compressed feature space. In another embodiment, the content feature vector z is obtained by a pre-trained autoencoder extracting features from the monitoring signals of the currently selected sensor channels. After pre-training in the offline phase, the autoencoder is deployed decoupled from the deep reinforcement learning agent as an independent feature extraction module, and is used to provide the content feature vector z to the agent during the state construction phase.

[0060] Finally, at each decision-making moment, the content feature vector z is compared with the binary mask vector m. t The state vector s is formed by concatenating the dimensions. t As one of the inputs to a deep reinforcement learning agent network, it is used to simultaneously characterize the current sensor data features and the channel selection history.

[0061] In this implementation, a state construction method using statistical / autoencoder content features and channel selection masks is used to encode data content information and decision history information in a unified vector space. This provides the agent with a state description that simultaneously reflects signal quality and remaining selectable channels, enhancing the policy learning process's ability to perceive the configuration space structure. This helps the agent converge to a better sensor configuration strategy in a complex configuration space.

[0062] In the above implementation, preferably, the sequential decision-making performed by the deep reinforcement learning agent in the deep reinforcement learning environment is divided into M+1 rounds of discrete action selection process, wherein the first M rounds are used only to select a new channel from the set of candidate sensor channels that have not been selected at present, and the last round is used to select a target global sampling rate from the set of preset global sampling rates.

[0063] Specifically, M can be set to the desired number of sensor channels (e.g., 4), and the action space is divided into a "channel selection action subspace" and a "sampling rate selection action subspace": In the first to the Mth rounds of decision-making, the agent determines the current state vector s based on the action vector s.t The action outputs a channel index from the set of unselected channel indices, and the environment updates the set of selected channels and the mask vector m accordingly. t In the (M+1)th round of decision-making, the agent selects one from a preset set of global sampling rates (such as a set of several downsampling coefficients) as the current global sampling rate. Thus, a complete sensor configuration process corresponds to M+1 action selections.

[0064] In this implementation, "channel selection" and "sampling rate selection" are decoupled in the action space and executed in a fixed round sequence, making the decision-making process clear and the action dimensions limited. This reduces the combinatorial complexity of the action space, thereby improving the stability and convergence efficiency of deep reinforcement learning training. It is beneficial to obtain a stable configuration strategy under limited training resources.

[0065] In the above implementation, preferably, after the deep reinforcement learning agent completes M+1 rounds of decision-making for the current sensor configuration, the environment calculates the terminal reward R_terminal at the terminal time, and the calculation method is as follows:

[0066] R terminal =w acc ·R acc +w comp ·R comp ;

[0067] Among them, R acc The reward for diagnostic performance predicted by the teacher's CNN can be the diagnostic accuracy output by the teacher's CNN or its normalized form; R comp The cost-saving reward is calculated based on the compression ratio of the current subset of data and the historical full sensor dataset. The higher the compression ratio, the higher the R... comp The larger; w acc and w comp These are preset weighting coefficients used to adjust the trade-off between diagnostic performance and cost savings.

[0068] The terminal reward is fed back to the deep reinforcement learning agent once at the end of each configuration sequence. By maximizing the learning objective of long-term cumulative rewards, the agent gradually learns how to make reasonable trade-offs between different channel combinations and sampling rate configurations, thereby achieving joint optimization of sensor configuration.

[0069] In this implementation, a multi-objective terminal reward function consisting of diagnostic performance rewards and cost-saving rewards is used to characterize the quantitative relationship between the two objectives of improving diagnostic accuracy and reducing data costs. This guides the deep reinforcement learning agent to take into account both accuracy and cost during policy updates, avoiding the pursuit of only a single indicator and improving the practicality and engineering promotion value of the optimization results.

[0070] In the above embodiments, preferably, the deep reinforcement learning agent adopts a neural network structure based on the Actor-Critic architecture, which includes a dual-stream input layer, a shared feature extraction backbone, and a multi-head output layer.

[0071] In the dual-stream input layer, one input is used to receive the content feature vector z from the state vector, and the other input is used to receive the binary mask vector m. t The two inputs are processed through independent pre-processing layers (such as several fully connected layers or one-dimensional convolutional layers) to extract suitable primary features, which are then fused along the channel dimension. The fused features are fed into a shared feature extraction backbone consisting of a one-dimensional convolutional network and gated recurrent units (GRUs). The one-dimensional convolutional network is responsible for extracting local temporal patterns, while the GRUs are used to capture temporal dependencies across decision steps.

[0072] A multi-head output layer is set up at the shared backbone output end, including two action output heads and one value output head: the first action output head outputs the probability distribution for sensor channel selection, which is used to provide the action distribution of candidate channels in the channel selection round; the second action output head outputs the probability distribution for global sampling rate selection, which is used to provide the sampling rate action distribution in the sampling rate selection round; the value output head outputs the value estimate of the current state, which is used as the benchmark signal in the Actor-Critic algorithm to guide the policy gradient update.

[0073] In this implementation, the PSRONet network structure with dual-stream input, shared CNN+GRU backbone and multi-head output is used to achieve collaborative modeling of state content features, decision history information and sequence decision value within the same network architecture. This enhances the agent's ability to express the dependencies between complex time series signals and action sequences, which is beneficial for learning higher-quality sensor configuration strategies in a high-dimensional configuration space.

[0074] In the above implementation, preferably, a convolutional neural network (CNN) structure is used for offline training on a historical full-dataset of sensors. Specifically, during the training phase, the historical full-dataset of sensors is divided according to operating conditions and fault labels. The raw signals collected by all candidate sensor channels at the highest sampling frequency are sliced ​​into time windows and input into the CNN model. The CNN extracts temporal features through multi-layer one-dimensional convolution and pooling structures, and classifies fault types or health states at the output through fully connected layers. After training, the CNN model parameters are fixed and used as the teacher CNN to evaluate the diagnostic performance of any subset of sensor data.

[0075] In a deep reinforcement learning environment, a subset of data determined by the global sampling rate obtained from the currently selected sensor channels and the current decision is input into the teacher CNN. The teacher CNN outputs a diagnostic performance score (such as classification accuracy or confidence), which serves as the source of diagnostic performance indicators when constructing reward signals in the deep reinforcement learning environment.

[0076] In this implementation, a teacher CNN trained offline on the full historical data and with its parameters frozen is used as a performance evaluator. This allows the diagnostic performance evaluation process, which originally required long runs on real systems, to be migrated to an efficient model prediction process. The time for each configuration evaluation is reduced from minutes to milliseconds, significantly reducing the computational cost of reinforcement learning search in a large configuration space and making large-scale sensor configuration optimization computationally feasible.

[0077] In the above embodiments, preferably, the target equipment is a railway switch machine, such as a ZDJ9 type switch machine. During the online deployment phase, based on the set of selected sensor channels included in the converged target sensor configuration strategy, the specific installation position of each channel on the switch machine structure is determined. For example, channels 0 and 1 correspond to the vertical and longitudinal signals of the accelerometer installed on the reducer housing, channel 3 corresponds to the vertical signal of the accelerometer installed on the switch circuit controller, and channel 8 corresponds to the longitudinal signal of the accelerometer installed on the push rod.

[0078] On-site engineers only install sensors and complete signal wiring at the aforementioned locations. Simultaneously, they configure the data acquisition equipment to only acquire data from the channels specified by the target configuration strategy. In the sampling rate settings, the final sampling rate of these channels is uniformly set to the target global sampling rate. For example, if the highest sampling rate is 512Hz and the strategy provides a global downsampling factor of 2, the sampling rate is uniformly set to 256Hz. During system operation, only this optimized channel data is acquired and sent to the downstream fault diagnosis system for real-time health status monitoring.

[0079] In this implementation, in the specific scenario of railway switch machines, the abstract deep reinforcement learning configuration strategy is transformed into an executable sensor physical layout and sampling configuration scheme. This enables the construction of an online monitoring system with fewer sensor channels and a lower sampling rate while ensuring fault diagnosis capabilities. This significantly reduces sensor hardware costs, wiring complexity, and data maintenance costs, and improves the system's deployability and operational economy in the rail transit field environment.

[0080] In the above implementation, preferably, during the policy generation stage, a complete deterministic decision-making process is performed on the converged deep reinforcement learning agent. Specifically, the pre-trained PSRONet model is loaded, and the initial state consisting of the historical full sensor dataset is input into the agent. At each decision step, the action with the highest probability is selected as the current action based on the action probability distribution output by the policy. That is, the channel index with the highest probability is selected in the channel selection round, and the global sampling rate with the highest probability is selected in the sampling rate selection round.

[0081] By continuously executing M+1 rounds of deterministic action selection, a fixed and specific sensor configuration strategy is obtained, such as "select channels 0, 1, 3, and 8; global downsampling coefficient is 2". This strategy serves as a clear engineering configuration instruction for field engineers to execute. Under the premise of the same historical dataset and a well-trained model, each execution of the above deterministic strategy generation process will output the same target sensor configuration strategy.

[0082] In this implementation, a deterministic policy generation mechanism is adopted, which constrains the output of the deep reinforcement learning agent to a unique target configuration policy. This achieves the reproducibility and interpretability of the policy results, facilitates the formation of stable and unified deployment specifications in engineering practice, and avoids the uncertainty introduced by random sampling policies during the deployment phase.

[0083] This invention also proposes a sensor configuration optimization system based on deep reinforcement learning, which applies the sensor configuration optimization method based on deep reinforcement learning disclosed in any of the above embodiments, and adopts a centralized computing architecture, including:

[0084] The data acquisition module is used to acquire the historical full sensor dataset collected for the target device. The historical full sensor dataset includes monitoring signals collected by candidate sensor channels at a preset highest sampling frequency.

[0085] The teacher model training module is used to train a fault diagnosis teacher model based on the historical full sensor dataset.

[0086] The learning environment building module is used to build a deep reinforcement learning environment based on the historical full sensor dataset and the trained fault diagnosis teacher model;

[0087] The configuration strategy optimization module is used to construct the sensor configuration optimization process in the deep reinforcement learning environment. It takes the state vector formed by the currently selected sensor channels and their monitoring data as the environment state, and takes the decision on the selection of candidate sensor channels and the global sampling rate as the action. This enables the deep reinforcement learning agent to perform sequential decision-making in the deep reinforcement learning environment. Based on the updated environment state and reward signal of the environment output, the deep reinforcement learning agent is iteratively updated until convergence, and the sensor configuration optimization strategy is obtained.

[0088] The policy generation module is used to call the converged deep reinforcement learning agent during the policy generation stage and output the target sensor configuration policy, which includes the selected sensor channel set and the target global sampling rate, under preset constraints.

[0089] The online deployment control module is used to control the sensor acquisition system according to the target sensor configuration strategy during the online deployment phase, and only collect monitoring data of the sensor channels specified in the target sensor configuration strategy at the target global sampling rate.

[0090] In this implementation, an integrated system architecture was constructed, encompassing historical data acquisition, teacher model training, reinforcement learning environment construction, policy optimization, policy generation, and on-site deployment. This architecture enables the system-level implementation of the deep reinforcement learning-based sensor configuration optimization method in an engineering environment, establishing a link between the algorithm layer and the engineering application layer, and improving the automation level and engineering feasibility of the sensor configuration optimization scheme.

[0091] The sensor configuration optimization system based on deep reinforcement learning disclosed in the above embodiments has modules whose functions correspond to the steps of the sensor configuration optimization method based on deep reinforcement learning disclosed in the above embodiments. In the implementation process, the above embodiments are referred to for operation, and will not be repeated here.

[0092] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sensor configuration optimization method based on deep reinforcement learning, characterized in that, include: Obtain a historical full sensor dataset collected for the target device, the historical full sensor dataset including monitoring signals collected by candidate sensor channels at a preset highest sampling frequency; A fault diagnosis teacher model is trained based on the historical full sensor dataset, and a deep reinforcement learning environment is constructed based on the historical full sensor dataset and the trained fault diagnosis teacher model. In the deep reinforcement learning environment, a sensor configuration optimization process is constructed. The state vector formed by the currently selected sensor channels and their monitoring data is taken as the environment state. The decisions on the selection of candidate sensor channels and the selection of the global sampling rate are taken as actions. The deep reinforcement learning agent performs sequential decisions in the deep reinforcement learning environment. The deep reinforcement learning environment evaluates the current sensor configuration based on the actions and outputs the updated environment state and the reward signal used to train the deep reinforcement learning agent. Based on the reward signal, state construction, action selection and reward feedback are iteratively performed in the deep reinforcement learning environment to optimize the sensor configuration until convergence, thereby obtaining a sensor configuration optimization strategy. During the policy generation phase, the converged deep reinforcement learning agent is invoked to output a target sensor configuration policy, including the selected sensor channel set and the target global sampling rate, under preset constraints. During the online deployment phase, the sensor acquisition system is controlled according to the target sensor configuration strategy to collect only the monitoring data of the sensor channels specified in the target sensor configuration strategy at the target global sampling rate. The process by which the deep reinforcement learning environment evaluates the current sensor configuration and outputs the reward signal specifically includes: Based on the historical full sensor dataset, the monitoring data corresponding to the currently selected sensor channel is extracted to construct a data subset. The data subset is then downsampled according to the global sampling rate obtained from the current decision. The downsampled data is then input into the fault diagnosis teacher model to obtain diagnostic performance indicators used to characterize the diagnostic effect. Based on the data volume relationship between the data subset and the historical full sensor dataset, a cost-saving score index is determined to characterize data costs. A weighted fusion of the diagnostic performance index and the cost-saving score index is then performed to generate the reward signal used to train the deep reinforcement learning agent. Specifically, after the deep reinforcement learning agent completes the selection of sensor channels and global sampling rate for the current sensor configuration, the terminal reward R is calculated. terminal ,in, R terminal =w acc ·R acc +w comp ·R comp ; Among them, R acc R is the diagnostic performance reward output by the fault diagnosis teacher model. comp The cost-saving reward is determined based on the data compression rate of the subset of data and the historical full sensor dataset, w acc and w comp These are preset weighting coefficients used to control the trade-off between diagnostic performance and data cost; The deep reinforcement learning agent employs a neural network structure based on the Actor-Critic architecture. This neural network structure includes components for processing the global statistical feature vector z and the mask vector m in the state vector, respectively. t A dual-stream input layer; A shared feature extraction backbone composed of a one-dimensional convolutional network and gated recurrent units is used to extract local temporal patterns and temporal dependencies in the decision-making process; And a multi-head output layer connected to the shared feature extraction backbone, the multi-head output layer including a first action output head for outputting the sensor channel selection probability distribution, a second action output head for outputting the global sampling rate selection probability distribution, and a value output head for outputting the current state value assessment.

2. The sensor configuration optimization method based on deep reinforcement learning according to claim 1, characterized in that, In the step of constructing the sensor configuration optimization process in the deep reinforcement learning environment, the state vector includes: a global statistical feature vector z calculated based on the monitoring data of the currently selected sensor channels, and a binary mask vector m used to indicate the selection history of candidate sensor channels. t ; Wherein, the global statistical feature vector z is constructed based on the monitoring signals of the currently selected sensor channels, and includes at least the mean and variance of the monitoring signals of each of the selected sensor channels; or it is obtained by feature extraction of the monitoring signals of the currently selected sensor channels by a pre-trained autoencoder. The state vector is composed of the global statistical feature vector z and the binary mask vector m. t The data is spliced ​​together to simultaneously characterize the current sensor data features and channel selection.

3. The sensor configuration optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process by which the deep reinforcement learning agent performs the sequential decision-making in the deep reinforcement learning environment includes: In the preset number of decision rounds M+1, the first M decisions sequentially select a sensor channel from the set of currently unselected candidate sensor channels, and the last decision selects one from the preset set of global sampling rates as the global sampling rate corresponding to the current decision.

4. The sensor configuration optimization method based on deep reinforcement learning according to claim 1, characterized in that, The fault diagnosis teacher model is a fault diagnosis model built on a convolutional neural network. It is trained and its parameters are fixed in the offline stage using the historical full sensor dataset. In the deep reinforcement learning environment, it acts as a performance evaluator and outputs a diagnostic performance score for the subset of data determined by the global sampling rate obtained from the currently selected sensor channels and the current decision, in order to support the generation of the reward signal.

5. The sensor configuration optimization method based on deep reinforcement learning according to claim 1, characterized in that, The target device is a railway switch machine, and the specific process of controlling the sensor acquisition system according to the target sensor configuration strategy includes: Based on the selected sensor channel set, the sensor installation positions on different structural components of the railway switch machine are determined, and the data acquisition equipment is configured to collect monitoring data of the selected sensor channels only at the target global sampling rate. The collected monitoring data is then input into the downstream fault diagnosis system for real-time health status assessment.

6. The sensor configuration optimization method based on deep reinforcement learning according to claim 1, characterized in that, During the policy generation phase, a complete deterministic decision-making process is performed on the converged deep reinforcement learning agent. At each decision step, the action with the highest probability is selected based on the action probability distribution of the policy output. Sensor channel selection and global sampling rate selection are completed in sequence, thereby outputting a unique target sensor configuration policy given the historical full sensor dataset.

7. A sensor configuration optimization system based on deep reinforcement learning, characterized in that, The sensor configuration optimization method based on deep reinforcement learning as described in any one of claims 1 to 6 includes: The data acquisition module is used to acquire the historical full sensor dataset collected for the target device. The historical full sensor dataset includes monitoring signals collected by candidate sensor channels at a preset highest sampling frequency. The teacher model training module is used to train a fault diagnosis teacher model based on the historical full sensor dataset. The learning environment construction module is used to construct a deep reinforcement learning environment based on the historical full sensor dataset and the trained fault diagnosis teacher model; The configuration strategy optimization module is used to construct a sensor configuration optimization process in the deep reinforcement learning environment. It takes the state vector formed by the currently selected sensor channel and its monitoring data as the environment state, and takes the decision on the selection of candidate sensor channels and the selection of global sampling rate as actions. It enables the deep reinforcement learning agent to perform sequential decision-making in the deep reinforcement learning environment, and iteratively updates the deep reinforcement learning agent based on the updated environment state and reward signal of the environment output until convergence, so as to obtain the sensor configuration optimization strategy. The strategy generation module is used to call the converged deep reinforcement learning agent during the strategy generation stage and output a target sensor configuration strategy including the selected sensor channel set and the target global sampling rate under preset constraints. An online deployment control module is used to control the sensor acquisition system according to the target sensor configuration strategy during the online deployment phase, and to collect only the monitoring data of the sensor channels specified in the target sensor configuration strategy at the target global sampling rate. The process by which the configuration strategy optimization module evaluates the current sensor configuration and outputs the reward signal specifically includes: Based on the historical full sensor dataset, the monitoring data corresponding to the currently selected sensor channel is extracted to construct a data subset. The data subset is then downsampled according to the global sampling rate obtained from the current decision. The downsampled data is then input into the fault diagnosis teacher model to obtain diagnostic performance indicators used to characterize the diagnostic effect. Based on the data volume relationship between the data subset and the historical full sensor dataset, a cost-saving score index is determined to characterize data costs. A weighted fusion of the diagnostic performance index and the cost-saving score index is then performed to generate the reward signal used to train the deep reinforcement learning agent. Specifically, after the deep reinforcement learning agent completes the selection of sensor channels and global sampling rate for the current sensor configuration, the terminal reward R is calculated. terminal ,in, R terminal =w acc ·R acc +w comp ·R comp ; Among them, R acc R is the diagnostic performance reward output by the fault diagnosis teacher model. comp The cost-saving reward is determined based on the data compression rate of the subset of data and the historical full sensor dataset, w acc and w comp These are preset weighting coefficients used to control the trade-off between diagnostic performance and data cost; The deep reinforcement learning agent employs a neural network structure based on the Actor-Critic architecture. This neural network structure includes components for processing the global statistical feature vector z and the mask vector m in the state vector, respectively. t A dual-stream input layer; A shared feature extraction backbone composed of a one-dimensional convolutional network and gated recurrent units is used to extract local temporal patterns and temporal dependencies in the decision-making process; And a multi-head output layer connected to the shared feature extraction backbone, the multi-head output layer including a first action output head for outputting the sensor channel selection probability distribution, a second action output head for outputting the global sampling rate selection probability distribution, and a value output head for outputting the current state value assessment.

Images