Reinforcement learning-based environment monitoring decision optimization method, system, device and medium

Abstract

This application relates to a method, system, device, and medium for environmental monitoring decision optimization based on reinforcement learning. The method includes: acquiring multimodal raw monitoring data in a hazardous waste management scenario and performing feature fusion processing to generate a continuous state vector; calculating the current risk potential scalar value based on the continuous state vector and a constructed differentiable risk potential function; correcting the potential scalar value based on expert experience and semantics; constructing a continuous reward function and calculating an immediate reward value based on the continuous reward function and the total risk potential value; inputting the continuous state vector, the immediate reward value, and the next-time continuous state vector after the environmental feedback following the intervention action into a proximal policy optimization network for iterative training until the network converges to obtain the optimal policy. This method can significantly improve the real-time performance, foresight, and intelligent decision-making level of environmental risk monitoring, effectively curbing illegal and criminal activities related to solid waste and preventing environmental risks.

Description

Technical Field

[0001] This invention belongs to the field of environmental monitoring technology, and in particular relates to environmental monitoring decision optimization methods, systems, equipment and media based on reinforcement learning. Background Technology

[0002] With increasingly stringent requirements for ecological and environmental protection, environmental risk monitoring technologies for hazardous waste are also constantly evolving. Currently, integrated systems for solid waste and chemical management are being gradually implemented across various regions. By deploying video surveillance, IoT sensors, and electronic ledgers, they have initially achieved full-process tracking of hazardous waste generation, transfer, storage, and disposal. Simultaneously, artificial intelligence technology has been introduced into the field of environmental monitoring to automatically identify behaviors in video images, assisting regulatory authorities in detecting violations.

[0003] In traditional technologies, environmental monitoring decisions primarily rely on pre-set rule bases and post-event event detection mechanisms. The monitoring system collects video data through cameras deployed at fixed locations and uses computer vision models to identify clearly defined violations, such as illegal dumping, vehicles deviating from planned routes, or storage outside designated areas. When such events are detected, the system triggers an alarm and records the violation information for subsequent handling by law enforcement. This model is essentially a passive response to events that have already occurred; the decision-making logic is relatively simple, relying mainly on human experience or fixed thresholds for judgment.

[0004] However, current environmental monitoring decision-making methods have significant limitations. Because they can only identify violations that have already occurred, the feedback signals received by the decision-making models during the learning process are extremely sparse and severely delayed—a negative feedback is only obtained after a violation has actually occurred, while the system receives no effective learning signals during the long process of risk incubation and evolution. This "post-punishment" feedback mechanism makes it difficult for existing technologies to detect subtle signs of accumulating risk before a violation has occurred, and even more difficult to proactively intervene in the early stages of risk accumulation, thus missing the best opportunity to nip environmental risks in the bud. Summary of the Invention

[0005] Therefore, it is necessary to provide an environmental monitoring decision optimization method that can perceive the dynamic evolution of risks and achieve proactive intervention to address the above-mentioned technical problems.

[0006] Firstly, this application provides a reinforcement learning-based method for optimizing environmental monitoring decisions, including:

[0007] The system acquires multimodal raw monitoring data in hazardous waste management scenarios and performs feature fusion processing on the data to generate a continuous state vector. The multimodal raw monitoring data includes video surveillance time-series behavioral data, GPS trajectory point data of transport vehicles, and electronic ledger record data. The continuous state vector is used to characterize the current system risk status.

[0008] A differentiable risk potential energy function is constructed, and based on the differentiable risk potential energy function and the continuous state vector, the scalar value of the risk potential energy at the current moment is calculated; whereby the scalar value of the risk potential energy is used to quantify the current degree of risk accumulation.

[0009] The similarity between the received expert experience text and the continuous state vector is calculated to obtain a similarity score. Based on the similarity score, the risk potential scalar value is dynamically corrected to generate the total risk potential value at the current moment.

[0010] A continuous reward function is constructed, and the instantaneous reward value at the current moment is calculated based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time; wherein, the continuous reward function includes an instantaneous risk term, a risk trend term, and an event penalty term;

[0011] The continuous state vector, the immediate reward value, and the next-time continuous state vector of the environmental feedback after the intervention action are input into the proximal policy optimization network for iterative training until the network converges and the optimal policy is obtained; the optimal policy is used to guide environmental monitoring and intervention decisions.

[0012] Furthermore, multimodal raw monitoring data under hazardous waste management scenarios are acquired, and feature fusion processing is performed on the multimodal raw monitoring data to generate continuous state vectors, including:

[0013] Acquire multimodal raw monitoring data in hazardous waste management scenarios, and perform behavioral feature embedding processing on the video surveillance time-series behavioral data in the multimodal raw monitoring data to generate behavioral feature vectors;

[0014] From the GPS trajectory point data of transport vehicles in the multimodal raw monitoring data, trajectory geometric features and dwell features are extracted, and a trajectory feature vector is generated based on the trajectory geometric features and dwell features. Among them, the trajectory geometric features include trajectory curvature and sway amplitude, and the dwell features include dwell duration and dwell position.

[0015] The electronic ledger records in the multimodal raw monitoring data are encoded to obtain ledger feature vectors; the ledger feature vectors include waste generation, transfer, storage and reporting timestamps;

[0016] Align the behavioral feature vector, trajectory feature vector, and ledger feature vector along the time dimension to obtain the aligned multimodal feature sequence;

[0017] The aligned multimodal feature sequence is input into the cross-modal Transformer encoder to generate a continuous state vector.

[0018] Furthermore, a differentiable risk potential energy function is constructed, and based on the differentiable risk potential energy function and the continuous state vector, the scalar value of the risk potential energy at the current moment is calculated, including:

[0019] A neural network determined by parameter φ is constructed, and the initial risk potential value is calculated based on the neural network and the historical continuous state vector; where parameter φ is all trainable parameters.

[0020] Obtain a first set of state vectors marked as normal and a second set of state vectors marked as high-risk from historical data; wherein, high-risk states include states within a preset time window before the occurrence of the violation event;

[0021] Based on the first set of state vectors and the second set of state vectors, a loss function is constructed, and the neural network is trained based on the loss function and the initial risk potential value to obtain a trained neural network.

[0022] The trained neural network is defined as a differentiable risk potential function, and the risk potential scalar value at the current moment is calculated based on the differentiable risk potential function and the continuous state vector.

[0023] Furthermore, a similarity score is calculated between the received expert experience text and the continuous state vector to obtain a similarity score. Based on the similarity score, the risk potential scalar value is dynamically corrected to generate the total risk potential value at the current moment, including:

[0024] The received expert experience text is encoded to generate a text semantic vector; the expert experience text includes the inspection log records and the fuzzy semantic descriptions in the expert evaluation report.

[0025] Mapping continuous state vectors to the same semantic space as text semantic vectors yields state semantic vectors.

[0026] The cosine similarity between the text semantic vector and the state semantic vector is calculated, and based on the cosine similarity and the preset semantic risk cardinality, the fuzzy semantic correction term is calculated.

[0027] The fuzzy semantic correction term and the risk potential scalar value are superimposed to generate the total risk potential value at the current moment, which incorporates expert experience.

[0028] Furthermore, a continuous reward function is constructed, and based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time, the instantaneous reward value at the current moment is calculated, including:

[0029] Based on the total risk potential value and the total risk potential value at the previous moment, the derivative of the total risk potential value with respect to time is calculated.

[0030] Detect whether a clear violation event has occurred at the current moment and generate a violation event indicator function; the violation event indicator function takes a value of 1 when a violation event occurs, and a value of 0 when no violation event occurs;

[0031] A continuous reward function is constructed, and the total risk potential value, derivative, and violation event indicator function are substituted into the continuous reward function for calculation to generate the instantaneous reward value at the current moment; wherein, the continuous reward function is expressed in the following form:

[0032]

[0033] in, Let t be the continuous reward function, and t be the current time. This represents the total risk potential energy value. For the derivative, Let β1 be the instantaneous risk weight coefficient, β2 be the risk trend weight coefficient, and β3 be the event penalty weight coefficient.

[0034] Secondly, this application also provides an environment monitoring decision optimization system based on reinforcement learning, including:

[0035] The data acquisition module is used to acquire multimodal raw monitoring data in hazardous waste management scenarios, and to perform feature fusion processing on the multimodal raw monitoring data to generate a continuous state vector. The multimodal raw monitoring data includes video surveillance time-series behavior data, transport vehicle GPS trajectory point data, and electronic ledger record data. The continuous state vector is used to characterize the current system risk status.

[0036] The data calculation module is used to construct a differentiable risk potential energy function and, based on the differentiable risk potential energy function and the continuous state vector, calculate the risk potential energy scalar value at the current moment; wherein, the risk potential energy scalar value is used to quantify the current risk accumulation level.

[0037] The data correction module is used to calculate the similarity between the received expert experience text and the continuous state vector, obtain a similarity score, and dynamically correct the risk potential scalar value based on the similarity score to generate the total risk potential value at the current moment.

[0038] The instant reward calculation module is used to construct a continuous reward function and calculate the instant reward value at the current moment based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time. The continuous reward function includes an instantaneous risk term, a risk trend term, and an event penalty term.

[0039] The policy generation module is used to input the continuous state vector, the immediate reward value, and the next-time continuous state vector of the environmental feedback after the intervention action into the proximal policy optimization network for iterative training until the network converges and the optimal policy is obtained; the optimal policy is used to guide environmental monitoring and intervention decisions.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement any of the reinforcement learning-based environmental monitoring decision optimization methods described in the embodiments of this application.

[0041] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement any of the reinforcement learning-based environmental monitoring decision optimization methods described in the embodiments of this application.

[0042] The aforementioned reinforcement learning-based environmental monitoring decision optimization method, system, equipment, and medium fuse multimodal raw monitoring data in hazardous waste management scenarios to generate a continuous state vector; construct a differentiable risk potential energy function, and calculate the risk potential energy scalar value at the current moment by combining it with the continuous state vector; correct the potential energy scalar value based on expert experience semantics; design a continuous reward function that includes instantaneous risk, risk trend, and event penalty to calculate the immediate reward value at the current moment; and train the agent using a proximal policy optimization algorithm until convergence to obtain the optimal policy. This achieves a leap from traditional ex-post punitive regulation to predictive intervention, significantly improving the real-time performance, foresight, and intelligence level of environmental risk monitoring, effectively curbing illegal and criminal activities related to solid waste, and preventing environmental risks. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating an environmental monitoring decision optimization method based on reinforcement learning in one embodiment.

[0045] Figure 2This is a flowchart illustrating the steps of acquiring multimodal raw monitoring data in a hazardous waste management scenario and performing feature fusion processing on the multimodal raw monitoring data to generate a continuous state vector, as shown in one embodiment.

[0046] Figure 3 This is a schematic diagram of the structure of an environmental monitoring decision optimization system based on reinforcement learning in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] In one embodiment, a reinforcement learning-based environmental monitoring decision optimization method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through the interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0049] Step S101: Obtain multimodal raw monitoring data in the hazardous waste management scenario, and perform feature fusion processing on the multimodal raw monitoring data to generate a continuous state vector; wherein, the multimodal raw monitoring data includes video surveillance time-series behavior data, transport vehicle GPS trajectory point data and electronic ledger record data; the continuous state vector is used to characterize the current system risk status.

[0050] Feature fusion refers to combining features from different sources, dimensions, and levels into a new set of more comprehensive and distinctive features in a certain way. The system in the current system risk situation refers to the entire hazardous waste supervision object system consisting of hazardous waste generation nodes, storage facilities, transport vehicles, and related operations. This system covers the physical entities and their dynamic behaviors throughout the entire process from the source of waste generation to final disposal, including but not limited to the production status of waste-generating enterprises, the storage and movement of hazardous waste, the driving trajectory and stationary behavior of transport vehicles, and the changes in the flow and quantity of hazardous waste reflected in the ledger records.

[0051] For example, multimodal raw monitoring data is collected from IoT front-ends deployed at hazardous waste generation nodes, hazardous waste storage facilities, and transport vehicles. This data includes: time-series behavioral data output by a video surveillance system in conjunction with an AI visual recognition model, which records the identified behavioral categories and their confidence levels at fixed time intervals (e.g., every 15 minutes); trajectory point data uploaded in real-time by vehicle-mounted GPS terminals, including latitude and longitude, instantaneous speed, orientation angle, and timestamp; and electronic ledger records automatically synchronized by the enterprise's electronic ledger system or manually entered, covering waste generation, transfer, storage, and declaration timestamps. Feature extraction and feature fusion processing are performed on the above heterogeneous data to generate a high-dimensional continuous state vector, which compresses the system's risk situation information across multiple dimensions at the current moment. Among them, hazardous waste generation nodes refer to the source links that generate hazardous waste, such as production workshops, laboratories, and process points; hazardous waste storage warehouses refer to temporary storage sites specifically for hazardous waste, with requirements for seepage prevention, leak prevention, rain protection, and explosion prevention, and cannot be piled up arbitrarily; transport vehicles refer to qualified, dedicated, sealed, and leak-proof hazardous waste transfer vehicles used to transport hazardous waste from storage warehouses to disposal units; the Internet of Things (IoT) front end refers to the layer where people interact with IoT devices, the core of which is the visible, tangible, and operable part; IoT devices refer to physical objects that are connected to the Internet and can exchange data and communicate with other devices and networks; video surveillance systems are... This refers to an integrated system composed of cameras, transmission equipment, storage devices, and display devices, used for real-time image acquisition, transmission, recording, storage, and playback of specific areas or targets; an artificial intelligence visual recognition model refers to a computational model that uses artificial intelligence technologies such as deep learning and computer vision algorithms to automatically identify, classify, detect, or understand visual information such as objects, scenes, text, and faces in images or videos; behavioral categories include the frequency of hazardous waste bin movement and the duration of abnormal lingering by personnel; a vehicle-mounted global positioning system terminal refers to a terminal installed on a vehicle to receive signals from the Global Positioning System (GPS). GPS (Global Navigation Satellite System) is a dedicated electronic device that uses satellite signals to achieve positioning, navigation, track recording, and data transmission. GPS is a satellite navigation system developed and operated by the United States. Its core relies on the collaboration of satellites, ground stations, and terminals to achieve global, all-weather, three-dimensional positioning, velocity measurement, and time synchronization. An enterprise electronic ledger system is a business data management platform that digitizes and centrally manages various paper ledgers. Its core uses database and network technologies to achieve standardized collection, real-time tracking, access-based sharing, and intelligent analysis of business data, replacing traditional manual bookkeeping and supporting efficient operation and compliant control. Feature extraction refers to the process of identifying, selecting, or constructing variables (i.e., features) from raw data that are most discriminative, representative, or informative for a task (such as classification, regression, or clustering).

[0052] Step S102: Construct a differentiable risk potential energy function, and calculate the risk potential energy scalar value at the current moment based on the differentiable risk potential energy function and the continuous state vector; wherein, the risk potential energy scalar value is used to quantify the current risk accumulation level.

[0053] For example, construct a parameter Determining neural networks As a differentiable risk potential function, its input is a continuous state vector, and its output is a scalar value. The design goal of this network is to map normal states to low potential energy and high-risk states (i.e., states within a time window before a violation occurs) to high potential energy, with the potential energy changing smoothly in the state space. To achieve this goal, a set of first state vectors labeled as normal states is obtained from a historical database. and the set of second state vectors marked as high-risk states The network is trained using the following loss function:

[0054]

[0055] in, For loss function, This represents the expectation of the potential energy values ​​of all states in the first set of state vectors. This represents the expectation of the potential energy values ​​of all states in the second state vector set, and the potential energy of the two types of states is separated by minimizing the difference between them. This represents the expectation of the square of the difference between the gradient norm of the potential function and 1, and is the gradient penalty term. This is a balance coefficient used to constrain the local Lipschitz constant of the potential energy function in the state space, ensuring its smoothness. The network parameters are updated by minimizing this loss function using the backpropagation algorithm. Continue training until convergence. After training is complete, the converged neural network will be... It is determined to be a differentiable risk potential function. For a continuous state vector input in real time... Substituting this value into the function will yield the scalar value of the risk potential energy at the current moment. This value is used to quantitatively characterize the current level of risk accumulation in the system. Mapping refers to mapping a normal state to a lower, more stable state through a certain rule / mechanism. A historical database is a database system used to store and manage historical data, typically containing data accumulated over a past period that is no longer frequently updated but needs long-term preservation and retrieval; it is commonly used in data analysis, audit tracing, and trend research. Backpropagation is a core algorithm in artificial neural networks used for training models. Based on the chain rule, it calculates the gradient of the loss function with respect to parameters (such as weights and biases) at each layer, propagating the prediction error of the output layer from the back to the input layer layer by layer, thus guiding parameter updates. The chain rule is a fundamental differentiation rule in calculus used to calculate the derivative of a composite function. Its core idea is that the rate of change of a composite function is equal to the rate of change of the outer function with respect to intermediate variables. The rate of change is multiplied by the rate of change of the intermediate variable with respect to the independent variable; convergence refers to the property that the value of a sequence, function, or iterative process gradually approaches a specific value (limit) under certain conditions. For example, if a function takes a value that tends to a certain constant near a certain point, it is said that the function converges at that point; in numerical computation or optimization algorithms, convergence also refers to the iterative result gradually stabilizing and approaching the target solution; state space is a core concept in control theory, artificial intelligence, mathematical modeling, and other fields. It refers to the set of all possible states of a system, and each state is uniquely determined by a specific set of variables (called state variables) that are sufficient to completely describe the behavior of the system; the local Lipschitz constant is a positive number that characterizes the maximum amplification factor of the output change due to the state change of the function on the right-hand side of the state equation near a certain point. It is a quantitative index of local Lipschitz continuity.

[0056] Step S103: Calculate the similarity between the received expert experience text and the continuous state vector to obtain a similarity score. Based on the similarity score, dynamically correct the risk potential scalar value to generate the total risk potential value at the current moment.

[0057] Similarity calculation refers to the mathematical or algorithmic process of measuring the degree of similarity between two objects (such as text, images, vectors, etc.). Its core is to define a similarity function or distance metric (such as cosine similarity) to map objects into comparable numerical forms. The larger the value, the more similar they are, or the smaller the value, depending on the definition. Dynamic correction refers to automatically and continuously adjusting the value according to the real-time situation to make it more accurate and closer to the real situation.

[0058] For example, the similarity score between the received expert experience text and the continuous state vector is calculated, and the risk potential scalar value is dynamically corrected based on the similarity score to generate the total risk potential value at the current moment.

[0059] Step S104: Construct a continuous reward function, and calculate the instantaneous reward value at the current moment based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time; wherein, the continuous reward function includes an instantaneous risk term, a risk trend term, and an event penalty term.

[0060] For example, based on the obtained total risk potential value and its temporal changes, a continuous reward function is designed to provide dense learning signals for the reinforcement learning agent. The derivative of the total risk potential value with respect to time is calculated. Simultaneously, it monitors in real time whether a clear violation event occurs, such as illegal dumping being captured by a visual recognition model, generating a violation event indicator function. This function takes a value of 1 when the event occurs and 0 otherwise. Substituting the current total risk potential value, derivative, and violation event indicator function into this function yields the immediate reward value. Here, temporal changes refer to the continuous changes that occur over time.

[0061] Step S105: Input the continuous state vector, the immediate reward value, and the next-time continuous state vector of the environmental feedback after the intervention action into the proximal policy optimization network for iterative training until the network converges and the optimal policy is obtained; wherein, the optimal policy is used to guide environmental monitoring and intervention decisions.

[0062] Iterative training refers to gradually optimizing model parameters through repeated training processes. Each iteration typically includes calculating predicted values, measuring the difference between the prediction and the true label, calculating gradients, and updating parameters. The entire training process consists of multiple such iterations, which may occur multiple times within one epoch (traversing all training data once), especially when using mini-batch training, or continue across multiple epochs until the model performance converges or reaches a predetermined stopping condition. Proximal policy optimization (PPO) is a widely used reinforcement learning algorithm used to train agents to make sequential decisions in complex environments. Its core idea is to introduce a constraint mechanism during policy updates to ensure that each update does not cause the new policy to deviate too much from the old policy, thereby improving the stability and sample efficiency of training.

[0063] For example, a Markov Decision Process (MDP) is constructed, in which the state space is defined as the continuous state vector generated in real time in step S101. The action space is defined as the set of executable intervention instructions, such as sending early warnings to enterprises, increasing the frequency of video spot checks on enterprises, and temporarily closing suspected risk areas. The reward function uses the immediate reward calculated in step S104, and the state transition probability is determined by environmental dynamics, i.e., the execution of the action. Then, how does it evolve to the next state? It depends on real physical processes and business feedback, not on human-made models. Initialize the policy network. (Parameters are) ) and value network (Parameters are) ), where the policy network is state-based The input is the state, and the output is in the state. The probability distribution of each action taken; the value network is based on states. Given the input, the output is an estimate of the expected cumulative reward in that state. The parameters of the two networks are... and Initialize randomly. At each time step of the training loop. The agent, based on the current policy network Regarding the current state Perform motion sampling to obtain the motion And execute it. After the environment responds, it feeds back the status for the next moment. and corresponding instant rewards This empirical tuple Store the samples in the experience replay buffer. Once the buffer has accumulated enough samples, randomly sample a batch of experience tuples from it. For each sample, first, based on the current value network... Estimate state value and and in combination with rewards Calculate the time-series difference error. Based on this, the generalized advantage estimation (GAE) method can be used to calculate the advantage function estimate. This value is used to measure the state. Take action below The formula for calculating the degree of superiority or inferiority relative to the average level is as follows: ,in, As a discount factor, For smoothing parameters, To address temporal difference errors, the introduction of the advantage function allows policy updates to focus on actions that yield higher rewards. For example, a pruning substitution objective function is constructed for a near-end policy optimization algorithm to update the policy network parameters. The objective function is:

[0064]

[0065] in, In action for both old and new strategies The probability ratio above, This is the trimming parameter, which can usually be set to 0.2. The function limits the probability ratio to Within the specified interval. This objective function limits the step size of each update to prevent drastic policy fluctuations and ensure training stability. Simultaneously, it updates the value network parameters by minimizing the prediction error of the value network. The loss function can typically be the mean squared error. ,in This is the calculated cumulative discount reward. The process of sampling, calculating advantage, and updating the network is repeated until the loss function of the policy network converges or the preset number of iterations is reached. The resulting policy network after training... This is the optimal strategy. Among them,

[0066] Among them, MDP is the standard mathematical framework for agents to make sequential decisions in stochastic environments; environmental dynamics is an interdisciplinary field that studies the processes, mechanisms, and driving factors of changes in the Earth / ecological / social environment over time and space; random initialization refers to assigning a set of non-zero random values ​​to the model's parameters (such as weights and biases) at program startup, rather than setting them all to 0 or fixed values; an agent is a computational entity that can autonomously perceive, make decisions, and execute in the environment to achieve its goals, with autonomous closed-loop operation as its core, distinguishing it from traditional AI tools that require manual step-by-step instructions; action sampling means that the policy network does not directly specify an action, but outputs a probability distribution of the action space and then randomly selects an action from this distribution; experience regression slowdown. The buffer zone is a key technique in reinforcement learning used to store experiential data such as states, actions, rewards, and next states collected by an agent during its interaction with the environment. It is typically stored as tuples like (s, a, r, s'). Random sampling refers to randomly selecting a subset of samples from a whole (the population) without selection, ensuring each individual has an equal probability of being selected, to represent the overall characteristics. Sufficient samples are a pre-set threshold, such as 100 samples, filling a fixed memory size, or accumulating to 1 second. Generalized advantage estimation is a core technique in reinforcement learning policy gradient methods that balances the estimation bias and variance of the advantage function. It uses an exponentially weighted average of multi-step temporal difference (TD) errors to make policy learning more stable and efficient. Exponentially weighted averaging is a smoothing averaging method that assigns greater weight to recent data and less weight to older data. The preset number of iterations, also called the fixed number of iterations, is simply the total number of times a certain piece of code needs to be executed repeatedly when writing loop / iterative logic.

[0067] In this embodiment, a continuous state vector is generated by multimodal data fusion, a differentiable risk potential energy function is constructed to quantify the degree of risk accumulation, and the potential energy value is corrected using expert experience semantics. A continuous reward function including instantaneous risk, risk trend and event penalty is designed, and a proximal policy optimization algorithm is used to train the agent. This achieves a leap from traditional ex-post punitive supervision to predictive intervention, which can significantly improve the real-time performance, foresight and decision-making intelligence of environmental risk monitoring, effectively curb illegal and criminal activities related to solid waste and prevent environmental risks.

[0068] In one embodiment, such as Figure 2 As shown, multimodal raw monitoring data in a hazardous waste management scenario is acquired, and feature fusion processing is performed on the multimodal raw monitoring data to generate a continuous state vector, including:

[0069] Step S201: Obtain multimodal raw monitoring data in the hazardous waste management scenario, and perform behavioral feature embedding processing on the video monitoring time-series behavioral data in the multimodal raw monitoring data to generate behavioral feature vectors.

[0070] Among them, behavioral feature embedding processing refers to transforming the discrete behaviors and temporal interactions of users / entities into low-dimensional, dense, semantically rich, and computable fixed-length vectors.

[0071] For example, continuous video stream data is collected from video surveillance systems deployed at key locations such as hazardous waste generation nodes, hazardous waste storage facilities, and transport vehicles. These video streams are analyzed in real time by a pre-trained artificial intelligence visual model, which can identify and output a preset temporal behavior sequence, specifically including the frequency of hazardous waste bin movement, the duration of abnormal loitering by personnel in sensitive areas, the operating trajectory of forklifts, and other behavior categories and their corresponding recognition confidence levels. Using fixed time windows (e.g., every 15 minutes), multiple behavior categories and their confidence levels identified within each time period are combined into an original temporal behavior description. An embedding layer can be used to perform feature embedding processing on this description: one-hot encoding of the behavior categories is performed, and this encoding is concatenated with the confidence level value, mapped to a low-dimensional dense vector space through a linear transformation layer, ultimately generating the behavior feature vector for that time period. Pre-training refers to the initial training process of a model on a large amount of unlabeled data in machine learning and natural language processing. This stage is not specific to any particular task, but rather allows the model to learn the basic structure, syntax, semantics, and statistical rules of language, thereby acquiring general language representation capabilities. After pre-training, the model is usually further optimized on labeled data for specific downstream tasks (such as text classification) through fine-tuning. A pre-defined temporal sequence of actions refers to a series of actions or steps arranged in chronological order. Real-time analysis refers to processing, calculating, and judging data immediately upon its generation, without waiting or backlog, and providing results or decisions immediately to support scenarios requiring immediate response. An embedding layer is a type of network layer in deep learning, mainly used to embed discrete symbols (such as classes) into a single data set. The process of mapping discrete, high-dimensional, and irregular original features (such as categories) into continuous, low-dimensional real-valued vectors, i.e., embedding vectors, is called feature embedding. Feature embedding refers to mapping discrete, high-dimensional, and irregular original features (such as categories) into low-dimensional, continuous vectors with semantic relationships, so that the model can understand and compute them. Hot coding is a method to convert discrete categorical data into binary numerical forms that computers can process efficiently. Its core logic is to create an independent virtual column for each category, using 1 to indicate that the sample belongs to the category and 0 to indicate that it does not belong, and each row has only one 1. The linear transformation layer is a basic neural network layer in deep learning. Its core operation is to perform a linear mapping on the input vector or tensor. The linear mapping preserves the addition and scalar multiplication, and linearly maps the elements in the vector space without destroying the linear structure.

[0072] Step S202: Extract trajectory geometric features and dwell features from the GPS trajectory point data of transport vehicles in the multimodal raw monitoring data, and generate trajectory feature vectors based on the trajectory geometric features and dwell features; wherein, the trajectory geometric features include trajectory curvature and sway amplitude, and the dwell features include dwell duration and dwell position.

[0073] For example, trajectory point data uploaded in real time by an onboard GPS terminal is received. Each trajectory point includes latitude and longitude coordinates, instantaneous speed, direction angle, and timestamp. Feature extraction is performed on the sequence of trajectory points within a time window. In terms of geometric features, the curvature of the trajectory is calculated, i.e., the rate of change of direction angle per unit arc length, to reflect the degree of path curvature; abnormal detours or frequent turns will lead to increased curvature. Simultaneously, the sway amplitude is calculated, i.e., the root mean square error of the trajectory point deviating from the straight line between the start and end points, to characterize the tortuosity of the driving path. Regarding dwell features, dwell events are identified through speed threshold detection: when the speed of multiple consecutive trajectory points is lower than a preset threshold and the duration exceeds a set time, it is determined as a dwell time, and the dwell time and dwell position are recorded, with the geometric center of the trajectory points during the dwell period being taken. These extracted geometric features (curvature, sway amplitude) and dwell features (dwell time, dwell position coordinates) are concatenated and can be nonlinearly transformed through a multilayer perceptron to generate a trajectory feature vector within the time window. This vector is used to comprehensively reflect the operating status and abnormal dwelling behavior of hazardous waste transport vehicles on the road network. Among them, root mean square error (RMSE) refers to the square root of the square of the error between the predicted and actual values, used to measure the accuracy of the prediction; speed threshold detection refers to setting a speed threshold, judging in real time whether the current speed exceeds / below / equals this value, and triggering the corresponding action; preset threshold refers to a pre-set threshold / standard, which will trigger a certain action, judgment, or result when it is reached or exceeded; set duration refers to a pre-defined duration; multilayer perceptron (MLP) is a feedforward artificial neural network consisting of an input layer, one or more hidden layers, and an output layer. The layers are connected by full connections, that is, each neuron is connected to all neurons in the next layer, and nonlinear activation functions achieve complex nonlinear mapping capabilities; nonlinear transformation refers to mapping data with non-linear rules, making the originally linearly inseparable and too simple data into a more complex and easier-to-distinguish form.

[0074] Step S203: Encode the electronic ledger record data in the multimodal raw monitoring data to obtain the ledger feature vector; wherein, the ledger feature vector includes waste generation, transfer, storage and reporting timestamp.

[0075] Encoding refers to converting information into another form that is easier to store, transmit, or process, according to a set of rules.

[0076] For example, by synchronizing records from the enterprise's electronic ledger system, core business data for hazardous waste management can be obtained, including waste generation, transfer, storage, and corresponding declaration timestamps. A time-aware attention mechanism can be used to encode this data: the ledger records are treated as an irregular time series, and for the current moment... All ledger records from a past period (e.g., the past 72 hours) are selected. The time interval between each record and the current moment is calculated, and this time interval is transformed into an attention weight using a learnable time decay function, giving more attention to recent records. The numerical features (waste production, etc.) of each record are multiplied by its time weight, and aggregated into a fixed-length vector through weighted summation. This vector is then input into a fully connected layer for dimensionality transformation to obtain the ledger feature vector for the current moment. This vector dynamically incorporates historical ledger information to reflect the fluctuation trends between waste production and reported quantities, as well as potential inconsistencies. Among them, the core of the time-aware attention mechanism is that the allocation of attention resources directly regulates the subjective time experience, which is the core cognitive mechanism to explain the difference in subjective feelings when the objective duration is the same; the learnable time decay function refers to the time decay function that can be automatically optimized through training data; the time decay function is a function whose value / weight decreases as time goes by; weighted summation refers to multiplying each value by its weight and then adding up all the results; the fully connected layer (FC layer) is a layer in a neural network that connects all input nodes and output nodes in pairs, used to integrate and map the previously extracted features into the final result; dimensionality transformation refers to adjusting the structure or representation of data to change its number of dimensions or the size of each dimension.

[0077] Step S204: Align the behavioral feature vector, trajectory feature vector, and ledger feature vector in the time dimension to obtain the aligned multimodal feature sequence.

[0078] Alignment in the time dimension refers to unifying multiple objects, data, signals, or events onto the same time scale, time point, or time rhythm along a timeline.

[0079] For example, since the behavioral feature vector, trajectory feature vector, and ledger feature vector are generated based on different original data sources and different time windows, their sampling frequencies and time points on the time axis may be inconsistent. A unified baseline timestamp sequence can be set, for example, generating a time point at fixed time intervals (e.g., every 15 minutes). For each baseline time point, the feature vector closest to that baseline time point is selected from the time series of behavioral, trajectory, and ledger features as the representation of that modality at that time. If a modality has no valid data near a specific baseline time, interpolation or forward padding can be used to generate the features for that time. Through this alignment operation, a multimodal feature sequence arranged in chronological order is obtained, in which each time step contains feature vectors for the three modalities of behavior, trajectory, and ledger, and these vectors have a temporal correspondence. Interpolation is a mathematical method that estimates the value of an unknown data point among known data points. Its core idea is to construct a continuous function, such as a polynomial or spline function, based on existing discrete data, so that the function passes through or approximates all known data points, and can be used to calculate the approximate value at any position within the interval. Forward imputation is a data processing technique that fills a missing value with the nearest non-missing value before it.

[0080] Step S205: Input the aligned multimodal feature sequence into the cross-modal Transformer encoder to generate a continuous state vector.

[0081] The cross-modal Transformer encoder is an encoder that extends the standard Transformer and is specifically designed to process multimodal data such as text, images, and audio, and to achieve semantic fusion between modalities. The core of this encoder is the multi-head self-attention mechanism, which can learn the complex dependencies between different time steps within a sequence and between different modalities. The multi-head self-attention mechanism is a core component of the Transformer model. It learns the dependencies between different positions in the input sequence by running multiple independent self-attention sub-layers (i.e., multiple heads) in parallel.

[0082] For example, during the input phase, positional encoding is added to the three modal feature vectors at each time step to preserve temporal information. These vectors are then concatenated into a unified sequence and input into the Transformer encoder. The self-attention mechanism in the encoder calculates the attention weights of each position and all other positions, thereby automatically capturing cross-modal and cross-temporal correlation patterns such as "a sudden drop in ledger declarations followed by an increase in nighttime forklift activity." After being processed by stacked Transformer layers, the output vector at each position has been fused with global contextual information. Typically, the output vector of the last time step can be taken, or the outputs of all time steps can be averaged and pooled to obtain the final continuous state vector. This vector is a high-dimensional compressed representation used to comprehensively characterize the current system's risk posture across multiple dimensions. Positional encoding refers to assigning temporal location labels to features, allowing the model to understand the temporal location information of the features. The Transformer layer is a basic unit in the encoder / decoder, typically containing two core sub-layers: a multi-head self-attention mechanism and a feedforward neural network. Each sub-layer is followed by residual connections and layer normalization. Stacking processing refers to the combination of multiple layers / units, hierarchical processing, and unified output. Average pooling involves averaging all values ​​within a region of the feature map, using this average value to represent that region, thereby reducing the feature map size, lowering dimensionality, and preserving overall information.

[0083] In this embodiment, heterogeneous video behavior data, GPS trajectory data, and electronic ledger data in hazardous waste management scenarios are processed through a series of refined processes, including feature embedding, geometric feature extraction, time-aware encoding, time alignment, and cross-modal Transformer fusion, to generate a continuous state vector that comprehensively reflects the system's risk situation. This significantly improves the sensitivity and accuracy of risk perception, providing high-quality state input for subsequent forward-looking decision-making based on reinforcement learning, thereby effectively supporting the upgrade of environmental regulatory models from passive response to proactive prevention.

[0084] In one embodiment, a differentiable risk potential energy function is constructed, and based on the differentiable risk potential energy function and the continuous state vector, the scalar value of the risk potential energy at the current moment is calculated, including:

[0085] Step S301: Construct a neural network determined by parameter φ, and calculate the initial risk potential value based on the neural network and the historical continuous state vector; where parameter φ is all trainable parameters.

[0086] Among them, parameters This represents all trainable parameters in the network, including the weight matrix and bias vector of each layer.

[0087] For example, a neural network architecture for fitting a risk potential function is defined. This network takes a continuous state vector as input, passes through several fully connected layers and nonlinear activation functions layer by layer, and outputs a scalar value. The purpose of this network design is to learn a nonlinear mapping from a high-dimensional state space to a one-dimensional risk potential, so that different states can form an ordered distribution in the potential space according to their inherent risk level. In the initial state, the parameters... Random initialization can be used for assignment. Therefore, for any input continuous state vector, such as samples extracted from a historical database (i.e., historical continuous state vectors), the network calculates the initial risk potential value through forward propagation. It does not possess actual physical meaning; it is only used as the starting point and foundation for subsequent training and optimization processes. Among them, the nonlinear activation function is a mathematical function used in artificial neural networks to introduce nonlinear characteristics. It acts on the weighted input (i.e., linear combination) of neurons, mapping the input to the output, enabling the neural network to learn and represent complex nonlinear relationships and patterns. If only linear transformations are used, regardless of the number of layers in the network, the overall result is still equivalent to a single linear transformation, which cannot solve nonlinearly separable problems (such as the XOR problem). Layer-by-layer transformation refers to transforming data or states according to a hierarchical structure, through specific functions or rules at each layer, with the output serving as the input for the next layer, ultimately achieving a progressive mapping from the original representation to the target representation. Nonlinear mapping refers to mappings that do not satisfy linear relationships; the input and output are not proportional or linear. Forward propagation is the process by which the neural network calculates the input data layer by layer, ultimately outputting the predicted result.

[0088] Step S302: Obtain a first set of state vectors marked as normal and a second set of state vectors marked as high-risk from historical data; wherein, high-risk states include states within a preset time window before the occurrence of the violation event.

[0089] The preset time window refers to a fixed time period that is set in advance.

[0090] For example, two types of state vectors with different risk labels are retrieved and selected from the historical database for subsequent supervised training of the network. The first type is the set of normal state vectors, which correspond to stable operating periods during hazardous waste management where no anomalies occurred and no violations were subsequently discovered. These states can typically be labeled by domain experts through manual annotation or rule-based automated programs. The second type is the set of high-risk state vectors, defined in a way that reflects a deep understanding of the dynamic evolution of risk. This means that not only are the state vectors corresponding to the moment a violation is confirmed included in this set, but more importantly, all state vectors within a preset time window before the violation occurs (e.g., within 24 hours before the event) are also labeled as high-risk states. The rule-based automated program refers to a program that automatically executes tasks according to preset rules (such as conditional judgments). Its behavior is entirely driven by manually defined rules and does not rely on machine learning or data training.

[0091] Step S303: Based on the first set of state vectors and the second set of state vectors, construct a loss function, and train the neural network based on the loss function and the initial risk potential value to obtain a trained neural network.

[0092] For example, based on the initial risk potential value, the first set of state vectors, and the second set of state vectors, a specially designed loss function is constructed to optimize the network parameters. The loss function takes the following form:

[0093]

[0094] in, and Let represent the expected potential energy values ​​of all samples in the normal state set and the high-risk state set, respectively. By minimizing the difference between these two terms, i.e., forcing the first term to be as small as possible and the second term to be as large as possible, the network maps high-risk states to high potential energy values ​​and normal states to low potential energy values, thereby achieving effective separation of the two types of states in the potential energy space. This is a gradient penalty term. This is a balance coefficient. It calculates the gradient norm of the potential function with respect to the input state vector for samples sampled from the state space (typically a mixture of samples from two sets) and constrains it to be close to 1. This constraint ensures that the potential function is smooth and Lipschitz continuous throughout the state space, preventing drastic potential changes that could lead to training instability or decreased generalization ability. For example, mini-batch stochastic gradient descent can be used to calculate the loss value for each batch of samples and iteratively update the parameters using backpropagation. Repeat this process until the loss function converges, and you will finally obtain the trained neural network. Mini-batch stochastic gradient descent is a commonly used optimization algorithm for training machine learning models. In each iteration, a small subset, or mini-batch, is randomly selected from the training set. Typical sizes are 16, 32, 64, or 128 samples. The gradient of the loss function with respect to the model parameters is calculated based on this batch of data, and the parameters are updated accordingly.

[0095] Step S304: The trained neural network is determined as a differentiable risk potential function, and the risk potential scalar value at the current moment is calculated based on the differentiable risk potential function and the continuous state vector.

[0096] For example, after training is complete, the converged neural network is... It has been formally identified as a differentiable risk potential function. This function possesses two key properties: firstly, differentiability, meaning it can compute the gradient of its input state vector. Secondly, it has the ability to quantify risk, meaning that for any input state vector, its output value can quantitatively reflect the degree of risk accumulation under that state; the higher the value, the greater the current environmental risk. In actual monitoring, the generated real-time continuous state vector... Using the trained differentiable risk potential function as input, a forward propagation calculation can be performed to obtain the scalar value of the risk potential at the current moment. .

[0097] In this embodiment, by constructing and training a differentiable risk potential function, after training, this function can quickly infer from real-time continuous state vectors and output a physically meaningful, continuously changing risk potential scalar value. This makes the abstract risk accumulation process computable and comparable, providing dense and informative core inputs for the design of subsequent reinforcement learning reward functions, thereby effectively supporting the keen identification of early signs of risk and proactive intervention decisions.

[0098] In one embodiment, a similarity score is calculated between the received expert experience text and the continuous state vector to obtain a similarity score. Based on the similarity score, the risk potential scalar value is dynamically corrected to generate the total risk potential value at the current moment, including:

[0099] Step S401: Encode the received expert experience text to generate a text semantic vector; wherein, the expert experience text includes the inspection log records and the ambiguous semantic descriptions in the expert evaluation report.

[0100] For example, text data input by environmental regulators or experts is received. This text is typically presented in natural language, including rich qualitative judgments and vague descriptions, such as "suspected signs of mixed storage exist in a certain storage area" recorded in an inspection log, or "the enterprise's waste generation rhythm and transfer frequency are mismatched, posing a long-term chronic risk of violation" pointed out in an expert assessment report. A pre-trained language model, such as a bidirectional encoder representation based on the Transformer architecture, can be invoked to encode each input text. This language model, through a multi-layer self-attention mechanism, can capture the contextual dependencies between words in the text and compress the semantic information of the entire sentence or paragraph into a fixed-length vector. After forward propagation computation, the output is a text semantic vector corresponding to the expert's experience text. This vector is used to uniquely represent the meaning of the text in a high-dimensional semantic space. Pre-trained language models refer to artificial intelligence models pre-trained on large-scale text corpora. Their core objective is to learn the statistical regularities, grammatical structures, semantic relationships, and world knowledge of language. These models are typically trained using self-supervised learning, requiring no manual labeling. After training, the model possesses general language understanding and generation capabilities and can serve as a base model, fine-tuned to adapt to various downstream tasks such as text classification, machine translation, question answering, and summarization. Self-supervised learning is a machine learning paradigm whose core idea is to automatically construct supervisory signals, i.e., labels, using the structural information inherent in the data itself, thus enabling model training without manual labeling. The bidirectional encoder representation based on the Transformer architecture is BERT (Bidirectional Encoder Representations from...). The Chinese translation of the model name "Transformers" refers to a pre-trained language representation method that utilizes the bidirectional self-attention mechanism of Transformers to perform deep contextual modeling of text. Multi-layer self-attention mechanism refers to the structure of stacked self-attention layers in deep learning models (especially the Transformer architecture). Each layer of self-attention mechanism allows the model to dynamically calculate the relevance weights between positions in a sequence when processing sequential data (such as text), thereby capturing long-distance dependencies. Multi-layer design enables the model to extract multi-level semantic features from local to global and from surface to abstract layers. Compression refers to the process of reducing data volume through specific algorithms or techniques. "Environmental regulators" refers to professionals who legally engage in environmental protection supervision and management. Their responsibilities include supervising enterprises, institutions, and other producers and operators to comply with environmental protection laws and regulations, monitoring environmental quality, investigating and punishing environmental violations, conducting environmental law enforcement inspections, and promoting pollution control and ecological protection.

[0101] Step S402: Map the continuous state vector to the same semantic space as the text semantic vector to obtain the state semantic vector.

[0102] For example, a state semantic encoding network is pre-built and trained. This network is typically composed of multiple fully connected neural networks, and its input is a continuous state vector. The output is a state semantic vector with the same dimensions as the text semantic vector. The training process of this network is based on the principle of contrastive learning: during the training phase, using a set of state vectors and text description pairs with known matching relationships, the InfoNCE loss can be optimized to improve the accuracy of the matching. and In the semantic space, distances are shortened, while the distances between mismatched sample pairs are widened. After training, this state semantic encoding network has the ability to map any input state vector to the semantic space, and the mapped position can reflect the similarity relationship between the state and various semantic descriptions. For the continuous state vector at the current time step... By performing a forward propagation through this network, its corresponding state semantic vector can be obtained. Among them, State Semantic Encoding Networks are a type of neural network that fuses and encodes the state and semantic information of a dynamic system. The core is to map temporal / state data to an interpretable and computable semantic vector space, taking into account both state evolution and semantic understanding. Fully Connected Neural Networks are an artificial neural network structure where each neuron in each layer is connected to all neurons in the previous layer (i.e., fully connected). They typically do not contain special structures such as convolution, recursion, or attention. Their basic unit is a perceptron or a neuron with a non-linear activation function, achieving layer-by-layer abstraction and non-linear transformation of features through multi-layer stacking. InfoNCE (Information Noise Contrastive Estimation) loss is the core loss in contrastive learning / self-supervised learning. Essentially, it is cross-entropy with a temperature coefficient, aiming to bring positive samples closer together, push away negative samples, and maximize mutual information between samples.

[0103] Step S403: Calculate the cosine similarity between the text semantic vector and the state semantic vector, and calculate the fuzzy semantic correction term based on the cosine similarity and the preset semantic risk basis.

[0104] For example, calculating the state semantic vector and text semantic vectors The cosine similarity between (newly input expert texts) is calculated using the following formula:

[0105]

[0106] Similarity score The value ranges from -1 to 1; a higher value indicates a semantic similarity between the current system state and the specific risk scenario described by the expert. The similarity score... and the preset semantic risk base Multiplying them together yields the fuzzy semantic correction term. The fuzzy semantic correction term dynamically reflects the extent to which the current state aligns with the expert's risk prediction. When the state and the high-risk description are highly similar, the correction term takes a larger value; when the state and the text description are unrelated or opposite, the correction term approaches zero or takes a negative value. The preset semantic risk base is used to represent the baseline risk weight corresponding to each expert experience text, and its value can be preset according to the severity of the risk described in that experience.

[0107] Step S404: The fuzzy semantic correction term and the risk potential scalar value are superimposed to generate the total risk potential value at the current moment that incorporates expert experience.

[0108] For example, the fuzzy semantic correction term and the risk potential scalar value are... Linear superposition is performed to incorporate the fuzzy empirical knowledge of experts into the quantitative risk measurement, generating the final total risk potential value:

[0109]

[0110] in, This is the total risk potential value, which retains the objectivity of data-driven approaches while incorporating the wisdom of expert experience to more comprehensively and accurately reflect the current level of real environmental risks.

[0111] In this embodiment, unstructured textual experience is encoded into semantic vectors by constructing a pre-trained language model. Structured state vectors are mapped to the same semantic space through contrastive learning. Then, cosine similarity is used to dynamically quantify the semantic correlation between the current state and the expert description. Based on this, a fuzzy semantic correction term is generated and superimposed with a data-driven risk potential value. This effectively solves the problem of traditional models struggling to understand and utilize textual experience, achieving an organic combination of objective data patterns and subjective expert wisdom, significantly improving the robustness and foresight of the risk quantification model.

[0112] In one embodiment, a continuous reward function is constructed, and based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time, the instantaneous reward value at the current moment is calculated, including:

[0113] Step S501: Based on the total risk potential energy value and the total risk potential energy value at the previous moment, calculate the derivative of the total risk potential energy value with respect to time.

[0114] For example, obtain the current time Total risk potential value and the previous moment Total risk potential value The potential energy value from the previous time step can be read from the historical cache. Since environmental monitoring data is typically sampled at discrete times, a first-order backward difference approximation can be used to calculate the derivative of the total risk potential energy value with respect to time, i.e.

[0115]

[0116] This derivative is used to reflect the rate of change of the degree of risk accumulation at the current moment: if the derivative is positive, it means that the risk is accelerating; if the derivative is negative, it means that the risk is easing.

[0117] Step S502: Detect whether a clear violation event has occurred at the current moment, and generate a violation event indicator function; wherein, the violation event indicator function takes a value of 1 when a violation event occurs, and takes a value of 0 when no violation event occurs.

[0118] Among them, the violation event indicator function is a Boolean discrete signal used to represent the most direct ex-post punishment basis in the traditional regulatory system. This ensures that even when continuous risk guidance fails or a sudden serious violation occurs, the reward function can still provide strong negative feedback, thus guaranteeing the safety bottom line of the decision-making model.

[0119] For example, a pre-defined violation detection mechanism is run, typically based on an AI visual recognition model, sensor alarm system, or business rule engine deployed at the edge. For instance, video analytics models can identify clearly defined violations such as illegal dumping of hazardous waste, vehicles severely deviating from their routes, or unauthorized personnel entering storage areas; or electronic ledger comparisons can reveal significant discrepancies between waste generation and transfer volumes. Once any such violation is confirmed, the current time... Corresponding violation event indicator function The value is set to 1; if no violation is detected, it is set to 0. The preset violation detection mechanism is an automated risk control logic that pre-sets rules, keywords, features, and model thresholds to automatically identify, intercept, and warn of violations. The sensor alarm system refers to an automated security protection system that uses sensors to monitor physical or environmental parameters in real time, automatically triggering alarms in the form of sound, light, signals, and network notifications when the detected value exceeds a preset safety threshold or an abnormal state occurs. The business rule engine is a software system or component used to define, deploy, execute, and manage enterprise business rules. The video analysis model refers to an artificial intelligence model specifically designed to process, understand, and parse video data. It typically combines computer vision and sequence modeling techniques to extract spatiotemporal features from consecutive frames of video, completing tasks such as action recognition, target tracking, event detection, scene understanding, and abnormal behavior analysis.

[0120] Step S503: Construct a continuous reward function, and substitute the total risk potential value, derivative, and violation event indicator function into the continuous reward function for calculation to generate the instantaneous reward value at the current moment; wherein, the continuous reward function is expressed in the following form:

[0121]

[0122] in, Let t be the continuous reward function, and t be the current time. This represents the total risk potential energy value. For the derivative, Let β1 be the instantaneous risk weight coefficient, β2 be the risk trend weight coefficient, and β3 be the event penalty weight coefficient.

[0123] Among them, the weighting coefficient , , It is a pre-set value. Typically assigned a much larger and The value of the reward function is used to highlight the severe punishment for clear violations; the construction principle of the continuous reward function is based on the following: the first term This is used to encourage the agent to consistently maintain the system state in a low-potential region; the second term This is used to guide agents to intervene early in the process of risk escalating, even if the current potential energy has not yet reached a dangerous threshold; the third item This is used to ensure that the agent receives a very strong negative signal for any serious violation that has occurred.

[0124] For example, the calculated derivative Compare with 0 and take the maximum value. This ensures that only an upward trend in risk incurs a penalty, while a downward trend in risk (negative derivative) does not contribute a reward. The total risk potential value at the current moment is then calculated. Processed risk trend items and violation event indicator functions Multiply each value by its corresponding weighting coefficient, then sum them after removing the negative sign to obtain the final instant reward value. .

[0125] In this embodiment, the instantaneous reward value at the current moment is calculated by fusing the abstract risk potential value and its time derivative with the explicit violation event signal into a structured continuous reward function. This enables the reinforcement learning agent to obtain rich learning signals in its interaction with the environment: it learns to proactively reduce current risks, and also learns to provide timely warnings and interventions in the early stages of accelerated risk accumulation, while strictly avoiding crossing the absolute violation red line.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a reinforcement learning-based environmental monitoring decision optimization system for implementing the reinforcement learning-based environmental monitoring decision optimization method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more reinforcement learning-based environmental monitoring decision optimization system embodiments provided below can be found in the limitations of the reinforcement learning-based environmental monitoring decision optimization method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 3 As shown, an environment monitoring decision optimization system 300 based on reinforcement learning is provided, including:

[0129] The data acquisition module 301 is used to acquire multimodal raw monitoring data in hazardous waste management scenarios and perform feature fusion processing on the multimodal raw monitoring data to generate a continuous state vector. The multimodal raw monitoring data includes video surveillance time-series behavior data, transport vehicle GPS trajectory point data, and electronic ledger record data. The continuous state vector is used to characterize the current system risk status.

[0130] The data calculation module 302 is used to construct a differentiable risk potential energy function and, based on the differentiable risk potential energy function and the continuous state vector, calculate the risk potential energy scalar value at the current moment; wherein, the risk potential energy scalar value is used to quantify the current risk accumulation level.

[0131] The data correction module 303 is used to calculate the similarity between the received expert experience text and the continuous state vector, obtain a similarity score, and dynamically correct the risk potential scalar value based on the similarity score to generate the total risk potential value at the current moment.

[0132] The instant reward calculation module 304 is used to construct a continuous reward function and calculate the instant reward value at the current moment based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time; wherein, the continuous reward function includes an instantaneous risk term, a risk trend term, and an event penalty term;

[0133] The policy generation module 305 is used to input the continuous state vector, the immediate reward value, and the next-time continuous state vector of the environmental feedback after the intervention action into the proximal policy optimization network for iterative training until the network converges and the optimal policy is obtained; the optimal policy is used to guide environmental monitoring and intervention decisions.

[0134] In one embodiment, the data acquisition module 301 is further configured to:

[0135] Acquire multimodal raw monitoring data in hazardous waste management scenarios, and perform behavioral feature embedding processing on the video surveillance time-series behavioral data in the multimodal raw monitoring data to generate behavioral feature vectors;

[0136] From the GPS trajectory point data of transport vehicles in the multimodal raw monitoring data, trajectory geometric features and dwell features are extracted, and a trajectory feature vector is generated based on the trajectory geometric features and dwell features. Among them, the trajectory geometric features include trajectory curvature and sway amplitude, and the dwell features include dwell duration and dwell position.

[0137] The electronic ledger records in the multimodal raw monitoring data are encoded to obtain ledger feature vectors; the ledger feature vectors include waste generation, transfer, storage and reporting timestamps;

[0138] Align the behavioral feature vector, trajectory feature vector, and ledger feature vector along the time dimension to obtain the aligned multimodal feature sequence;

[0139] The aligned multimodal feature sequence is input into the cross-modal Transformer encoder to generate a continuous state vector.

[0140] In one embodiment, the data calculation module 302 is further configured to:

[0141] A neural network determined by parameter φ is constructed, and the initial risk potential value is calculated based on the neural network and the historical continuous state vector; where parameter φ is all trainable parameters.

[0142] Obtain a first set of state vectors marked as normal and a second set of state vectors marked as high-risk from historical data; wherein, high-risk states include states within a preset time window before the occurrence of the violation event;

[0143] Based on the first set of state vectors and the second set of state vectors, a loss function is constructed, and the neural network is trained based on the loss function and the initial risk potential value to obtain a trained neural network.

[0144] The trained neural network is defined as a differentiable risk potential function, and the risk potential scalar value at the current moment is calculated based on the differentiable risk potential function and the continuous state vector.

[0145] In one embodiment, the data correction module 303 is further configured to:

[0146] The received expert experience text is encoded to generate a text semantic vector; the expert experience text includes the inspection log records and the fuzzy semantic descriptions in the expert evaluation report.

[0147] Mapping continuous state vectors to the same semantic space as text semantic vectors yields state semantic vectors.

[0148] The cosine similarity between the text semantic vector and the state semantic vector is calculated, and based on the cosine similarity and the preset semantic risk cardinality, the fuzzy semantic correction term is calculated.

[0149] The fuzzy semantic correction term and the risk potential scalar value are superimposed to generate the total risk potential value at the current moment, which incorporates expert experience.

[0150] In one embodiment, the instant reward calculation module 304 is further configured to:

[0151] Based on the total risk potential value and the total risk potential value at the previous moment, the derivative of the total risk potential value with respect to time is calculated.

[0152] Detect whether a clear violation event has occurred at the current moment and generate a violation event indicator function; the violation event indicator function takes a value of 1 when a violation event occurs, and a value of 0 when no violation event occurs;

[0153] A continuous reward function is constructed, and the total risk potential value, derivative, and violation event indicator function are substituted into the continuous reward function for calculation to generate the instantaneous reward value at the current moment; wherein, the continuous reward function is expressed in the following form:

[0154]

[0155] in, Let t be the continuous reward function, and t be the current time. This represents the total risk potential energy value. For the derivative, Let β1 be the instantaneous risk weight coefficient, β2 be the risk trend weight coefficient, and β3 be the event penalty weight coefficient.

[0156] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the reinforcement learning-based environmental monitoring decision optimization method as described above.

[0157] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0158] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0159] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A reinforcement learning-based method for optimizing environmental monitoring decisions, characterized in that, The method includes: Multimodal raw monitoring data in a hazardous waste management scenario is acquired, and feature fusion processing is performed on the multimodal raw monitoring data to generate a continuous state vector; wherein, the multimodal raw monitoring data includes video surveillance time-series behavior data, transport vehicle GPS trajectory point data, and electronic ledger record data; the continuous state vector is used to characterize the current system risk status; A differentiable risk potential energy function is constructed, and based on the differentiable risk potential energy function and the continuous state vector, the risk potential energy scalar value at the current moment is calculated; wherein, the risk potential energy scalar value is used to quantify the current risk accumulation level; The similarity score is calculated between the received expert experience text and the continuous state vector to obtain a similarity score. Based on the similarity score, the risk potential scalar value is dynamically corrected to generate the total risk potential value at the current moment. A continuous reward function is constructed, and the instantaneous reward value at the current moment is calculated based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time; wherein, the continuous reward function includes an instantaneous risk term, a risk trend term, and an event penalty term; The continuous state vector, the immediate reward value, and the next-time continuous state vector of the environmental feedback after the intervention action are input into the proximal policy optimization network for iterative training until the network converges to obtain the optimal policy; wherein, the optimal policy is used to guide environmental monitoring and intervention decisions.

2. The method according to claim 1, characterized in that, The process of acquiring multimodal raw monitoring data in a hazardous waste management scenario and performing feature fusion processing on the multimodal raw monitoring data to generate a continuous state vector includes: Acquire multimodal raw monitoring data in hazardous waste management scenarios, and perform behavioral feature embedding processing on the video surveillance time-series behavioral data in the multimodal raw monitoring data to generate behavioral feature vectors; From the GPS trajectory point data of the transport vehicle in the multimodal raw monitoring data, trajectory geometric features and dwell features are extracted, and a trajectory feature vector is generated based on the trajectory geometric features and the dwell features; wherein, the trajectory geometric features include trajectory curvature and oscillation amplitude, and the dwell features include dwell duration and dwell position; The electronic ledger record data in the multimodal raw monitoring data is encoded to obtain a ledger feature vector; wherein, the ledger feature vector includes waste generation, transfer, storage and reporting timestamp; Align the behavior feature vector, the trajectory feature vector, and the ledger feature vector in the time dimension to obtain an aligned multimodal feature sequence; The aligned multimodal feature sequence is input into the cross-modal Transformer encoder to generate a continuous state vector.

3. The method according to claim 1, characterized in that, The process of constructing a differentiable risk potential function and calculating the scalar value of the risk potential energy at the current moment based on the differentiable risk potential function and the continuous state vector includes: A neural network determined by parameter φ is constructed, and the initial risk potential value is calculated based on the neural network and the historical continuous state vector; wherein, parameter φ is all trainable parameters; Obtain a first set of state vectors marked as normal and a second set of state vectors marked as high-risk from historical data; wherein, the high-risk state includes the state within a preset time window before the violation event occurs; Based on the first set of state vectors and the second set of state vectors, a loss function is constructed, and based on the loss function and the initial risk potential value, the neural network is trained to obtain a trained neural network. The trained neural network is determined as a differentiable risk potential function, and the risk potential scalar value at the current moment is calculated based on the differentiable risk potential function and the continuous state vector.

4. The method according to claim 1, characterized in that, The step of calculating the similarity between the received expert experience text and the continuous state vector to obtain a similarity score, and dynamically correcting the risk potential scalar value based on the similarity score to generate the total risk potential value at the current moment, includes: The received expert experience text is encoded to generate a text semantic vector; wherein, the expert experience text includes the fuzzy semantic descriptions in the inspection log records and expert evaluation reports; The continuous state vector is mapped to the same semantic space as the text semantic vector to obtain the state semantic vector; The cosine similarity between the text semantic vector and the state semantic vector is calculated, and based on the cosine similarity and a preset semantic risk basis, a fuzzy semantic correction term is calculated. The fuzzy semantic correction term and the risk potential scalar value are superimposed to generate the total risk potential value at the current moment, which incorporates expert experience.

5. The method according to claim 1, characterized in that, The construction of a continuous reward function, and the calculation of the instantaneous reward value at the current moment based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time, includes: Based on the total risk potential energy value and the total risk potential energy value at the previous moment, the derivative of the total risk potential energy value with respect to time is calculated. Detect whether a clear violation event has occurred at the current moment, and generate a violation event indicator function; wherein, the violation event indicator function takes a value of 1 when a violation event occurs, and takes a value of 0 when no violation event occurs; A continuous reward function is constructed, and the total risk potential value, the derivative, and the violation event indicator function are substituted into the continuous reward function for calculation to generate the instantaneous reward value at the current moment; wherein, the continuous reward function is expressed in the following form: in, Let be the continuous reward function, and t be the current time. This represents the total risk potential value. The derivative is... Let β1 be the instantaneous risk weight coefficient, β2 be the risk trend weight coefficient, and β3 be the event penalty weight coefficient.

6. An environment monitoring and decision optimization system based on reinforcement learning, characterized in that, The system includes: The data acquisition module is used to acquire multimodal raw monitoring data in hazardous waste management scenarios, and to perform feature fusion processing on the multimodal raw monitoring data to generate a continuous state vector; wherein, the multimodal raw monitoring data includes video surveillance time-series behavior data, transport vehicle GPS trajectory point data, and electronic ledger record data; the continuous state vector is used to characterize the current system risk status; The data calculation module is used to construct a differentiable risk potential energy function, and calculate the risk potential energy scalar value at the current moment based on the differentiable risk potential energy function and the continuous state vector; wherein, the risk potential energy scalar value is used to quantify the current risk accumulation level; The data correction module is used to calculate the similarity between the received expert experience text and the continuous state vector to obtain a similarity score, and to dynamically correct the risk potential scalar value based on the similarity score to generate the total risk potential value at the current moment. An instant reward calculation module is used to construct a continuous reward function and calculate the instant reward value at the current moment based on the continuous reward function, the total risk potential value, and the derivative of the total risk potential value with respect to time; wherein, the continuous reward function includes an instantaneous risk term, a risk trend term, and an event penalty term; The strategy generation module is used to input the continuous state vector, the immediate reward value, and the next-time continuous state vector of the environmental feedback after the intervention action into the proximal policy optimization network for iterative training until the network converges and the optimal policy is obtained; wherein, the optimal policy is used to guide environmental monitoring and intervention decisions.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images