Video monitoring analysis method and system for operation behavior of thermal power plant

Abstract

The application provides a video monitoring analysis method and system for the operation behavior of a thermal power plant. First, continuous monitoring video data of the operation area of the thermal power plant is obtained, which is composed of multiple operation scene video clips arranged in chronological order. Then, the continuous monitoring video data is analyzed for features to obtain the operation behavior features of the operators and the spatial configuration features of the operation environment. Next, the operation behavior features and the spatial configuration features are input into a pre-trained operation compliance detection model to generate compliance detection results for the operation scene video clips. Based on the compliance detection results, non-compliant operation events are identified, and their time occurrence ranges and spatial occurrence positions are extracted. Finally, operation warning instructions containing time-space correlation information are generated based on the time occurrence ranges and the spatial occurrence positions, and are sent to the safety monitoring terminal of the thermal power plant to trigger intervention responses, thereby achieving real-time and accurate monitoring of the operation safety of the thermal power plant.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring technology for thermal power plants, and more specifically, to a video monitoring and analysis method and system for the operational behavior of thermal power plants. Background Technology

[0002] Operational safety is paramount in the daily operation of thermal power plants. The working environment of thermal power plants is complex, involving numerous high-risk work scenarios, such as operating high-temperature and high-pressure equipment and electrical work. Traditional methods of monitoring operational safety in thermal power plants mainly rely on manual inspections and periodic safety checks. Manual inspections are not only labor-intensive but also difficult to achieve 24 / 7, all-round real-time monitoring of the work area. Periodic safety checks are lagging and cannot promptly detect violations during operations.

[0003] With the development of video surveillance technology, thermal power plants have widely installed video surveillance systems. However, most existing systems only have simple video recording and playback functions, lacking intelligent analysis capabilities for video content. They cannot automatically identify whether workers' actions are compliant, nor can they assess the spatial configuration of the work environment, making it difficult to meet the needs of thermal power plants for real-time and accurate monitoring of operational safety. Therefore, a method is needed to analyze video surveillance of operational behavior in thermal power plants to achieve timely detection and early warning of violations. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a video surveillance analysis method for the operational behavior of thermal power plants, the method comprising:

[0005] Acquire continuous monitoring video data of the operating area of ​​a thermal power plant, wherein the continuous monitoring video data consists of multiple video clips of operating scenes arranged in chronological order;

[0006] The continuous monitoring video data is subjected to feature analysis processing to obtain the operational behavior features of the workers in each work scene video segment and the spatial configuration features of the work environment in the work scene video segment;

[0007] The operational behavior features and spatial configuration features are input into a pre-trained operation compliance detection model to generate compliance detection results for the operation scene video clip;

[0008] Based on the compliance detection results, identify the non-compliant operation events in the continuous monitoring video data, and extract the temporal range and spatial location of the non-compliant operation events;

[0009] Based on the time range and spatial location of the occurrence, an operation warning instruction containing spatiotemporal correlation information is generated, and the operation warning instruction is sent to the power plant safety monitoring terminal to trigger an intervention response.

[0010] In another aspect, embodiments of the present invention also provide a video monitoring and analysis system for the operation of thermal power plants, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the machine-readable storage medium to implement the above-described method.

[0011] Based on the above, this embodiment of the invention acquires continuous monitoring video data of the working area of ​​a thermal power plant and performs feature analysis processing on the continuous monitoring video data. This enables the comprehensive and accurate acquisition of the operational behavior characteristics of the workers and the spatial configuration characteristics of the working environment in each video segment of the working scene. The operational behavior characteristics and the spatial configuration characteristics of the working environment are input into a pre-trained operation compliance detection model to generate compliance detection results for the video segments of the working scene. These results comprehensively consider multiple factors such as operational behavior and spatial configuration, greatly improving the accuracy and reliability of compliance detection. Based on the compliance detection results, violations of work regulations are identified, and their temporal range and spatial location are extracted, making the location of violations of work regulations more accurate. The final generated work warning instructions containing spatiotemporal correlation information can be sent to the thermal power plant safety monitoring terminal in a timely and accurate manner, triggering corresponding intervention responses, effectively preventing and reducing the occurrence of work accidents in thermal power plants, ensuring the safe and stable operation of thermal power plants, and thus improving the intelligence level and efficiency of work safety monitoring in thermal power plants. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the execution flow of the video monitoring and analysis method for the operation behavior of thermal power plants provided in the embodiments of the present invention.

[0013] Figure 2 This is a schematic diagram of exemplary hardware and software components of a video monitoring and analysis system for the operational behavior of thermal power plants provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a video monitoring and analysis method for the operational behavior of thermal power plants, provided in one embodiment of the present invention. The following is a detailed description of this video monitoring and analysis method for the operational behavior of thermal power plants.

[0015] Step S110: Obtain continuous monitoring video data of the thermal power plant's operating area, wherein the continuous monitoring video data consists of multiple video clips of operating scenarios arranged in chronological order.

[0016] In this embodiment, multiple surveillance cameras can be strategically deployed in various key operational areas of the thermal power plant to ensure comprehensive and continuous monitoring of these areas. These operational areas include, but are not limited to, boiler areas, turbine areas, generator areas, and fuel storage and transportation areas. Each surveillance camera possesses high resolution and good low-light performance to adapt to different operating environments within the thermal power plant.

[0017] Surveillance cameras continuously collect video data at preset time intervals, and this video data is transmitted to a data storage server in real time. The server performs preliminary processing on the received video data, such as compressing the video to reduce storage space usage, and numbering and classifying the videos according to time sequence. Continuous monitoring video data consists of multiple video clips of different work scenarios, representing the work situation within different time periods. For example, during boiler ignition operations, the surveillance camera records the entire process from ignition preparation to successful ignition; this process corresponds to one work scenario video clip. During routine turbine inspections, the recorded video constitutes another work scenario video clip. These work scenario video clips are arranged in chronological order to form the continuous monitoring video data.

[0018] Step S120: Perform feature analysis processing on the continuous monitoring video data to obtain the operational behavior features of the workers in each work scene video segment and the spatial configuration features of the work environment in the work scene video segment.

[0019] Next, in order to extract useful information from the continuous surveillance video data, it is necessary to perform feature parsing processing, which can be further subdivided into several sub-steps.

[0020] Step S121: Perform frame-by-frame decomposition processing on the continuous monitoring video data to obtain multiple operation scene video frames with temporal continuity.

[0021] Continuous surveillance video data is essentially composed of a series of consecutive video frames played at a set frame rate, thus forming a dynamic video image. For ease of subsequent analysis, the continuous surveillance video data needs to be decomposed into individual video frames. In this embodiment, video processing software is used to decompose the continuous surveillance video data frame by frame. This software can break down the video data into independent work scene video frames according to the video's frame rate. For example, if the video's frame rate is 25 frames per second, then a 10-second video will be decomposed into 250 work scene video frames. These video frames are arranged in chronological order, exhibiting strict temporal continuity, with a fixed time interval between each adjacent frame.

[0022] Step S122: Perform worker identification processing on the video frame of the work scene, locate the outline region of the worker in the video frame of the work scene using a target detection algorithm, and mark the boundary coordinate points of the outline region of the worker.

[0023] In this embodiment, after obtaining the video frames of the work scene, it is necessary to identify the workers within them. This embodiment employs advanced object detection algorithms, such as the YOLO (You Only Look Once) series of algorithms based on deep learning or the Faster R-CNN (Region-based Convolutional Neural Networks) algorithm. These algorithms have been trained on a large amount of labeled data and are capable of accurately identifying the workers in the video frames of the work scene.

[0024] The object detection algorithm searches for the outline region of a worker within the video frame. When a worker is detected, the algorithm draws a bounding box around them, roughly outlining their shape. Then, by processing the bounding box, the boundary coordinates of the worker's outline region can be obtained. Specifically, the coordinates of the four vertices of the bounding box—the top-left, top-right, bottom-left, and bottom-right corners—can be recorded. These coordinates, based on the pixel coordinate system of the video frame, are used to accurately describe the worker's position within the video frame.

[0025] Step S123: Based on the position change trajectory of the boundary coordinate point in the continuous operation scene video frame, extract the limb movement direction parameter and action completion parameter of the operator as operation behavior features. The limb movement direction parameter reflects the movement direction of the operator's limb in space, and the action completion parameter reflects the degree of matching between the operator's action and the standard operation procedure.

[0026] Furthermore, based on the positional changes of the boundary coordinates of the worker's outline region in continuous video frames of the work scene, the worker's operational behavior features are extracted. This process can be broken down into the following sub-steps.

[0027] Step S1231: Establish a pixel coordinate system in the video frame of the work scene and record the key point coordinate sequence of the outline area of ​​the worker. The key points include the coordinate points of the head, shoulders, elbows and hands.

[0028] To accurately describe the position and actions of workers within video frames, a pixel coordinate system needs to be established within the video frames of the work scene. Typically, the top-left corner of the video frame is used as the origin, with the positive X-axis pointing horizontally to the right and the positive Y-axis pointing vertically downwards. In this pixel coordinate system, each pixel has a unique coordinate value.

[0029] After determining the outline region of the worker, key points such as the head, shoulders, elbows, and hands are further identified. These key points reflect the worker's limb movements. By processing the worker's outline region using pose estimation algorithms, such as OpenPose or AlphaPose, the coordinates of these key points in the pixel coordinate system can be obtained. In consecutive video frames of the work scene, the coordinate sequence of these key points is recorded, with each coordinate sequence representing the position of the corresponding key point at different times.

[0030] Step S1232: Calculate the coordinate difference of corresponding key points in adjacent video frames of the work scene to obtain the single motion displacement vector of the key point. The single motion displacement vector includes a motion direction component and a motion distance component.

[0031] After obtaining the coordinate sequence of key points, the coordinate difference of corresponding key points in adjacent video frames of the work scene is calculated. For example, for a hand key point, if its coordinates in the nth frame are (Xn, Yn) and its coordinates in the (n+1)th frame are (Xn+1, Yn+1), then the X-direction component of the single-movement displacement vector of the hand key point is Xn+1-Xn, and the Y-direction component is Yn+1-Yn. These two components determine the direction of movement of the hand key point between adjacent frames.

[0032] The motion distance component can be calculated from the X and Y direction components. For example, the Pythagorean theorem can be used to add the squares of the X and Y direction components and then take the square root to obtain the motion distance. This single motion displacement vector can accurately describe the motion of keypoints between adjacent frames.

[0033] Step S1233: Statistically analyze the trend of the change of the direction component of the single motion displacement vector of the key point within the preset time window, and generate the limb motion direction parameters of the operator. The limb motion direction parameters are represented by the consistency of the direction components.

[0034] To more comprehensively describe the direction of worker's limb movements, it is necessary to statistically analyze the changing trend of the directional components of the displacement vector of a single movement at a key point within a preset time window. The size of the preset time window can be adjusted according to the actual situation, for example, it can be set to 5 seconds or 10 seconds.

[0035] Within a preset time window, the directional components of the single-movement displacement vector of each key point are summarized. If the directional component of a key point's single-movement displacement vector changes little within this time window, it indicates that the movement direction of that key point is relatively consistent; conversely, if the directional component changes significantly, it indicates that the movement direction is relatively dispersed. By quantifying the consistency of the directional components, the limb movement direction parameters of the worker can be obtained. For example, the standard deviation of the directional components can be calculated; the smaller the standard deviation, the higher the consistency of the directional components and the more stable the limb movement direction.

[0036] Step S1234: Obtain the standard action trajectory template corresponding to the standard operation procedure of the thermal power plant. The standard action trajectory template contains the standard coordinate sequence of key points at continuous time points.

[0037] To assess the completion rate of operators' actions, it is necessary to obtain standard action trajectory templates corresponding to the standard operating procedures of thermal power plants. These templates are obtained through detailed analysis and recording of the standard operating procedures of thermal power plants.

[0038] In standard operating procedures, each action can be broken down to determine the standard coordinate sequence of key points (such as the head, shoulder, elbow, and hand) at consecutive time points. For example, in the operation of opening and closing a valve, the standard coordinates of key hand points at different moments during the valve opening and closing process can be recorded. These standard coordinate sequences constitute a standard action trajectory template. The standard action trajectory template can be stored in a database for subsequent querying and use.

[0039] Step S1235: Calculate the average feature distance between the key point coordinate sequence of the operator and the coordinate sequence of the standard motion trajectory template, and generate the action completion parameter of the operator. The action completion parameter is represented by the reciprocal of the average feature distance to indicate the degree of matching.

[0040] The key point coordinate sequence of the operator is compared with the coordinate sequence of the standard motion trajectory template, and the average characteristic distance between them is calculated. The characteristic distance can be calculated using various methods, such as Euclidean distance or Manhattan distance.

[0041] For each keypoint, at the same time point, the characteristic distance between the operator's keypoint coordinates and the corresponding keypoint coordinates in the standard motion trajectory template is calculated. Then, the characteristic distances of all keypoints at different time points are summarized, and the average value is calculated. This average value reflects the overall degree of difference between the operator's actions and the standard actions.

[0042] The action completion parameter is represented by the reciprocal of the average feature distance. The smaller the average feature distance, the closer the operator's action is to the standard action, and the higher the action completion parameter, i.e., the better the action completion. Conversely, the larger the average feature distance, the lower the action completion parameter, and the worse the action completion.

[0043] Step S124: Perform environmental element recognition processing on the video frame of the work scene, segment the equipment area, channel area and warning sign area in the video frame of the work scene using a semantic segmentation algorithm, and mark the boundary contour lines of the equipment area, channel area and warning sign area.

[0044] In addition to the operational behavior characteristics of the workers, it is also necessary to extract the spatial configuration features of the work environment from the video frames of the work scene. First, environmental element recognition processing is performed on the video frames of the work scene.

[0045] In this embodiment, semantic segmentation algorithms, such as U-Net (U-shaped Convolutional Neural Networks) or DeepLab series algorithms, are used to segment the device area, channel area, and warning sign area in the video frame of the work scene. Semantic segmentation algorithms can classify each pixel in the video frame into different categories, thereby separating the device area, channel area, and warning sign area from the background.

[0046] Once these regions are segmented, their boundary contours can be marked. Specifically, by processing the segmentation results using edge detection algorithms, such as the Canny edge detection algorithm, the boundary pixels of the regions can be found. Then, these boundary pixels are connected to form boundary contours. These boundary contours can accurately describe the shape and location of device areas, channel areas, and warning sign areas.

[0047] Step S125: Based on the distribution relationship of the boundary contour line in the video frame of the work scene, extract the adjacency distance parameter between the equipment area and the channel area and the visible coverage parameter of the warning sign area as spatial configuration features, wherein the adjacency distance parameter reflects the spatial interval between the equipment and the passage path, and the visible coverage parameter reflects the display range of the warning sign in the video screen.

[0048] Furthermore, based on the distribution relationship of the boundary outlines of the equipment area, passage area, and warning sign area in the video frames of the work scene, the spatial configuration features of the work environment are extracted. This process can be further divided into the following sub-steps.

[0049] Step S1251: Perform closure detection processing on the boundary contour lines of the equipment area and the channel area to filter out valid equipment areas and valid channel areas with complete boundary contour lines.

[0050] After obtaining the boundary contours of the equipment area and the passage area, a closure test is required. The purpose of the closure test is to ensure that the boundary contours completely enclose the equipment area or the passage area.

[0051] Algorithms can be used to detect the closure of boundary contours, such as checking whether the start and end points of the boundary contours coincide, or calculating the relationship between the perimeter of the boundary contour and the area of ​​the enclosed region. If the closure of the boundary contour meets the set conditions, the boundary contour of the region is considered complete, and the region is a valid region; otherwise, the boundary contour of the region is considered incomplete, and further processing or exclusion is required.

[0052] Step S1252: Extract the outer edge coordinate points of the boundary contour line of the effective device area and the inner edge coordinate points of the boundary contour line of the effective channel area, calculate the minimum pixel distance between any outer edge coordinate point of the device and the inner edge coordinate point of the channel, and generate the adjacency distance parameters between the device area and the channel area.

[0053] For the selected valid device region and valid channel region, extract the outer coordinate points of the boundary contour lines of the valid device region and the inner coordinate points of the boundary contour lines of the valid channel region. The outer coordinate points are the pixels farthest from the device center in the boundary contour lines of the device region, and the inner coordinate points are the pixels closest to the channel center in the boundary contour lines of the channel region.

[0054] Calculate the pixel distance between any coordinate point on the outer edge of the device and the coordinate point on the inner edge of the channel. Euclidean or Manhattan distance can be used to calculate the pixel distance. Then, find the minimum value among all pixel distances; this minimum value is the adjacency distance parameter between the device area and the channel area. The adjacency distance parameter reflects the spatial separation between the device and the passageway; a larger adjacency distance indicates a greater distance between the device and the channel, and higher passage security.

[0055] Step S1253: Perform connectivity analysis on the boundary outline of the warning sign area to identify continuously visible sub-regions within the warning sign area.

[0056] The purpose of performing connectivity analysis on the boundary outline of the warning sign area is to identify continuously visible sub-regions within the warning sign area. Because warning signs may be partially obscured or damaged in the video frames of the work scene, they may appear discontinuous in the video footage.

[0057] Connectivity analysis can be implemented using the connected component algorithm in graph theory. Consider the boundary outline of the warning sign area as a graph, the pixels on the boundary outline as nodes in the graph, and the connections between adjacent pixels as edges. By analyzing the connected components of the graph, continuously visible sub-regions within the warning sign area can be identified.

[0058] Step S1254: Calculate the pixel area value of the continuous visible sub-region, count the area ratio of the continuous visible sub-region in the total pixel area value of the warning sign area, and generate the visible coverage range parameter of the warning sign area.

[0059] For each identified continuous visible sub-region, its pixel area value is calculated. This can be obtained by counting the number of pixels within the sub-region. Simultaneously, the total pixel area value of the warning sign area is calculated, which represents the total number of pixels occupied by the warning sign area in the video frame.

[0060] Next, the percentage of the pixel area of ​​the continuously visible sub-regions in the total pixel area of ​​the warning sign area is calculated. This percentage is the visible coverage parameter of the warning sign area. The visible coverage parameter reflects the display range of the warning sign in the video frame; the higher the percentage, the better the visibility of the warning sign and the better it can serve its warning function.

[0061] Step S1255: Analyze the relative positional relationship of the effective device area, effective channel area, and continuous visible sub-area in the video frame of the work scene, and generate auxiliary spatial configuration information that includes the region overlap and directional distribution characteristics. The auxiliary spatial configuration information is used to supplement the description of the spatial configuration characteristics of the work environment.

[0062] In addition to the adjacency distance and visible coverage parameters, it is also necessary to analyze the relative positional relationships of the effective device area, effective channel area, and continuous visible sub-regions within the video frames of the operational scene. These relative positional relationships include regional overlap and directional distribution characteristics.

[0063] Region overlap refers to whether there are overlapping areas between the effective device area, the effective channel area, and consecutive visible sub-regions. If overlap exists, the area and location of the overlapping portion need to be recorded. Orientation distribution characteristics refer to the directional distribution of these areas in the video frame, such as whether the device area is located to the left or right of the channel area, or whether the warning sign area is above or below the device area.

[0064] By analyzing and recording these relative positional relationships, auxiliary spatial configuration information is generated. This auxiliary spatial configuration information can supplement the spatial configuration characteristics of the work environment, providing a more comprehensive description of the spatial layout of the work scenario.

[0065] Step S130: Input the operation behavior features and the spatial configuration features into the pre-trained operation compliance detection model to generate the compliance detection results of the operation scene video clip.

[0066] The extracted operational behavior features (limb movement direction parameters and action completion parameters) and spatial configuration features (adjacency distance parameters, visibility coverage parameters, and auxiliary spatial configuration information) are input into a pre-trained job compliance detection model to generate compliance detection results for job scene video clips. This process can be broken down into the following sub-steps.

[0067] Step S131: Perform feature dimension alignment processing on the operation behavior features and the spatial configuration features, and integrate the limb movement direction parameters, action completion parameters, adjacency distance parameters and visible coverage parameters into a feature vector of a unified dimension through feature splicing operation.

[0068] Since operational behavior features and spatial configuration features may have different dimensions, they need to be aligned in terms of feature dimensions. The purpose of feature dimension alignment is to integrate different types of features into a unified dimensional space for subsequent model processing.

[0069] In this embodiment, feature concatenation is used to align feature dimensions. The limb movement direction parameter, action completion parameter, adjacency distance parameter, and visible coverage parameter are concatenated in a predetermined order to form a feature vector with a unified dimension. For example, the limb movement direction parameter and action completion parameter can be concatenated first, and then the adjacency distance parameter and visible coverage parameter can be concatenated sequentially. The concatenated feature vector contains information on all operational behavior features and spatial configuration features, possessing a unified dimension.

[0070] Step S132: Input the feature vector into the feature encoding layer of the operation compliance detection model, and perform time series encoding processing on the feature vector through a bidirectional long short-term memory network to generate time-series encoded features containing historical feature information and current feature information.

[0071] The integrated feature vector is then input into the feature encoding layer of the job compliance detection model. The feature encoding layer employs a Bidirectional Long Short-Term Memory (Bi-LSTM) network. Bi-LSTM is a recurrent neural network capable of processing sequential data and capturing long-term dependencies within the sequence.

[0072] In the feature encoding layer, Bi-LSTM performs time-series encoding on the feature vectors. Since the video clips of the work scene are arranged chronologically, each feature vector corresponds to a specific time point. Bi-LSTM considers both past and future feature information simultaneously, generating time-series encoded features that include both historical and current feature information through forward and backward recursive calculations. These time-series encoded features better reflect the changes in the work scene over time.

[0073] Step S133: Input the time-series coding features into the correlation analysis layer of the operation compliance detection model, calculate the mutual information value between the operation behavior features and the spatial configuration features, and generate a correlation weight matrix that reflects the correlation strength between the features.

[0074] After inputting the temporal coding features into the correlation analysis layer of the operation compliance detection model, the mutual information values ​​of the operation behavior features and spatial configuration features are calculated and the correlation weight matrix is ​​generated through the following sub-steps.

[0075] Step S1331: Decompose the temporal coding features into operational behavior sub-features and spatial configuration sub-features, wherein the operational behavior sub-features include coded representations of limb movement direction parameters and action completion parameters, and the spatial configuration sub-features include coded representations of adjacency distance parameters and visible coverage parameters.

[0076] In the association analysis layer, the input temporal coding features are first decomposed. Since the temporal coding features are the result of encoding both operational behavior features and spatial configuration features, they need to be separated into operational behavior sub-features and spatial configuration sub-features. The operational behavior sub-features mainly reflect information related to the operator's operational behavior. Specifically, the encoded representation of the limb movement direction parameter reflects the encoded form of the operator's limb movement direction information in space, while the encoded representation of the action completion parameter reflects the encoded content of the degree to which the operator's actions match the standard operating procedure. The spatial configuration sub-features contain information related to the spatial configuration of the work environment. The encoded representation of the adjacency distance parameter reflects the encoded result of the spatial interval information between the equipment and the passageway, and the encoded representation of the visible coverage parameter reflects the encoded display range information of the warning signs in the video frame.

[0077] Step S1332: Calculate the joint probability distribution and marginal probability distribution of the operation behavior sub-feature and the spatial configuration sub-feature, and calculate the mutual information value based on the joint probability distribution and the marginal probability distribution. The mutual information value represents the degree of information sharing between the two sub-features.

[0078] After obtaining the operational behavior sub-features and spatial configuration sub-features, their joint probability distribution and marginal probability distribution need to be calculated. The joint probability distribution describes the probability of both operational behavior sub-features and spatial configuration sub-features occurring simultaneously, while the marginal probability distribution describes the probability of each sub-feature occurring individually. Through statistical analysis of a large amount of training data, their joint probability distribution and marginal probability distribution are determined based on the values ​​of the operational behavior sub-features and spatial configuration sub-features.

[0079] After obtaining the joint probability distribution and marginal probability distribution, the mutual information value is calculated based on them. The mutual information value is an indicator that measures the degree of information sharing between the operational behavior sub-feature and the spatial configuration sub-feature. If the mutual information value between two sub-features is large, it indicates that there is a strong correlation between them, and the information of one sub-feature can greatly help predict the information of the other sub-feature; conversely, if the mutual information value is small, it indicates that there is a weak correlation between them.

[0080] Step S1333: Generate an initial correlation weight matrix based on the mutual information value, wherein each element value in the initial correlation weight matrix is ​​positively correlated with the mutual information value of the corresponding feature dimension.

[0081] An initial association weight matrix is ​​generated based on the calculated mutual information values. The size of the initial association weight matrix is ​​related to the dimensions of the operational behavior sub-features and the spatial configuration sub-features. Each element in the matrix corresponds to a combination of feature dimensions of the operational behavior sub-features and the spatial configuration sub-features. The value of each element is positively correlated with the mutual information value of the corresponding feature dimension; that is, the larger the mutual information value, the larger the corresponding matrix element value. In this way, the initial association weight matrix reflects the association strength between the various dimensions of the operational behavior sub-features and the spatial configuration sub-features.

[0082] Step S1334: Normalize the initial correlation weight matrix to generate a correlation weight matrix that reflects the correlation strength between features.

[0083] To improve the comparability and stability of the elements in the initial association weight matrix, it needs to be normalized. The purpose of normalization is to map the element values ​​in the matrix to a defined range, typically [0, 1]. Normalization eliminates potential scale differences between different feature dimensions, allowing the association weight matrix to more accurately reflect the strength of the association between features. After normalization, the association weight matrix reflecting the strength of the association between features is obtained.

[0084] Step S1335: Perform matrix multiplication on the correlation weight matrix and the temporal coding features to generate weighted coding features on the feature dimensions. In the weighted coding features, feature dimensions with higher correlation strength are enhanced, while feature dimensions with lower correlation strength are suppressed.

[0085] The association weight matrix is ​​multiplied with the temporal encoded features. During the matrix multiplication, the elements of the association weight matrix weight each feature dimension of the temporal encoded features. Since the element values ​​in the association weight matrix reflect the correlation strength between features, feature dimensions with higher correlation strength are given greater weight in the multiplication operation, thus being enhanced in the weighted encoded features; conversely, feature dimensions with lower correlation strength have relatively smaller weights, and their effect in the weighted encoded features is correspondingly suppressed. Through this method, weighted encoded features are generated along the feature dimensions, highlighting the correlation information between features.

[0086] Step S134: Use the association weight matrix to weight the temporal coding features to generate enhanced coding features containing feature association information.

[0087] The association weight matrix is ​​multiplied with the temporal coding features. Through this weighting process, each feature dimension in the temporal coding features is adjusted according to its association strength with other features. Feature dimensions with higher association strength are given greater weight during the calculation, thus becoming more prominent in enhancing the coding features; while feature dimensions with lower association strength have relatively smaller weights, and their role in enhancing the coding features is correspondingly weakened.

[0088] Specifically, each row of the association weight matrix corresponds to a feature dimension in the temporal coding feature, and the element values ​​in the matrix represent the association strength between that feature dimension and other feature dimensions. During matrix multiplication, each element of the association weight matrix is ​​multiplied one by one with the element of the corresponding column in the temporal coding feature, and these products are then added together to obtain the element value at the corresponding position in the enhanced coding feature. This completes the weighted processing of the temporal coding feature, generating an enhanced coding feature that contains feature association information.

[0089] Compared to the original temporal coding features, enhanced coding features highlight the correlation between operational behavior features and spatial configuration features. This correlation is crucial for determining the compliance of work scenarios because, in thermal power plant operations, operator behavior is often closely related to the spatial configuration of the work environment. For example, if the adjacent distance between equipment areas and passageway areas is too close, it may affect operator behavior and increase the risk of non-compliant operations. Enhanced coding features can better capture this correlation information, improving the accuracy of operational compliance detection.

[0090] Step S135: Input the enhanced encoding features into the classification output layer of the operation compliance detection model, perform compliance probability prediction processing through a fully connected neural network, and generate a compliance detection result containing a compliance probability value and a violation type label. The compliance probability value represents the probability that the current operation scene video segment complies with safety regulations, and the violation type label represents the specific violation category.

[0091] Next, the enhanced encoded features containing feature association information are input into the classification output layer of the job compliance detection model. The classification output layer uses a fully connected neural network, a common neural network structure in which each neuron is connected to all neurons in the previous layer.

[0092] In the classification output layer, the fully connected neural network further processes and analyzes the enhanced encoded features, mapping them to a probability space through a series of nonlinear transformations. Specifically, based on the input of the enhanced encoded features, the fully connected neural network calculates the probability that the current work scene video segment complies with safety regulations; this probability value is the compliance probability value. The compliance probability value ranges from 0 to 1; the closer the value is to 1, the greater the likelihood that the work scene video segment complies with safety regulations; the closer the value is to 0, the greater the likelihood of non-compliance.

[0093] Simultaneously, the fully connected neural network also determines the specific violation category based on the enhanced encoding features and outputs the corresponding violation type label. The violation type label can be of various categories, such as improper operation, improper equipment placement, or missing warning signs. These labels clearly identify the specific violations in the work scenario, facilitating subsequent processing and rectification.

[0094] Step S140: Identify the non-compliant operation events in the continuous monitoring video data based on the compliance detection results, and extract the temporal range and spatial location of the non-compliant operation events.

[0095] After obtaining the compliance test results, it is necessary to identify non-compliant operation events in the continuous monitoring video data based on these results, and extract the temporal range and spatial location of the non-compliant operation events. This process can be broken down into the following sub-steps.

[0096] Step S141: Analyze the compliance probability value in the compliance detection result and select video clips of work scenarios with compliance probability values ​​lower than a preset threshold as candidate violation clips.

[0097] First, the compliance probability value in the compliance inspection results is analyzed. The preset threshold is a pre-defined standard used to determine whether a video clip of a work scenario violates regulations. When the compliance probability value is lower than the preset threshold, it indicates that the video clip of the work scenario has a high risk of violation, and it is filtered out as a candidate for violation.

[0098] The preset threshold needs to be adjusted based on the actual conditions and safety requirements of the thermal power plant. If the preset threshold is set too high, some minor violations may be missed; if the preset threshold is set too low, too many false positives may occur. Therefore, a suitable preset threshold needs to be found through extensive experiments and data analysis to balance the missed detection rate and the false positive rate.

[0099] Step S142: Perform timestamp backtracking processing on the candidate violation segments to determine the earliest start time and latest end time of the candidate violation segments in the continuous monitoring video data, and generate the time range of the violation operation event.

[0100] For the selected candidate violation segments, timestamp backtracking processing is performed. Each video segment of the work scenario has a corresponding timestamp, recording the time position of the segment in the continuous monitoring video data. By analyzing the timestamps of the candidate violation segments, the earliest start time and latest end time in the continuous monitoring video data are found.

[0101] The time period between the earliest start time and the latest end time constitutes the time frame of the violation incident. This time frame clearly identifies when the violation began and ended, aiding in subsequent investigations and handling. For example, based on the time frame, surveillance video within the relevant time period can be reviewed to understand the specific process of the violation.

[0102] Step S143: Extract the spatial configuration features of the corresponding work scene video frames in the candidate violation segments, and filter out the target areas associated with the violation type tags. The target areas include one or more of the following: equipment area, channel area, or warning sign area.

[0103] Spatial configuration features of the corresponding work scene video frames are extracted from the candidate violation segments, including the adjacency distance parameters between the equipment area and the passage area, the visible coverage parameters of the warning sign area, and auxiliary spatial configuration information. Then, based on the violation type label, the target area associated with that label is selected.

[0104] Different violation type labels may be associated with different target areas. For example, if the violation type label is "improper equipment placement," the target area might be the equipment area; if the violation type label is "obstructed passageway," the target area might be the passageway area; and if the violation type label is "missing warning signs," the target area might be the warning sign area. By filtering out the target area, violations can be analyzed and handled more effectively.

[0105] Step S144: Perform spatial coordinate mapping processing on the target area to convert the pixel coordinates of the target area into the geographic coordinates of the actual operating area of ​​the thermal power plant. The geographic coordinates are calculated using the calibration parameters of the monitoring camera and the spatial projection model.

[0106] The target area is subjected to spatial coordinate mapping processing to convert its pixel coordinates into the geographic coordinates of the actual operating area of ​​the thermal power plant. This process includes the following sub-steps.

[0107] Step S1441: Obtain the intrinsic and extrinsic parameters of the surveillance camera corresponding to the continuous monitoring video data. The intrinsic and extrinsic parameters include the camera's focal length, principal point coordinates, rotation matrix, and translation vector.

[0108] To convert the pixel coordinates of a target area into geographic coordinates, it is first necessary to obtain the intrinsic and extrinsic parameters of the surveillance camera corresponding to the continuous surveillance video data. These intrinsic and extrinsic parameters are crucial information describing the camera's imaging characteristics and its position and orientation in space. The camera's focal length determines the degree to which the camera magnifies or reduces the image, the principal point coordinates are the coordinates of the center point on the image plane, the rotation matrix describes the camera's rotation direction in space, and the translation vector represents the camera's translational position in space. By calibrating the surveillance camera, the accurate values ​​of these intrinsic and extrinsic parameters can be obtained.

[0109] Step S1442: Based on the intrinsic and extrinsic parameters, establish a projection model from pixel coordinates to three-dimensional spatial coordinates. The projection model describes the transformation relationship between pixel coordinates and three-dimensional spatial coordinates using the pinhole camera model formula.

[0110] Based on the acquired intrinsic and extrinsic parameters of the camera, a projection model from pixel coordinates to 3D spatial coordinates is established. The pinhole camera model is a commonly used model to describe the imaging principle of a camera, assuming that light enters the camera through a small hole and forms an image on the imaging plane. The pinhole camera model formula describes the transformation relationship between pixel coordinates and 3D spatial coordinates. When establishing the projection model, the camera's focal length, principal point coordinates, rotation matrix, translation vector, and other intrinsic and extrinsic parameters are substituted into the pinhole camera model formula to obtain a specific projection model that can convert the pixel coordinates of the target area into 3D spatial coordinates.

[0111] Step S1443: Extract the boundary pixel coordinates of the target area, input the boundary pixel coordinates into the projection model for three-dimensional coordinate calculation, and obtain the corresponding three-dimensional spatial coordinates.

[0112] Boundary pixel coordinates are extracted from the target region. These coordinates accurately describe the shape and size of the target region. The extracted boundary pixel coordinates are then input into a pre-built projection model, and 3D coordinate calculations are performed based on the transformation relationships of the projection model. Through the calculations of the projection model, each boundary pixel coordinate is converted into a corresponding 3D spatial coordinate, which represents the position of the target region in 3D space.

[0113] Step S1444: Obtain the transformation parameters between the geographic coordinate system and the three-dimensional spatial coordinate system of the thermal power plant's operating area. The transformation parameters include the origin offset and the rotation angle of the coordinate axes.

[0114] To convert 3D spatial coordinates into geographic coordinates within the power plant's geographic coordinate system, it's necessary to obtain the transformation parameters between the geographic coordinate system and the 3D spatial coordinate system of the power plant's operating area. The geographic coordinate system describes the actual location of the power plant's operating area, while the 3D spatial coordinate system is established based on camera data. The origin offset in the transformation parameters represents the positional difference between the origins of the two coordinate systems, and the coordinate axis rotation angle represents the rotational relationship between the coordinate axes. Accurate values ​​for these transformation parameters can be obtained by measuring and analyzing the power plant's operating area.

[0115] Step S1445: Based on the transformation parameters, convert the three-dimensional spatial coordinate points into geographic coordinate points in the power plant geographic coordinate system. The geographic coordinate points include longitude, latitude, and altitude information.

[0116] Based on the acquired transformation parameters, the three-dimensional spatial coordinate points are converted into geographic coordinate points in the power plant's geographic coordinate system. During the transformation process, the three-dimensional spatial coordinate points are translated and rotated according to the origin offset and coordinate axis rotation angles, transforming them from the three-dimensional spatial coordinate system to the geographic coordinate system. The transformed geographic coordinate points contain longitude, latitude, and altitude information, accurately representing the location of the target area within the actual operating area of ​​the power plant.

[0117] Step S1446: Perform error correction processing on the geographic coordinate points, cross-verify the coordinate points of the overlapping areas of multiple surveillance cameras, and generate the error-corrected geographic coordinate points as the actual geographic coordinate representation of the target area.

[0118] In practical applications, the calibration and spatial projection models of surveillance cameras may contain certain errors, necessitating error correction of geographic coordinate points. Cross-validation using coordinate points from the overlapping areas of multiple surveillance cameras is an effective error correction method. Within the overlapping areas of multiple cameras, multiple geographic coordinate points for the same target area can be obtained. Since the error profiles of different cameras may vary, analyzing and processing these coordinate points—such as calculating the average and median—can eliminate some errors, resulting in more accurate geographic coordinate points. Using these error-corrected geographic coordinate points as the actual geographic coordinate representation of the target area improves the accuracy of spatial coordinate mapping.

[0119] Step S145: Based on the geographic coordinates, determine the actual coverage area of ​​the target area in the thermal power plant work site, extract the boundary coordinate points and center coordinate points of the actual coverage area, and generate the spatial location of the violation operation event. The spatial location is represented by the polygon set of boundary coordinate points and the coordinate values ​​of the center coordinate points.

[0120] Based on the error-corrected geographic coordinates, the actual coverage area of ​​the target region within the power plant's operating site is determined. The actual coverage area can be determined by the distribution of geographic coordinates; connecting these coordinates forms a polygonal region, which represents the actual coverage area of ​​the target region.

[0121] Extract the boundary and center coordinates of the actual coverage area. The boundary coordinates are the vertex coordinates of the polygonal region, which accurately describe the shape and size of the actual coverage area. The center coordinates can be obtained by calculating the geometric center of the polygonal region, representing the center position of the target area.

[0122] The spatial location of a violation incident is represented by a polygon set of boundary coordinate points and the coordinate values ​​of the center coordinate point. This representation clearly shows the specific location and extent of the violation incident within the power plant's work area.

[0123] Step S150: Generate an operation warning instruction containing spatiotemporal correlation information based on the time range and the spatial location of the occurrence, and send the operation warning instruction to the power plant safety monitoring terminal to trigger an intervention response.

[0124] After determining the temporal range and spatial location of the violation, an early warning instruction containing spatiotemporal correlation information needs to be generated and sent to the power plant's safety monitoring terminal to trigger an intervention response. This process can be broken down into the following sub-steps.

[0125] Step S151: Extract the start and end time points of the time occurrence range, and calculate the time difference between the start and end time points as the duration parameter of the violation.

[0126] Extract the start and end times from the time range of the violation event. The start time represents the moment the violation began, and the end time represents the moment it ended. Calculate the time difference between the start and end times; this time difference is the violation duration parameter. The violation duration parameter reflects the length of time the violation event lasted and is crucial for assessing its severity and scope of impact.

[0127] Step S152: Extract the boundary coordinates and center coordinates of the location where the violation occurred, and generate a spatial coverage area descriptor for the violation event. The spatial coverage area descriptor contains a polygon sequence of boundary coordinates and the coordinate values ​​of the center coordinates.

[0128] Boundary and center coordinate points are extracted from the spatial location of the violation incident. The boundary coordinate points are arranged in a predetermined order to form a polygon sequence, which accurately describes the actual coverage area of ​​the violation incident within the power plant's work site. The center coordinate point represents the central location of the violation incident.

[0129] By combining the polygon sequence of boundary coordinate points and the coordinate values ​​of the center coordinate point, a spatial coverage area descriptor for the violation event is generated. This spatial coverage area descriptor clearly displays the spatial extent and location information of the violation event.

[0130] Step S153: Perform time-space association binding processing on the time occurrence range, violation duration parameter and spatial coverage area descriptor to generate a spatiotemporal association data group containing timestamp alignment, wherein each time point in the spatiotemporal association data group corresponds to a spatial coverage area descriptor.

[0131] The time range of occurrence, duration of violation, and spatial coverage area descriptor are subjected to time-space correlation binding. Specifically, the time range of occurrence is divided according to a set time interval, with each time interval corresponding to a time point. For each time point, it is associated with the corresponding spatial coverage area descriptor to form a spatiotemporal correlation data group.

[0132] In the spatiotemporal correlation data set, each time point corresponds to a spatial coverage area descriptor, thus achieving alignment of temporal and spatial information. Through the spatiotemporal correlation data set, one can clearly understand the changes in the spatial location and extent of unauthorized work events at different time points.

[0133] Step S154: Obtain the violation type label from the compliance test results, query the thermal power plant safety standard database to obtain the standard intervention measures corresponding to the violation type label, the standard intervention measures include personnel attendance requirements, equipment shutdown instructions or warning information.

[0134] Extract violation type tags from the compliance inspection results. Based on these tags, query the thermal power plant safety regulations database. This database stores standard intervention measures corresponding to various violation types, formulated according to the safety requirements and relevant regulations for thermal power plants.

[0135] Standard intervention measures may include personnel arrival requirements, such as requiring personnel in specific positions to immediately arrive at the site of the violation to handle it; equipment shutdown instructions, such as requiring the relevant equipment to be shut down to prevent accidents; and warning messages, such as issuing warning messages at the work site to remind workers to pay attention to safety. Standard intervention measures corresponding to the violation type tag can be obtained by querying the database.

[0136] Step S155: Perform information fusion processing on the spatiotemporal correlation data group and the standard intervention measures to generate a structured early warning data body containing the time range of occurrence, spatial location of occurrence, and intervention measures.

[0137] The spatiotemporal correlation data set is fused with standard intervention measures. Specifically, the time and space information in the spatiotemporal correlation data set is integrated with the personnel attendance requirements, equipment shutdown instructions, or warning information in the standard intervention measures to form a structured early warning data body.

[0138] The structured early warning data contains crucial information such as the temporal range, spatial location, and corresponding intervention measures of the violation incident. This structured data format facilitates subsequent transmission and processing, ensuring that safety monitoring terminals accurately acquire and understand relevant information about violation incidents.

[0139] Step S156: Perform format conversion processing on the structured early warning data body to generate a communication protocol format compatible with the input interface of the thermal power plant safety monitoring terminal. The communication protocol format includes a data header, a data body, and a checksum field.

[0140] To transmit structured early warning data to the power plant's safety monitoring terminal, it needs to undergo format conversion to generate a communication protocol format compatible with the terminal's input interface. This communication protocol format typically includes a header, data body, and a checksum field.

[0141] The data header identifies the data type and source, the data body contains the specific content of the structured early warning data, and the checksum field verifies the data's integrity and accuracy. By converting the structured early warning data to conform to the input interface requirements of the security monitoring terminal, the system ensures that the data can be transmitted and received correctly.

[0142] Step S157: Use the structured early warning data body after format conversion as an operation early warning instruction containing spatiotemporal correlation information.

[0143] The structured early warning data, after format conversion, is used as an operational early warning instruction containing spatiotemporal correlation information. This operational early warning instruction includes important information such as the temporal range of the violation event, its spatial location, and corresponding intervention measures, and conforms to the input interface requirements of the thermal power plant safety monitoring terminal.

[0144] The work warning instruction is sent to the power plant's safety monitoring terminal. Upon receiving the instruction, the terminal can trigger corresponding intervention responses based on the information within. For example, it may notify relevant personnel to handle the violation at the work site, shut down related equipment, or issue warning messages at the site. Timely intervention effectively reduces the impact of violations on the power plant's safe production, ensuring its normal operation.

[0145] For example, in order for the operational compliance detection model to accurately detect compliance in thermal power plant operation scenarios, it needs to be pre-trained. The pre-training process involves several specific steps.

[0146] Step S211: Collect sample operation video data of thermal power plants as a basic training dataset. The basic training dataset includes compliant operation video clips that conform to safety regulations and non-compliant operation video clips that violate safety regulations.

[0147] Video data of sample operations at thermal power plants was collected, derived from surveillance camera recordings in different work areas of the plant. During collection, it was crucial to ensure the diversity and representativeness of the sample data, covering various work scenarios and operational behaviors. The basic training dataset consisted of two parts: one part comprised compliant video clips demonstrating workers following standard operating procedures; the other part contained video clips of non-compliant operations, including various violations such as improper operation and equipment placement.

[0148] Step S212: Perform feature extraction processing on the basic training dataset to form a training sample set containing feature vectors, and label the training sample set with compliance labels, wherein the compliance labels include compliance labels and specific violation type labels.

[0149] Feature extraction is performed on the basic training dataset using a method similar to that in steps S121-S125. This extracts the operational behavior features of the workers and the spatial configuration features of the work environment from the video clips, forming feature vectors. Each feature vector represents the feature information of a single video clip from a work scene.

[0150] The training sample set is labeled with compliance tags. For compliant video clips, a compliance tag is added; for non-compliant video clips, a specific violation type tag is added, such as improper operation or improper equipment placement. These tags provide supervisory information for the training of the compliance detection model, enabling it to learn the relationship between different feature vectors and compliance.

[0151] Step S213: Construct the initial network structure of the operation compliance detection model. The initial network structure includes a feature encoding layer, an association analysis layer, and a classification output layer. The feature encoding layer is used to perform temporal encoding processing on the feature vectors, the association analysis layer is used to calculate the association strength between features, and the classification output layer is used to output the compliance probability value and the violation type label.

[0152] The initial network structure for the operational compliance detection model is constructed. This structure consists of a feature encoding layer, a correlation analysis layer, and a classification output layer.

[0153] The feature encoding layer employs a bidirectional long short-term memory (Bi-LSTM) network, which performs temporal encoding on the input feature vector. Because the job video data has time-series characteristics, Bi-LSTM can capture the changes in the feature vector over time, generating temporally encoded features that include both historical and current feature information.

[0154] The association analysis layer is used to calculate the association strength between features. In this layer, operational behavior features and spatial configuration features can be analyzed, their mutual information values ​​can be calculated, and an association weight matrix can be generated to highlight the association relationships between features.

[0155] The classification output layer uses a fully connected neural network, which receives the enhanced encoded features output by the association analysis layer. Through a series of nonlinear transformations, it outputs compliance probability values ​​and violation type labels to determine the compliance of the work scenario.

[0156] Step S214: Input the training sample set into the initial network structure, and calculate the predicted compliance probability value and the predicted violation type label through forward propagation.

[0157] The training sample set containing feature vectors and compliance labels is input into the initial network structure of the job compliance detection model. In the model, data is processed sequentially from the input layer through the feature encoding layer, the association analysis layer, and the classification output layer, undergoing forward propagation computation.

[0158] In the feature encoding layer, the feature vectors are converted into temporal encoded features; in the association analysis layer, the temporal encoded features are weighted to generate enhanced encoded features; in the classification output layer, the enhanced encoded features are further processed to obtain the predicted compliance probability value and the predicted violation type label.

[0159] Step S215: Calculate the classification loss value based on the predicted compliance probability value and the actual compliance label, and calculate the type loss value based on the predicted violation type label and the actual violation type label.

[0160] The classification loss is calculated based on the predicted compliance probability value output by the model and the compliance label in the actual compliance label. The classification loss value measures the degree of difference between the compliance probability predicted by the model and the actual compliance situation. When the actual compliance label indicates that the operation scenario is compliant, but the model predicts a low compliance probability, the classification loss value will be large; conversely, if the model prediction is accurate, the classification loss value will be small.

[0161] When calculating the classification loss, the predicted compliance probability and the true compliance label of all training samples can be considered together. For each training sample, the predicted compliance probability is compared with the true compliance label, and the loss value of that sample is calculated based on the comparison result. Then, the loss values ​​of all samples are summed to calculate the classification loss value.

[0162] Simultaneously, a type loss value is calculated based on the predicted violation type label and the actual violation type label. The type loss value is used to evaluate the accuracy of the model's violation type judgment. When the model's predicted violation type is inconsistent with the actual violation type, the type loss value will increase; if the prediction is accurate, the type loss value will be small.

[0163] When calculating the type loss value, each training sample is analyzed individually. For a violation training sample, the model's predicted violation type label is compared with the true violation type label, and the type loss value for that sample is calculated based on the difference between them. The type loss values ​​for all violation samples are then summed and calculated to obtain the final type loss value.

[0164] Step S216: Standardize and weight the classification loss value and the type loss value to generate the total model loss value.

[0165] To comprehensively consider the impact of classification loss and type loss on model training, the classification loss value and type loss value need to be standardized and weighted to generate the total model loss value.

[0166] First, the classification loss and categorical loss values ​​are standardized. The purpose of standardization is to adjust the range of the two loss values ​​to a similar scale, preventing one loss from dominating the total loss due to excessively large scale differences. Standardization can be achieved using common methods, such as subtracting the mean from the loss value and then dividing by its standard deviation.

[0167] After standardization, appropriate weights are assigned to the classification loss and type loss values. The weight settings need to be adjusted based on the specific circumstances, as different weights will affect the model's emphasis on classification accuracy and violation type judgment accuracy during training. For example, if more emphasis is placed on the model's accurate judgment of violation types, a larger weight can be assigned to the type loss value; if more attention is paid to the overall compliance classification accuracy, a larger weight can be assigned to the classification loss value.

[0168] The standardized classification loss value is multiplied by its corresponding weight, and the standardized type loss value is multiplied by its corresponding weight. The two results are then added together to obtain the total model loss value. The total model loss value reflects the overall error of the model in classification and violation type determination.

[0169] Step S217: The backpropagation algorithm is used to adjust the network parameters of the initial network structure according to the total loss value of the model until the total loss value of the model converges to a preset threshold range. Then, the performance of the trained job compliance detection model is verified using a validation dataset. The validation dataset contains historical job video clips that were not used in the training. The effectiveness of the model is evaluated by the compliance detection accuracy of the validation samples.

[0170] The backpropagation algorithm is used to adjust the network parameters of the initial network structure based on the total loss value of the model. Backpropagation is a commonly used neural network training algorithm that determines how to adjust the network parameters to reduce the loss value by calculating the gradient of the total loss value relative to the network parameters.

[0171] Specifically, starting from the model's output layer, the gradient of each layer's network parameters is calculated based on the model's total loss value. The gradient represents the rate of change of the loss value relative to the parameters; it indicates the direction in which parameter adjustments can reduce the loss. Then, the network parameters are updated based on the gradient and a preset learning rate. The learning rate controls the step size of each parameter update; an excessively large learning rate may prevent the job compliance detection model from converging, while an excessively small learning rate will slow down the training speed.

[0172] Repeat the above process, continuously adjusting the network parameters, until the total model loss converges to a preset threshold range. Convergence means that the total model loss no longer decreases significantly after multiple iterations; at this point, the network parameters are considered to have been adjusted to a relatively stable state.

[0173] After the job compliance detection model is trained, its performance is validated using a validation dataset. The validation dataset contains historical job video clips that were not used in the training. These clips have also undergone feature extraction and annotation, and have similar features and labels to the training dataset.

[0174] The feature vectors from the validation dataset are input into the trained job compliance detection model, which outputs predicted compliance probabilities and violation type labels. These predictions are then compared with the true compliance labels in the validation dataset to calculate the compliance detection accuracy of the validation samples. Compliance detection accuracy refers to the proportion of samples where the job compliance detection model correctly identifies compliance or violation and the violation type out of the total number of validation samples. This accuracy is used to evaluate the effectiveness of the job compliance detection model. A high accuracy indicates that the job compliance detection model performs well even on unseen data and has strong generalization ability.

[0175] Step S218: Use the verified job compliance detection model as the pre-trained job compliance detection model.

[0176] When the performance of the operation compliance detection model on the validation dataset meets the expected requirements, i.e., the compliance detection accuracy meets the set standard, the operation compliance detection model is considered to have passed the validation. This validated operation compliance detection model is then used as a pre-trained operation compliance detection model for subsequent compliance detection of video clips from thermal power plant operation scenarios.

[0177] In practical applications, this pre-trained operation compliance detection model can continuously receive new video data of thermal power plant operations and perform real-time detection of the compliance of operation scenarios. Simultaneously, with the availability of new sample data and labels, the operation compliance detection model can be periodically updated and retrained to improve its performance and adaptability, enabling it to better adapt to changes in the thermal power plant operating environment and newly emerging violations.

[0178] Throughout the video surveillance and analysis of all operations at the thermal power plant, data collection and processing must adhere to strict privacy protection and leak prevention principles. For surveillance video data, encryption technology is used for encrypted storage and transmission to ensure that the data is not illegally accessed during storage and transmission. During feature extraction and annotation, sensitive data related to the identity of operators, such as personnel, is obfuscated or anonymized, retaining only non-sensitive information related to operational behavior and environmental configuration. Simultaneously, strict access permissions are set, allowing only authorized personnel to access and process this data to prevent the leakage of privacy data.

[0179] In summary, the above series of steps enables video surveillance and analysis of operations at thermal power plants. From acquiring continuous monitoring video data, to feature analysis, compliance detection, and violation event identification, to generating operational warning instructions and triggering intervention responses, and the pre-training process of the operational compliance detection model, a complete solution is formed. This solution can effectively monitor the compliance of operations at thermal power plants, promptly detect violations, and take corresponding intervention measures to ensure the safe production of thermal power plants.

[0180] Figure 2 The illustration shows exemplary hardware and software components of a video surveillance and analysis system 100 for thermal power plant operations, which can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used in the video surveillance and analysis system 100 for thermal power plant operations and to perform the functions described in this application.

[0181] The video monitoring and analysis system 100 for thermal power plant operations can be a general-purpose server or a special-purpose server; both can be used to implement the video monitoring and analysis method for thermal power plant operations described in this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0182] For example, a video surveillance and analysis system 100 for monitoring and analyzing the operation of a thermal power plant may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the video surveillance and analysis system 100 for monitoring and analyzing the operation of a thermal power plant may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The video surveillance and analysis system 100 for monitoring and analyzing the operation of a thermal power plant also includes an I / O interface 150 between the computer and other input / output devices.

[0183] For ease of explanation, only one processor is described in the video monitoring and analysis system 100 for thermal power plant operations. However, it should be noted that the video monitoring and analysis system 100 for thermal power plant operations in this application may also include multiple processors. Therefore, the steps performed by one processor as described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the video monitoring and analysis system 100 for thermal power plant operations performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0184] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned video monitoring and analysis method for the operation behavior of thermal power plants is implemented.

[0185] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A video surveillance analysis method for operational behavior in thermal power plants, characterized in that, The method includes: Acquire continuous monitoring video data of the operating area of ​​a thermal power plant, wherein the continuous monitoring video data consists of multiple video clips of operating scenes arranged in chronological order; The continuous monitoring video data is subjected to feature analysis processing to obtain the operational behavior features of the workers in each work scene video segment and the spatial configuration features of the work environment in the work scene video segment; The operational behavior features and spatial configuration features are input into a pre-trained operation compliance detection model to generate compliance detection results for the operation scene video clip; Based on the compliance detection results, identify the non-compliant operation events in the continuous monitoring video data, and extract the temporal range and spatial location of the non-compliant operation events; Based on the time range and spatial location of occurrence, an operation warning instruction containing spatiotemporal correlation information is generated, and the operation warning instruction is sent to the power plant safety monitoring terminal to trigger an intervention response; The step of performing feature analysis processing on the continuous monitoring video data to obtain the operational behavior features of the workers in each work scene video segment and the spatial configuration features of the work environment in the work scene video segment includes: The continuous monitoring video data is decomposed frame by frame to obtain multiple video frames of the operation scene with temporal continuity; The video frames of the work scene are processed to identify workers. The outline regions of workers in the video frames of the work scene are located by the target detection algorithm, and the boundary coordinate points of the outline regions of workers are marked. Based on the position change trajectory of the boundary coordinate points in the continuous operation scene video frames, the limb movement direction parameters and action completion parameters of the operator are extracted as operation behavior features. The limb movement direction parameters reflect the movement direction of the operator's limbs in space, and the action completion parameters reflect the degree of matching between the operator's actions and the standard operation procedures. The video frame of the work scene is processed for environmental element recognition. The equipment area, channel area and warning sign area in the video frame of the work scene are segmented by semantic segmentation algorithm, and the boundary contour lines of the equipment area, channel area and warning sign area are marked. Based on the distribution relationship of the boundary contour lines in the video frames of the work scene, the adjacency distance parameter between the equipment area and the passage area and the visible coverage parameter of the warning sign area are extracted as spatial configuration features. The adjacency distance parameter reflects the spatial interval between the equipment and the passage path, and the visible coverage parameter reflects the display range of the warning sign in the video frame.

2. The video surveillance and analysis method for thermal power plant operations according to claim 1, characterized in that, The method of extracting the limb movement direction parameters and action completion parameters of the operator as operational behavior features based on the position change trajectory of the boundary coordinate points in continuous operation scene video frames includes: A pixel coordinate system is established in the video frame of the work scene to record the key point coordinate sequence of the outline area of ​​the worker. The key points include the coordinate points of the head, shoulders, elbows and hands. Calculate the coordinate difference of corresponding key points in adjacent video frames of the work scene to obtain the single motion displacement vector of the key point. The single motion displacement vector includes a motion direction component and a motion distance component. The trend of the change of the directional component of the single motion displacement vector of the key point within the preset time window is statistically analyzed to generate the limb motion direction parameters of the operator. The limb motion direction parameters are represented by the consistency of the directional components. Obtain the standard action trajectory template corresponding to the standard operating procedure of a thermal power plant. The standard action trajectory template contains a standard coordinate sequence of key points at consecutive time points. The average feature distance between the key point coordinate sequence of the operator and the coordinate sequence of the standard motion trajectory template is calculated to generate the action completion parameter of the operator. The action completion parameter is represented by the reciprocal of the average feature distance.

3. The video surveillance and analysis method for thermal power plant operations according to claim 1, characterized in that, The method of extracting the adjacency distance parameters between the equipment area and the channel area and the visible coverage parameters of the warning sign area as spatial configuration features based on the distribution relationship of the boundary contour lines in the video frames of the work scene includes: The boundary contours of the equipment area and the channel area are subjected to closure detection processing to filter out valid equipment areas and valid channel areas with complete boundary contours; Extract the outer edge coordinates of the boundary contour of the effective device area and the inner edge coordinates of the boundary contour of the effective channel area, calculate the minimum pixel distance between any outer edge coordinates of the device and the inner edge coordinates of the channel, and generate the adjacency distance parameters between the device area and the channel area; Connectivity analysis is performed on the boundary outline of the warning sign area to identify continuously visible sub-regions within the warning sign area; Calculate the pixel area value of the continuous visible sub-region, and calculate the area ratio of the continuous visible sub-region in the total pixel area value of the warning sign area to generate the visible coverage range parameter of the warning sign area; The relative positional relationships of the effective device area, effective channel area, and continuous visible sub-region in the video frame of the work scene are analyzed to generate auxiliary spatial configuration information that includes regional overlap and directional distribution characteristics. The auxiliary spatial configuration information is used to supplement the description of the spatial configuration characteristics of the work environment.

4. The video surveillance and analysis method for thermal power plant operations according to claim 1, characterized in that, The step of inputting the operational behavior features and the spatial configuration features into a pre-trained job compliance detection model to generate compliance detection results for the job scene video clip includes: The operation behavior features and the spatial configuration features are aligned in terms of feature dimensions. The limb movement direction parameters, action completion parameters, adjacency distance parameters and visible coverage parameters are integrated into a feature vector of a unified dimension through feature splicing operations. The feature vector is input into the feature encoding layer of the operation compliance detection model, and the feature vector is processed by a bidirectional long short-term memory network to generate a time-series encoded feature containing historical feature information and current feature information. The temporal coding features are input into the correlation analysis layer of the operation compliance detection model to calculate the mutual information value between the operation behavior features and the spatial configuration features, and generate a correlation weight matrix that reflects the correlation strength between the features. The temporal coding features are weighted using the association weight matrix to generate enhanced coding features containing feature association information. The enhanced encoding features are input into the classification output layer of the operation compliance detection model, and compliance probability prediction is performed through a fully connected neural network to generate a compliance detection result containing a compliance probability value and a violation type label. The compliance probability value represents the probability that the current operation scenario video segment complies with safety regulations, and the violation type label represents the specific violation category.

5. The video surveillance and analysis method for thermal power plant operations according to claim 4, characterized in that, The step of inputting the temporal coding features into the correlation analysis layer of the operation compliance detection model, calculating the mutual information value between the operational behavior features and the spatial configuration features, and generating a correlation weight matrix reflecting the correlation strength between features includes: The temporal coding features are decomposed into operational behavior sub-features and spatial configuration sub-features, wherein the operational behavior sub-features include coded representations of limb movement direction parameters and action completion parameters, and the spatial configuration features include coded representations of adjacency distance parameters and visible coverage parameters; Calculate the joint probability distribution and marginal probability distribution of the operational behavior sub-feature and the spatial configuration sub-feature, and calculate the mutual information value based on the joint probability distribution and the marginal probability distribution. The mutual information value represents the degree of information sharing between the two sub-features. An initial association weight matrix is ​​generated based on the mutual information value, wherein each element value in the initial association weight matrix is ​​positively correlated with the mutual information value of the corresponding feature dimension; The initial correlation weight matrix is ​​normalized to generate a correlation weight matrix that reflects the strength of the correlation between features. The correlation weight matrix is ​​multiplied with the temporal coding features to generate weighted coding features on the feature dimensions. In the weighted coding features, feature dimensions with higher correlation strength are enhanced, while feature dimensions with lower correlation strength are suppressed.

6. The video surveillance and analysis method for thermal power plant operations according to claim 4, characterized in that, The step of identifying non-compliant operation events in the continuous monitoring video data based on the compliance detection results, and extracting the temporal range and spatial location of the non-compliant operation events, includes: The compliance probability value in the compliance detection results is analyzed, and video clips of work scenarios with compliance probability values ​​lower than a preset threshold are selected as candidate violation clips. The candidate violation segments are processed by timestamp backtracking to determine the earliest start time and latest end time of the candidate violation segments in the continuous monitoring video data, thereby generating the time range of the violation operation event. Extract the spatial configuration features of the corresponding video frames of the work scene from the candidate violation segments, and filter out the target areas associated with the violation type tags. The target areas include one or more of the equipment area, the channel area, or the warning sign area. The target area is subjected to spatial coordinate mapping processing to convert the pixel coordinates of the target area into the geographic coordinates of the actual operating area of ​​the thermal power plant. The geographic coordinates are calculated using the calibration parameters of the monitoring camera and the spatial projection model. Based on the geographic coordinates, the actual coverage area of ​​the target area in the thermal power plant work site is determined. The boundary coordinate points and center coordinate points of the actual coverage area are extracted to generate the spatial location of the violation event. The spatial location is represented by a polygon set of boundary coordinate points and the coordinate values ​​of the center coordinate point.

7. The video surveillance and analysis method for thermal power plant operations according to claim 6, characterized in that, The step of performing spatial coordinate mapping processing on the target area, converting the pixel coordinates of the target area into the geographic coordinates of the actual operating area of ​​the thermal power plant, includes: Obtain the intrinsic and extrinsic parameters of the surveillance camera corresponding to the continuous monitoring video data. The intrinsic and extrinsic parameters include the camera's focal length, principal point coordinates, rotation matrix, and translation vector. Based on the aforementioned intrinsic and extrinsic parameters, a projection model is established to transform pixel coordinates into three-dimensional spatial coordinates. The projection model describes the transformation relationship between pixel coordinates and three-dimensional spatial coordinates using the pinhole camera model formula. Extract the boundary pixel coordinates of the target region, input the boundary pixel coordinates into the projection model for three-dimensional coordinate calculation, and obtain the corresponding three-dimensional spatial coordinates. Obtain the transformation parameters between the geographic coordinate system and the three-dimensional spatial coordinate system of the thermal power plant's operating area. The transformation parameters include the origin offset and the rotation angle of the coordinate axes. Based on the transformation parameters, the three-dimensional spatial coordinate points are converted into geographic coordinate points in the geographic coordinate system of the thermal power plant, and the geographic coordinate points include longitude, latitude and altitude information; Error correction processing is performed on the geographic coordinate points, and cross-validation is performed using the coordinate points of the overlapping areas of multiple surveillance cameras to generate error-corrected geographic coordinate points as the actual geographic coordinate representation of the target area.

8. The video surveillance and analysis method for thermal power plant operations according to claim 1, characterized in that, The process of generating a work early warning instruction containing spatiotemporal correlation information based on the time range and spatial location of occurrence includes: Extract the start and end time points of the time range, and calculate the time difference between the start and end time points as the duration parameter of the violation; Extract the boundary coordinates and center coordinates of the location where the violation occurred, and generate a spatial coverage area descriptor for the violation event. The spatial coverage area descriptor contains a polygon sequence of boundary coordinates and the coordinate values ​​of the center coordinates. The time occurrence range, violation duration parameter, and spatial coverage area descriptor are subjected to time-space association binding processing to generate a spatiotemporal association data group containing timestamp alignment. Each time point in the spatiotemporal association data group corresponds to a spatial coverage area descriptor. Obtain the violation type label from the compliance test results, query the thermal power plant safety specification database to obtain the standard intervention measures corresponding to the violation type label, and the standard intervention measures include personnel on-site requirements, equipment shutdown instructions or warning information; The spatiotemporal correlation data set is fused with the standard intervention measures to generate a structured early warning data body containing the time range of occurrence, spatial location of occurrence, and intervention measures; The structured early warning data body is processed by format conversion to generate a communication protocol format compatible with the input interface of the thermal power plant safety monitoring terminal. The communication protocol format includes a data header, a data body, and a checksum field. The structured early warning data body after format conversion is used as an operation early warning instruction containing spatiotemporal correlation information.

9. A video surveillance and analysis system for the operational behavior of thermal power plants, characterized in that, The device includes a processor and a memory, the memory being connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the memory to implement the video monitoring and analysis method for the operation behavior of thermal power plants as described in any one of claims 1-8.

Images