Regional abnormal condition real-time early warning method based on high-point panoramic intelligent inspection

Abstract

The invention relates to the technical field of intelligent inspection, and discloses a regional abnormal condition real-time early warning method based on high-point panoramic intelligent inspection, which comprises the following steps: collecting visible light and infrared thermal imaging video streams of a target region to form a panoramic video sequence, and carrying out intelligent analysis, feature extraction and analysis, construction of a spatio-temporal topological graph and detection of an abnormal behavior mode. Predicting environmental risks, constructing a spatio-temporal evolution model, and further generating graded early warning information of regional abnormal conditions; the method effectively solves the problems of large panoramic inspection data volume, exception complexity, easy environmental influence on target detection and the like in a large-scale scene, significantly improves the accuracy and real-time performance of exception early warning, and guarantees the regional safety.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent patrol inspection, and more specifically, to a method for real-time early warning of regional abnormal conditions based on high-point panoramic intelligent patrol inspection. Background Art

[0002] The panoramic patrol inspection technology based on high points (such as monitoring towers, fixed high-position cameras, etc.), with its advantage of being able to obtain large-scale real-time video data, has become an indispensable means in the field of intelligent monitoring. It greatly improves the monitoring efficiency and enables managers to grasp the dynamic situation of the monitored area in real time. However, with the continuous expansion of application scenarios and the increasing complexity of requirements, the traditional intelligent patrol inspection technology faces many severe challenges.

[0003] Chinese patent application with publication number CN114693490A discloses a method and system for intelligent patrol inspection of digital parks. The system accesses the digital park equipment information in a three-dimensional simulation panoramic map with the help of a building integrated management system cloud platform to realize fault early warning and intelligent maintenance of the digital twins of park equipment. At the same time, it conducts real-time monitoring and heat analysis of vehicle flow and pedestrian flow, and then carries out safety maintenance and crisis early warning work. This existing technology has certain limitations. Firstly, it mainly targets digital park scenarios, and its applicability is greatly reduced for more complex and diverse large-scale scenarios such as urban management, transportation hubs, border security, etc. In these scenarios, not only does the data volume increase exponentially, but the types and complexity of abnormal conditions far exceed those of digital parks. The method of this existing technology is difficult to handle such a large and complex data processing requirement, nor can it accurately identify and process various complex abnormal conditions. Secondly, in terms of data fusion, this existing technology only relies on park equipment information and vehicle flow and pedestrian flow data, and fails to fully consider the depth and breadth of multi-source data fusion. In the actual large-scale scenario patrol inspection, the data collected by various sensors such as visible light cameras and infrared thermal imagers are crucial for accurately analyzing abnormal conditions. The technical means of this existing technology in this regard are relatively weak, and it cannot achieve the efficient fusion and collaborative analysis of multi-source data, resulting in limited accuracy in the judgment and early warning of abnormal conditions.

[0004] The Chinese patent application with the publication number CN119562031A provides an intelligent patrol warning method and system based on panoramic video fusion, which obtains the first-level alarm information of equipment failures through a panoramic video monitoring component and conducts fault traceability analysis based on the upstream and downstream topological information of the equipment. It also has some deficiencies. On the one hand, the existing technology mainly focuses on the warning and traceability of equipment failures, and the scope of identifying abnormal situations is relatively narrow, unable to cover various abnormal situations such as abnormal human behaviors, fires, and illegal intrusions, which are crucial in large-scale scenarios. On the other hand, in the face of the problems of large data volume and difficult real-time processing in large-scale scenarios, the existing technology lacks efficient data processing algorithms and architectures.

[0005] The existing technology cannot effectively solve the problems of large panoramic patrol data volume and difficult real-time processing in large-scale scenarios, and it is also difficult to achieve the efficient fusion and collaborative analysis of multi-source data, resulting in the accuracy and real-time performance of abnormal warnings being difficult to meet actual needs. Summary of the Invention

[0006] To overcome the above defects of the existing technology, the present invention provides a method for real-time warning of regional abnormal situations based on high-point panoramic intelligent patrol, which constructs a complete abnormal analysis system by collecting visible light and infrared thermal imaging video streams. This method can accurately extract various abnormal features, construct a spatio-temporal topology map and combine environmental risk prediction to construct a spatio-temporal evolution model, thereby generating accurate hierarchical warning information, significantly improving the accuracy and real-time performance of abnormal warnings, providing a strong guarantee for the safety monitoring of large-scale scenarios, effectively reducing the occurrence of safety accidents, and reducing potential losses.

[0007] The present invention is applicable to various large-scale scenarios such as urban management, industrial parks, transportation hubs, border defense and security. In urban management, it can monitor the activities of people in public places in real time and detect potential fire hazards in a timely manner; in industrial parks, it can effectively monitor the operation status of equipment and prevent safety accidents; in transportation hubs, it helps to ensure the safety of passengers and maintain good order; in the field of border defense and security, it can quickly detect abnormal situations such as illegal intrusions.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A method for real-time warning of regional abnormal situations based on high-point panoramic intelligent patrol, including:

[0010] Collect the visible light video stream and infrared thermal imaging video stream of the target area to form a panoramic video sequence of the target area; perform intelligent analysis on the panoramic video sequence to obtain an abnormal feature enhanced video sequence; generate target feature description information according to the abnormal feature enhanced video sequence;

[0011] Construct a spatio-temporal topology graph among the targets in the area according to the target feature description information, analyze the spatio-temporal correlation characteristics among the targets, and detect the abnormal behavior patterns in the area; Embed the detected abnormal behavior patterns in the area as regional abnormal features into the spatio-temporal topology graph; Obtain the environmental risk prediction result based on the target feature description information; Construct a spatio-temporal evolution model based on the spatio-temporal topology graph integrating the regional abnormal behavior patterns and the environmental risk prediction result;

[0012] Generate a hierarchical early warning information for the abnormal situation in the area according to the spatio-temporal evolution model.

[0013] Further, the obtaining of the abnormal feature enhanced video sequence includes:

[0014] Analyze each frame of the visible light video stream, extract the flame area in the video, and generate a flame abnormal enhanced video stream; Perform moving target segmentation on the visible light video stream, extract the smoke area in the video, and generate a smoke abnormal enhanced video stream; Perform target detection and tracking on the visible light video stream, extract the skeleton key point coordinates of the target and the abnormal behaviors of the people in the video, and generate a personnel abnormal behavior enhanced video stream; Perform foreground target segmentation on the infrared thermal imaging video stream, extract the hot spot target area, and generate a hot spot abnormal enhanced video stream;

[0015] The flame abnormal enhanced video stream, the smoke abnormal enhanced video stream, the personnel abnormal behavior enhanced video stream, and the hot spot abnormal enhanced video stream constitute the abnormal feature enhanced video sequence.

[0016] Further, the generating of the target feature description information according to the abnormal feature enhanced video sequence includes:

[0017] Input the flame abnormal enhanced video stream, the smoke abnormal enhanced video stream, and the hot spot abnormal enhanced video stream into a pre-trained deep convolutional neural network to generate target category information reflecting the target type attributes;

[0018] Generate target motion trajectory information reflecting the dynamic movement law of the target according to the personnel abnormal behavior enhanced video stream;

[0019] Obtain target behavior feature information reflecting the abnormal behavior of the target according to the target motion trajectory information;

[0020] Obtain environmental abnormal feature information reflecting the abnormal state of the environment according to the target category information, the target motion trajectory information, and the infrared thermal imaging video stream;

[0021] The target category information, the target motion trajectory information, the target behavior feature information, and the environmental abnormal feature information constitute the target feature description information.

[0022] Further, the generating of the target motion trajectory information reflecting the dynamic movement law of the target includes:

[0023] Extract the skeletal key point coordinates of the target in each frame of the enhanced video stream of abnormal human behavior; generate the human movement trajectory curve of each target in the entire video based on the skeletal key point coordinates of the target; parameterize the human movement trajectory curve, extract the position coordinates of the target at key time steps as spatio-temporal anchor points, and construct a movement trajectory feature matrix; construct a spatio-temporal graph network based on the movement trajectory feature matrix, with each spatio-temporal anchor point as a node of the spatio-temporal graph network;

[0024] Introduce a spatio-temporal coordinate position encoding mechanism, map the discrete position coordinates of the nodes to a continuous periodic feature space through sine and cosine functions, and obtain a spatio-temporal graph network incorporating position encoding information; generate target movement trajectory information reflecting the dynamic movement law of the target according to the spatio-temporal graph network incorporating position encoding information.

[0025] Further, the obtaining of the target behavior feature information reflecting the abnormal behavior of the target according to the target movement trajectory information includes:

[0026] Perform slicing processing on the target movement trajectory information to obtain a movement segment data set containing n1 movement segments; design a temporal variational autoencoder including an encoder and a decoder, and train the temporal variational autoencoder on the movement segment data set; use the encoder in the trained temporal variational autoencoder to extract the mean vector of the new movement segment as the movement behavior feature vector of the movement segment;

[0027] Concatenate the movement behavior feature vectors extracted from each target on different movement segments into a behavior feature matrix, where each row of the behavior feature matrix corresponds to a target and each column corresponds to a behavior category; based on the behavior feature matrix, perform abnormal behavior discrimination on the behavior features of each target to generate an abnormal behavior confidence score matrix; according to a preset abnormal determination threshold, binarize the abnormal behavior confidence score matrix to obtain a target abnormal behavior detection matrix, where each element value of the target abnormal behavior detection matrix is 0 or 1, and define the target abnormal behavior detection matrix as the target behavior feature information.

[0028] Further, the slicing processing of the target movement trajectory information includes:

[0029] Set a sliding time window with a fixed length of L', slide in the time dimension with a step size of S, and cut the target movement trajectory information into a series of movement segments with a length of L'.

[0030] Further, the construction of the spatio-temporal topology graph between targets within the region includes:

[0031] Input the target category information and target motion trajectory information into a graph neural network to construct a spatial topology relationship graph among targets within a region; based on the spatial topology relationship graph, introduce the time dimension to construct a spatio-temporal topology graph; in the spatio-temporal topology graph, the states of the same target at different times are connected by temporal edges, and the relationships between different targets at the same time are connected by spatial edges.

[0032] Further, the obtaining of the environmental risk prediction result includes:

[0033] Input the environmental anomaly feature information into a pre-constructed gated recurrent unit network including an encoder part, extract the temporal feature representation of the environmental state through the encoder part of the gated recurrent unit network, and define it as the environmental state feature; perform dynamic weighting on the environmental state features at different times to obtain the significant environmental state features; based on the significant environmental state features, introduce an autoregressive structure in the encoder part of the gated recurrent unit network, and perform rolling estimation on the environmental risk state at future times through iterative prediction to obtain the environmental risk prediction result; the environmental risk prediction result includes a sequence of environmental risk prediction values within a preset future time.

[0034] Further, the construction of the spatio-temporal evolution model includes:

[0035] Based on the spatio-temporal topology graph and the environmental risk prediction result that integrate the regional abnormal behavior patterns, adopt the pyramid pooling method to generate a multi-scale spatio-temporal pyramid representation of the regional abnormal situation from bottom to top;

[0036] According to the multi-scale spatio-temporal pyramid representation of the regional abnormal situation, obtain the multi-level spatio-temporal correlation characteristics of the regional abnormal situation;

[0037] Based on the multi-level spatio-temporal correlation characteristics of the regional abnormal situation, construct a spatio-temporal evolution model.

[0038] Further, the generation of the graded early warning information for the regional abnormal situation according to the spatio-temporal evolution model includes: based on the spatio-temporal evolution model, construct a graded early warning framework for abnormal situations; under the graded early warning framework for abnormal situations, generate the graded early warning information for the regional abnormal situation in real time.

[0039] Further, the construction of the graded early warning framework for abnormal situations includes:

[0040] According to the regional abnormal behavior patterns in the spatio-temporal evolution model, set different levels of abnormal behavior early warning thresholds and construct an abnormal behavior graded early warning model;

[0041] According to the environmental risk prediction result in the spatio-temporal evolution model, set different levels of environmental risk early warning thresholds and construct an environmental risk graded early warning model;

[0042] Fuse the abnormal behavior grading and early warning model and the environmental risk grading and early warning model to construct an abnormal situation grading and early warning framework covering regional abnormal behaviors and environmental risks; the abnormal situation grading and early warning framework divides the early warning levels of abnormal situations into three levels: general warning, warning, and severe warning.

[0043] Furthermore, the early warning thresholds for different levels of abnormal behaviors include a low behavior risk threshold Risk1 and a high behavior risk threshold Risk2, where 0 < Risk1 < Risk2 < 1.

[0044] Furthermore, the real-time generation of the grading and early warning information for regional abnormal situations includes:

[0045] According to the target feature description information, calculate the real-time regional abnormal behavior risk value and the real-time environmental risk value at the current moment; input the real-time regional abnormal behavior risk value and the real-time environmental risk value into the abnormal situation grading and early warning framework to judge the early warning level of the regional abnormal situation at the current moment.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] The present invention forms a complete and efficient system from data collection, processing to abnormal early warning. In the data collection stage, visible light and infrared thermal imaging video streams are used to obtain multi-dimensional information, providing a rich data basis for subsequent analysis. Intelligent parsing of panoramic video sequences can effectively extract various abnormal features and generate detailed target feature description information, enhancing the perception ability of abnormal situations. Constructing a spatio-temporal topology map and combining environmental risk prediction to construct a spatio-temporal evolution model can deeply analyze the spatio-temporal correlation characteristics between targets, accurately detect abnormal behavior patterns, and predict environmental risks in advance. Finally, the grading and early warning information generated based on the spatio-temporal evolution model is intuitive and targeted. The overall solution effectively overcomes the deficiencies of traditional methods in data processing, abnormal recognition, environmental adaptability, and multi-source data fusion through multi-link collaboration, greatly improving the accuracy and real-time performance of abnormal early warning, providing a strong guarantee for the safety monitoring of large-scale scenarios such as urban management, industrial parks, transportation hubs, and border security, reducing the occurrence of safety accidents and potential losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1It is the principle flowchart of the real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection in the present invention;

[0050] Figure 2 It is the method flowchart for obtaining an abnormal feature enhanced video sequence in the real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection of the present invention;

[0051] Figure 3 It is the schematic diagram of the principle of a specific embodiment of the present invention;

[0052] Figure 4 It is the method flowchart for generating target feature description information in the real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection of the present invention;

[0053] Figure 5 It is the method flowchart for generating target motion trajectory information reflecting the dynamic movement law of the target in the real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection of the present invention;

[0054] Figure 6 It is the method flowchart for obtaining the environmental risk prediction result in the real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection of the present invention;

[0055] Figure 7 It is the method flowchart for constructing an abnormal situation grading early warning framework in the real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection of the present invention;

[0056] Figure 8 It is the functional module diagram of the real-time early warning system for regional abnormal situations based on high-point panoramic intelligent inspection in the present invention. Specific embodiments

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0058] Embodiment 1

[0059] Please refer to Figure 1 As shown, this embodiment provides a real-time early warning method for regional abnormal situations based on high-point panoramic intelligent inspection, including:

[0060] Step S1000, collect the visible light video stream and the infrared thermal imaging video stream of the target area to form a panoramic video sequence of the target area; perform intelligent analysis on the panoramic video sequence to obtain an enhanced abnormal feature video sequence; generate target feature description information based on the enhanced abnormal feature video sequence;

[0061] Further, step S1000 includes:

[0062] Step S1100, collect the visible light video stream and the infrared thermal imaging video stream of the target area to form a panoramic video sequence of the target area;

[0063] Specifically, use a dedicated panoramic view observation device, which has characteristics such as high-resolution imaging and wide-angle shooting, and can meet the observation requirements of large-scale scenes. When installing and deploying, it is necessary to select an appropriate height, position and shooting angle according to the characteristics of the specific scene. For example, in the traffic hub scene, install the panoramic view observation device on the monitoring tower, and set the height to a position that can cover the entire waiting hall and the main passage, and adjust the shooting angle to clearly capture the activity areas of people, luggage and vehicles.

[0064] The visible light video stream is mainly used to obtain the appearance morphological characteristics of the targets in the area. For example, in the urban management scene, through the visible light video stream, the appearance of buildings, the driving status of vehicles on the road, the behavior of pedestrians, etc. can be clearly seen. The infrared thermal imaging video stream is used to obtain the surface temperature distribution characteristics of the targets in the area. In the industrial park scene, the infrared thermal imaging video stream can be used to monitor the heat generation of equipment and timely detect abnormal temperature increases caused by equipment failures.

[0065] Through the collaborative observation of multi-source heterogeneous sensors (i.e., visible light cameras and infrared thermal imagers), this step comprehensively obtains multi-dimensional feature information such as vision and thermal radiation of the target area. The beneficial effects of this process are significant: First, the multi-dimensional information complements each other, improving the perception ability of the target area. For example, in the fire monitoring scene, the visible light video stream can detect the shape of the flame, and the infrared thermal imaging video stream can determine the temperature distribution of the flame. The combination of the two can more accurately judge the fire situation. Second, it provides a rich data basis for subsequent abnormal analysis, making subsequent abnormal detection and analysis more accurate and comprehensive. The fusion of multi-source data can reduce the errors and uncertainties of a single data source and enhance the adaptability and reliability of the system to complex scenes.

[0066] Step S1200, perform intelligent analysis on the panoramic video sequence to obtain an enhanced abnormal feature video sequence;

[0067] The purpose of step S1200 is to perform intelligent parsing on the panoramic video sequence to obtain an anomaly feature enhanced video sequence, so as to more clearly highlight the hidden anomaly features in the original data and create favorable conditions for subsequent semantic understanding and analysis.

[0068] Further, as Figure 2 shown, step S1200 includes:

[0069] Step S1210, using a deep learning-based flame recognition algorithm to analyze each frame of the visible light video stream, extract the flame regions in the video, and generate a flame anomaly enhanced video stream;

[0070] Specifically, deep learning is a type of machine learning technology based on artificial neural networks. By constructing a neural network model with multiple layers, the computer can automatically learn features and patterns from a large amount of data. In the flame recognition algorithm used in this step, the Convolutional Neural Network (CNN) structure is often used. CNN includes components such as convolutional layers, pooling layers, and fully connected layers. The convolutional layer performs convolutional operations by sliding the convolutional kernel over the image to extract local features in the image, such as the color, texture, etc. of the flame. Different convolutional kernels can extract different types of features. By stacking multiple convolutional layers, more advanced and abstract features can be gradually extracted. The pooling layer is used to downsample the features extracted by the convolutional layer, reduce the amount of data, and at the same time retain the main features, improving the computational efficiency and generalization ability of the model. The fully connected layer integrates the features after pooling and outputs the final classification result.

[0071] In the actual training process, a large amount of visible light video data containing flames and non-flames needs to be collected as the training set. These data cover flame situations in different scenarios, such as flames generated by different fuels, flames under different lighting conditions, etc., to ensure the wide applicability of the model. During training, the video data is input into the convolutional neural network frame by frame, and the model learns based on the input image features, continuously adjusting various parameters in the network (such as the weights of the convolutional kernels, etc.), so that the output result of the model is as close as possible to the actual label (whether it is a flame region). The training process usually uses the backpropagation algorithm to calculate the gradient of the loss function with respect to the network parameters and updates the parameters according to the gradient to minimize the loss function. The loss function generally uses cross-entropy loss, etc., to measure the difference between the model prediction result and the true label.

[0072] Take fire monitoring in urban management as an example. In the high-rise areas of the city, visible light video streams are obtained through surveillance cameras installed at high points. When a fire occurs, the flame recognition algorithm analyzes the video frame by frame. The algorithm first extracts the color features of the flame through the convolution layer, such as the red and orange tones of the flame; at the same time, it extracts the texture features of the flame, such as the irregular texture presented by the flickering edge of the flame. In addition, the motion features of the flame are also combined. The flame usually has an upward fluttering and swaying motion. Through spatiotemporal context information, that is, considering the position change and feature coherence of the flame in consecutive frames, the algorithm can more accurately identify the flame. For example, if a red area is detected in a certain frame, it may not be possible to determine whether it is a flame based on the color alone, but combined with the motion trajectory and feature changes of the area in the previous frames, if it is found that it presents the typical fluttering and flickering features of flames, it can be more accurately determined as a flame area.

[0073] The beneficial effects of this algorithm are: First, it improves the accuracy and timeliness of flame detection. Compared with traditional flame detection methods, deep learning algorithms can identify flames more accurately and reduce misjudgments and missed judgments. For example, in urban high-rise fire monitoring, traditional methods may not be able to accurately determine the location of the flame due to factors such as smoke obstruction, while deep learning algorithms can accurately detect the flame area by learning the flame characteristics under complex backgrounds. Second, it has strong adaptability. The deep learning algorithm can adapt to the changes in flame characteristics in different scenarios by continuously updating training data, such as the differences in flame characteristics produced by the combustion of different fuels. Third, it has good real-time performance and can meet the real-time monitoring needs of abnormal flame conditions, detect fire hazards in a timely manner, and buy time for fire rescue.

[0074] Step S1220, using a smoke recognition algorithm based on optical flow field analysis to segment the visible light video stream into moving targets, extracting the smoke area in the video, and generating a smoke anomaly enhanced video stream;

[0075] Specifically, the optical flow field refers to the motion vector field of pixel points caused by the movement of objects in an image, which reflects the motion information of objects in the image. The smoke recognition algorithm based on optical flow field analysis utilizes the distribution characteristics of the optical flow field of smoke movement, and realizes the accurate detection of the smoke area through multi-scale optical flow pyramid matching and adaptive threshold segmentation. Multi-scale optical flow pyramid matching is a method of constructing an image into a pyramid structure with different resolutions, and performing optical flow calculation and matching at different scales, which can take into account both the global and local information of the image and improve the ability to capture the motion characteristics of smoke. Adaptive threshold segmentation automatically determines the threshold for segmenting the smoke area according to the local characteristics of the image, improving the accuracy of segmentation. For example, when a fire occurs in a transportation hub, the smoke may spread rapidly, and this algorithm can accurately segment the smoke area from the complex background by analyzing the optical flow field generated by the smoke movement. Its beneficial effects are as follows: First, it can accurately identify smoke. Even when the smoke concentration is low and the contrast with the background is not high, the presence of smoke can be detected through the feature analysis of the optical flow field. Second, through multi-scale optical flow pyramid matching and adaptive threshold segmentation, the adaptability of the algorithm to smoke movement in different scenarios is improved, and it can work effectively both in indoor enclosed spaces and outdoor open areas. Third, it provides an important basis for fire warning and personnel evacuation. Detecting smoke in a timely manner allows relevant personnel to take measures in advance and reduce the losses caused by fires.

[0076] Step S1230, use a human behavior recognition algorithm based on skeleton point tracking to perform target detection and tracking on the visible light video stream, extract the skeleton key point coordinates of the target and the abnormal behaviors of the people in the video, and generate an enhanced video stream of human abnormal behaviors;

[0077] Specifically, first use a convolutional neural network (CNN) to achieve the precise positioning of human skeleton points. As mentioned above, CNN automatically extracts features in the image through components such as convolutional layers, pooling layers, and fully connected layers. In the task of human skeleton point positioning, the training data includes a large number of human body images in different poses and different scenarios, as well as the corresponding accurate skeleton key point annotations. During the training process, CNN learns features such as the shape, structure of the human body in the image and the spatial relationship between the skeleton key points. By continuously adjusting the network parameters, the position of the skeleton key points output by the network is made as close as possible to the actual annotation. For example, in the surveillance video of a transportation hub, CNN can accurately identify the positions of the skeleton key points such as the head, shoulders, elbows, and knees of each human body from the complex crowd scene. After obtaining the coordinates of the human skeleton key points, the abnormal behavior patterns of the people, such as staying, running, falling, etc., are identified through temporal correlation analysis. Temporal correlation analysis analyzes the coordinates of the skeleton key points of a person at different time points, and judges the behavior pattern according to their movement trajectory and posture change. Taking the judgment of a person staying as an example, such as Figure 3As shown in the figure, at time t, the coordinates of the skeleton key points of a certain person are (x1, y1, z1). After Δt time, the coordinates of the skeleton key points of this person are (x2, y2, z2). After calculation, the position change of the skeleton key points of this person is Δs. If Δs is lower than the preset position change threshold s1, it can be determined that this person is in a staying state. For the running behavior, the movement speed and trajectory change of the skeleton key points will show the characteristics of rapid movement and relatively continuity; while for the falling behavior, it is manifested as a sudden drastic change in the spatial relationship between the skeleton key points, such as a rapid drop in the body center of gravity and abnormal changes in the positions of the limbs, etc.

[0078] This algorithm has many beneficial effects. First, it can monitor human behavior in real time, detect abnormal behaviors in a timely manner, and ensure the safety of personnel. In transportation hubs, where there are a large number of people, if someone falls and it cannot be detected in time, it may lead to stampede accidents. This algorithm can quickly detect the falling behavior and issue an alarm to notify relevant personnel for rescue. Second, it improves the level of scene safety management. In places such as industrial parks, by identifying abnormal behaviors of personnel, such as entering a dangerous area without authorization and staying there, potential safety accidents can be prevented in advance. Third, it provides accurate information for subsequent emergency handling. When an abnormal behavior occurs, the system can quickly formulate corresponding emergency strategies according to the detected type of abnormal behavior and the position of the personnel, etc., improving the efficiency of emergency handling.

[0079] Step S1240, use an object feature extraction algorithm based on saliency detection to segment the foreground object in the infrared thermal imaging video stream, extract the hot spot target area, and generate a hot spot abnormal enhanced video stream;

[0080] Specifically, saliency detection is a technique for extracting salient objects (i.e., objects that attract human attention) from images. In this algorithm, by calculating the local contrast and global sparsity of infrared images, a saliency probability map of hot spot objects is adaptively constructed, and then the abnormal high-temperature target area is segmented. The local contrast reflects the degree of difference between a certain area in the image and the surrounding areas. For hot spot objects in an infrared thermal imaging video stream, their temperature is different from the surrounding environment, which is manifested as a difference in gray values or colors (false color infrared images) in the infrared image. By calculating the contrast between the pixel values in a local area and the pixel values in the surrounding neighborhood, areas with a large difference from the surrounding environment can be highlighted, and these areas are likely to be the locations of hot spot objects. For example, in an industrial park, the temperature of normally operating equipment is relatively stable, while equipment with a fault may experience local overheating, and its local contrast in the infrared image will be significantly higher than that of the surrounding normal equipment areas. The global sparsity measures the sparse distribution of objects in the entire image. In an infrared image, hot spot objects usually only occupy a small part of the entire image and have a certain degree of sparsity. By calculating the global sparsity, the location of hot spot objects can be further determined. When a certain area has a high local contrast and a sparse feature globally, then this area is more likely to be a real hot spot object. Based on the local contrast and global sparsity calculated above, the algorithm adaptively constructs a saliency probability map of hot spot objects. Specifically, the local contrast and global sparsity are weighted and fused, and appropriate weights are set according to the characteristics of different scenarios. For example, in some scenarios that are more sensitive to temperature changes, the weight of the local contrast may be appropriately increased; while in some scenarios with more interference factors, more attention may be paid to the weight of the global sparsity. A probability map is generated through the fused result, and the value of each pixel point in the map represents the probability that the point belongs to a hot spot object. Finally, foreground object segmentation is performed according to the saliency probability map. A suitable threshold is set, and when the probability value of a certain pixel point in the probability map is greater than this threshold, the area where the pixel point is located is determined as the hot spot object area.

[0081] The beneficial effects of this step are as follows: First, it can quickly locate hot spot objects and timely discover potential faults or dangers. In an industrial park, overheating of equipment may cause serious accidents such as fires. Through this algorithm, overheated equipment can be detected in time to avoid accidents. Second, by adaptively constructing a saliency probability map, the detection ability of hot spot objects in different scenarios is improved. Whether it is during the day or at night, indoors or outdoors, this algorithm can work effectively. Third, it provides important clues for equipment maintenance and fault troubleshooting. By accurately detecting the hot spot object area, maintenance personnel can quickly determine the location of the fault and improve the efficiency of equipment maintenance.

[0082] Step S1250, the flame abnormally enhanced video stream, the smoke abnormally enhanced video stream, the abnormal human behavior enhanced video stream, and the hotspot abnormally enhanced video stream constitute an abnormal feature enhanced video sequence.

[0083] Specifically, the flame abnormally enhanced video stream, the smoke abnormally enhanced video stream, the abnormal human behavior enhanced video stream, and the hotspot abnormally enhanced video stream together constitute an abnormal feature enhanced video sequence. These different types of abnormal feature enhanced video streams complement each other and comprehensively display the abnormal conditions in the target area. For example, in a fire accident, the flame abnormally enhanced video stream shows the location and scope of the fire, the smoke abnormally enhanced video stream reflects the spread of the smoke, the abnormal human behavior enhanced video stream presents the evacuation or abnormal actions of people, and the hotspot abnormally enhanced video stream highlights the high-temperature areas at the fire scene. The beneficial effects of this combination method are as follows: First, it provides a more comprehensive and intuitive display of abnormal information, facilitating operators to quickly understand the abnormal conditions in the target area. For example, in the border security scenario, the duty personnel can quickly grasp the abnormal conditions on the border line, including abnormal human activities and possible fire sources, by viewing the abnormal feature enhanced video sequence. Second, it provides a rich information basis for subsequent semantic understanding and analysis. Different types of abnormal features corroborate each other, improving the accuracy and reliability of abnormal analysis.

[0084] Step S1300, generate target feature description information according to the abnormal feature enhanced video sequence;

[0085] Furthermore, as Figure 4 shown, step S1300 includes:

[0086] Step S1310, input the flame abnormally enhanced video stream, the smoke abnormally enhanced video stream, and the hotspot abnormally enhanced video stream into a pre-trained deep convolutional neural network to generate target category information reflecting the target type attributes;

[0087] The deep convolutional neural network adopts a residual network structure; the target category information is an N-dimensional vector, where N is the total number of target types in the area.

[0088] Specifically, the deep convolutional neural network in step S1310 adopts a residual network structure. Taking ResNet-50 as an example, its network is stacked by multiple residual blocks. Each residual block contains two convolutional layers and a shortcut connection. The convolutional layer performs convolution operations by sliding the convolutional kernel on the image to extract features in the image, such as the color and shape of the flame, the form of the smoke, the thermal radiation characteristics of the hot spot target, etc. The shortcut connection directly connects the input to the output, avoiding the problem of gradient disappearance, enabling the network to learn deeper feature representations, and effectively alleviating the problem of network degradation. During the training process, this network is pre-trained on a large-scale multi-class abnormal target dataset. These datasets contain a large amount of image data of flames, smoke, hot spot anomalies and other targets in different scenarios. Through learning these data, the network can extract robust abnormal target feature representations. During training, the image data is input into the network, and the network learns according to the target features in the image, continuously adjusting the weights and other parameters of the convolutional layer in the network to make the result output by the network as close as possible to the actual target class label. Generally, the cross-entropy loss function is used to measure the difference between the network output and the true label. The gradient of the loss function with respect to the network parameters is calculated through the backpropagation algorithm, and the parameters are updated according to the gradient to minimize the loss function.

[0089] Taking the fire warning scenario in urban management as an example, after the video stream obtained from the high-point monitoring camera is pre-processed to generate a flame, smoke, hot spot anomaly enhanced video stream, it is input into the pre-trained ResNet-50 network. The input layer of the network receives these video stream data. After feature extraction by multiple residual blocks, the features of the flame, smoke, and hot spot targets are abstracted and integrated layer by layer. Then, the output of the network undergoes global average pooling to compress the information in the spatial dimension, obtaining a feature vector. Then, through the softmax layer, this feature vector is transformed into target class information in probability form, and an N-dimensional vector is output, where N is the total number of target types in the region. Each dimension of the vector corresponds to a target type, and the dimension value is the probability of the occurrence of that type of target. For example, the output vector [0.8, 0.1, 0.05, 0.05] indicates that there is an 80% probability of a flame in the current region, a 10% probability of smoke, a 5% probability of a hot spot anomaly, and a 5% probability of no abnormal target.

[0090] This step has many beneficial effects. First, it improves the accuracy of target category recognition. Through pre-training on large-scale data sets, the network has learned rich target features, can accurately distinguish different types of abnormal targets, and reduce misjudgments. In complex urban environments, there may be a variety of interference factors such as flames and smoke. The network can accurately identify real abnormal targets. Second, it enhances adaptability to different scenarios. Since the data set covers target data in a variety of scenarios, the deep convolutional neural network can still stably identify target categories when facing different lighting, weather and other conditions. Third, it provides a basis for subsequent abnormal analysis and processing. Accurate target category information helps to further analyze the nature and scope of the abnormal situation. For example, in an industrial park, after determining the type of abnormal target, the corresponding emergency response process can be initiated in a targeted manner.

[0091] Step S1320, enhancing the video stream according to the abnormal behavior of the personnel, and generating target motion trajectory information reflecting the dynamic movement law of the target;

[0092] Furthermore, if Figure 5 As shown, step S1320 includes:

[0093] Step S1321, extracting the skeleton key point coordinates of the target in each frame of the enhanced video stream of abnormal behavior of personnel;

[0094] Step S1322, based on the skeleton key point coordinates, through time series association and kinematic analysis, generate a motion trajectory curve of each target in the entire video;

[0095] Step S1323, parameterizing the motion trajectory curve of the personnel, extracting the position coordinates of the target at the key time step as the spatiotemporal anchor point, and constructing a motion trajectory feature matrix;

[0096] Step S1324, constructing a space-time graph network based on the motion trajectory feature matrix, with each space-time anchor point as a node of the space-time graph network;

[0097] Step S1325, introducing a spatiotemporal coordinate position coding mechanism, mapping the discrete position coordinates of the nodes to a continuous periodic feature space through sine and cosine functions, and obtaining a spatiotemporal graph network incorporating position coding information;

[0098] Step S1326, generating target motion trajectory information reflecting the dynamic movement law of the target based on the spatiotemporal graph network integrated with the position coding information.

[0099] The target motion trajectory information is:

[0100]

[0101] Among them, T is the target motion trajectory information, xij Denotes the abscissa in the position coordinates of the j-th target at the i-th key time step, y ij Denotes the ordinate in the position coordinates of the j-th target at the i-th key time step, t i Denotes the i-th key time step. The number of rows of matrix T is m×n, where m is the number of key time steps and n is the number of targets extracted at each time step. The number of columns is 3, which are the x coordinate, y coordinate, and timestamp respectively.

[0102] Specifically, temporal correlation refers to the temporal tracking of the corresponding skeleton points of the same human target in different frames to analyze the position changes in the time dimension. Kinematic analysis combines human kinematic constraints, such as the range of joint motion and the coordination of human movement, to optimize the spatial positions of the skeleton points. For example, in the urban square scenario, when a person is walking, the positions of the skeleton key points will change regularly over time. Through temporal correlation, the coordinates of the skeleton key points of this person in each frame are connected in chronological order to form a preliminary motion trajectory. Then, the Kalman filtering algorithm is used to smooth this preliminary trajectory. This algorithm can predict the current state based on the previous state and correct it by combining the observed value at the current moment, effectively reducing the influence of noise on the trajectory. At the same time, according to human kinematic constraints, the spatial positions of the skeleton points are adjusted to ensure that the motion trajectory conforms to the actual situation of human movement, and finally a smooth and continuous human motion trajectory curve is formed. The beneficial effect of this step is that the generated smooth and continuous motion trajectory curve can more accurately depict the dynamic behavior changes of the human target, providing a more reliable basis for subsequent analysis of normal and abnormal behaviors of the person. Through accurate trajectory description, the motion trend and behavior pattern of the person can be understood more clearly, which helps to detect abnormal behaviors in a timely manner. For example, in the border security scenario, accurate trajectory analysis can help identify the action routes of suspicious persons.

[0103] Step S1323 uses a parameterized curve model such as a B-spline curve to smoothly interpolate the discrete trajectory sampling points to obtain a continuous motion trajectory function. The B-spline curve is a piecewise defined polynomial curve with good local control characteristics and smoothness, and can fit discrete trajectory points well to obtain a continuous motion trajectory function. In the time dimension, the key time steps are uniformly or adaptively sampled according to actual needs. For example, when analyzing the flow of personnel in a transportation hub, the key time steps can be adaptively selected according to the change in personnel flow. At these key time steps, the position coordinates of the personnel target are extracted as time-space anchor points, and arranged in chronological order to form a motion trajectory feature matrix. Taking the waiting hall of a railway station as an example, during the peak hours when people enter and exit the station, the key time steps are encrypted and the position changes of the personnel are recorded in more detail; when there are fewer people, the sampling frequency is appropriately reduced. The motion trajectory feature matrix constructed in this way contains information about the personnel at different time and space positions, and provides a structured data representation for the subsequent analysis of the characteristics and laws of personnel movement, which helps to perform data analysis and behavior pattern recognition more efficiently.

[0104] In the spatiotemporal graph network, the node feature is the coordinate embedding representation of the spatiotemporal point. By converting the position coordinates into feature vectors suitable for network processing, the position information is retained while facilitating network learning and analysis. The edges between nodes are divided into temporal edges and spatial edges. The temporal edges connect the position nodes of the same person at adjacent time steps, which are used to model the motion correlation of a single target and reflect the continuity of the movement of the person in the time dimension. For example, in an industrial park, a worker walks from one workshop to another, and the temporal edges can reflect the relationship of his position change at different time points. The spatial edges connect the position nodes of different people at the same time step, which are used to model the interactive correlation of multiple targets and show the spatial position relationship and potential interactive behavior of different people at the same time. For example, in the security inspection area of ​​a transportation hub, multiple people undergo security inspection at the same time, and the spatial edges can reflect the relative positions and possible interactions between these people. By constructing such a spatiotemporal graph network, the movement trajectory information of people is converted into graph structure data, which facilitates the use of graph neural network and other technologies for more in-depth analysis and mining the spatiotemporal correlation characteristics and potential abnormal behavior patterns in the movement of people.

[0105] Step S1325 introduces a spatio-temporal coordinate position encoding mechanism, which maps the discrete position coordinates of nodes to a continuous periodic feature space through sine and cosine functions, obtaining a spatio-temporal graph network integrated with position encoding information. Since the original position coordinates are discrete numerical values, directly using them for network learning may not fully utilize the continuity and periodicity characteristics of position information. Sine and cosine functions have the characteristic of periodicity. Taking the position coordinates of nodes (such as the abscissa x and the ordinate y) as the independent variables of the sine and cosine functions respectively, the corresponding sine and cosine values are obtained. For example, for the abscissa x, calculate sin(x) and cos(x), and add these values as new features to the node features. In this way, the position encoding not only retains the numerical information of the position but also introduces the relative relationship and periodicity characteristics of the position, enabling the network to better understand the position information of nodes in space and the position differences between different nodes. In the urban management scenario, the personnel flow in different regions has certain periodic patterns, such as certain sections of the road being crowded during the peak commuting hours. Through position encoding, the network can better capture these patterns, providing richer information for analyzing the personnel flow pattern and predicting personnel behavior, enhancing the network's learning ability of spatio-temporal features, and improving the accuracy of analyzing the personnel movement trajectory.

[0106] Step S1326 generates target motion trajectory information reflecting the dynamic movement law of the target based on the spatio-temporal graph network integrated with position encoding information. In the spatio-temporal graph network, the information of nodes and edges includes the positions of personnel at different times and spaces and their correlation relationships. By analyzing the spatio-temporal graph network, the key information of the personnel movement trajectory can be extracted. For example, the position coordinates of each person at different key time steps can be obtained from the spatio-temporal graph network, and these coordinates are arranged in chronological order to form the target motion trajectory information as shown in matrix T.

[0107] Step S1330 obtains target behavior feature information reflecting the abnormal behavior of the target based on the target motion trajectory information;

[0108] Furthermore, Step S1330 includes:

[0109] Step S1331 performs slicing processing on the target motion trajectory information to obtain a motion segment data set containing n1 motion segments;

[0110] Set a sliding time window with a fixed length of L', which slides at a step size of S in the time dimension, and cuts the target motion trajectory information into a series of motion segments with a length of L'.

[0111] Specifically, taking the waiting hall of a railway station as an example, the movement trajectory information of people is recorded in a time series. Assuming that the length L' of the time window is set to 10 seconds and the step size S is set to 1 second, then the movement trajectory every 10 seconds is divided into a segment. Each movement segment represents the movement subsequence of a target within L' consecutive time steps, and there can be overlap between adjacent segments. This is done to enhance the context information. For example, in a scenario with complex personnel flow, the movement of a person within a certain time period may span multiple segments. The segment overlap can ensure that the coherence information of their movement is not missed during analysis, making the subsequent analysis of personnel behavior patterns more accurate. Through this slicing process, the continuous movement trajectory information is transformed into a segment data set that is convenient for analysis, laying a foundation for subsequent mining of personnel movement behavior characteristics. Its beneficial effect is that the complex continuous movement trajectory is decomposed into multiple shorter movement segments, reducing the complexity of analysis. At the same time, the segment overlap preserves the context information of the movement, helping to capture the behavior patterns of people more comprehensively and avoiding misjudgments caused by information loss. For example, in the border security scenario, after slicing the movement trajectories of people on the border line, even if the movement trajectories of people are interrupted or have complex changes, their behavior intentions can be accurately analyzed through the overlapping segments.

[0112] Step S1332: Design a temporal variational autoencoder including an encoder and a decoder, and train the temporal variational autoencoder on the movement segment data set;

[0113] Specifically, after obtaining the motion segment dataset, in step S1332, a temporal variational autoencoder including an encoder and a decoder is designed and trained on this dataset. The encoder uses a multi-layer gated recurrent unit (GRU) to sequentially encode the motion segments and extract their temporal features. GRU is a variant of the recurrent neural network (RNN). It controls the flow of information through a gating mechanism and can effectively handle the long-term dependence problem in time series data. In this step, the multi-layer GRU processes the data of each time step in the motion segment in sequence to learn the variation characteristics of the motion segment in the time dimension. Based on the hidden state of the last time step, the mean vector and the diagonal covariance matrix of the probability distribution in the latent space are predicted through two fully connected layers respectively. The fully connected layer is a common layer in the neural network. It connects all neurons in the previous layer to each neuron in the next layer and can integrate the input feature information. The decoder also uses a multi-layer GRU, takes the sampled latent variable as the initial hidden state, and decodes and generates the original motion sequence step by step in an autoregressive manner. The autoregressive manner means that when generating the output of the current time step, it depends on the output results of the previous time steps. During the training process, the stochastic gradient descent algorithm is adopted. The model learns the low-dimensional latent space representation of the motion behavior by minimizing the mean square error between the reconstructed motion segment and the original segment, and minimizing the KL divergence between the latent space distribution and the standard normal distribution. The mean square error is used to measure the similarity between the reconstruction result and the original data. By minimizing the mean square error, it is ensured that the latent variable can reconstruct the original motion segment well; the KL divergence measures the difference between two probability distributions. By minimizing the KL divergence, the latent space distribution is close to the smooth prior distribution, endowing the model with good generation ability. The beneficial effect of this step is that through the learning of the temporal variational autoencoder, the key features of the motion segment can be extracted and mapped to the low-dimensional latent space, reducing the data dimension and improving the subsequent analysis efficiency. At the same time, the good reconstruction ability and generation ability enable the model to better capture the laws of motion behavior, which is of great significance for the detection of abnormal behaviors.

[0114] Step S1333, use the encoder in the trained temporal variational autoencoder to extract the mean vector of the new motion segment as the motion behavior feature vector of the motion segment;

[0115] Specifically, for the new motion segment data, it is input into the trained temporal variational autoencoder, and the mean vector and covariance matrix of the latent space distribution are obtained through forward calculation. Then, the mean vector is extracted as the behavioral feature description of the motion segment, and the covariance matrix reflects the uncertainty of this feature representation. For example, in the scenario of a transportation hub, a new segment of personnel motion data is obtained and input into the encoder. The encoder processes this segment through the previously learned feature extraction method to obtain the relevant parameters of the latent space distribution. The mean vector contains the main feature information of the motion segment, such as the trend information of the speed and direction of personnel movement, and can represent the behavioral characteristics of the motion segment. The covariance matrix reflects the correlation and uncertainty between these features. For example, if a certain element value in the covariance matrix is large, it indicates that there is a strong correlation between the corresponding two features, and at the same time, it also reflects that there is a certain degree of uncertainty in this feature representation. In this way, the complex motion segment data is converted into a concise feature vector, which is convenient for subsequent analysis and processing. The beneficial effect is that the extracted mean vector, as the motion behavior feature vector, can effectively represent the core features of the motion segment, reduce data redundancy, and improve the efficiency of abnormal behavior analysis. The uncertainty information provided by the covariance matrix helps to more comprehensively evaluate the reliability of the features and can more accurately grasp when judging abnormal behaviors.

[0116] Step S1334, splicing the motion behavior feature vectors extracted from each target on different motion segments into a behavior feature matrix, where each row of the behavior feature matrix corresponds to a target and each column corresponds to a behavior category;

[0117] Specifically, assume that in the monitoring of a certain area of urban management, there are a total of M2 targets (such as M2 pedestrians) and K predefined behavior categories (such as staying, running, falling, etc.). The motion sequences of each target are divided into motion segments, and a D-dimensional motion behavior feature vector is extracted from each segment. Taking a certain pedestrian as an example, his motion sequence is divided into multiple segments, and a D-dimensional motion behavior feature vector is extracted from each segment through step S1333. The motion behavior feature vectors of all segments of this pedestrian are averaged to obtain the final motion behavior feature vector of this pedestrian, which comprehensively reflects the motion behavior characteristics of this pedestrian at different time periods. The behavior feature vectors of all M2 targets are concatenated row by row to form a behavior feature matrix of M2×D. Finally, through a pre-trained behavior classifier, the D-dimensional features are mapped to the probability responses of K behavior categories, obtaining an abnormal behavior probability matrix of M2×K. The pre-trained behavior classifier is trained based on a large number of labeled abnormal behavior samples and can judge the probability of belonging to different behavior categories according to the input feature vector. For example, by processing the behavior feature vector of a certain pedestrian through this classifier, the probability of belonging to the "staying" behavior category is 0.8, and the probability of belonging to the "normal walking" behavior category is 0.2, etc. The beneficial effect of this step is that by constructing the behavior feature matrix and the abnormal behavior probability matrix, the motion behavior features of each target are systematically integrated and quantified, facilitating the subsequent unified analysis and judgment of the abnormal behaviors of a large number of targets. In the industrial park, the management personnel can quickly understand the behavior of each worker through these matrices, timely discover potential abnormal behaviors, and ensure production safety.

[0118] Step S1335: Based on the behavior feature matrix, perform abnormal behavior discrimination on the behavior features of each target to generate an abnormal behavior confidence score matrix;

[0119] Specifically, a binary or multi-class support vector machine model (SVM) is adopted and trained on manually labeled abnormal behavior samples to learn the discriminant hyperplanes of different behavior classes. SVM is a commonly used machine learning model that separates data of different classes by finding an optimal classification hyperplane. In this step, the SVM is trained with a large number of manually labeled abnormal behavior samples (such as samples labeled as "normal behavior" and "abnormal behavior", or samples specifically labeled as different abnormal behavior classes like "loitering" and "running") so that it learns the boundary features between different behavior classes. The behavior feature vector of each target is input into the SVM model, the distance between it and the hyperplane of each behavior class is calculated, and it is converted into a confidence score of abnormal behavior. For example, for the behavior feature vector of a target, the distance from it to the hyperplane of the "loitering" behavior class is d1, and the distance from it to the hyperplane of the "normal walking" behavior class is d2. These distances are converted into confidence scores through a specific conversion function (such as a distance-based normalization function). Finally, an M2×K abnormal behavior confidence score matrix is output, reflecting the likelihood of each target belonging to various abnormal behavior classes. Its beneficial effect is that by using the classification ability of the SVM model, it can accurately discriminate the abnormal behavior of the target's behavior features, and the generated confidence score matrix provides a quantitative basis for the judgment of abnormal behavior. In the border security scenario, suspicious persons can be quickly screened out through this matrix, improving the security efficiency.

[0120] Step S1336, according to a preset abnormal determination threshold, binarize the abnormal behavior confidence score matrix to obtain a target abnormal behavior detection matrix. Each element of the target abnormal behavior detection matrix takes a value of 0 or 1, and the target abnormal behavior detection matrix is defined as target behavior feature information.

[0121] Specifically, different scenarios have different requirements for the sensitivity and detection accuracy of abnormal behaviors, and appropriate thresholds need to be set according to specific circumstances. For example, in urban management, for the detection of abnormal behaviors of people in important places, a relatively low threshold may be set to ensure the timely discovery of potential risks; while in some ordinary areas, the threshold can be appropriately increased to reduce false alarms. Threshold processing is independently performed on each column (corresponding to a behavior category) of the abnormal behavior confidence score matrix. Elements with a confidence higher than the threshold are assigned a value of 1, indicating that the corresponding target has this type of abnormal behavior; otherwise, they are assigned a value of 0, indicating that the abnormal behavior does not exist. For example, in an industrial park, for the abnormal behavior category of "staying in a dangerous area without authorization", the threshold is set to 0.6. If the confidence score of a certain worker's abnormal behavior is 0.7, the corresponding element in the target abnormal behavior detection matrix is assigned a value of 1; if the score is 0.5, it is assigned a value of 0. After binarization, a 0-1 matrix of M2×K is obtained. Each row represents the abnormal behavior detection vector of a target, and each element represents whether the target has the corresponding abnormal behavior. This matrix is the final target abnormal behavior recognition result. Its beneficial effect is that through binarization processing, complex confidence information is converted into a simple and intuitive 0-1 matrix, which is convenient for quickly identifying and counting the abnormal behaviors of targets.

[0122] Step S1340, according to the target category information, target motion trajectory information, and infrared thermal imaging video stream, obtain environmental abnormal feature information reflecting the abnormal state of the environment;

[0123] Specifically, step S1340 aims to obtain environmental anomaly feature information reflecting the abnormal state of the environment based on the target category information, target motion trajectory information, and infrared thermal imaging video stream. First, the target category information clarifies the types of targets existing in the area and their corresponding probabilities. For example, in an industrial park scenario, it may show that the probability of the existence of a fire is 80%, indicating a potential risk of fire in this area; the target motion trajectory information records the position coordinates of the target at different time steps. For instance, in a transportation hub, by analyzing the movement trajectories of people, it can be determined whether people stay in non-designated areas or gather rapidly. For the infrared thermal imaging video stream, it can reflect the surface temperature distribution characteristics of objects. By integrating these pieces of information and using a temperature anomaly analysis algorithm, the temperature distribution patterns in different areas under normal circumstances are compared, as well as the correlation between the target motion trajectory and the temperature anomaly area. If it is found that the temperature in a certain area rises abnormally, and there is abnormal aggregation in the movement trajectories of people around this area, combined with the probability of the existence of high-temperature objects (such as fire sources) in the target category information, it can be inferred that there may be a fire hazard in this area. Then, information such as the location of the temperature anomaly area, the temperature change trend, the degree of anomaly, and the correlation with the target motion trajectory is used as the environmental anomaly feature information. This way of obtaining environmental anomaly feature information can comprehensively and accurately perceive the environmental state. By integrating multi-source information, the limitations of single-information judgment are avoided, and the recognition accuracy of the abnormal environmental state is improved. Taking urban management as an example, when there is abnormal aggregation of people and accompanied by abnormal temperature rise in a certain area, accurate environmental anomaly feature information can enable managers to promptly determine whether a fire or other dangerous events have occurred, so as to take corresponding measures in advance to ensure the safety of personnel and the normal operation of facilities. At the same time, accurate environmental anomaly feature information is also helpful for the subsequent analysis and processing of abnormal events, provides strong support for formulating targeted solutions, and improves the overall management efficiency and safety.

[0124] Step S1350 constitutes the target feature description information by using the target category information, target motion trajectory information, target behavior feature information, and environmental anomaly feature information.

[0125] Specifically, the target category information is obtained by analyzing the abnormal enhanced video stream such as flames, smoke, and hot spots through a pre-trained deep convolutional neural network, and is presented in the form of a vector. Each dimension of the vector corresponds to a target type and its occurrence probability. For example, in the urban management scenario, through this information, it can be determined that the probability of the existence of flames in a certain area is 80%, and the probability of the existence of smoke is 10%, etc. This enables the operator to quickly understand the possible abnormal target types in the area. The target movement trajectory information is generated based on the video stream of abnormal human behavior through a series of operations such as extracting the coordinates of the skeleton key points, temporal correlation analysis, and parametric representation. The target behavior feature information is obtained through steps such as slicing the target movement trajectory information, training a temporal variational autoencoder, and constructing a behavior feature matrix. The environmental anomaly feature information is generated after processing the target category information, target movement trajectory information, and infrared thermal imaging video stream by a multi-scale semantic segmentation network, and is used to describe the abnormal state of the environment. Integrating these information to form the target feature description information can comprehensively and meticulously depict the abnormal situation in the area. For example, in the urban management scenario, combining the target category information, it can be known that there is a fire risk (high probability of flames) in a certain area. The target movement trajectory information shows that the surrounding people are quickly evacuating. The target behavior feature information indicates that some people have panic behavior. The environmental anomaly feature information points out that the fire has caused the surrounding temperature to rise and smoke to spread. These information are interrelated and complementary, providing a complete data basis for subsequent abnormal correlation analysis and early warning. Its beneficial effects are as follows. On the one hand, it provides a unified information framework for the comprehensive analysis of regional abnormal situations. Integrating abnormal information from different aspects together avoids the dispersion and isolation of information, and helps to more deeply understand the essence and scope of influence of abnormal events. In the border security scenario, comprehensively analyzing various information can more accurately judge whether the abnormal situation on the border is ordinary personnel activities or illegal border crossing behavior. On the other hand, it improves the efficiency of abnormal early warning and processing. Through a comprehensive target feature description information, relevant personnel can quickly obtain comprehensive abnormal information, make decisions in a timely manner, and take corresponding measures. For example, when an abnormality occurs at a transportation hub, resources can be quickly dispatched to ensure the safety and order of personnel.

[0126] Step S2000: According to the target feature description information, construct a spatio-temporal topology graph among the targets in the area, analyze the spatio-temporal correlation characteristics among the targets, and detect the regional abnormal behavior patterns; Embed the detected regional abnormal behavior patterns into the spatio-temporal topology graph as regional abnormal features; Construct a temporal abnormal detection framework for the regional environmental state to obtain the environmental risk prediction result; Based on the spatio-temporal topology graph and the environmental risk prediction result that incorporate the regional abnormal behavior patterns, construct a spatio-temporal evolution model;

[0127] Further, step S2000 includes:

[0128] Step S2100: Based on the target feature description information, construct a spatio-temporal topology graph among the targets within the region, analyze the spatio-temporal correlation characteristics among the targets, and detect the abnormal behavior patterns in the region; Embed the detected abnormal behavior patterns in the region as regional abnormal features into the spatio-temporal topology graph.

[0129] Further, step S2100 includes:

[0130] Step S2110: Input the target category information and the target motion trajectory information into a graph neural network to construct a spatial topology relationship graph among the targets within the region.

[0131] Step S2120: On the basis of the spatial topology relationship graph, introduce the time dimension to construct a spatio-temporal topology graph; In the spatio-temporal topology graph, the states of the same target at different times are connected by temporal edges, and the relationships among different targets at the same time are connected by spatial edges.

[0132] Step S2130: Apply a graph convolutional neural network on the spatio-temporal topology graph to detect the abnormal behavior patterns in the region.

[0133] Step S2140: Embed the detected abnormal behavior patterns in the region as regional abnormal features into the spatio-temporal topology graph.

[0134] Specifically, in step S2110, the graph neural network uses a message passing mechanism to model the interactions among the targets. The message passing mechanism is a core concept in graph neural networks, which updates the feature representations of nodes by transmitting information between nodes. In this step, the target category information and the target motion trajectory information provide the initial data for the graph neural network. For example, in the scenario of a transportation hub, the target category information includes different target types such as people and vehicles, and the target motion trajectory information records their position changes in space. Each target serves as a node in the graph, and the initial features of the node are determined by the target category and location information. The edges between nodes represent the potential relationships among the targets. The message passing mechanism enables each node to receive information from adjacent nodes and update its own features based on this information. For instance, the node of a vehicle can receive information transmitted by the nodes of surrounding vehicles and pedestrians, thereby updating its own feature representation regarding the surrounding traffic conditions. In this way, the graph neural network can learn the spatial topology relationships among the targets and construct a spatial topology relationship graph. The beneficial effect of this step is to transform complex target information into a graph structure, facilitating the subsequent analysis of the interaction relationships among the targets.

[0135] Based on the spatial topology relation graph constructed in step S2110, step S2120 introduces the time dimension to construct a spatio-temporal topology graph. In the spatio-temporal topology graph, the states of the same target at different times are connected by temporal edges, and the relationships between different targets at the same time are connected by spatial edges. Taking an industrial park as an example, assume that at a certain moment, multiple workers are active in different working areas, and the positions and behaviors of these workers constitute a spatial topology relation, which is connected by spatial edges. As time goes by, the position of each worker changes, and the same worker nodes at different times are connected by temporal edges. In this way, the spatio-temporal topology graph not only contains the distribution and interaction information of the targets in space, but also incorporates the change information in the time dimension. In this way, the dynamic behavior of the targets can be described more comprehensively. For example, when analyzing the work process of workers, the activity trajectories of workers in each working area at different time points can be viewed through the spatio-temporal topology graph to determine whether they operate according to the specified process. The beneficial effect is that the spatio-temporal topology graph provides an intuitive model for analyzing the spatio-temporal correlation characteristics of the targets.

[0136] In step S2130, a graph convolutional neural network is applied to the spatio-temporal topology graph to detect regional abnormal behavior patterns. The graph convolutional neural network is a neural network specifically designed for processing graph data, and it extracts features on the graph structure through convolutional operations. In the spatio-temporal topology graph, the graph convolutional neural network jointly learns the spatio-temporal dependence relationships between targets in an end-to-end manner to mine regional abnormal behavior patterns, such as abnormal aggregation, sudden events, etc. Abnormal aggregation refers to the formation of a highly correlated sub-graph structure by multiple targets on the spatio-temporal topology graph. Taking the square scene in urban management as an example, under normal circumstances, the distribution of people in the square is relatively uniform, and the nodes and edges in the spatio-temporal topology graph show a certain regularity. When abnormal aggregation occurs, multiple person nodes are closely connected in a certain area at a certain moment, forming a highly correlated sub-graph structure. The graph convolutional neural network can identify such abnormal structures by learning a large amount of spatio-temporal topology graph data in normal and abnormal situations. It slides the convolutional kernel on the graph, aggregates and transforms the features of the nodes and their neighborhoods, and extracts the features reflecting abnormal behaviors. For example, by analyzing the movement trajectories and position information of people, it is judged whether there is a situation where a large number of people suddenly gather at a certain location and stay for a long time. The beneficial effect of this step is that it can automatically detect abnormal behavior patterns in complex scenarios and improve the accuracy and efficiency of anomaly detection.

[0137] When the graph convolutional neural network detects abnormal behavior patterns, such as multiple workers gathering in a non-working area for a long time (abnormal gathering pattern), or abnormal frequent personnel activities around a certain device (abnormal pattern related to equipment failure), these abnormal behavior patterns will be converted into specific feature representations. Then these features are embedded into the spatio-temporal topology graph. For example, a special mark or attribute is given to the sub-graph structure representing abnormal gathering, so that it can be clearly distinguished in the spatio-temporal topology graph. The beneficial effect of doing so is to make the spatio-temporal topology graph more prominent in abnormal information, facilitating subsequent analysis and decision-making. In urban management, staff can quickly locate abnormal areas and abnormal behaviors by viewing the spatio-temporal topology graph embedded with abnormal features and take corresponding measures in a timely manner.

[0138] Step S2200, based on the environmental abnormal feature information in the target feature description information, obtain the environmental risk prediction result;

[0139] Furthermore, as Figure 6 shown, step S2200 includes:

[0140] Step S2210, input the environmental abnormal feature information into the pre-constructed gated recurrent unit network including the encoder part, and extract the temporal feature representation of the environmental state through the encoder part of the gated recurrent unit network, defined as the environmental state feature;

[0141] Step S2220, dynamically weight the environmental state features at different times to obtain the significant environmental state features;

[0142] Step S2230, based on the significant environmental state features, introduce an autoregressive structure in the encoder part of the gated recurrent unit network, and perform rolling estimation on the environmental risk state at future times through iterative prediction to obtain the environmental risk prediction result; the environmental risk prediction result includes a sequence of environmental risk prediction values within a future preset time.

[0143] Specifically, in step S2210, the gated recurrent unit network is a special type of recurrent neural network (RNN) that can effectively handle the long-term dependence problem in time series data. In this step, the environmental anomaly feature information is obtained after the multi-scale semantic segmentation network processes the target category information, target motion trajectory information, and infrared thermal imaging video stream in the previous steps. This information contains the characteristics of various abnormal situations in the regional environment. The encoder of the GRU network controls the flow and memory of information through gating mechanisms, including the reset gate and the update gate. The reset gate determines how to combine the newly input information with past memories, and the update gate controls the influence degree of past memories and current inputs on the output. When processing the environmental anomaly feature information, the GRU network processes the input data step by step in time, and extracts the characteristics of the environmental state changing over time by continuously updating the hidden state. For example, in the industrial park scenario, the environmental anomaly feature information may include data such as abnormal changes in equipment temperature and changes in smoke concentration. The encoder of the GRU network will learn features such as the trend of abnormal increase in equipment temperature and the time pattern of the appearance and spread of smoke based on the time series of this data. These features are integrated into the environmental state features. In this way, key temporal features can be extracted from complex environmental anomaly data, providing strong support for subsequent environmental risk analysis. Its beneficial effect is that, compared with traditional time series analysis methods, the GRU network can better capture the long-term dependence relationships in environmental data, improving the ability to understand and grasp the changing trends of the environmental state.

[0144] Step S2220 dynamically weights the environmental state features at different times to obtain the significant environmental state features. Here, the attention mechanism is adopted, which assigns higher weights to significant features by calculating the similarity between environmental state features. In practical applications, the influence degrees of environmental state features at different times on environmental risks are different. For example, in the transportation hub scenario, during the morning and evening rush hours, abnormal changes in traffic flow have a more significant impact on environmental risks; while during the off-peak hours, abnormalities caused by some equipment failures may have a greater impact on environmental risks. The attention mechanism can automatically identify the features that are more important for the current environmental risk assessment by calculating the similarity between environmental state features at each time. Specifically in the calculation, the environmental state features are usually mapped to a high-dimensional space, and the similarity scores between them are calculated through operations such as dot product, and then the softmax function is used to convert the scores into weight values. These weight values are used to perform weighted summation on the environmental state features at different times, thus highlighting the significance of critical moments. The features after weighting are the significant environmental state features, which more concentratedly reflect the information that has an important impact on environmental risks. This dynamic weighting method can more accurately focus on key environmental changes and improve the accuracy of environmental risk assessment.

[0145] The autoregressive structure is a time series prediction model that uses the historical data of a time series to predict future values, taking the prediction result at the current moment as the input for the next moment's prediction. In this step, based on the significant features of the environmental state obtained previously, the encoder of the gated recurrent unit network starts to work. The significant features of the environmental state may include information such as the temperature change trend in a specific area, the change in smoke concentration, and the operating status of equipment. During the iterative prediction process, first, the significant features of the environmental state at the current moment are processed through the encoder of the gated recurrent unit network to obtain the predicted value of the environmental risk at the current moment. For example, at a certain moment, based on the rising temperature trend and smoke concentration data in the chemical industrial area of an industrial park, a preliminary value of the fire risk in this area within the next half hour is predicted. Then, this predicted value is used as the input for the next moment's prediction, combined with the newly obtained significant features of the environmental state at the next moment, and processed again through the encoder of the gated recurrent unit network to obtain the predicted value of the environmental risk at the next time step. This cycle continues, constantly updating the predicted value to achieve a rolling estimation of the environmental risk state within a preset future time, and finally obtaining a sequence of environmental risk predicted values. This rolling estimation method based on the autoregressive structure has many beneficial effects. From the perspective of time continuity, by continuously taking the prediction result of the previous moment as the input for subsequent predictions, it fully considers the time correlation of the environmental risk state, making the prediction result more in line with the dynamic changes of the actual situation. In the urban management scenario, the personnel flow and facility operating status at transportation hubs are constantly changing, and the autoregressive structure can continuously track these changes to improve the accuracy of prediction. From the perspective of risk prevention, by obtaining a sequence of predicted values of environmental risks within a preset future time in advance, managers can take corresponding preventive measures according to the prediction results. For example, in the border security scenario, if it is predicted that there may be a risk of abnormal personnel gathering in a certain area within the next few hours, police deployment can be arranged in advance to prevent potential security incidents. In addition, for the formulation of management decisions in large-scale scenarios, a detailed sequence of predicted values of environmental risks provides data support for formulating scientific and reasonable response strategies, helps optimize resource allocation, improve management efficiency, and reduce losses caused by potential risks.

[0146] Step S2300, based on the spatio-temporal topology graph integrating regional abnormal behavior patterns and the environmental risk prediction results, use the pyramid pooling method to generate a multi-scale spatio-temporal pyramid representation of the regional abnormal situation from bottom to top;

[0147] Specifically, pyramid pooling is an image processing and feature extraction technique that generates multi-scale feature representations by downsampling data at different scales. In this step, the spatio-temporal topology map and the environmental risk prediction results are used as the basic data. For example, in a commercial area of a city, the spatio-temporal topology map shows the movement trajectories of people and vehicles and the relationships between them, and the environmental risk prediction results may include the fire risk, crowd congestion risk, etc. in this area. Starting from the most detailed data at the bottom layer, following the rules of pyramid pooling, the resolution of the data is gradually reduced, and the small-scale local information is merged into large-scale global information. For example, the information of people and vehicles in the shooting areas of several adjacent cameras is merged to generate more macroscopic regional features. Through this bottom-up method, spatio-temporal pyramid representations at different scales are generated, and each scale contains information on regional anomalies at different levels. The beneficial effect of this method is that it can observe regional anomalies from multiple scales, taking into account both local details and global features. Small-scale information can help detect local abnormal behaviors, such as vehicle congestion or abnormal aggregation of people at a certain intersection; large-scale information can grasp the overall situation of the entire area, such as the overall pedestrian flow density and distribution in the commercial area. Through the multi-scale spatio-temporal pyramid representation, the regional anomalies can be understood more comprehensively, providing richer data support for subsequent analysis and decision-making.

[0148] Step S2400, according to the multi-scale spatio-temporal pyramid representation of the regional anomalies, obtain the multi-level spatio-temporal correlation characteristics of the regional anomalies;

[0149] Specifically, at each layer of the pyramid, the abnormal correlation characteristics within the local scope of the region are modeled through a graph attention network. The graph attention network is a graph neural network based on the attention mechanism. It can automatically assign different weights according to the correlation between nodes, highlighting the information of important nodes and edges. In the traffic hub scenario, taking the spatio-temporal pyramid data at a certain level as an example, the graph attention network will analyze the relationship between each node (such as the position of a vehicle or pedestrian at a certain moment) and the surrounding nodes. For example, if a vehicle suddenly decelerates, the graph attention network will pay attention to the driving states of other vehicles around the vehicle, as well as the distance and speed differences between these vehicles. By calculating the similarity and importance between nodes, weights are assigned to different edges, thereby modeling the abnormal correlation characteristics within the local scope. Cross-level fusion of the hierarchical features of the pyramid realizes the correlated understanding of regional abnormal situations at different spatio-temporal scales through two-way information propagation from top to bottom and from bottom to top. Top-down propagation transmits macroscopic information from the high layer to the low layer of the pyramid, helping the low layer to better understand the overall trend; bottom-up propagation transmits detailed local information from the low layer to the high layer, enriching the high layer's cognition of details. For example, when the high layer discovers a congestion trend in a certain area (macroscopic information), through top-down propagation, the low layer can more specifically analyze the specific behaviors of vehicles and pedestrians within that area; when the low layer discovers abnormal vehicle aggregation at a certain intersection (local information), through bottom-up propagation, the high layer can further evaluate the impact of this local anomaly on the entire traffic hub. This analysis method of multi-level spatio-temporal correlation characteristics has important beneficial effects. First, it can more accurately identify the root cause and scope of influence of abnormal situations. By analyzing the correlation between different nodes and the fusion of information at different scales, it can be clearly judged whether an abnormal behavior is an isolated event or a key event that will trigger a chain reaction. Second, it helps to detect potential abnormal risks in advance. By analyzing the spatio-temporal correlation characteristics, some abnormal situations that seem independent but actually have potential correlations can be discovered, so as to take preventive measures in advance. From the perspective of the reasoning process, the graph attention network weights nodes and edges through the attention mechanism, highlighting key information and making the local abnormal correlation characteristics more obvious; while the two-way information propagation realizes a comprehensive understanding of regional abnormal situations at different spatio-temporal scales through information interaction between different levels, thus improving the analysis and prediction ability of abnormal situations.

[0150] Step S2500, construct a spatio-temporal evolution model based on the multi-level spatio-temporal correlation characteristics of regional abnormal situations.

[0151] Specifically, the multi-level spatio-temporal correlation characteristics are obtained through the analysis of regional anomalies at different spatio-temporal scales. At each layer of the pyramid, a graph attention network is used to model the local anomaly correlation characteristics within the region, and abnormal patterns such as target aggregation and abnormal movement trajectories are mined at different spatio-temporal scales. At the same time, through two-way information propagation from top to bottom and from bottom to top, the hierarchical features of the pyramid are fused across levels to achieve an associated understanding of regional anomalies at different spatio-temporal scales and obtain comprehensive and detailed anomaly information. When constructing a spatio-temporal evolution model based on these multi-level spatio-temporal correlation characteristics, the abnormal correlation characteristics at different levels are used as input elements of the model. For example, in the urban management scenario, when it is detected that there is a personnel gathering in a certain area within a short period of time (obtained through the analysis of local anomaly correlation characteristics at the spatial scale), and the environmental temperature in this area (the environmental anomaly characteristics obtained from the infrared thermal imaging video stream, which is part of the spatio-temporal correlation characteristics) also shows an abnormal increase, and at the same time, combined with the occurrence pattern of similar situations in this area in the past at the time scale (the time correlation information obtained through cross-level fusion), these information are integrated into the model. The construction of this spatio-temporal evolution model has many beneficial effects. First, it describes the propagation and evolution law of abnormal events within the region from two dimensions of time and space, providing a comprehensive and accurate basis for subsequent anomaly early warning and decision-making. By integrating the spatio-temporal correlation characteristics at different levels, the model can more accurately capture the characteristics and development trends of abnormal events, avoiding the limitations of single-dimensional analysis. Second, it helps to improve the management and monitoring efficiency of large-scale scenarios. For scenarios such as urban management and industrial parks, resource allocation can be planned in advance. For example, when it is predicted that there may be a fire risk in a certain area, fire trucks and rescue personnel can be arranged to standby in advance to reduce the losses caused by disasters.

[0152] Step S3000, generate graded early warning information for regional abnormal situations according to the spatio-temporal evolution model.

[0153] Furthermore, step S3000 includes:

[0154] Step S3100, construct a graded early warning framework for abnormal situations based on the spatio-temporal evolution model;

[0155] Furthermore, step S3100 includes:

[0156] Step S3110, set early warning thresholds for different levels of abnormal behaviors according to the regional abnormal behavior patterns in the spatio-temporal evolution model, and construct a graded early warning model for abnormal behaviors;

[0157] The early warning thresholds for different levels of abnormal behaviors include a low behavior risk threshold Risk1 and a high behavior risk threshold Risk2, where 0 < Risk1 < Risk2 < 1.

[0158] Specifically, regional abnormal behavior patterns include abnormal aggregation, sudden events, etc. The early warning threshold for abnormal behavior is used to divide the risk levels of abnormal behavior at different degrees, where the low behavior risk threshold Risk1 and the high behavior risk threshold Risk2 satisfy 0 < Risk1 < Risk2 < 1. When setting these two thresholds, various factors need to be considered comprehensively. First, conduct statistical analysis on historical data to obtain relevant data indicators under different abnormal behavior patterns. For example, count data such as the occurrence frequency, duration, and impact of personnel aggregation events at different levels in a railway station over a certain period of time. Then, determine the thresholds in combination with the actual situation and management requirements of the current scenario. If Risk1 is set too low, it may lead to frequent issuance of unnecessary early warnings, increasing management costs; if set too high, some potential risks may not be detected in time. Suppose through data analysis, it is found that when the proportion of the number of people with abnormal aggregation in a certain area within a unit time to the total number of people in that area is in the range of 0 - 5%, the impact on the overall order is relatively small. At this time, Risk1 can be set to 5%. That is, when the occurrence frequency of abnormal behavior is in the interval [0, 5%), it is determined as a low behavior risk. At this risk level, although there are abnormal behaviors, the overall risk is relatively low. For example, in a railway station, it may only be that a small number of people show short-term abnormal behaviors, which have not had an obvious impact on the overall order.

[0159] For the setting of the high behavior risk threshold Risk2, it is also based on the analysis of historical data and the actual scenario. If the occurrence frequency of abnormal behavior reaches or exceeds Risk2, it may cause relatively serious consequences. Continuing with the example of the railway station, when the proportion of the number of people with abnormal aggregation within a unit time reaches 15% or more, it may lead to serious situations such as passage blockage and crowd panic. At this time, Risk2 can be set to 15%. That is, when the occurrence frequency of abnormal behavior is in the interval [15%, 1], it is determined as a high behavior risk. And when the occurrence frequency of abnormal behavior is in the interval [5%, 15%), it is a medium behavior risk. At this time, the abnormal behavior has had a certain impact on the scenario, and it is necessary to pay close attention and take corresponding measures for intervention. The abnormal behavior grading early warning model is constructed based on the set low behavior risk threshold Risk1 and high behavior risk threshold Risk2. This model divides the risk levels of abnormal behavior into three levels: low behavior risk, medium behavior risk, and high behavior risk. Its purpose is to quantitatively evaluate and classify the management of abnormal behaviors in the region so that relevant personnel can take corresponding measures according to different risk levels.

[0160] In terms of beneficial effects, first of all, the model can improve the accuracy and pertinence of risk early warning. Through clear threshold division, misjudgment and omission of abnormal behaviors are avoided. For example, in urban management, for the monitoring of crowd activities in squares, the abnormal behavior grading early warning model can accurately judge the risk level of crowd behaviors. When the behavior risk is low, managers can arrange a small number of personnel for observation and guidance; while when the high behavior risk is reached, the emergency plan can be quickly activated to allocate a large number of resources for handling, effectively ensuring public safety and the normal order of the venue. Secondly, it helps to reasonably allocate management resources. For low-risk areas, fewer human and material resources can be allocated for monitoring and management; for high-risk areas, concentrated superior resources can be used for key prevention and handling to improve resource utilization efficiency. Moreover, the abnormal behavior grading early warning model is convenient for long-term statistics and analysis of abnormal behaviors. Through the accumulation and analysis of data such as the occurrence frequency and development trend of abnormal behaviors at different risk levels, the laws of abnormal behaviors in the region can be deeply understood, providing strong data support for subsequent optimization of management strategies and improvement of monitoring systems. For example, by analyzing, it is found that high-risk abnormal behaviors are more likely to occur in a certain time period or specific area, so targeted monitoring and management measures can be strengthened in this time period or area to reduce the possibility of risks.

[0161] Step S3120: According to the environmental risk prediction results in the spatio-temporal evolution model, set environmental risk early warning thresholds at different levels, and construct an environmental risk grading early warning model;

[0162] Furthermore, step S3120 includes:

[0163] Step S3121: Extract the sequence of environmental risk prediction values {R i' |i' = 1, 2,..., n2} for the next n2 time steps from the spatio-temporal evolution model;

[0164] Step S3122: Calculate the maximum value Rmax, minimum value Rmin, and mean value Rmean of the sequence of environmental risk prediction values {R i' |i' = 1, 2,..., n2};

[0165] Step S3123: According to the maximum value Rmax, minimum value Rmin, and mean value Rmean, set the low environmental risk threshold Risk3 and the high environmental risk threshold Risk4, where 0 < Risk3 < Risk4 < 1.

[0166] Specifically, step S3121 needs to extract the sequence of environmental risk prediction values {R i'|i' = 1, 2, ..., n2}. The spatio-temporal evolution model analyzes the environmental anomaly feature information, extracts the temporal feature representation of the environmental state using the encoder of the gated recurrent unit network, and performs iterative prediction in combination with the autoregressive structure to obtain the environmental risk prediction values at different future time steps. Taking an industrial park as an example, assume that by analyzing the environmental data such as temperature and smoke concentration at each monitoring point in the park, combining historical data and the current anomaly trend, the fire risk values in different areas of the park within the next 5 hours (i.e., n2 = 5, each time step is 1 hour) are predicted, and these risk values constitute the environmental risk prediction value sequence {R i' |i' = 1, 2, 3, 4, 5}. The purpose of this step is to obtain the predicted data of environmental risks within a future period of time, providing basic data support for setting the early warning threshold in the following steps. Its beneficial effect is that by obtaining the predicted values of environmental risks in advance, it enables managers to have a forward-looking understanding of the future environmental conditions. For example, in an industrial park, if it is known in advance that the fire risk in certain areas may increase within the next few hours, personnel can be arranged in advance to inspect these areas to eliminate fire hazards, or fire-fighting equipment can be prepared in advance to improve the ability to respond to fires, thereby reducing potential losses.

[0167] Step S3122 is to calculate the maximum value Rmax, minimum value Rmin, and mean value Rmean of the environmental risk prediction value sequence {R i' |i' = 1, 2, ..., n2}. Still taking the fire risk prediction value sequence in the industrial park as an example, assume that the fire risk prediction values for these 5 hours are 0.2, 0.5, 0.3, 0.7, and 0.4 respectively. Then the maximum value Rmax = 0.7, the minimum value Rmin = 0.2, and the mean value Rmean = (0.2 + 0.5 + 0.3 + 0.7 + 0.4) ÷ 5 = 0.42. The purpose of calculating these statistical values is to more comprehensively understand the distribution of environmental risk prediction values. The maximum value Rmax reflects the highest risk level that may occur during the prediction period, the minimum value Rmin represents the lowest risk level, and the mean value Rmean reflects the overall average risk degree. This is very crucial for setting a reasonable early warning threshold. For example, the maximum value Rmax can help determine the lower limit of the high-risk early warning to ensure that an early warning can be issued in a timely manner when the highest risk may occur; the minimum value Rmin can be used as a reference for the low-risk early warning to avoid misjudging normal low-risk situations as anomalies; the mean value Rmean can be used as an intermediate reference value for balancing the division of different risk levels. Through these statistical values, the early warning threshold can be set more scientifically, improving the accuracy and reliability of the early warning.

[0168] Setting the low environmental risk threshold Risk3 and high environmental risk threshold Risk4 according to the maximum value Rmax, minimum value Rmin, and mean value Rmean includes:

[0169]

[0170] Among them, r1 is the first weight, which is used to adjust the influence degrees of the minimum value Rmin and the mean value Rmean on the low environmental risk threshold; r2 is the second weight, which is used to adjust the influence degrees of the minimum value Rmin and the mean value Rmean on the high environmental risk threshold; r2 > r1;

[0171] In the calculation formula of the low environmental risk threshold Risk3, for the part Rmin + r1×(Rmean - Rmin), Rmin is the minimum value, that is, the lower limit of the risk, and r1×(Rmean - Rmin) represents an offset of a certain proportion (r1, such as 0.3) between the minimum value and the mean value to appropriately increase the low environmental risk threshold and avoid being overly sensitive to some minor risks due to too low a threshold. The denominator is used to adjust the numerator, reflects the relative relationship between the difference between the maximum value and the mean value and the difference between the mean value and the minimum value. When the difference between the maximum value and the mean value is large, the denominator increases, making the growth of Risk3 relatively gentle and avoiding too high a threshold. In the calculation formula of the high environmental risk threshold Risk4, in Rmax + r2×(Rmean - Rmin), Rmax is the maximum value, that is, the upper limit of the risk, and r2×(Rmean - Rmin) subtracts a certain proportion (r2, such as 0.7) of the difference between the mean value and the minimum value from the maximum value to appropriately reduce the high risk threshold and prevent ignoring some actual high-risk situations due to too high a threshold. The denominator is also functioning similarly to that in the Risk3 formula, used to adjust the threshold to make it more in line with the actual risk distribution.

[0172] When Rmin increases, in the numerator part, Rmin directly increases, and at the same time, the term r1×(Rmean - Rmin) will decrease because of the increase in Rmin (since Rmean is relatively fixed and the increase in Rmin makes Rmean - Rmin decrease), but overall Risk3 will increase. When Rmean increases, Rmean - Rmin increases, r1×(Rmean - Rmin) in the numerator increases, and the denominator will also change, and overall Risk3 will increase. When Rmax increases, the denominator increases, the change in the numerator is relatively small, and Risk3 will decrease. When Rmax increases, Rmax in the numerator directly increases, the change in r2×(Rmean - Rmin) is relatively small, and Risk4 will increase. When Rmean increases, Rmean - Rmin increases, r2×(Rmean - Rmin) in the numerator increases, and the denominator will also change, and generally Risk4 will increase. When Rmin increases, the denominator changes, and r2×(Rmean - Rmin) in the numerator will decrease, so Risk4 will decrease.

[0173] This formula comprehensively considers the maximum, minimum, and mean values of the environmental risk prediction values, and can more comprehensively reflect the overall situation of environmental risks. Compared with simply setting thresholds based on experience or a single indicator, it can more accurately divide the environmental risk levels. For example, in the air quality monitoring of urban transportation hubs, the thresholds set in this way can more accurately judge the risk degree of air quality and avoid misjudgments caused by ignoring some key data. The parameters in the formula are dynamically calculated according to the actual prediction data and can adapt to the changes in environmental risks in different scenarios. In industrial parks, the environmental risk characteristics are different in different production stages or seasons. This formula can adjust the thresholds according to the real-time risk prediction values to improve the accuracy and effectiveness of early warnings. A reasonable threshold setting helps to improve the reliability of environmental risk early warnings. An accurate early warning enables relevant personnel to take timely measures. For example, in a chemical industrial park, when the environmental risk reaches the high-risk threshold, the emergency plan can be activated in advance to evacuate personnel, stop dangerous operations, etc., to ensure the safety of personnel and the normal operation of facilities and equipment.

[0174] Step S3130, fuse the abnormal behavior grading early warning model and the environmental risk grading early warning model to construct an abnormal situation grading early warning framework covering regional abnormal behaviors and environmental risks; the abnormal situation grading early warning framework divides the abnormal situation warning levels into three levels: general warning (low risk), warning warning (medium risk), and severe warning (high risk).

[0175] Further, step S3130 includes:

[0176] Step S3131, based on the low behavior risk threshold Risk1, high behavior risk threshold Risk2, low environmental risk threshold Risk3, and high environmental risk threshold Risk4, construct a two-dimensional risk matrix based on abnormal behavior risks and environmental risks;

[0177] Step S3132, each cell of the two-dimensional risk matrix corresponds to a behavior-environment risk combination. According to the severity of the combination of behavior risks and environmental risks, divide the two-dimensional risk matrix into three risk areas: low risk, medium risk, and high risk;

[0178] Step S3133, map the low-risk, medium-risk, and high-risk areas of the two-dimensional risk matrix to the three early warning levels of general warning, warning warning, and severe warning of abnormal situation early warning respectively.

[0179] Specifically, abnormal behaviors and environmental risks are often interrelated and coexist simultaneously. Warning a single type of risk alone cannot comprehensively reflect the true safety status of the region. By integrating the abnormal behavior grading warning model and the environmental risk grading warning model, various risk factors can be more comprehensively considered, improving the accuracy and reliability of the warning. For example, in an industrial park, the abnormal gathering behavior of personnel (belonging to the risk of abnormal behavior) may increase the probability of a fire occurring (belonging to environmental risk). Combining the two for assessment can more accurately judge the severity of the risk, and corresponding measures can be taken accordingly. Dividing the warning level of abnormal situations into three levels helps managers adopt different response strategies according to different risk levels. General warning (low risk) indicates that both the abnormal behavior and environmental risks in the current region are at a relatively low level, but still need to be monitored; Warning warning (medium risk) means that the risk has increased, and certain measures need to be taken for intervention to prevent the risk from further expanding; Severe warning (high risk) indicates that the region faces a greater safety threat, and immediate emergency measures must be taken to ensure the safety of personnel and the normal operation of the region. This grading method makes the warning information more intuitive and clear, facilitating managers to make quick decisions.

[0180] Dividing the warning level of abnormal situations into three levels helps managers adopt different response strategies according to different risk levels. General warning (low risk) indicates that both the abnormal behavior and environmental risks in the current region are at a relatively low level, but still need to be monitored; Warning warning (medium risk) means that the risk has increased, and certain measures need to be taken for intervention to prevent the risk from further expanding; Severe warning (high risk) indicates that the region faces a greater safety threat, and immediate emergency measures must be taken to ensure the safety of personnel and the normal operation of the region. This grading method makes the warning information more intuitive and clear, facilitating managers to make quick decisions.

[0181] Step S3131 constructs a two-dimensional risk matrix based on abnormal behavior risk and environmental risk, based on the low behavior risk threshold Risk1, high behavior risk threshold Risk2, low environmental risk threshold Risk3, and high environmental risk threshold Risk4. Among them, the rows of the two-dimensional risk matrix represent the abnormal behavior risk levels, including low behavior risk (Low_Behavior), medium behavior risk (Medium_Behavior), and high behavior risk (High_Behavior); the columns represent the environmental risk levels, including low environmental risk

[0182] (Low_Environment), medium environmental risk (Medium_Environment), and high environmental risk (High_Environment). The purpose of constructing a two-dimensional risk matrix is to systematically integrate abnormal behavior risks and environmental risks, so as to more intuitively analyze and evaluate different risk combinations. This matrix form can clearly show the cross relationships between different risk levels, providing a clear framework for subsequent risk area division and warning level determination. Through this matrix, managers can quickly understand the overall risk situation under different combinations of abnormal behavior risks and environmental risks. For example, at a certain moment, the behavior of people in the square is at low risk, but the air quality is at high risk. Through the two-dimensional risk matrix, the corresponding cell can be directly located to clarify the severity of this risk combination. Its beneficial effects are as follows: on the one hand, it provides a structured method for risk assessment, avoiding fuzzy judgment of risks. In urban management, in the face of complex and changing situations, the two-dimensional risk matrix can help managers quickly sort out risk information and make more accurate decisions. On the other hand, it helps to improve the efficiency of risk management. By classifying and integrating risks, managers can formulate corresponding plans for different risk combinations. When the risk situation changes, they can quickly find the corresponding response strategies, saving decision-making time and improving the ability to respond to risks. In addition, this matrix form is also convenient for data statistics and analysis. By analyzing the frequency and consequences of different risk combinations, the setting of risk thresholds and warning mechanisms can be further optimized.

[0183] Taking the abnormal behavior risk level as the rows of the matrix and the environmental risk level as the columns of the matrix, a 3×3 two-dimensional risk matrix is formed as follows:

[0184]

[0185] Each cell in the matrix represents a combination of an abnormal behavior risk and an environmental risk. The letters in the brackets represent the behavior risk level and the environmental risk level respectively.

[0186] Next, according to the severity of the risk combination, the matrix is divided into three areas: low risk (LowRisk), medium risk (MediumRisk), and high risk (HighRisk):

[0187] Low risk (LowRisk): {(L,L),(L,M),(M,L)}, corresponding to the green area;

[0188] Medium risk (MediumRisk): {(L,H),(M,M),(H,L)}, corresponding to the yellow area;

[0189] High Risk: {(M, H), (H, M), (H, H)}, corresponding to the red area.

[0190] The basis for the division is that the higher the levels of behavioral risk and environmental risk, the greater the severity of the risk combination. The LowRisk area represents combinations with relatively low behavioral and environmental risks. The HighRisk area represents combinations with high behavioral risk or high environmental risk. The MediumRisk area lies between the two.

[0191] Finally, establish a correspondence between the divided risk matrix areas and the early warning levels for abnormal situations: for the LowRisk area, the early warning level is a general warning; for the MediumRisk area, the early warning level is a warning; for the HighRisk area, the early warning level is a severe warning.

[0192] For the LowRisk area {(L, L), (L, M), (M, L)}, it represents combinations with relatively low behavioral and environmental risks. In the airport scenario, when passengers behave properly without abnormal behaviors such as a large number of people gathering or running (low behavioral risk), and environmental indicators such as air quality and temperature are within the normal range (low environmental risk or medium environmental risk), it belongs to this combination. In this case, the airport operation is basically in a safe and stable state, with little impact on passengers and staff. The MediumRisk area {(L, H), (M, M), (H, L)} means there is a certain degree of risk. For example, if there is a short-term gathering of people in some areas of the airport (medium behavioral risk), but the environmental risk is still at a medium level, or the personnel behavior is normal (low behavioral risk), but some environmental indicators such as noise levels are slightly exceeded (high environmental risk), it belongs to the medium-risk combination. At this time, although it has not seriously affected the normal operation of the airport, relevant staff need to pay close attention and take some preventive measures, such as strengthening personnel guidance and adjusting equipment operation parameters, to prevent the risk from escalating further. The HighRisk area {(M, H), (H, M), (H, H)} represents combinations with high behavioral risk or high environmental risk. In the airport, if there is a sudden gathering of a large number of passengers and chaos occurs (high behavioral risk), and at the same time there are abnormal situations such as smoke in the environment (high environmental risk), this combination will pose a serious threat to the safety and normal operation of the airport, possibly resulting in flight delays, passenger panic, etc.

[0193] The purpose of dividing the two-dimensional risk matrix into different risk regions is to more intuitively identify and manage risks of different levels. Through this division method, managers can quickly judge the severity of the current risk portfolio and take corresponding measures. For example, when it is determined that the risk is in the low-risk region, it can be operated according to the regular management process; if it is in the medium-risk region, the inspection frequency needs to be increased and some emergency plans need to be activated; while for the high-risk region, a comprehensive emergency response mechanism must be immediately activated and a large amount of resources need to be allocated for handling. The beneficial effects brought by this division are significant. First of all, it improves the accuracy and efficiency of risk judgment. In places such as transportation hubs where people and equipment are concentrated, quickly and accurately judging the risk level is crucial. Through clear risk region division, managers can make correct decisions in a short time and avoid the expansion of risks caused by misjudgment. Secondly, it helps to allocate resources reasonably. According to the characteristics of different risk regions, human, material and financial resources can be allocated targeted. For the low-risk region, fewer resources can be allocated for daily management; for the medium-risk region, the resource investment can be appropriately increased; while for the high-risk region, superior resources are concentrated for response to improve the utilization efficiency of resources. Moreover, it is convenient to track and evaluate risks. As time goes by and the situation changes, the risk portfolio may shift between different regions. Through the division of risk regions, the changing trend of risks can be clearly observed, and management strategies and measures can be adjusted in a timely manner to ensure that risks are always under control.

[0194] If the risk portfolio in the current region is in the low-risk region, that is, {(L,L),(L,M),(M,L)}, it is mapped to the general warning level; if the risk portfolio is in the medium-risk region, that is, {(L,H),(M,M),(H,L)}, it is mapped to the warning level; if the risk portfolio is in the high-risk region, that is, {(M,H),(H,M),(H,H)}, it is mapped to the severe warning level. The purpose of mapping the risk regions to the warning levels is to achieve a standardized correspondence between risks and warnings, making the warning information clearer and more explicit, and facilitating the understanding and implementation by relevant personnel. This mapping relationship makes the warning system more scientific and standardized, avoiding chaos and misjudgment caused by unclear warning criteria. This mapping relationship helps to improve the timeliness and effectiveness of emergency response. Once the risk level is determined, the corresponding warning level can be quickly conveyed to relevant personnel, enabling them to quickly take correct response measures, reducing the response time and minimizing the losses caused by risks. At the same time, it helps to improve the overall risk management level. Through the clear mapping relationship, the response measures under different warning levels can be standardized for management; in addition, this mapping relationship also facilitates subsequent risk assessment and improvement. By statistically analyzing the risk events under different warning levels, the setting of risk thresholds and the warning mechanism can be continuously optimized to improve the accuracy and reliability of the warning system.

[0195] Step S3200, under the framework of hierarchical early warning for abnormal situations, generate hierarchical early warning information for regional abnormal situations in real time;

[0196] Further, step S3200 includes:

[0197] Step S3210, calculate the real-time regional abnormal behavior risk value and the real-time environmental risk value at the current moment according to the target feature description information;

[0198] Further, step S3210 includes:

[0199] Step S3211, calculate the real-time regional abnormal behavior risk value according to the target category information, target movement trajectory information, and target behavior feature information in the target feature description information;

[0200] Further, step S3211 includes:

[0201] Step S32111, set the abnormal behavior risk weight vectors for different target categories, and calculate the target category abnormal behavior risk component at the current moment according to the target category information and the abnormal behavior risk weight vectors;

[0202] Step S32112, perform abnormal trajectory detection on the target movement trajectory information to obtain a trajectory anomaly value vector, and calculate the trajectory abnormal behavior risk component based on the trajectory anomaly value vector;

[0203] Step S32113, calculate the abnormal behavior risk component according to the target abnormal behavior detection matrix;

[0204] Step S32114, calculate the real-time regional abnormal behavior risk value at the current moment according to the target category abnormal behavior risk component, the trajectory abnormal behavior risk component, and the abnormal behavior risk component.

[0205] Specifically, different target categories have different potential risk levels, and setting the abnormal behavior risk weight vector is to reflect this difference. For example, in the urban management scenario, the target categories in crowded places such as railway stations and squares may include ordinary pedestrians, staff, and suspicious persons. The risk of ordinary pedestrians is relatively low during normal activities, but if a large number of ordinary pedestrians gather abnormally, it may also trigger risks; suspicious persons themselves have relatively high potential risks. Therefore, different weights are assigned to different target categories. Suppose the abnormal behavior risk weight of ordinary pedestrians is set as w1, that of staff is w2, and that of suspicious persons is w3, and w1 < w2 < w3. The target category information is an N-dimensional vector, with each dimension corresponding to a target type, and the dimension value being the probability of the occurrence of the target type. For example, in the monitoring of a certain urban square, the target category information vector is [0.7, 0.2, 0.1], indicating that the probability of the occurrence of ordinary pedestrians is 70%, that of staff is 20%, and that of suspicious persons is 10%. Multiply each probability value in the target category information by the corresponding abnormal behavior risk weight and then sum them up to obtain the abnormal behavior risk component of the target category. Suppose w1 = 0.1, w2 = 0.3, w3 = 0.8, then the abnormal behavior risk component Rt1 of the target category = 0.7×0.1 + 0.2×0.3 + 0.1×0.8 = 0.21. The purpose of this step is to quantify the contribution degree of different target categories to the abnormal behavior risk. By setting the weight vector, the risk differences of different target categories can be more reasonably reflected, and the influence brought by ignoring the target category factor during risk assessment can be avoided. Its beneficial effect is to improve the accuracy and pertinence of risk assessment. In urban management, clarifying the risk weights of different target categories can more accurately identify high-risk areas and time periods.

[0206] In the actual panoramic inspection scenario, the target movement trajectory contains rich information. Taking a transportation hub as an example, under normal circumstances, the movement trajectory of passengers has a certain regularity, such as going to the waiting area, ticket checkpoint, etc. according to the guiding signs. If the movement trajectory of a passenger deviates from the normal route or lingers in a certain area for a long time, it may indicate abnormal behavior. By analyzing the target movement trajectory information and using a specific algorithm (such as an anomaly detection model trained based on historical trajectory data) to judge whether each trajectory is abnormal. For each trajectory, if it is determined to be abnormal, the corresponding element in the trajectory anomaly value vector is set to 1, otherwise it is set to 0. For example, in the monitoring of a certain airport, the movement trajectories of n targets are detected, and the trajectory anomaly value vector A = [a1, a2, …, a n , where a j'Denote the trajectory outlier of the j'-th target, taking values of 0 or 1, where 1 ≤ j' ≤ n. When calculating the risk component of trajectory abnormal behavior, sum all the elements in the trajectory outlier vector and then divide by the number of targets n. Suppose A = [0, 1, 0, 1, 0] and n = 5, then the risk component of trajectory abnormal behavior The purpose of this step is to evaluate the risk of abnormal behavior from the perspective of the target motion trajectory. Abnormalities in the motion trajectory are often closely related to abnormal behavior. By detecting trajectory abnormalities and calculating the risk component, potential abnormal behavior risks can be discovered in a timely manner. Its beneficial effect is that it can quickly locate the area and target where abnormal behavior occurs. In transportation hubs, through the analysis of the motion trajectories of a large number of people, once a high risk component of trajectory abnormal behavior is found in a certain area, key attention can be quickly paid to this area, and corresponding measures can be taken, such as guiding people to evacuate and strengthening security inspections, to ensure the normal operation of the transportation hub and the safety of personnel.

[0207] The target abnormal behavior detection matrix is obtained by analyzing the target behavior characteristics and judging whether each target has abnormal behavior. Each element of this matrix takes values of 0 or 1, indicating whether the corresponding target has a certain abnormal behavior. In the monitoring scenario of an industrial park, suppose M2 targets are detected under K types of behavior categories, and the target abnormal behavior detection matrix B is obtained. When calculating the risk component of abnormal behavior, first sum all the elements in matrix B and then divide by the total number of matrix elements M2 × K. For example, if B is a 3×4 matrix, then the risk component Rt3 of abnormal behavior is:[[]]

[0208]

[0209] The purpose of step S32113 is to directly start from the detection results of target abnormal behavior, quantify the distribution of abnormal behavior in the entire target group, and thus obtain the risk component of abnormal behavior. Its beneficial effect is to intuitively reflect the prevalence of abnormal behavior in the area. In an industrial park, by calculating the risk component of abnormal behavior, managers can clearly understand the severity of the current abnormal behavior. If this component is high, it indicates that there are more abnormal behaviors, and corresponding measures need to be taken in a timely manner, such as strengthening safety training and adjusting management strategies, to reduce safety risks and ensure the normal progress of production activities in the park.

[0210] Step S32114 calculates the real-time regional abnormal behavior risk value at the current moment based on the target category abnormal behavior risk component, the trajectory abnormal behavior risk component, and the abnormal behavior risk component. In practical applications, these three risk components reflect the abnormal behavior risk situation within the region from different perspectives. Combining them can obtain a more comprehensive and accurate real-time regional abnormal behavior risk value. When calculating the real-time regional abnormal behavior risk value Rt, a weighted summation method is used. The purpose of this step is to integrate abnormal behavior risk information from multiple aspects and obtain a comprehensive risk assessment value. Its beneficial effect is to provide a unified indicator to measure the abnormal behavior risk status within the region. In scenarios such as urban management, industrial parks, transportation hubs, border security, etc., managers can quickly judge the risk level of the current region based on this real-time regional abnormal behavior risk value, and then decide what countermeasures to take. When the risk value is low, the normal monitoring and management intensity can be maintained; when the risk value is high, the emergency plan should be activated in a timely manner, allocate more resources for processing, and improve the efficiency and effect of dealing with abnormal behaviors.

[0211] Step S3212 calculates the real-time environmental risk value according to the environmental abnormal feature information in the target feature description information.

[0212] Specifically, the environmental abnormal feature information is obtained from the preprocessing of the panoramic video sequence, which includes information such as the hot target area extracted from the infrared thermal imaging video stream. These information reflect the possible abnormal temperature changes, heat source distributions, etc. in the environment. For the extracted hot target area, it is necessary to quantitatively analyze its relevant parameters. For example, calculate the temperature deviation value of the hot target area. Suppose that under normal circumstances, the temperature range of a certain area is [T min , T max , and the average temperature of the hot target area detected by the infrared thermal imaging video stream is T hot , then the temperature deviation value The larger this temperature deviation value is, the higher the temperature abnormality degree of the area, and the greater the potential environmental risk. Then, consider the area size of the hot target area. Convert the number of pixels N hot of the hot target area into the actual area S hot (which can be converted through the pre-calibrated image scale relationship). A large-area hot area may mean more serious environmental problems, such as fire hazards. An area weight coefficient w S can be set and determined according to the risk assessment experience of the actual scenario. At the same time, analyze the temperature change trend of the hot target area. By analyzing consecutive frames of infrared thermal imaging videos, calculate the change rate of the temperature of the hot target area If the temperature shows a rapid upward trend, it indicates that the potential risk is intensifying. Set a weight coefficient w for the temperature change rate. T Considering the above factors, the real-time environmental risk value R env can be calculated by weighted summation, that is where w M is the weight coefficient of the temperature deviation value, and like w S , w T , it is determined based on the importance of environmental risk factors in different scenarios through a large amount of historical data and practical experience.

[0213] This method of calculating the real-time environmental risk value has beneficial effects in many aspects. From the perspective of risk assessment accuracy, it comprehensively considers multiple factors such as temperature deviation, hot spot area, and temperature change trend, and can more comprehensively reflect the degree of environmental abnormality. For example, in an industrial park, only focusing on temperature deviation may ignore a large area of low-temperature abnormal areas (such as the expansion of low-temperature areas caused by cold storage leakage), and considering the area factor comprehensively can more accurately assess the risk. From the perspective of timeliness, by analyzing the temperature change trend, the development trend of potential risks can be detected in a timely manner. In the urban management scenario, if the temperature in a certain area rises rapidly, it can timely warn of possible emergencies such as fires, and gain time for subsequent emergency handling. From the perspective of reasonable resource allocation, accurate risk value calculation helps managers reasonably allocate human and material resources according to the risk levels of different regions. At transportation hubs, priority is given to arranging inspections and maintenance for high-risk areas to improve resource utilization efficiency.

[0214] Step S3220, input the real-time regional abnormal behavior risk value and the real-time environmental risk value into the abnormal situation classification and early warning framework, and judge the early warning level of the regional abnormal situation at the current moment;

[0215] Specifically, the abnormal situation classification and early warning framework is formed by integrating the abnormal behavior classification and early warning model and the environmental risk classification and early warning model. This framework divides the early warning level of abnormal situations into three levels: general warning (low risk), warning warning (medium risk), and severe warning (high risk), and realizes the judgment of the early warning level by constructing a two-dimensional risk matrix based on abnormal behavior risk and environmental risk. Compare the real-time regional abnormal behavior risk value R beh and the real-time environmental risk value R env with the thresholds in the two-dimensional risk matrix. Determine their respective risk levels according to the values of R beh and R env . For example, if R beh = 0.6, Risk1 = 0.4, Risk2 = 0.7, then R beh belongs to the medium behavior risk level; if Renv＝ 0.5, Risk3 = 0.3, Risk4 = 0.6, then R env It belongs to the medium environmental risk level. Then, find the corresponding cell in the two-dimensional risk matrix, and the risk area where this cell is located corresponds to the current abnormal situation warning level. Taking the previous example, the cell corresponding to medium behavior risk and medium environmental risk is (M, M), which belongs to the medium risk area in the risk matrix division. Therefore, the abnormal situation warning level at the current moment is determined as a warning. The beneficial effect of this judgment method is significant. From the perspective of risk visualization, the two-dimensional risk matrix intuitively shows the warning levels corresponding to different combinations of abnormal behavior risks and environmental risks, facilitating managers to quickly understand the risk status of the area. When conducting risk assessments in multiple industrial parks, using the same risk matrix and threshold criteria can ensure the consistency and comparability of the assessment results. From the perspective of warning timeliness, once the real-time risk value is obtained, the warning level can be quickly located in the risk matrix, and a warning can be issued in a timely manner.

[0216] Step S3230, if the regional abnormal situation warning level at the current moment reaches the set warning trigger condition, then generate a regional abnormal situation warning information of the corresponding level.

[0217] Specifically, the early warning trigger conditions are set according to the early warning levels in the hierarchical early warning framework for abnormal situations. When the early warning level reaches the warning (medium risk) or severe warning (high risk), the generation of early warning information is triggered. The generated early warning information includes key contents such as the early warning level, early warning time, early warning location, and early warning reason. The early warning level directly corresponds to the general warning, warning, and severe warning in the hierarchical early warning framework for abnormal situations, clearly informing relevant personnel of the severity of the risk. The early warning time records the exact moment when the early warning information is generated, accurate to the second, facilitating subsequent backtracking and analysis of the event timeline. The early warning location is determined through the positioning information of abnormal targets in the panoramic video sequence. For example, in urban management, by using the geographical coordinate information of high-point cameras and the target location information in video images, combined with map data, the specific location of the abnormality, such as the specific street and building, can be accurately determined. The early warning reason is determined based on the abnormal behaviors and environmental abnormal features in the target feature description information. If the early warning is caused by the abnormal gathering behavior of people and the abnormal rise in temperature in the hot spot area, the early warning reason is detailed as "abnormal gathering of people is found and the temperature in this area rises abnormally". This way of generating early warning information has many beneficial effects. From the perspective of emergency response, detailed and accurate early warning information enables relevant departments to quickly take targeted response measures. When a fire early warning occurs in an industrial park, the fire department can quickly rush to the scene according to the early warning location and prepare corresponding fire-fighting equipment according to the early warning reason. From the perspective of information transmission, the standardized early warning information format facilitates the efficient transmission of information among different departments and personnel. In border defense security, personnel in different positions can quickly understand the key contents of the early warning and cooperate to carry out work. From the perspective of event traceability, complete early warning information provides detailed data for subsequent event investigation and analysis. In urban management, if an emergency occurs, lessons can be learned by analyzing the early warning information to optimize the early warning system.

[0218] Step S3300, adaptively adjust and optimize the hierarchical early warning information for regional abnormal situations.

[0219] Specifically, the adaptive adjustment and optimization are mainly based on multi-source data obtained in real time. On the one hand, continuously collect panoramic video stream data, including visible light video stream and infrared thermal imaging video stream. As time goes by, the situation in the scene may change. For example, in an industrial park, the temperature of the original hot spot area may decrease, or the abnormal behavior patterns of personnel may change. By analyzing the new video stream data, re-evaluate the abnormal behavior risk value and the environmental risk value. On the other hand, collect data from other sensors, such as air quality sensors, sound sensors, etc. In the urban management scenario, when the air quality sensor detects an increase in the concentration of harmful gases in a certain area, which may affect the environmental risk assessment, this information needs to be taken into account and the real-time environmental risk value is recalculated. Compare the risk value calculated based on the new data with the original warning level. If the risk value changes significantly, resulting in a change in the warning level, the warning information is adjusted accordingly. For example, if a certain area was originally in a warning state (medium risk), and due to new data showing that the abnormal behavior has intensified and the environmental risk has further increased, the recalculated risk value raises the warning level to a severe warning (high risk), then the content such as the warning level and warning reason in the warning information needs to be updated.

[0220] This way of adaptively adjusting and optimizing warning information has important effects. From the aspect of accurate warning, it can timely adjust the warning level and content according to the real-time changing situation, improving the accuracy of warning. In an industrial park, it will not cause excessive resource waste in low-risk areas due to inaccurate warnings, nor will it cause high-risk areas not to be dealt with in a timely manner due to untimely warnings. From the aspect of collaborative work, the warning information optimized for different recipients helps different departments better cooperate to deal with abnormal situations.

[0221] Embodiment 2

[0222] Based on Embodiment 1, this embodiment provides a real-time warning system for regional abnormal situations based on high-point panoramic intelligent inspection, as Figure 8 shown, including:

[0223] Video stream analysis module: used to collect the visible light video stream and infrared thermal imaging video stream of the target area, form a panoramic video sequence of the target area; perform intelligent analysis on the panoramic video sequence to obtain an abnormal feature enhanced video sequence; generate target feature description information according to the abnormal feature enhanced video sequence;

[0224] Space-time evolution model construction module: According to the target feature description information, construct a space-time topology graph among the targets in the area, analyze the space-time correlation characteristics among the targets, and detect the abnormal behavior patterns in the area; Embed the detected abnormal behavior patterns in the area as regional abnormal features into the space-time topology graph; Based on the target feature description information, obtain the environmental risk prediction result; Based on the space-time topology graph integrated with the regional abnormal behavior patterns and the environmental risk prediction result, construct a space-time evolution model;

[0225] Hierarchical warning module: Generate hierarchical warning information for the abnormal situation in the area according to the space-time evolution model.

[0226] The methods and systems of the present application may be implemented in many ways. For example, the methods and systems of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps for the method is only for illustration, and the steps of the method of the present application are not limited to the above specific described order unless otherwise specifically stated.

[0227] In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the corresponding technical solutions in the prior art in terms of implementation principles are not described in detail to avoid excessive elaboration.

[0228] As described above in the specific embodiments, the purpose, technical solutions, and beneficial effects of the present invention have been further described in detail. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time warning method for regional abnormal conditions based on high-point panoramic intelligent inspection, characterized in that, The method includes: Collecting the visible light video stream and the infrared thermal imaging video stream of the target area to form a panoramic video sequence of the target area; performing intelligent analysis on the panoramic video sequence to obtain an enhanced abnormal feature video sequence; generating target feature description information according to the enhanced abnormal feature video sequence; Constructing a spatio-temporal topology graph among the targets within the area according to the target feature description information, analyzing the spatio-temporal correlation characteristics among the targets, and detecting the abnormal behavior patterns of the area; embedding the detected abnormal behavior patterns of the area as area abnormal features into the spatio-temporal topology graph; obtaining an environmental risk prediction result based on the target feature description information; constructing a spatio-temporal evolution model based on the spatio-temporal topology graph integrated with the area abnormal behavior patterns and the environmental risk prediction result; Generating a hierarchical early warning information for the abnormal situation of the area according to the spatio-temporal evolution model.

2. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 1, characterized in that The obtaining of the enhanced abnormal feature video sequence includes: Performing frame-by-frame analysis on the visible light video stream, extracting the flame area in the video to generate a flame abnormal enhanced video stream; performing moving target segmentation on the visible light video stream, extracting the smoke area in the video to generate a smoke abnormal enhanced video stream; performing target detection and tracking on the visible light video stream, extracting the skeleton key point coordinates of the target and the abnormal behaviors of the personnel in the video to generate a personnel abnormal behavior enhanced video stream; performing foreground target segmentation on the infrared thermal imaging video stream, extracting the hot spot target area to generate a hot spot abnormal enhanced video stream; The flame abnormal enhanced video stream, the smoke abnormal enhanced video stream, the personnel abnormal behavior enhanced video stream, and the hot spot abnormal enhanced video stream constitute the enhanced abnormal feature video sequence.

3. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 2, characterized in that The generating of the target feature description information according to the enhanced abnormal feature video sequence includes: Inputting the flame abnormal enhanced video stream, the smoke abnormal enhanced video stream, and the hot spot abnormal enhanced video stream into a pre-trained deep convolutional neural network to generate target category information reflecting the target type attributes; Generating target motion trajectory information reflecting the dynamic movement law of the target according to the personnel abnormal behavior enhanced video stream; Obtaining target behavior feature information reflecting the abnormal behavior of the target according to the target motion trajectory information; Obtaining environmental abnormal feature information reflecting the abnormal state of the environment according to the target category information, the target motion trajectory information, and the infrared thermal imaging video stream; The target category information, the target motion trajectory information, the target behavior feature information, and the environmental abnormal feature information constitute the target feature description information.

4. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 3, characterized in that, The generating of the target motion trajectory information reflecting the dynamic movement law of the target includes: Extracting the skeleton key point coordinates of the target in each frame of the personnel abnormal behavior enhanced video stream; generating the personnel motion trajectory curve of each target in the whole video based on the skeleton key point coordinates of the target; performing parametric representation on the personnel motion trajectory curve, extracting the position coordinates of the target at the key time steps as spatio-temporal anchor points, and constructing a motion trajectory feature matrix; constructing a spatio-temporal graph network based on the motion trajectory feature matrix, with each spatio-temporal anchor point as a node of the spatio-temporal graph network; Introduce a spatio-temporal coordinate position encoding mechanism, map the discrete position coordinates of nodes to a continuous periodic feature space through sine and cosine functions, and obtain a spatio-temporal graph network incorporating position encoding information; generate target motion trajectory information reflecting the dynamic movement law of the target according to the spatio-temporal graph network incorporating position encoding information.

5. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 4, characterized in that, The obtaining of target behavior feature information reflecting target abnormal behavior according to the target motion trajectory information includes: Perform slicing processing on the target motion trajectory information to obtain a motion segment data set containing n1 motion segments; design a temporal variational autoencoder including an encoder and a decoder, and train the temporal variational autoencoder on the motion segment data set; use the encoder in the trained temporal variational autoencoder to extract the mean vector of a new motion segment as the motion behavior feature vector of the motion segment. Concatenate the motion behavior feature vectors extracted from each target on different motion segments into a behavior feature matrix, where each row of the behavior feature matrix corresponds to a target and each column corresponds to a behavior category; based on the behavior feature matrix, perform abnormal behavior discrimination on the behavior features of each target to generate an abnormal behavior confidence score matrix; according to a preset abnormal determination threshold, binarize the abnormal behavior confidence score matrix to obtain a target abnormal behavior detection matrix, where each element of the target abnormal behavior detection matrix takes a value of 0 or 1, and define the target abnormal behavior detection matrix as the target behavior feature information.

6. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 5, wherein, The performing of slicing processing on the target motion trajectory information includes: Set a sliding time window with a fixed length of L', slide at a step size of S in the time dimension, and cut the target motion trajectory information into a series of motion segments with a length of L'.

7. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 6, characterized in that, The constructing of a spatio-temporal topology graph among targets within a region includes: Input the target category information and the target motion trajectory information into a graph neural network to construct a spatial topology relationship graph among targets within the region; on the basis of the spatial topology relationship graph, introduce the time dimension to construct a spatio-temporal topology graph; in the spatio-temporal topology graph, the states of the same target at different times are connected by temporal edges, and the relationships among different targets at the same time are connected by spatial edges.

8. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 7, characterized in that, The obtaining of the environmental risk prediction result includes: Input the environmental anomaly feature information into a pre-constructed gated recurrent unit network including an encoder part, extract the temporal feature representation of the environmental state through the encoder part of the gated recurrent unit network, and define it as the environmental state feature; perform dynamic weighting on the environmental state features at different times to obtain the environmental state significant feature; based on the environmental state significant feature, introduce an autoregressive structure in the encoder part of the gated recurrent unit network, and perform rolling estimation on the environmental risk state at future times in an iterative prediction manner to obtain the environmental risk prediction result; the environmental risk prediction result includes a sequence of environmental risk prediction values within a preset future time.

9. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 8, characterized in that, The constructing of the spatio-temporal evolution model includes: On the basis of the spatio-temporal topology graph and the environmental risk prediction result that fuse the regional abnormal behavior pattern, adopt the pyramid pooling method to generate a multi-scale spatio-temporal pyramid representation of the regional abnormal situation from bottom to top. Based on the multi-scale spatio-temporal pyramid representation of regional anomaly situations, the multi-level spatio-temporal correlation characteristics of regional anomaly situations are obtained; Based on the multi-level spatio-temporal correlation characteristics of regional anomaly situations, a spatio-temporal evolution model is constructed.

10. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 9, characterized in that The generation of hierarchical early warning information for regional anomaly situations according to the spatio-temporal evolution model includes: constructing a hierarchical early warning framework for anomaly situations based on the spatio-temporal evolution model; under the hierarchical early warning framework for anomaly situations, generating hierarchical early warning information for regional anomaly situations in real time.

11. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 10, characterized in that, The construction of the hierarchical early warning framework for anomaly situations includes: According to the regional anomaly behavior patterns in the spatio-temporal evolution model, setting early warning thresholds for different levels of anomaly behaviors, and constructing a hierarchical early warning model for anomaly behaviors; According to the environmental risk prediction results in the spatio-temporal evolution model, setting early warning thresholds for different levels of environmental risks, and constructing a hierarchical early warning model for environmental risks; Fusing the hierarchical early warning model for anomaly behaviors and the hierarchical early warning model for environmental risks to construct a hierarchical early warning framework for anomaly situations covering regional anomaly behaviors and environmental risks; the hierarchical early warning framework for anomaly situations divides the early warning levels of anomaly situations into three levels: general early warning, warning early warning, and severe early warning.

12. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 11, wherein The early warning thresholds for different levels of anomaly behaviors include a low behavior risk threshold Risk1 and a high behavior risk threshold Risk2, where 0 < Risk1 < Risk2 < 1.

13. The real-time early warning method for regional abnormal conditions based on high-point panoramic intelligent inspection according to claim 12, characterized in that, The real-time generation of hierarchical early warning information for regional anomaly situations includes: According to the target feature description information, calculating the real-time regional anomaly behavior risk value and the real-time environmental risk value at the current moment; inputting the real-time regional anomaly behavior risk value and the real-time environmental risk value into the hierarchical early warning framework for anomaly situations to judge the early warning level of the regional anomaly situation at the current moment.

Images