Intelligent decision-making and response method and system based on data analysis

Abstract

The present application relates to the technical field of data analysis, and discloses an intelligent decision-making and response method and system based on data analysis, the method comprising: collecting multi-scene monitoring video data based on security cameras and performing preprocessing to obtain preprocessed image data; performing multi-scale feature extraction and L2 regularization feature selection to obtain a target multi-scale feature set; performing scene classification and object recognition to obtain scene classification results and object recognition results; performing three-dimensional scene reconstruction and scene graph sequence modeling to obtain a three-dimensional scene model and scene change prediction results; performing multi-dimensional scene error analysis and reinforcement learning to obtain multiple candidate monitoring action strategies; performing adaptive parameter calibration of spatiotemporal fusion to generate target monitoring action parameters of security cameras, thereby effectively reducing misidentification and false detection in complex environments and improving the system's anti-interference ability.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to an intelligent decision-making and response method and system based on data analysis. Background Art

[0002] Traditional security monitoring systems mainly rely on manual operation and simple image processing technology. When faced with complex and ever-changing monitoring scenarios, it is often difficult to timely and accurately identify potential threats and abnormal situations. In addition, the collection and processing of large-scale video data also brings huge computing and storage pressure to the system, limiting the real-time and scalability of the monitoring system.

[0003] In recent years, with the rapid development of artificial intelligence and big data technology, intelligent security systems have gradually become a research hotspot. However, existing intelligent security solutions still have some problems: object recognition and scene understanding in multiple scenes are not high enough, especially in the case of lighting changes, occlusion and complex backgrounds; secondly, the accuracy of three-dimensional scene reconstruction and change prediction needs to be improved, and it is difficult to accurately capture the dynamic changes and potential risks in the scene; the existing system lacks an adaptive decision-making mechanism and cannot automatically adjust the monitoring strategy and parameters according to scene changes. Summary of the invention

[0004] The present application provides an intelligent decision-making and response method and system based on data analysis, thereby effectively reducing misidentification and false detection in complex environments and improving the system's anti-interference ability.

[0005] The first aspect of the present application provides an intelligent decision-making and response method based on data analysis, and the intelligent decision-making and response method based on data analysis includes:

[0006] Collect multi-scene monitoring video data based on security cameras and perform preprocessing to obtain preprocessed image data;

[0007] Performing multi-scale feature extraction and L2 regularization feature selection on the pre-processed image data to obtain a target multi-scale feature set;

[0008] Respectively performing scene classification and object recognition on the target multi-scale feature set to obtain a scene classification result and an object recognition result;

[0009] Based on the scene classification result and the object recognition result, three-dimensional scene reconstruction and scene graph sequence modeling are performed to obtain a three-dimensional scene model and a scene change prediction result;

[0010] Perform multi-dimensional scene error analysis and reinforcement learning based on the three-dimensional scene model and the scene change prediction result to obtain multiple candidate monitoring action strategies;

[0011] The multiple candidate monitoring action strategies are subjected to spatiotemporal fusion adaptive parameter calibration to generate target monitoring action parameters of the security camera.

[0012] The second aspect of the present application provides an intelligent decision-making and response system based on data analysis, and the intelligent decision-making and response system based on data analysis includes:

[0013] A preprocessing module is used to collect and preprocess multi-scene monitoring video data based on security cameras to obtain preprocessed image data;

[0014] A feature selection module, used for performing multi-scale feature extraction and L2 regularization feature selection on the pre-processed image data to obtain a target multi-scale feature set;

[0015] A recognition module, used to perform scene classification and object recognition on the target multi-scale feature set, respectively, to obtain a scene classification result and an object recognition result;

[0016] A modeling module, used to perform three-dimensional scene reconstruction and scene graph sequence modeling based on the scene classification result and the object recognition result, to obtain a three-dimensional scene model and a scene change prediction result;

[0017] An analysis module, used for performing multi-dimensional scene error analysis and reinforcement learning based on the three-dimensional scene model and the scene change prediction result to obtain multiple candidate monitoring action strategies;

[0018] A generation module is used to perform adaptive parameter calibration of the spatiotemporal fusion of the multiple candidate monitoring action strategies to generate target monitoring action parameters of the security camera.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned intelligent decision-making and response method based on data analysis.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the above-mentioned intelligent decision-making and response method based on data analysis.

[0021] Compared with the prior art, the present application has the following beneficial effects: through multi-scale feature extraction and L2 regularized feature selection, combined with the GateNet network and the Faster R-CNN model, the accuracy of scene classification and object recognition in complex scenes is greatly improved. Three-dimensional scene reconstruction and scene graph sequence modeling are performed based on the scene classification results and object recognition results, achieving more accurate three-dimensional scene model construction and scene change prediction. Through multi-dimensional scene error analysis and reinforcement learning, the system can automatically generate and optimize monitoring action strategies according to scene changes, thereby improving the intelligence level of the monitoring system. The adaptive parameter calibration method using spatiotemporal fusion can quickly generate target monitoring action parameters for security cameras, ensuring the real-time response capability of the system. By introducing conditional random field models and multiple optimization strategies, misidentification and false detection in complex environments are effectively reduced, and the system's anti-interference ability is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0024] Figure 1 It is a flowchart of an intelligent decision-making and response method based on data analysis provided by an embodiment of the present invention;

[0025] Figure 2 It is a schematic block diagram of the structure of an intelligent decision-making and response system based on data analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 , an embodiment of the intelligent decision-making and response method based on data analysis in the embodiments of the present application includes:

[0030] Step 100: Collect multi-scene monitoring video data based on security cameras and perform preprocessing to obtain preprocessed image data;

[0031] It is understandable that the execution subject of the present application may be an intelligent decision-making and response system based on data analysis, or a terminal or a server, which is not specifically limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0032] Specifically, surveillance video data of multiple different scenes are collected by security cameras, covering different geographical locations, time periods and lighting conditions. The surveillance video data is frame extracted to obtain an original image sequence containing multiple scenes. Noise detection is performed on each image in the original image sequence. By analyzing various noise sources in the image, such as Gaussian noise, salt and pepper noise, etc., the type and distribution of noise are identified, and a noise feature map is generated to describe the distribution and intensity of noise in the image. Based on the noise feature map, the original image sequence is denoised, and appropriate algorithms and parameters are used to reduce noise to eliminate unnecessary interference and noise in the image, and a denoised image sequence is obtained. Considering the optical distortion problems existing in the installation and use of security cameras, such as lens distortion, tilt angle, etc., distortion analysis is performed on the denoised image sequence based on the internal parameters of the camera. By calculating the degree and type of image distortion, the corresponding distortion correction parameters are obtained to describe the distortion and offset of the geometric shape of the image, and can guide the system to perform geometric correction on the denoised image sequence to obtain a distortion-free image sequence. The illumination condition analysis is performed on the distortion-free image sequence. By calculating the illumination intensity and color temperature of each scene, the illumination balance parameters are obtained to reflect the distribution of light in the image and the change of color temperature. Based on the illumination balance parameters, the color of the distortion-free image sequence is adjusted to eliminate the effects of uneven illumination and color temperature offset, so that the image is more visually balanced, the color is more natural and accurate, and a color-balanced image sequence is obtained. Contrast analysis is performed on the color-balanced image sequence. By calculating local and global contrast indicators, the areas with insufficient or excessive contrast in the image are identified, and the contrast of the image is optimized according to the analysis results. Contrast optimization processing aims to enhance the layering and detail expression of the image, making the outline and texture of the object clearer, and finally obtaining the preprocessed image data.

[0033] Step 200, performing multi-scale feature extraction and L2 regularization feature selection on the pre-processed image data to obtain a target multi-scale feature set;

[0034] Specifically, the preprocessed image data is processed by Gaussian pyramid and Laplacian pyramid respectively to generate multiple sub-image sequences of different scales. Gaussian pyramid generates a series of image levels with gradually decreasing resolution by downsampling and smoothing the original image layer by layer, and retains the image feature information at different scales. Laplacian pyramid, on the basis of Gaussian pyramid, extracts multi-scale features of edges and details in the image by calculating the image difference between levels, and obtains sub-image sequences that can represent details of different scales. For sub-image sequences of different scales, gray level co-occurrence matrix features of 4 directions and 3 distances, local binary pattern features of 8 modes, and gradient histogram features of 9 directions are calculated respectively to obtain low-level feature sets. Gray level co-occurrence matrix features capture the spatial distribution law of gray values ​​in the image, and can reveal the directionality, contrast and consistency of image texture by analyzing the relationship between pixel pairs. Local binary pattern features are a local texture descriptor that generates a binary coding pattern by comparing the gray value of each pixel with its neighboring pixels, effectively representing the local structural information in the image. The gradient histogram feature describes the directional distribution characteristics of the edge in the image by counting the image gradient direction. These three features describe the texture, edge and local structure characteristics of the image from different angles, and together constitute a low-level feature set. The Hessian matrix extreme point detection is performed on multiple sub-image sequences of different scales to identify the feature points in the image. These feature points are usually located at key positions such as corners and edges of the image. The 64-dimensional descriptor is calculated using the Haar wavelet response to describe the local texture information around the feature points to form an intermediate feature set. At the same time, high-level semantic features are extracted from multiple sub-image sequences of different scales. Through deep learning models or other high-level feature extraction methods, complex semantic information in the image, such as scene categories and object types, is extracted to form a high-level feature set. The low-level feature set, the intermediate feature set and the high-level feature set are spliced ​​to form the original multi-scale feature set. The original multi-scale feature set is subjected to principal component dimensionality reduction processing. Through linear transformation, the original multi-dimensional data is projected into a new low-dimensional space, while retaining most of the variance information in the data to obtain a reduced dimensionality feature set. The L2 norm is calculated for each feature vector of the reduced dimension feature set, and each feature vector is normalized by dividing it by its corresponding modulus to obtain a normalized feature set. The normalized feature set is L2 regularized to obtain a regularized feature set. L2 regularization imposes additional constraints on the feature vector to prevent certain feature weights from being too large, thereby suppressing the occurrence of overfitting. Based on the mutual information value between the regularized feature set and the target object, a recursive feature elimination algorithm is used to iteratively remove the features with the smallest contribution. The recursive feature elimination algorithm removes the 10% of features with the smallest contribution each time and retrains the model until the number of remaining features reaches 50% of the original number of features, thereby obtaining the target multi-scale feature set.

[0035] Step 300: performing scene classification and object recognition on the target multi-scale feature set to obtain a scene classification result and an object recognition result;

[0036] It should be noted that the target multi-scale feature set is input into the GateNet network, and the importance weight of each scale feature is calculated through the sigmoid gate unit. By inputting the features of each scale into a single-layer fully connected network, calculating the output value, and substituting it into the sigmoid function, the importance weight of each scale feature is obtained, which reflects the contribution of different scale features in the subsequent classification task and generates a weighted feature set. The multi-scale feature fusion module is applied to the weighted feature set. The spatial attention map is calculated, and after all scale features are spliced, they are input into a single-layer fully connected network, and the final spatial attention map is obtained through the sigmoid function. The attention map is multiplied by the features of each scale, and cross-scale connections are performed, and the products are summed to obtain the fused feature set. The fused feature set is input into the fully connected layer, and the probability of each scene category is calculated through the softmax function. The calculation process of the softmax function includes taking the exponential value of each output of the fully connected layer, adding these exponential values, and dividing each exponential value by the sum to obtain the initial classification result of each scene category. A first conditional random field model is constructed, in which the observed variable is the initial classification result, and the hidden variable represents the true scene label. In order to make the classification results closer to the true label, the model defines a potential function, considers the spatial relationship between adjacent scenes, and solves by maximizing the posterior probability to obtain the scene classification result. While classifying the scene, a Faster R-CNN model is constructed based on the target multi-scale feature set for object recognition. The Faster R-CNN model consists of two main parts: a region proposal network and a classification network. The region proposal network consists of a 3x3 convolutional layer and two 1x1 convolutional layers to generate preliminary candidate regions; while the classification network uses ResNet-101 as the backbone network to further classify and regress these candidate regions. The target multi-scale feature set is input into the region proposal network to generate multiple anchor boxes of different scales and aspect ratios, and the objectness score and bounding box regression parameters are calculated for each anchor box. To ensure the quality of the candidate regions, the non-maximum suppression algorithm is used to screen the anchor boxes, retaining high-quality candidate regions with an intersection-over-union (IoU) threshold greater than 0.7, and obtaining the first candidate region set. The SENet module is applied to the first candidate region set to calculate the channel attention weight. This module performs global average pooling and maximum pooling on the feature map of each candidate region, inputs the pooling results into a two-layer fully connected network, and calculates the weight of each channel through the sigmoid function. These weights are multiplied element-by-element with the feature map in the first candidate region set to obtain the second candidate region set after channel attention optimization. The second candidate region set is input into the classification network, and a feature pyramid is constructed using the feature pyramid network structure. The construction of the feature pyramid includes adding the feature map of each layer to the feature map of the previous layer through upsampling operation, and then obtaining object features of different scales through convolution operation.Focal Loss is applied as the classification loss function for multi-scale object features. Focal Loss can dynamically adjust the loss weights of easy-to-classify samples and difficult-to-classify samples, so that the model pays more attention to the learning of difficult-to-classify samples. In this process, an online difficult example mining strategy is introduced. According to the loss value ranking calculated by Focal Loss, the top N difficult samples with the highest loss value are selected for training to improve the model's recognition ability for complex objects and obtain the initial recognition results. In order to optimize the initial recognition results, the second conditional random field model is applied. The model defines a unary potential function and a binary potential function, where the unary potential function represents the label probability of a single pixel, and the binary potential function represents the relationship between the labels of adjacent pixels. By minimizing the energy function containing these two functions, the model can optimize the boundary information of the object, reduce false detections, and finally obtain accurate object recognition results.

[0037] A random variable is assigned to each pixel in the initial recognition result, and a pixel-level conditional random field graph structure is constructed to generate a second conditional random field model. In this model, each pixel is regarded as a node, and the connection between adjacent pixels represents their association relationship. Based on the category probability distribution of each pixel in the initial recognition result, the log likelihood ratio is calculated as the univariate potential function value, and a univariate potential function set is obtained. The univariate potential function value represents the confidence that a pixel belongs to a certain category, that is, the possibility that the pixel is a specific label. By calculating these confidences, a basic category distribution is determined for each pixel. The category labels of adjacent pixel pairs in the initial recognition result are compared, and the similarity of each pair of pixels is calculated and used as the binary potential function value to generate a binary potential function set. The binary potential function is used to describe the relationship between adjacent pixels and reflect the label consistency between adjacent pixels. If the category labels of two adjacent pixels are the same, their similarity is high and the binary potential function value is also high, and vice versa. In this way, the binary potential function can help the model take into account the spatial consistency of adjacent pixels in the image, thereby better smoothing the classification results overall. The unary potential function set and the binary potential function set are combined to construct the conditional random field energy function. The classification problem is transformed into an energy minimization problem, that is, the optimal pixel classification result is found by minimizing the total energy of the conditional random field. The label distribution of each pixel is optimized by an iterative update method. Based on the conditional random field energy function, the marginal probability distribution of each pixel belonging to each category is calculated, and the classification label of each pixel is continuously updated until the energy function of the entire conditional random field model reaches the minimum value or convergence state. This process can not only eliminate the misclassification of isolated pixels, but also optimize the object boundary globally to improve the accuracy and consistency of the classification results. The connected domain analysis is performed on the optimized pixel-level classification results, and the adjacent similar pixels are merged to form the object area to obtain the preliminary optimized object boundary. Based on the preliminary optimized object boundary, the small area connected domain is filtered. The filtering process is based on the area threshold to remove the connected domains with too small area and possible noise to reduce the impact of false detection. The irregular boundaries are smoothed to remove the jagged edges and noise of the object edges, so that the outline of the object is smoother and more natural, and the optimized object recognition result is obtained.

[0038] Step 400: Perform 3D scene reconstruction and scene graph sequence modeling based on the scene classification result and the object recognition result to obtain a 3D scene model and a scene change prediction result;

[0039] Specifically, feature extraction and matching are performed on the scene classification results and object recognition results. By analyzing the scene and object features under different viewing angles, the feature correspondence across views is found. This correspondence can connect the same scene elements or objects in different camera viewing angles to form an association between multiple viewing angles. Based on the cross-view feature correspondence, the relative position and posture information between different cameras is calculated. This process involves the calculation of the essential matrix or basic matrix and the solution of the camera extrinsic parameters. Thus, the camera parameter matrix is ​​obtained, which includes information such as the position, orientation and focal length of each camera. According to the camera parameter matrix and the feature correspondence across views, the three-dimensional spatial coordinates of each feature point in the scene are calculated by triangulation to obtain the initial three-dimensional point cloud data. The initial three-dimensional point cloud data is density enhanced and smoothed to obtain an optimized three-dimensional point cloud. Density enhancement increases the density of the point cloud by interpolation or multi-view fusion, so that the distribution of the three-dimensional points is more uniform; while smoothing is to remove noise points in the point cloud, so that the surface of the point cloud is smoother and more continuous. The optimized three-dimensional point cloud data is converted into a continuous surface representation to obtain a three-dimensional scene model. The construction of the 3D scene model uses surface reconstruction algorithms, such as Poisson surface reconstruction or Delaunay triangulation. These algorithms can transform discrete 3D points into continuous 3D surfaces and generate 3D scene models with details and textures. An object relationship graph is constructed based on the 3D scene model. The object relationship graph is a way to represent the objects in the scene and the spatial relationships between them as a graph structure. The nodes of the graph represent each object in the scene, such as people, vehicles, buildings, etc., while the edges of the graph represent the spatial relationships between these objects, such as relative position, distance, orientation, etc. The graph structure can effectively express the static relationship between the objects in the scene and obtain a static scene graph. The static scene graphs at multiple time points are analyzed in time series, and a time series model of scene changes is established to generate a dynamic scene graph. The dynamic scene graph includes the object relationship at each time point and can also reflect the changing trends and laws of the relationships between objects. By analyzing the dynamic scene graph, possible scene changes in the future are predicted. Based on these predictions, a probability weight is assigned to each possible scene change, that is, the probability of occurrence is set for each possible future scene. The probability weight can help the system make intelligent decisions in an uncertain environment. The prediction results based on the dynamic scene graph are aggregated to form the scene change prediction results.

[0040] The static scene graphs at multiple time points are timestamped and aligned to a unified time axis to ensure that the dynamic changes of the scene can be correctly reflected in the time series analysis, forming a time-aligned scene graph sequence. Based on the time-aligned scene graph sequence, the change features of the nodes and edges in the scene graph are extracted to generate a scene change feature sequence. The change features of the nodes include the appearance or disappearance of the object, the change of the position, etc., while the change features of the edges involve the establishment, release or change of the relationship between the objects. These change features can reflect the evolution of the objects and their relationships in the scene over time. In the process of feature extraction, the states of the nodes and edges and their change patterns are comprehensively considered to ensure that the generated scene change feature sequence can fully represent the dynamic changes of the scene. The scene change feature sequence is segmented into time windows, and the entire scene change feature sequence is divided according to a fixed time length to form multiple feature subsequences. Each feature subsequence represents the scene change pattern within a fixed time length. The feature subsequence is input into a recurrent neural network for modeling. The recurrent neural network is suitable for processing time series data and can use its hidden state to capture the time dependency in the sequence data. By modeling the feature subsequence, the time series pattern of scene changes is learned, and the time series model of scene changes is obtained. Generate a dynamic scene graph based on the time series model. The dynamic scene graph is a model that reflects the changes of the scene over time on the time axis, and can predict the state of the scene at a future time point. The scene state information output by the time series model is mapped to the nodes and edges of the dynamic scene graph, and the future states of these nodes and edges are probabilistically predicted to obtain the state distribution of the scene graph at future moments, which indicates the possible state of each node and edge at a certain moment in the future and its corresponding probability. In order to analyze possible scene changes in the future, the scene graph state distribution is sampled, and multiple possible future scene graphs are generated by sampling to form a set of candidate future scenes. Based on the set of candidate future scenes, the probability of occurrence of each scene is calculated to obtain the scene probability distribution. The scene probability distribution reflects the importance of each future scene graph in the candidate set, that is, the possibility of occurrence of different future scenes. In order to optimize the prediction results, according to the scene probability distribution, the top K scenes with the highest probability are selected as the scene change prediction results.

[0041] Step 500: Perform multi-dimensional scene error analysis and reinforcement learning based on the three-dimensional scene model and the scene change prediction results to obtain multiple candidate monitoring action strategies;

[0042] Specifically, the three-dimensional scene model is spatially divided. The entire three-dimensional scene is divided into multiple monitoring sub-areas, each of which represents a relatively independent monitoring unit in physical space. The spatial division is based on the structural characteristics of the scene, the coverage of the monitoring camera, and the monitoring requirements. The security risk level of each monitoring sub-area is analyzed to evaluate the potential security risks in the area, such as the density of personnel gathering, the frequency of suspicious activities, the incidence of historical security events, etc. By combining these factors, a security risk level is assigned to each sub-area, and a regional risk distribution map is generated to show the risk level of different areas in the entire scene. Based on the regional risk distribution map and the scene change prediction results, the dynamic risk index of each monitoring sub-area is calculated. The dynamic risk index reflects the risk change trend of each monitoring sub-area in different time periods. The calculation process is based on the risk factors and event probabilities at different time points in the scene change prediction. By weighted summation of these factors, a risk index matrix that can reflect spatiotemporal changes is obtained, namely, the spatiotemporal risk matrix. The matrix contains the risk index of each monitoring sub-area at different time points and is a two-dimensional spatiotemporal mapping. The error analysis of the spatiotemporal risk matrix is ​​performed, and the confidence of the risk prediction is calculated by comparing the differences between historical data and predicted data. The confidence of risk prediction is used to measure the credibility of the system's prediction of future risk changes in the current scenario, helping the system identify possible uncertainties or deviations in the prediction. According to the confidence of risk prediction, the high-risk areas and time periods that need to be focused on are determined, and a list of priority monitoring targets is generated. Based on this list, the state space and action space of the monitoring environment are constructed. The state space includes the security status of all sub-areas in the current monitoring scene, camera parameter settings, and dynamic information of scene changes, while the action space contains all monitoring strategies that the system can execute, such as camera angle adjustment, focal length change, resolution setting, etc. In order to optimize the monitoring strategy, a reward function for the security monitoring effect is defined, which comprehensively considers multiple factors such as the system's monitoring coverage, event detection accuracy, and response time. When the system executes a monitoring strategy, the reward function will feedback the effect of the strategy, thereby helping the system evaluate and select the optimal monitoring solution. In this way, a reinforcement learning environment model is obtained, which contains elements such as state space, action space, and reward function. The reinforcement learning environment model is input into the deep reinforcement learning algorithm, and the optimal monitoring strategy is learned through interaction with the environment. The reinforcement learning algorithm continuously tries different monitoring strategies and adjusts the parameters of the policy network according to the feedback of the reward function, gradually converging to an optimal policy network that can maximize the reward. Multiple sets of camera parameter adjustment schemes are generated based on the initial policy network. Each set of schemes includes parameter settings such as the camera angle, focal length, and resolution, which determine the specific behavior of the camera in different monitoring sub-areas.By adjusting these parameters, high-risk areas can be monitored more effectively in different time periods, improving the detection rate and response speed of events. Multiple candidate monitoring action strategies are generated.

[0043] Step 600: Perform spatiotemporal fusion adaptive parameter calibration on multiple candidate monitoring action strategies to generate target monitoring action parameters of the security camera.

[0044] Specifically, a time series analysis is performed on multiple candidate monitoring action strategies to extract the time features of each candidate monitoring action strategy, including the execution status, change trend and corresponding response effect of the monitoring action strategy at different time points. By extracting the time features, a time feature matrix is ​​formed. Each row in the matrix represents the time feature of a candidate strategy, and each column corresponds to a specific feature value at a certain time point or a certain time period. Based on the time feature matrix, a spatial correlation analysis is performed on each candidate monitoring action strategy to evaluate the spatial similarities and differences between different monitoring strategies. For example, different camera angle adjustment strategies will produce overlapping monitoring areas in space, and different focal length settings will affect the clarity and coverage of the monitoring scene. By calculating the spatial distance between each candidate monitoring action strategy, their spatial differences are quantified to obtain a spatial correlation graph. The nodes of the spatial correlation graph represent each candidate monitoring action strategy, and the edges represent the spatial similarity or distance between them. The time feature matrix and the spatial correlation graph are used as input and imported into the spatiotemporal fusion model to generate an initial fusion strategy. The spatiotemporal fusion model can comprehensively analyze the information of the time and space dimensions and generate a preliminary monitoring strategy fusion solution. The parameter sensitivity analysis of the initial fusion strategy is performed to evaluate the influence of each parameter on the monitoring effect. By fine-tuning different parameters and observing the changes in the monitoring effect, the importance of each parameter is quantified, and a parameter sensitivity vector is generated to reflect the weight of each parameter in the overall strategy. Based on the parameter sensitivity vector, an adaptive parameter adjustment function is constructed. The adaptive parameter adjustment function can dynamically adjust the parameter configuration in the initial fusion strategy according to the sensitivity of the parameter to optimize the overall monitoring strategy. Through dynamic adjustment, an adaptive adjustment strategy is obtained. Based on the adaptive adjustment strategy, the target parameters are extracted and mapped to the control instructions of the security camera. The mapping process converts the abstract monitoring strategy parameters into specific camera control commands, such as the rotation angle, zoom focal length, resolution setting, etc. of the camera. Each control command corresponds to the specific implementation details of the monitoring action. Through this mapping, complex strategy parameters are converted into operation instructions directly executed by the camera, and the target monitoring action parameters of the security camera are generated.

[0045] In the embodiment of the present application, through multi-scale feature extraction and L2 regularization feature selection, combined with the GateNet network and the Faster R-CNN model, the accuracy of scene classification and object recognition in complex scenes is greatly improved. Based on the scene classification results and object recognition results, three-dimensional scene reconstruction and scene graph sequence modeling are performed to achieve more accurate three-dimensional scene model construction and scene change prediction. Through multi-dimensional scene error analysis and reinforcement learning, the system can automatically generate and optimize monitoring action strategies according to scene changes, thereby improving the intelligence level of the monitoring system. The adaptive parameter calibration method using time-space fusion can quickly generate target monitoring action parameters for security cameras, ensuring the real-time response capability of the system. By introducing conditional random field models and multiple optimization strategies, misidentification and false detection in complex environments are effectively reduced, and the system's anti-interference ability is improved.

[0046] In a specific embodiment, the process of executing step 100 may specifically include the following steps:

[0047] Collect multi-scene surveillance video data based on security cameras, and perform frame extraction on the multi-scene surveillance video data to obtain original image sequences of multiple scenes;

[0048] Perform noise detection on each image in the original image sequence, identify the noise type and distribution in the image, obtain a noise feature map, and perform denoising on the original image sequence based on the noise feature map to obtain a denoised image sequence;

[0049] Based on the internal parameters of the security camera, the denoised image sequence is subjected to distortion analysis, the degree and type of distortion are calculated, the distortion correction parameters are obtained, and the denoised image sequence is subjected to geometric correction according to the distortion correction parameters to obtain a distortion-free image sequence;

[0050] Analyze the illumination conditions of the distortion-free image sequence, calculate the illumination intensity and color temperature of each scene, obtain illumination balance parameters, and perform color adjustment on the distortion-free image sequence based on the illumination balance parameters to obtain a color-balanced image sequence;

[0051] The contrast analysis is performed on the color-balanced image sequence, the local and global contrast indexes are calculated, the analysis results are obtained, and the contrast is optimized according to the analysis results to obtain the preprocessed image data.

[0052] Specifically, security cameras are used to collect video data of multiple scenes. These video data usually cover different monitoring areas, such as parking lots, building entrances, corridors and other scenes. Frames are extracted from the monitoring video to obtain high-frequency image sequences. Each frame of the image represents the picture at a certain moment in the video stream. After continuous frame extraction, the original image sequence of multiple scenes is obtained. Noise detection is performed on each original image. The type of noise in the image is identified, such as Gaussian noise, salt and pepper noise, or Poisson noise. The distribution of noise is calculated by calculating the local variance of the image. To identify, Represents the pixel coordinates of the image, and the noise detection formula is:

[0053] ;

[0054] in, is the first The pixel value of the frame, is the mean value of the pixel area, is the number of images involved in the detection. By calculating the local variance, a noise feature map is generated to identify the areas with higher noise in the image. Based on the noise feature map, the original image sequence is denoised. Non-local mean filtering or bilateral filtering is used to reduce noise interference in the image. After denoising, a denoised image sequence is obtained. Processing geometric distortion in the image. Due to the nonlinear characteristics of the camera lens, barrel distortion or pincushion distortion appears in the surveillance video. In order to correct these distortions, the denoised image sequence is subjected to distortion analysis based on the internal parameters of the camera (including focal length, optical center coordinates, distortion coefficient, etc.). Distortion correction is achieved by calculating the difference between the actual position of the pixel and the ideal position. The distortion correction parameters are calculated by the following formula:

[0055] ;

[0056] in, represents the pixel radius after distortion, is the undistorted ideal pixel radius, is the radial distortion coefficient of the lens. Through this formula, the degree and type of distortion of each pixel are calculated, and the denoised image sequence is geometrically corrected based on the distortion correction parameters to obtain a distortion-free image sequence. The illumination conditions of the distortion-free image sequence are analyzed to calculate the illumination intensity and color temperature of each scene. The illumination intensity is estimated by the grayscale histogram of the image, and the color temperature is derived by the relative proportion of the RGB channels. Calculate the illumination intensity The formula is as follows:

[0057] ;

[0058] in, The images are The pixel values ​​of the red, green, and blue channels on the coordinates. The above formula is used to obtain the illumination intensity of each pixel, and the illumination balance parameters are calculated based on the global and local illumination conditions. Based on these parameters, the color of the undistorted image sequence is adjusted to ensure that the colors in the image are more natural and consistent, and a color-balanced image sequence is generated. The image is contrast optimized. The contrast of the image is analyzed by calculating the local and global contrast indicators. Local contrast It is expressed as the difference between the grayscale values ​​of each pixel in the image and its neighboring pixels, and the calculation formula is:

[0059] ;

[0060] in, is the gray value of the current pixel, and are the mean and standard deviation of the pixel neighborhood, respectively. Global contrast The calculation is based on the grayscale histogram of the entire image. According to these analysis results, the image contrast is optimized by using methods such as adaptive histogram equalization, making the details in the image clearer and the brightness levels more distinct, and finally generating pre-processed high-quality image data.

[0061] In a specific embodiment, the process of executing step 200 may specifically include the following steps:

[0062] Perform Gaussian pyramid and Laplacian pyramid processing on the preprocessed image data to generate multiple sub-image sequences of different scales;

[0063] For multiple sub-image sequences of different scales, gray-level co-occurrence matrix features of four directions and three distances, local binary pattern features of eight modes, and nine directional gradient histogram features are calculated to obtain a low-level feature set.

[0064] Perform Hessian matrix extreme point detection on multiple sub-image sequences of different scales, and use Haar wavelet response to calculate 64-dimensional descriptors to obtain intermediate feature sets, and perform high-level semantic feature extraction on multiple sub-image sequences of different scales to obtain high-level feature sets;

[0065] Perform feature concatenation on the low-level feature set, the intermediate feature set and the high-level feature set to obtain the original multi-scale feature set, and perform principal component dimensionality reduction on the original multi-scale feature set to obtain a reduced dimensionality feature set;

[0066] Calculate the L2 norm of each feature vector of the reduced dimension feature set, and divide each feature vector of the reduced dimension feature set by the corresponding modulus length for normalization to obtain a normalized feature set;

[0067] The normalized feature set is subjected to L2 regularization to obtain a regularized feature set. Based on the mutual information value between the regularized feature set and the target object, a recursive feature elimination algorithm is used to iteratively remove the features with the smallest contribution, removing 10% of the features each time until the number of remaining features reaches 50% of the original number of features, thus obtaining the target multi-scale feature set.

[0068] Specifically, the preprocessed image data is processed by Gaussian pyramid and Laplacian pyramid. The Gaussian pyramid is generated by Gaussian blurring and downsampling the image layer by layer, and the resolution of each layer is half lower than the previous layer. Its construction process is expressed by the following formula.

[0069] ;

[0070] in, Represents the pyramid The image pixel values ​​of the layer, Indicates The image pixel values ​​of the layer, is the weight of the Gaussian convolution kernel, Indicates the radius of the convolution kernel. Through layer-by-layer downsampling and Gaussian smoothing, a multi-level Gaussian pyramid is generated. Based on the generated Gaussian pyramid, the Laplacian pyramid is calculated. The Laplacian pyramid is constructed by the difference between two adjacent layers of the Gaussian pyramid, which can effectively extract the edge information in the image. Its construction process is expressed by the following formula:

[0071] ;

[0072] in, It is the Laplace pyramid The image pixel value of the layer is The Gaussian pyramid image after upsampling The difference between the layers of Gaussian pyramid images is calculated. Through Gaussian pyramid and Laplacian pyramid processing, multiple sub-image sequences of different scales are generated. A variety of low-level image features are calculated for sub-image sequences of different scales. The first is the grayscale co-occurrence matrix feature. The grayscale co-occurrence matrix describes the image texture features by calculating the co-occurrence of grayscale values ​​of pixel pairs in the image. For each sub-image sequence, the grayscale co-occurrence matrix features are calculated in 4 directions (0 degrees, 45 degrees, 90 degrees, 135 degrees) and 3 distances (1, 2, 3 pixels). The calculation formula of the grayscale co-occurrence matrix is ​​as follows:

[0073] ;

[0074] in, Indicates the gray value is and The pixel pairs are at a distance and angle The co-occurrence probability under and is in the direction The pixel offset under is the number of pixel pairs that meet the conditions, and are the width and height of the image respectively. The gray-level co-occurrence matrix can effectively capture the texture features of the image and generate a multi-dimensional texture descriptor through the combination of different directions and distances. Calculate the local binary pattern (LBP) feature. LBP is an important method for describing the local texture of an image. It generates a binary pattern by comparing the gray value relationship between each pixel and its neighboring pixels. For each pixel , if the gray value of one of its neighboring pixels is greater than or equal to the pixel, then output 1, otherwise output 0. The formula is as follows:

[0075] ;

[0076] in, is the first pixel values, is a symbolic function defined as:

[0077] ;

[0078] By calculating the LBP features in different modes, a local binary pattern feature set containing 8 modes is generated to effectively describe the local texture information in the image. The directional gradient histogram features are calculated to describe the edge and shape features by counting the gradient direction of the image. The image is divided into several small blocks, the gradient direction is calculated for the pixels in each small block, and the gradient value is quantized and stored in the histogram. The formula for calculating the gradient direction is as follows:

[0079] ;

[0080] in, The image is The gradient direction at and are the gradients of the image in the horizontal and vertical directions respectively. HOG features are calculated in 9 directions to obtain a feature vector that can describe the edge distribution of the image. For mid-level feature extraction, the Hessian matrix is ​​used for extreme point detection, and the Haar wavelet response is used to calculate the 64-dimensional descriptor. The Hessian matrix is ​​used to detect corners and edges in the image, and the extreme points are determined by calculating the second-order derivatives of the image in different directions. Its expression is:

[0081] ;

[0082] in, and are the second-order derivatives of the image in the horizontal and vertical directions, respectively. is a mixed second-order derivative. Extreme point detection based on the Hessian matrix can locate corner points in the image, while the Haar wavelet response is used to describe the local features around these extreme points, generating a 64-dimensional descriptor to form an intermediate feature set. For high-level feature extraction, a deep learning model is used to process multi-scale image sequences. The deep learning model extracts high-level semantic features of the image, such as object category, scene background, etc., through a convolutional neural network. The low-level features, intermediate features, and high-level features are concatenated to form the original multi-scale feature set. Principal component analysis is used to perform dimensionality reduction. The main components of the data are selected by constructing a covariance matrix and calculating its eigenvectors and eigenvalues ​​to reduce the dimension of the features. Through dimensionality reduction, a reduced dimensionality feature set is obtained. On the reduced dimensionality feature set, the L2 norm is calculated for each eigenvector and normalized. The L2 norm is defined as

[0083] ;

[0084] in, The eigenvector Quantity, is the dimension of the feature vector. Each feature vector is divided by its L2 norm to achieve normalization and obtain a normalized feature set. The normalized feature set is L2 regularized and the feature weight is suppressed by adding an L2 regularization term to the loss function to obtain a regularized feature set. Based on the mutual information value between the regularized feature set and the target object, a recursive feature elimination algorithm is used to iteratively remove the feature with the smallest contribution. Mutual information value Defined as:

[0085] ;

[0086] in, is a variable and The joint probability distribution of and They are their marginal distributions respectively. The 10% of features with the smallest contribution are removed each time until the number of remaining features reaches 50% of the original number of features, and the target multi-scale feature set is obtained.

[0087] In a specific embodiment, the process of executing step 300 may specifically include the following steps:

[0088] The target multi-scale feature set is input into the GateNet network, and the importance weight of each scale feature is calculated through the sigmoid gating unit, where the importance weight is equal to the result of the sigmoid function acting on the output of a single-layer fully connected network to obtain a weighted feature set;

[0089] The multi-scale feature fusion module is applied to the weighted feature set. First, the spatial attention map is calculated, that is, all scale features are concatenated and then passed through a single-layer fully connected network and a sigmoid function. Then, cross-scale connection is performed, that is, the attention map is multiplied with each scale feature and summed to obtain the fused feature set.

[0090] The fused feature set is input into the fully connected layer, and then the probability of each scene category is calculated through the softmax function, which includes taking the exponent of each output of the fully connected layer and dividing it by the sum of all output exponents to obtain the initial classification result;

[0091] Construct the first conditional random field model, where the observed variable is the initial classification result, the hidden variable is the real scene label, and the potential function is defined to consider the spatial relationship between adjacent scenes. The scene classification result is obtained by maximizing the posterior probability.

[0092] Based on the target multi-scale feature set, a Faster R-CNN model is constructed, including a region proposal network and a classification network. The region proposal network consists of a 3x3 convolutional layer and two 1x1 convolutional layers, and the classification network uses ResNet-101 as the backbone network.

[0093] Input the target multi-scale feature set into the region proposal network to generate multiple anchor boxes of different scales and aspect ratios, calculate the objectness score and bounding box regression parameters of each anchor box, and use the non-maximum suppression algorithm to retain high-quality candidate regions with an intersection-over-union ratio threshold greater than 0.7 to obtain the first candidate region set;

[0094] Apply the SENet module to the first candidate region set and calculate the channel attention weights, which includes global average pooling and maximum pooling of the feature maps of the candidate regions. The results are input into a two-layer fully connected network and then obtained through a sigmoid function. The weights are then multiplied by the first candidate region set to obtain the second candidate region set.

[0095] The second candidate region set is input into the classification network, and a feature pyramid is constructed using a feature pyramid network structure, specifically including adding the feature map of each layer to the upsampled feature map of the previous layer and performing a convolution operation to obtain multi-scale object features;

[0096] Focal Loss is applied as the classification loss function for multi-scale object features, where the loss value is related to the prediction probability and difficulty level. An online difficult example mining strategy is used to select the top N difficult samples with the highest loss value for training to obtain the initial recognition results.

[0097] The second conditional random field model is applied to the initial recognition results, and a unary potential function and a binary potential function are defined. The unary potential function represents the label probability of a single pixel, and the binary potential function represents the relationship between adjacent pixel labels. The object boundary is optimized and false detections are eliminated by minimizing the energy function containing these two functions to obtain the object recognition result.

[0098] Specifically, the target multi-scale feature set is input into the GateNet network, and the GateNet network calculates the importance weight of each scale feature through the sigmoid gate unit. The input of the GateNet network is the feature representation of different scales, which comes from different resolutions or different spatial feature maps. In order to capture the importance of features at each scale, the weight of the feature is calculated through a single-layer fully connected network, and the result is substituted into the sigmoid function to limit the weight between 0 and 1. The formula is expressed as:

[0099] ;

[0100] in, Indicates The weight of the scale feature, is the weight matrix of the fully connected layer, It is The characteristics of the scale, is the bias term, Represents the sigmoid function. This process generates a weighted feature set, which is represented as the linear weighted result of all scale features, and can dynamically adjust the importance of each scale feature. Apply the multi-scale feature fusion module to the weighted feature set. Improve the feature representation capability through the spatial attention mechanism. Concatenate the features of all scales together to construct a global feature representation. Then generate the spatial attention map through a single-layer fully connected network and the sigmoid function, and the formula is expressed as:

[0101]

[0102] in, is the generated spatial attention map, is the weight matrix of the fully connected layer, is the concatenated multi-scale feature, is the bias term. The attention map is multiplied by each scale feature and weighted summed across scales to generate a fused feature set. The generated fused feature set is input into the fully connected layer, and the probability of each scene category is calculated by the softmax function. The softmax function is used to map the fused feature to the probability distribution of scene classification, and its calculation formula is as follows:

[0103] ;

[0104] in, Indicates that the scene belongs to The probability of the class, is the output of the fully connected layer, is the number of scene categories. Softmax obtains the initial classification result by normalizing the exponential output of each category to a probability distribution. A first conditional random field model is constructed to optimize the classification result. The conditional random field is a graph model that smoothes the classification result by modeling the spatial relationship between adjacent pixels. Its definition potential function takes into account the spatial association between adjacent scenes, and the formula is expressed as:

[0105] ;

[0106] in, is the energy function of the conditional random field, is the potential function of a single pixel, indicating the confidence of the pixel classification It is a binary potential function between adjacent pixels, which indicates the label relationship between adjacent pixels. By maximizing the posterior probability, the optimized scene classification result is obtained. Based on the target multi-scale feature set, the Faster R-CNN model is constructed for object detection. The Faster R-CNN model consists of two parts: the region proposal network and the classification network. The region proposal network consists of a 3x3 convolutional layer and two 1x1 convolutional layers, which are used to generate candidate boxes. The formula is expressed as:

[0107] ;

[0108] in, It is The objectness score of the candidate boxes, represents the convolution operation, is the input feature map, is the weight parameter of the candidate box. By evaluating candidate boxes of different scales and aspect ratios, multiple anchor boxes are generated, and the bounding box regression parameters of each anchor box are calculated. In order to optimize the candidate regions, the non-maximum suppression algorithm is used to filter out candidate boxes with high overlap, and only high-quality candidate regions with an intersection over union (IoU) greater than 0.7 are retained to obtain the first candidate region set. The SENet module is applied to the first candidate region set, and SENet further improves the feature representation by calculating the channel attention weights. Global average pooling and maximum pooling are performed on the feature map of each candidate region, and then the pooling results are input into a two-layer fully connected network, and the weights are generated by the sigmoid function. The formula is expressed as

[0109] ;

[0110] in, Indicates The attention weights of the channels, GAP and GMP represent global average pooling and maximum pooling respectively. and is the weight matrix of the fully connected layer. The weights are used to adjust the feature importance of each channel and finally generate the second candidate region set. The second candidate region set is input into the classification network, and a multi-scale feature pyramid is constructed using the feature pyramid network structure. Each layer of the feature pyramid is added to the upsampled feature map of the previous layer and then convolved to obtain multi-scale object features with higher resolution. For the classification loss function, Focal Loss is used to deal with the problem of class imbalance. Focal Loss introduces an adjustment factor into the loss function so that difficult-to-classify samples contribute more to the loss. Its formula is:

[0111] ;

[0112] in, represents the predicted probability of the correct category, is the balancing factor, is a regulating factor. Focal Loss can reduce the weight of simple samples and increase the attention to difficult-to-classify samples. During the training process, an online difficult example mining strategy is used to select the top N difficult samples with the highest loss value for training, improve the robustness of the model, and obtain the initial recognition results. In order to optimize the initial recognition results, the second conditional random field model is applied to eliminate false detections and optimize object boundaries. The conditional random field model uses unary potential functions and binary potential functions to model the label relationship between pixels. The unary potential function represents the confidence that a single pixel belongs to a certain category, and the binary potential function represents the consistency of adjacent pixel labels. The energy function is as follows:

[0113] ;

[0114] in, is a univariate potential function, It is a binary potential function that optimizes the pixel-level classification results by minimizing the energy function, ensuring the accuracy of the object boundary, and eliminating noise and false targets in the detection, and finally obtains the object recognition result.

[0115] In a specific embodiment, the execution step applies a second conditional random field model to the initial recognition result, defines a unary potential function and a binary potential function, wherein the unary potential function represents the label probability of a single pixel, and the binary potential function represents the relationship between adjacent pixel labels, and optimizes the object boundary and eliminates false detections by minimizing an energy function containing these two functions, and the process of obtaining the object recognition result can specifically include the following steps:

[0116] Assign a random variable to each pixel in the initial recognition result, construct a pixel-level conditional random field graph structure, obtain a second conditional random field model, and calculate the log-likelihood ratio as a univariate potential function value based on the category probability distribution of each pixel in the initial recognition result to obtain a univariate potential function set;

[0117] The category labels of adjacent pixel pairs in the initial recognition results are compared, and the similarity of the pixel pairs is calculated as the binary potential function value to obtain a binary potential function set, and the univariate potential function set and the binary potential function set are combined to construct an energy function, where the energy function is a weighted sum of the univariate potential function and the binary potential function, and a conditional random field energy function is obtained;

[0118] According to the conditional random field energy function, each pixel is iteratively updated, the edge probability distribution of each category is calculated, and the optimized pixel-level classification results are obtained. The connected domain analysis is performed on the optimized pixel-level classification results, and adjacent pixels of the same type are merged to form an object area, and the preliminary optimized object boundary is obtained;

[0119] Based on the preliminary optimized object boundaries, small connected domains are filtered and irregular boundaries are smoothed to obtain object recognition results.

[0120] Specifically, a random variable is assigned to each pixel, and a pixel-level conditional random field graph structure is constructed to form a second conditional random field model. This model further refines the initial recognition results by considering the spatial relationship between pixels. Each pixel in the initial recognition result is assigned a random variable, which represents the label state of the pixel. For example, for a pixels of the image, and the labels of all pixels are represented as a vector , where each Indicates The class label of each pixel. In order to calculate the energy function of the conditional random field, a univariate potential function and a binary potential function are defined. The univariate potential function reflects the confidence that a single pixel belongs to a certain class. In the conditional random field model, the univariate potential function is expressed as a log-likelihood ratio, which is the logarithmic form of the class probability distribution of each pixel in the initial recognition result. pixels, and its unary potential function value is expressed as:

[0121] ;

[0122] in, It is The log-likelihood ratio of pixels is the value of the univariate potential function, It is The probability that a pixel is classified as the current category. By calculating the log-likelihood ratio of each pixel, a set of univariate potential functions is generated to reflect the classification confidence of each pixel in the initial recognition result. The binary potential function is calculated to describe the relationship between adjacent pixels. The binary potential function represents the label consistency of adjacent pixels, that is, if adjacent pixels have the same category label, the value of the binary potential function will be smaller, otherwise it will be larger. Let and pixels are adjacent pixel pairs, and their binary potential function is defined as:

[0123] ;

[0124] in, It is and The binary potential function value between pixels, It's pixels and The similarity function between them is calculated by color, gradient and other information. is an indicator function, when The value is 1 when , otherwise it is 0. By calculating the binary potential function values ​​of all adjacent pixel pairs in the image, a binary potential function set is obtained, which reflects the similarity and label consistency of different pixel pairs in the image. The unary potential function set and the binary potential function set are combined together to construct the energy function of the conditional random field model. The definition of the energy function is as follows:

[0125] ;

[0126] in, is the conditional random field energy function of the entire image, is a univariate potential function, is the binary potential function, is the sum of the unary potential functions of all pixels, Represents the sum of the binary potential functions of all adjacent pixel pairs. Energy function The size of reflects the “cost” of the pixel label distribution of the entire image. The goal is to optimize the pixel-level classification results by minimizing the energy function. In order to minimize the conditional random field energy function, an iterative optimization method is used, such as maximizing the posterior probability or a graph-cut based algorithm. The label distribution of each pixel is updated in each iteration, and the label assignment is adjusted according to the current energy function value. For each pixel , and its update rules are as follows:

[0127] ;

[0128] in, It's pixels The updated label probability distribution, L is the set of all possible labels, Represents pixels and the sum of the binary potential functions of its adjacent pixels. Through iterative updates, the label distribution of each pixel is gradually optimized to obtain the optimized pixel-level classification results. Connected domain analysis is performed on the optimized pixel-level classification results. Adjacent pixels of the same type are merged into object regions. Traverse all pixels in the image and classify adjacent pixels with the same label into a connected domain. These connected domains represent different object regions in the image. Through connected domain analysis, a preliminary optimized object boundary is formed. Based on the preliminary optimized object boundary, small-area connected domains are filtered. Small-area connected domains represent noise or non-target areas, so an area threshold is set to remove connected domains with an area smaller than the threshold from the results. Smooth irregular boundaries to eliminate jagged edges or protrusions in the boundaries, making the object contour smoother and more natural. Smoothing is achieved through Gaussian smoothing or morphological operations (such as erosion and dilation). For example, the following formula is used to smooth the object boundary:

[0129] ;

[0130] in, is the smoothed boundary image, is the original boundary image, is the smoothing window, is the weight of the smoothing window and is in the form of a Gaussian kernel. Through the smoothing operation, the boundary of the object can be made more continuous and natural. After the optimization of the conditional random field model, the connected domain analysis and the boundary smoothing processing, the refined object recognition result is obtained.

[0131] In a specific embodiment, the process of executing step 400 may specifically include the following steps:

[0132] Perform feature extraction and matching on the scene classification results and object recognition results to obtain cross-view feature correspondences, and calculate the relative position and posture between cameras based on the cross-view feature correspondences to obtain the camera parameter matrix;

[0133] According to the camera parameter matrix and feature correspondence, the coordinates of the three-dimensional space points are calculated to obtain the initial three-dimensional point cloud data, and the initial three-dimensional point cloud data is density enhanced and smoothed to obtain the optimized three-dimensional point cloud;

[0134] Convert the optimized 3D point cloud into a continuous surface representation to obtain a 3D scene model;

[0135] Construct an object relationship graph based on the 3D scene model, represent the objects in the scene and the spatial relationships between them as a graph structure, and obtain a static scene graph;

[0136] Perform time series analysis on static scene graphs at multiple time points, establish a time series model of scene changes, obtain a dynamic scene graph, and based on the dynamic scene graph, predict possible scene changes in the future, assign probability weights to each change situation, and obtain scene change prediction results.

[0137] Specifically, feature extraction and matching are performed on the scene classification results and object recognition results. For each image from each perspective, global features are extracted from the scene classification results, such as the category label of the scene, the spatial distribution of the main objects, etc.; local features are extracted from the object recognition results, such as the object's bounding box, category, key point position, etc. These features are obtained through various feature extraction algorithms, such as scale-invariant feature transformation or accelerated robust features. Suppose the features under perspective A and perspective B are and ,in and Respectively represent different objects in the perspective and In order to establish the feature correspondence across views, and Perform feature matching. Use the Euclidean distance metric to measure the similarity between two feature vectors. Suppose the feature vector and The Euclidean distance between them is:

[0138] ;

[0139] in, represents the dimension of the feature vector, and The eigenvectors are and No. By calculating the distance between all feature vector pairs, we find the most similar feature pairs and establish feature correspondences across views. Based on the feature correspondences across views, we calculate the relative position and posture between cameras. Assuming the view angle The camera coordinate system is , viewing angle The camera coordinate system is In order to calculate the relative position and posture between the two, the relationship between the two coordinate systems is described by the basic matrix or the essential matrix. The calculation formula is as follows:

[0140] ;

[0141] in, and Perspective and perspective The internal parameter matrix of is the fundamental matrix. The intrinsic matrix describes the intrinsic properties of the camera, such as focal length and principal point offset, while the fundamental matrix captures the geometric relationship between the perspectives. By decomposing the essential matrix, we get the relative rotation matrix between the cameras. and translation vector , thereby constructing the camera parameter matrix:

[0142] ;

[0143] in, is the camera parameter matrix, is the internal parameter matrix, is the rotation matrix, is the translation vector. This matrix describes the complete projection relationship of the camera, mapping the 3D world coordinates to the 2D image plane. According to the camera parameter matrix and the feature correspondence, the coordinates of the 3D space point are calculated using triangulation. Assuming that the viewing angle and perspective In the figure, the coordinates of the feature points of the same object on the image plane are and . The coordinates of the 3D point Calculated by the following formula:

[0144] ;

[0145] Among them, triangulate(·) represents the triangulation process, and is the camera parameter matrix, and are the image coordinates of the feature points in two perspectives. The initial three-dimensional point cloud data is obtained by triangulating all matched feature point pairs. The initial three-dimensional point cloud data is density enhanced and smoothed. Density enhancement uses a multi-view stereo matching algorithm (such as PatchMatch Stereo) to increase the density of the point cloud through interpolation and multi-view fusion. Smoothing removes noise and outliers through point cloud filtering (such as voxel grid filtering, statistical filtering, etc.). The optimized three-dimensional point cloud is converted into a continuous surface representation to generate a three-dimensional scene model. Surface reconstruction uses algorithms such as Poisson reconstruction or Delaunay triangulation to convert discrete point cloud data into a continuous three-dimensional surface. Assume that each point in the point cloud is , whose surface is represented as:

[0146] ;

[0147] in, is the implicit function of the surface model, is the shape function, describing the point Contributions to surface models, Yes By solving the Cainite function, a smooth and continuous 3D scene model is generated. Based on the constructed 3D scene model, an object relationship graph is constructed to represent the objects in the scene and the spatial relationships between them as a graph structure. The nodes of the object relationship graph represent each object in the scene, and the edges represent the spatial relationships between them, such as relative distance, direction, etc. Suppose there are objects, respectively, , their relationship diagram is represented as ,in is a collection of objects, is a set of relationships between objects. The spatial relationship between objects is expressed by the following formula:

[0148] ;

[0149] in, Representation Object and The strength of the relationship between is the Euclidean distance between them, is a scale parameter that controls the attenuation of the relationship strength. Through this formula, a static scene graph is constructed to represent the spatial layout and relationship of objects in the scene. A time series analysis is performed on the static scene graphs at multiple time points, and a time series model of scene changes is established to obtain a dynamic scene graph. The time series model captures the changes in the object relationship in the scene over time. Assume that at time point The scene diagram is , then the time series model is expressed as ,in is the time step. The time series model is modeled by graph convolutional networks or recurrent neural networks to capture dependencies in the time dimension. Based on the dynamic scene graph, possible scene changes in the future are predicted, and a probability weight is assigned to each change. By inferring the time series model, the scene graph at a future time point is predicted. , and calculate the probability of each scene change. Suppose the probability of future scene changes is , then all possible scene changes are sampled to obtain the scene change prediction results.

[0150] In a specific embodiment, the execution step performs a time series analysis on the static scene graph at multiple time points, establishes a time series model of scene changes, obtains a dynamic scene graph, and predicts possible future scene changes based on the dynamic scene graph, and assigns a probability weight to each change situation. The process of obtaining the scene change prediction result can specifically include the following steps:

[0151] Perform timestamp alignment on static scene graphs at multiple time points to obtain a time-aligned scene graph sequence, and based on the time-aligned scene graph sequence, extract the change features of nodes and edges in the scene graph to obtain a scene change feature sequence;

[0152] The scene change feature sequence is segmented into time windows to obtain multiple feature subsequences of fixed length, and the feature subsequences are input into a recurrent neural network to model the scene changes and obtain a time series model of the scene changes;

[0153] Generate a dynamic scene graph based on the time series model, perform probability prediction on the nodes and edges in the dynamic scene graph, obtain the state distribution of the scene graph at the future moment, and sample the state distribution of the scene graph to generate multiple possible future scene graphs to obtain a set of candidate future scenes;

[0154] Based on the set of candidate future scenes, the occurrence probability of each scene is calculated to obtain the scene probability distribution, and according to the scene probability distribution, the top K scenes with the highest probability are selected as the scene change prediction results.

[0155] Specifically, the timestamps of static scene graphs at multiple time points are aligned. Assume that there are multiple scene graphs at different time points , each scene graph By node set and edge sets They respectively represent the objects in the scene and the relationships between objects. In order to align timestamps, the scene graphs are arranged in chronological order and the time differences between different scene graphs are eliminated. All scene graphs are mapped to a unified time axis through methods such as linear interpolation or synchronous sampling to obtain a time-aligned scene graph sequence. Based on the time-aligned scene graph sequence, the change features of the nodes and edges in each scene graph are extracted to form a scene change feature sequence. The change feature of the node is the state change of the object at different time points, such as position, speed, direction, etc.; the change feature of the edge describes the change in the relationship between objects, such as relative distance, interaction frequency, etc. Suppose at time point Moment, Node The state vector of , then the node At the point in time and The change characteristics between them are expressed as:

[0156] ;

[0157] in, Representation Node At the point in time Similarly, for the edge , its change characteristics are defined as:

[0158] ;

[0159] in, It is the edge At the point in time The amount of change, Representation Object and Between the time points By calculating the change features of all nodes and edges, a scene change feature sequence containing multiple time points is obtained. The scene change feature sequence is segmented into time windows. The long feature sequence is divided into multiple feature subsequences of fixed length for modeling analysis. Assume that the time window length is set to , then the form of each feature subsequence is ,in, Indicates at a point in time The feature subsequence is input into the recurrent neural network or long short-term memory network for modeling. Suppose the input sequence is , the hidden state update formula of the recurrent neural network is as follows:

[0160] ;

[0161] in, Indicates at a point in time The hidden state of is the weight matrix of the hidden state, is the weight matrix of the input features, is the bias term, is a nonlinear activation function. Through this formula, the recurrent neural network gradually transmits information over time and learns a time series model of scene changes. Based on the trained time series model, a dynamic scene graph is generated. The dynamic scene graph contains the state distribution of nodes and edges at each time point, indicating the changing trend of objects and relationships in the scene. In order to predict the state of the scene graph at a future time point, probabilistic predictions are made for the nodes and edges in the dynamic scene graph. Assume that at time point Node The state probability distribution of , then the future time point The state distribution at is expressed as:

[0162] ;

[0163] in, Is a node The state transition probability represents the transition from time point arrive The state changes of the scene graph. Through recursive prediction, the state distribution of the scene graph at the future moment is obtained. In order to generate multiple possible future scene graphs, the state distribution of the scene graph is sampled to generate a set of candidate future scenes. In the set of candidate future scenes, each scene represents a possible future state. In order to quantify the probability of each scene occurring, the state of each candidate scene is calculated probabilistically. Let the candidate scene The state probability is , then the calculation formula is

[0164] ;

[0165] in, and Represent the state probabilities of nodes and edges respectively. By calculating the probability distribution of each candidate scene, we get the scene probability distribution. According to the scene probability distribution, we select the top scene with the highest probability. The scene is taken as the scene change prediction result.

[0166] In a specific embodiment, the process of executing step S500 may specifically include the following steps:

[0167] The three-dimensional scene model is spatially divided to obtain a plurality of monitoring sub-areas, and the security risk level of each monitoring sub-area is analyzed to obtain a regional risk distribution map;

[0168] Based on the regional risk distribution map and the scene change prediction result, the dynamic risk index of each monitored sub-area is calculated to obtain a spatiotemporal risk matrix, and an error analysis is performed on the spatiotemporal risk matrix to obtain a risk prediction confidence;

[0169] According to the risk prediction confidence, high-risk areas and time periods that need to be focused on are determined to obtain a priority monitoring target list, and based on the priority monitoring target list, the state space and action space of the monitoring environment are constructed, and the reward function of the security monitoring effect is defined to obtain a reinforcement learning environment model;

[0170] The reinforcement learning environment model is input into a deep reinforcement learning algorithm, and the optimal monitoring strategy is learned by interacting with the environment to obtain an initial strategy network. The initial strategy network is used to generate multiple sets of camera parameter adjustment schemes, including angle, focal length and resolution, to obtain multiple candidate monitoring action strategies.

[0171] Specifically, the 3D scene model is spatially divided. The 3D scene model consists of a series of dense 3D point clouds or 3D grids, which divide the 3D scene into multiple monitoring sub-areas. The spatial division is based on the physical structure of the scene (such as buildings, roads, parking lots, etc.), or based on the coverage of the camera. Assume that the set of all point cloud data points in the 3D scene model is , each point cloud data point Contains the three-dimensional coordinates of the point. Through space partitioning algorithms (such as octree partitioning or grid partitioning), the system converts point cloud data Divide into Monitoring sub-areas , where each monitoring sub-area Contains the subset of point cloud data belonging to this area The goal of the division is to make each monitoring sub-area relatively independent in space, and each area can be effectively covered by at least one camera. Analyze the security risk level of each monitoring sub-area to obtain a regional risk distribution map. The security risk level is evaluated based on multiple factors, including historical event data, real-time monitoring information (such as personnel density, abnormal behavior detection results, etc.) and environmental characteristics of the scene (such as the number of entrances and exits, visual obstacles, etc.). Suppose a monitoring sub-area The risk level is , and its calculation formula is:

[0172] ;

[0173] in, Indicates Risk factors in the region The value of is the corresponding weight coefficient, Represents the number of all risk factors. By combining the impact of different risk factors, the risk level of each sub-area is calculated The risk level results of all sub-areas are combined to form a regional risk distribution map, which shows the risk distribution in the entire monitoring scenario. Based on the regional risk distribution map and the scenario change prediction results, the dynamic risk index of each monitoring sub-area is calculated. The dynamic risk index considers the risk level at the current moment and reflects the risk change trend in the future. Assume that at time point Monitoring sub-area The dynamic risk index is , and its calculation formula is as follows:

[0174] ;

[0175] in, is the risk level at the current moment, In the future The risk change at is a decay coefficient, indicating that the influence weight of future moments decreases over time. is the prediction time window. Through this formula, the dynamic risk index of each monitoring sub-area at different time points is obtained to form a spatiotemporal risk matrix , each element represents the dynamic risk index of each monitoring sub-area. In order to evaluate the reliability of the spatiotemporal risk matrix, the spatiotemporal risk matrix is ​​subjected to error analysis and the confidence of risk prediction is calculated. The confidence indicates the reliability of the system for the current risk prediction results. The confidence is calculated by comparing the risk prediction error with the historical actual risk. Assuming that a monitoring sub-area The prediction error is , then its confidence is expressed as:

[0176] ;

[0177] in, It is the monitoring sub-area The confidence level, is the prediction error in this region, is the standard deviation of the error. By calculating the confidence of all sub-areas, we get the risk prediction confidence matrix , indicating the credibility of the system's risk prediction for each monitored sub-area. Based on the confidence level of risk prediction, determine the high-risk areas and time periods that need to be focused on. Select a dynamic risk index Higher and more confident The larger sub-areas are regarded as high-risk areas, and the high-risk areas and time periods are combined to form a priority monitoring target list. Based on the priority monitoring target list, the state space and action space of the monitoring environment are constructed. State space Contains all possible monitoring scene states, each state consists of information such as the risk level of the monitoring sub-area, the dynamic risk index, and the working status of the camera. Action Space This includes all possible monitoring actions, such as camera angle adjustment, focal length setting, resolution change, etc. In order to evaluate the effects of different monitoring strategies, a reward function for security monitoring effect is defined. , whose expression is:

[0178] ;

[0179] in, is in state Next action Rewards received, Coverage Indicates the coverage of the current monitoring strategy, Detection represents the accuracy of event detection, Cost is the cost of performing the monitoring action, is the weight coefficient. Through the reward function, the coverage effect, detection effect and execution cost of the monitoring strategy are balanced to evaluate the pros and cons of different strategies. The reinforcement learning environment model is input into the deep reinforcement learning algorithm, such as the deep Q network or the deep deterministic policy gradient algorithm, to learn the optimal monitoring strategy through interaction with the environment. In the reinforcement learning process, different monitoring strategies are tentatively executed, and the parameters of the policy network are adjusted according to the feedback of the reward function. Assume that the state Next action The policy network output is , then the goal of the strategy update is to minimize the following loss function:

[0180] ;

[0181] in, is the loss function, It’s an instant reward. is the discount factor, is the value function of the current strategy, is the parameter of the policy network. By minimizing the loss function, the parameters of the policy network are continuously optimized, and finally an initial policy network is obtained. The initial policy network is used to generate multiple sets of camera parameter adjustment schemes, including the camera angle, focal length, and resolution. Each set of schemes represents a candidate monitoring action strategy for different monitoring sub-areas and time periods. In this way, the system can flexibly adjust the monitoring strategy in different scenarios and time periods to maximize the effect of security monitoring.

[0182] In a specific embodiment, the process of executing step S600 may specifically include the following steps:

[0183] Performing time series analysis on the multiple candidate monitoring action strategies, extracting the time characteristics of each candidate monitoring action strategy, obtaining a time characteristic matrix, and performing spatial correlation analysis on each candidate monitoring action strategy based on the time characteristic matrix, calculating the spatial distance between the candidate monitoring action strategies, and obtaining a spatial correlation graph;

[0184] Input the time feature matrix and the spatial correlation graph into the spatiotemporal fusion model to generate an initial fusion strategy, and perform parameter sensitivity analysis on the initial fusion strategy to calculate the influence of each parameter on the monitoring effect and obtain a parameter sensitivity vector;

[0185] Based on the parameter sensitivity vector, an adaptive parameter adjustment function is constructed to dynamically adjust the parameters of the initial fusion strategy to obtain an adaptive adjustment strategy, and based on the adaptive adjustment strategy, the target parameters are extracted and mapped to the control instructions of the security camera to generate the target monitoring action parameters of the security camera.

[0186] Specifically, the temporal features of each candidate monitoring action strategy are extracted. Candidate monitoring action strategies refer to various operation plans set by the system for the camera at different time periods and spatial positions, including specific parameter settings such as angle, focal length, and resolution. In order to capture the changing rules of these strategies in the time dimension, a time series analysis is performed on each monitoring action strategy. Assume that at time point At this point, the parameters of the candidate monitoring action strategy are set to , where each Indicates Monitoring parameters at time point The values ​​of the camera, such as the focal length and pitch angle of the camera. In order to obtain the time feature matrix, the change of each parameter in the entire time series is statistically analyzed. For example, the statistical characteristics such as the mean value, variance, and change rate of each parameter are calculated. In order to better reflect the change trend of the parameter, the change rate of the parameter is calculated. , which is defined as:

[0187] ;

[0188] in, is the parameter at time point and By calculating the time characteristics of all candidate monitoring action strategies, we can get the time feature matrix , each row represents the time feature of a strategy, and each column represents the value of a certain time statistical feature. Finally, the spatial correlation analysis of each candidate monitoring action strategy is performed to evaluate the spatial similarities and differences of different monitoring strategies. Each candidate monitoring action strategy covers different monitoring sub-areas in space, and their spatial correlation is quantified by calculating the spatial distance between strategies. The spatial characteristics of the monitoring action strategy are , where each This means that the strategy The coverage or risk level of each monitoring sub-area. Candidate strategies and The spatial distance between them is calculated by the following formula:

[0189] ;

[0190] in, Indicates and The spatial distance between candidate monitoring action strategies, and Respectively represent strategies and In the The spatial characteristic value of the monitoring sub-area, is the number of all monitored sub-areas. By calculating the spatial distance between all candidate strategies, we can get the spatial correlation graph , each element of which Indicates the spatial similarity of two strategies. The spatial correlation diagram can intuitively show the spatial distribution of different strategies. and spatial correlation plot As input, the spatiotemporal fusion model is imported. The goal of the spatiotemporal fusion model is to comprehensively consider the information of the two dimensions of time and space to generate an optimal initial fusion strategy. The spatiotemporal fusion model is implemented by methods such as multi-layer perceptron (MLP) or graph convolutional network (GCN). Suppose the input of the spatiotemporal fusion model is , the output is the fusion strategy vector , where each Indicates The weight or importance of each monitoring parameter in the initial fusion strategy. The calculation formula of the fusion strategy vector is as follows:

[0191] ;

[0192] Among them, MLP represents a multi-layer perceptron, is the weight matrix of the model, is the bias term, is a nonlinear activation function. Through the formula, the spatiotemporal information is mapped to the fusion strategy space to obtain an initial fusion strategy vector. The parameter sensitivity analysis of the initial fusion strategy is performed to evaluate the influence of each parameter on the monitoring effect and find out the key parameters that have the greatest impact on the result. Suppose each parameter in the initial fusion strategy vector is , its sensitivity to monitoring effect is expressed as the following formula:

[0193] ;

[0194] in, Representation parameters sensitivity, It is a fusion strategy By calculating the sensitivity system of each parameter, we get a parameter sensitivity vector , where each Represents the sensitivity value of the corresponding parameter. Based on the parameter sensitivity vector , construct an adaptive parameter adjustment function. The adaptive parameter adjustment function can dynamically adjust the parameters of the initial fusion strategy according to the sensitivity of the parameters to improve the overall monitoring effect. Suppose the adaptive adjustment function is , and its calculation formula is as follows:

[0195] ;

[0196] in, represents the adjusted parameter value, is the adjustment step factor, which is used to control the amplitude of the adjustment. Through the adaptive adjustment function, the initial strategy is optimized according to the sensitivity value of each parameter to obtain the adaptive adjustment strategy . Extract target parameters based on the adjusted strategy and map these parameters to the control instructions of the security camera. Adaptive adjustment strategy Each element in represents the control parameters of the camera in a specific time period and spatial position, such as the rotation angle of the camera, focal length adjustment, etc. The target parameters are mapped through the mapping function Converted into camera control instructions, such as:

[0197] ;

[0198] in, Indicates the horizontal rotation angle of the camera. Indicates the vertical rotation angle, represents the focal length, Represents the resolution. Through this mapping, the abstract monitoring strategy is transformed into specific camera control instructions to generate the target monitoring action parameters of the security camera.

[0199] The above describes the intelligent decision-making and response method based on data analysis in the embodiment of the present application. The following describes the intelligent decision-making and response system 10 based on data analysis in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of the intelligent decision-making and response system 10 based on data analysis includes:

[0200] A preprocessing module 11 is used to collect and preprocess multi-scene monitoring video data based on security cameras to obtain preprocessed image data;

[0201] A feature selection module 12 is used to perform multi-scale feature extraction and L2 regularization feature selection on the pre-processed image data to obtain a target multi-scale feature set;

[0202] The recognition module 13 is used to perform scene classification and object recognition on the target multi-scale feature set to obtain a scene classification result and an object recognition result;

[0203] A modeling module 14 is used to perform 3D scene reconstruction and scene graph sequence modeling based on the scene classification results and object recognition results to obtain a 3D scene model and scene change prediction results;

[0204] An analysis module 15 is used to perform multi-dimensional scene error analysis and reinforcement learning based on the three-dimensional scene model and the scene change prediction results to obtain multiple candidate monitoring action strategies;

[0205] The generation module 16 is used to perform adaptive parameter calibration of the spatiotemporal fusion of multiple candidate monitoring action strategies to generate target monitoring action parameters of the security camera.

[0206] Through the collaboration of the above components, multi-scale feature extraction and L2 regularized feature selection, combined with the GateNet network and Faster R-CNN model, the accuracy of scene classification and object recognition in complex scenes has been greatly improved. Based on the scene classification results and object recognition results, three-dimensional scene reconstruction and scene graph sequence modeling are carried out to achieve more accurate three-dimensional scene model construction and scene change prediction. Through multi-dimensional scene error analysis and reinforcement learning, the system can automatically generate and optimize monitoring action strategies according to scene changes, improving the intelligence level of the monitoring system. The adaptive parameter calibration method of spatiotemporal fusion can quickly generate the target monitoring action parameters of the security camera, ensuring the real-time response capability of the system. By introducing the conditional random field model and multiple optimization strategies, the misidentification and false detection in complex environments are effectively reduced, and the anti-interference ability of the system is improved.

[0207] The present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the intelligent decision-making and response method based on data analysis in the above-mentioned embodiments.

[0208] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the intelligent decision-making and response method based on data analysis.

[0209] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0210] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0211] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent decision-making and response method based on data analysis, characterized in that: The method comprises: Collect multi-scene monitoring video data based on security cameras and pre-process them to obtain pre-processed image data; The pre-processed image data is subjected to multi-scale feature extraction and L2 regularization feature selection to obtain a target multi-scale feature set; specifically comprising: performing Gaussian pyramid and Laplacian pyramid processing on the pre-processed image data to generate a plurality of sub-image sequences of different scales; respectively calculating grayscale co-occurrence matrix features of 4 directions and 3 distances, local binary pattern features of 8 modes, and 9 directional gradient histogram features for the sub-image sequences of different scales to obtain a low-level feature set; performing Hessian matrix extreme point detection on the sub-image sequences of different scales, and using Haar wavelet response to calculate a 64-dimensional descriptor to obtain an intermediate feature set, and performing high-level semantic feature extraction on the sub-image sequences of different scales to obtain a high-level feature set; performing feature concatenation on the low-level feature set, the intermediate feature set and the high-level feature set to obtain an original multi-scale feature set, and performing principal component dimensionality reduction on the original multi-scale feature set to obtain a reduced dimensionality feature set; calculating an L2 norm for each feature vector of the reduced dimensionality feature set, and dividing each feature vector of the reduced dimensionality feature set by a corresponding modulus to perform normalization processing to obtain a normalized feature set; performing L2 regularization processing on the normalized feature set to obtain a regularized feature set, and based on the mutual information value between the regularized feature set and the target object, using a recursive feature elimination algorithm to iteratively remove the feature with the smallest contribution, removing 10% of the features each time, until the number of remaining features reaches 50% of the number of original features, to obtain a target multi-scale feature set; The target multi-scale feature set is subjected to scene classification and object recognition respectively to obtain scene classification results and object recognition results; specifically comprising: inputting the target multi-scale feature set into the GateNet network, calculating the importance weight of each scale feature through the sigmoid gate unit, wherein the importance weight is equal to the result of the sigmoid function acting on the output of a single-layer fully connected network, and obtaining a weighted feature set; applying a multi-scale feature fusion module to the weighted feature set, first calculating a spatial attention map, i.e., concatenating all scale features and obtaining the result through a single-layer fully connected network and a sigmoid function, and then performing cross-scale connection, i.e., multiplying the attention map with each scale feature and summing the results to obtain a fused feature set; inputting the fused feature set into the fully connected layer, and then calculating the probability of each scene category through the softmax function, specifically comprising taking the exponent of each output of the fully connected layer and dividing it by the sum of all output exponents to obtain an initial classification result; constructing a first conditional random field model, wherein the observed variable is the initial classification result, the hidden variable is the real scene label, defining a potential function to consider the spatial relationship between adjacent scenes, and obtaining a scene classification result by maximizing the posterior probability; based on the target The target multi-scale feature set is used to construct a FasterR-CNN model, including a region proposal network and a classification network, wherein the region proposal network is composed of a 3x3 convolutional layer and two 1x1 convolutional layers, and the classification network uses ResNet-101 as the backbone network; the target multi-scale feature set is input into the region proposal network, a plurality of anchor boxes of different scales and aspect ratios are generated, the objectness score and bounding box regression parameters of each anchor box are calculated, and a non-maximum suppression algorithm is used to retain high-quality candidate regions with an intersection-over-union ratio threshold greater than 0.7 to obtain a first candidate region set; the SENet module is applied to the first candidate region set to calculate the channel attention weight, specifically including global average pooling and maximum pooling of the feature map of the candidate region, the result is input into a two-layer fully connected network and then obtained by a sigmoid function, and then the weight is multiplied by the first candidate region set to obtain a second candidate region set; the second candidate region set is input into the classification network, and a feature pyramid is constructed using a feature pyramid network structure, specifically including adding the feature map of each layer to the upsampled feature map of the previous layer and performing a convolution operation to obtain multi-scale object features; the Focal Loss is used as a classification loss function, where the loss value is related to the prediction probability and difficulty level, and an online difficult example mining strategy is used to select the top N difficult samples with the loss value for training to obtain an initial recognition result; a second conditional random field model is applied to the initial recognition result to define a unary potential function and a binary potential function, where the unary potential function represents the label probability of a single pixel, and the binary potential function represents the relationship between adjacent pixel labels, and the object boundary is optimized and false detections are eliminated by minimizing an energy function containing these two functions to obtain an object recognition result; Based on the scene classification result and the object recognition result, three-dimensional scene reconstruction and scene graph sequence modeling are performed to obtain a three-dimensional scene model and a scene change prediction result; Perform multi-dimensional scene error analysis and reinforcement learning based on the three-dimensional scene model and the scene change prediction result to obtain multiple candidate monitoring action strategies; The multiple candidate monitoring action strategies are subjected to spatiotemporal fusion adaptive parameter calibration to generate target monitoring action parameters of the security camera.

2. The intelligent decision-making and response method based on data analysis according to claim 1 is characterized in that: The method of collecting multi-scene monitoring video data based on security cameras and preprocessing to obtain preprocessed image data includes: Collecting multi-scene monitoring video data based on security cameras, and performing frame extraction on the multi-scene monitoring video data to obtain original image sequences of multiple scenes; Performing noise detection on each image in the original image sequence, identifying the type and distribution of noise in the image, obtaining a noise feature map, and performing denoising processing on the original image sequence according to the noise feature map to obtain a denoised image sequence; Based on the internal parameters of the security camera, the denoised image sequence is subjected to distortion analysis, the degree and type of distortion are calculated, distortion correction parameters are obtained, and the denoised image sequence is subjected to geometric correction according to the distortion correction parameters to obtain a distortion-free image sequence; Performing illumination condition analysis on the distortion-free image sequence, calculating illumination intensity and color temperature of each scene to obtain illumination balance parameters, and performing color adjustment on the distortion-free image sequence based on the illumination balance parameters to obtain a color-balanced image sequence; Contrast analysis is performed on the color-balanced image sequence, local and global contrast indices are calculated to obtain analysis results, and contrast optimization is performed based on the analysis results to obtain preprocessed image data.

3. The intelligent decision-making and response method based on data analysis according to claim 1 is characterized in that: The second conditional random field model is applied to the initial recognition result to define a unary potential function and a binary potential function, wherein the unary potential function represents the label probability of a single pixel, and the binary potential function represents the relationship between adjacent pixel labels, and the object boundary is optimized and false detections are eliminated by minimizing an energy function containing the two functions to obtain an object recognition result, including: Assigning a random variable to each pixel in the initial recognition result, constructing a pixel-level conditional random field graph structure to obtain a second conditional random field model, and calculating a log-likelihood ratio as a univariate potential function value based on the category probability distribution of each pixel in the initial recognition result to obtain a univariate potential function set; Comparing the category labels of adjacent pixel pairs in the initial recognition result, calculating the similarity of the pixel pairs as the binary potential function value, obtaining a binary potential function set, and combining the unary potential function set with the binary potential function set to construct an energy function, wherein the energy function is a weighted sum of the unary potential function and the binary potential function, and obtaining a conditional random field energy function; According to the conditional random field energy function, each pixel is iteratively updated, the edge probability distribution of each category is calculated, and an optimized pixel-level classification result is obtained; and a connected domain analysis is performed on the optimized pixel-level classification result, adjacent pixels of the same type are merged to form an object region, and a preliminary optimized object boundary is obtained; Based on the initially optimized object boundary, small-area connected domains are filtered and irregular boundaries are smoothed to obtain object recognition results.

4. The intelligent decision-making and response method based on data analysis according to claim 1 is characterized in that: The three-dimensional scene reconstruction and scene graph sequence modeling are performed based on the scene classification result and the object recognition result to obtain a three-dimensional scene model and a scene change prediction result, including: Performing feature extraction and matching on the scene classification result and the object recognition result to obtain a cross-view feature correspondence, and calculating the relative position and posture between cameras based on the cross-view feature correspondence to obtain a camera parameter matrix; Calculating the coordinates of the three-dimensional space points according to the camera parameter matrix and the feature correspondence to obtain initial three-dimensional point cloud data, and performing density enhancement and smoothing processing on the initial three-dimensional point cloud data to obtain an optimized three-dimensional point cloud; Converting the optimized three-dimensional point cloud into a continuous surface representation to obtain a three-dimensional scene model; Building an object relationship graph based on the three-dimensional scene model, representing the objects in the scene and the spatial relationships between them as a graph structure, and obtaining a static scene graph; A time series analysis is performed on the static scene graph at multiple time points, a time series model of scene changes is established, and a dynamic scene graph is obtained. Based on the dynamic scene graph, possible scene changes in the future are predicted, and a probability weight is assigned to each change situation to obtain a scene change prediction result.

5. The intelligent decision-making and response method based on data analysis according to claim 4 is characterized in that: The static scene graphs at multiple time points are subjected to time series analysis to establish a time series model of scene changes to obtain a dynamic scene graph, and based on the dynamic scene graph, possible future scene changes are predicted, and a probability weight is assigned to each change situation to obtain a scene change prediction result, including: Performing timestamp alignment on static scene graphs at multiple time points to obtain a time-aligned scene graph sequence, and extracting change features of nodes and edges in the scene graph based on the time-aligned scene graph sequence to obtain a scene change feature sequence; Performing time window segmentation on the scene change feature sequence to obtain a plurality of feature subsequences of fixed lengths, and inputting the feature subsequences into a recurrent neural network to model the scene change to obtain a time series model of the scene change; Generate a dynamic scene graph based on the time series model, perform probability prediction on nodes and edges in the dynamic scene graph to obtain a scene graph state distribution at a future moment, and sample the scene graph state distribution to generate multiple possible future scene graphs to obtain a candidate future scene set; Based on the candidate future scene set, the occurrence probability of each scene is calculated to obtain a scene probability distribution, and according to the scene probability distribution, the top K scenes with the highest probability are selected as the scene change prediction results.

6. The intelligent decision-making and response method based on data analysis according to claim 1 is characterized in that: The multi-dimensional scene error analysis and reinforcement learning are performed based on the three-dimensional scene model and the scene change prediction result to obtain multiple candidate monitoring action strategies, including: The three-dimensional scene model is spatially divided to obtain a plurality of monitoring sub-areas, and the security risk level of each monitoring sub-area is analyzed to obtain a regional risk distribution map; Based on the regional risk distribution map and the scene change prediction result, the dynamic risk index of each monitored sub-area is calculated to obtain a spatiotemporal risk matrix, and an error analysis is performed on the spatiotemporal risk matrix to obtain a risk prediction confidence; According to the risk prediction confidence, high-risk areas and time periods that need to be focused on are determined to obtain a priority monitoring target list, and based on the priority monitoring target list, the state space and action space of the monitoring environment are constructed, and the reward function of the security monitoring effect is defined to obtain a reinforcement learning environment model; The reinforcement learning environment model is input into a deep reinforcement learning algorithm, and the optimal monitoring strategy is learned by interacting with the environment to obtain an initial strategy network. The initial strategy network is used to generate multiple sets of camera parameter adjustment schemes, including angle, focal length and resolution, to obtain multiple candidate monitoring action strategies.

7. The intelligent decision-making and response method based on data analysis according to claim 1 is characterized in that: The adaptive parameter calibration of the spatiotemporal fusion of the multiple candidate monitoring action strategies to generate the target monitoring action parameters of the security camera includes: Performing time series analysis on the multiple candidate monitoring action strategies, extracting the time characteristics of each candidate monitoring action strategy, obtaining a time characteristic matrix, and performing spatial correlation analysis on each candidate monitoring action strategy based on the time characteristic matrix, calculating the spatial distance between the candidate monitoring action strategies, and obtaining a spatial correlation graph; Input the time feature matrix and the spatial correlation graph into the spatiotemporal fusion model to generate an initial fusion strategy, and perform parameter sensitivity analysis on the initial fusion strategy to calculate the influence of each parameter on the monitoring effect and obtain a parameter sensitivity vector; Based on the parameter sensitivity vector, an adaptive parameter adjustment function is constructed to dynamically adjust the parameters of the initial fusion strategy to obtain an adaptive adjustment strategy, and based on the adaptive adjustment strategy, the target parameters are extracted and mapped to the control instructions of the security camera to generate the target monitoring action parameters of the security camera.

8. An intelligent decision-making and response system based on data analysis, characterized in that: Used to execute the intelligent decision-making and response method based on data analysis as described in any one of claims 1 to 7, the intelligent decision-making and response system based on data analysis comprises: A preprocessing module is used to collect and preprocess multi-scene surveillance video data based on security cameras to obtain preprocessed image data; A feature selection module, used for performing multi-scale feature extraction and L2 regularization feature selection on the pre-processed image data to obtain a target multi-scale feature set; A recognition module, used to perform scene classification and object recognition on the target multi-scale feature set, respectively, to obtain a scene classification result and an object recognition result; A modeling module, used to perform three-dimensional scene reconstruction and scene graph sequence modeling based on the scene classification result and the object recognition result, to obtain a three-dimensional scene model and a scene change prediction result; An analysis module, used to perform multi-dimensional scene error analysis and reinforcement learning based on the three-dimensional scene model and the scene change prediction result to obtain multiple candidate monitoring action strategies; A generation module is used to perform adaptive parameter calibration of the multiple candidate monitoring action strategies for spatiotemporal fusion to generate target monitoring action parameters of the security camera.

Images