Radar-based danger alarm method, device, equipment and storage medium

Abstract

The present invention relates to the field of radar sensing technology, and discloses a radar sensing-based danger alarm method, device, equipment and storage medium, wherein the method comprises: preprocessing three-dimensional point cloud data collected by a laser radar and original image data collected by a security camera to obtain target point cloud data and target image data; constructing a target Voronoi diagram of a monitoring area; performing feature extraction and feature fusion to obtain a fused feature vector; inputting the fused feature vector into a deep learning model to perform target detection and classification to obtain an initial detection result of each potential dangerous target; performing a hazard level assessment to obtain a hazard level and confidence level of each potential dangerous target; optimizing monitoring parameters of the laser radar and the security camera, and outputting a device collaborative monitoring parameter combination, wherein the method optimizes the monitoring parameters of the laser radar and the security camera in real time, realizes collaborative monitoring of the equipment, and improves the overall monitoring efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the field of radar sensing technology, and in particular to a danger alarm method, device, equipment and storage medium based on radar sensing. Background Art

[0002] Traditional security monitoring systems mainly rely on fixed cameras, which have problems such as blind spots, poor environmental adaptability, and limited data processing capabilities. In recent years, the development of LiDAR technology has brought new possibilities to security systems. It can provide high-precision three-dimensional point cloud data, effectively making up for the shortcomings of traditional visual systems. However, how to effectively integrate the data of LiDAR and visual cameras to achieve the fusion and intelligent analysis of multimodal information is still a problem that needs to be solved.

[0003] In addition, with the rapid development of artificial intelligence technology, deep learning has made significant progress in target detection and scene understanding. However, in complex security scenarios, how to accurately assess the danger level of different types of potential threats and achieve coordinated scheduling of equipment still faces huge challenges. Existing security systems often lack the ability to dynamically adapt to environmental changes and are difficult to effectively deal with complex scenarios. Summary of the invention

[0004] The present invention provides a radar-sensing-based danger alarm method, device, equipment and storage medium, which are used to optimize the monitoring parameters of laser radar and security cameras in real time, realize the coordinated monitoring of equipment, and improve the overall monitoring efficiency of the system.

[0005] In a first aspect, the present invention provides a danger alarm method based on radar sensing, the danger alarm method based on radar sensing comprising:

[0006] Preprocess the three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera to obtain target point cloud data and target image data;

[0007] Constructing a target Voronoi diagram of a monitoring area based on the target point cloud data and the target image data;

[0008] According to the target Voronoi diagram, feature extraction and feature fusion are performed on the target point cloud data and the target image data to obtain a fused feature vector;

[0009] Inputting the fused feature vector into a deep learning model for target detection and classification to obtain an initial detection result for each potentially dangerous target;

[0010] Performing a hazard level assessment based on the initial detection results to obtain a hazard level and confidence level for each potential hazard target;

[0011] According to the danger level and the confidence level, the monitoring parameters of the laser radar and the security camera are optimized, and a device collaborative monitoring parameter combination is output.

[0012] In a second aspect, the present invention provides a danger alarm device based on radar sensing, the danger alarm device based on radar sensing comprising:

[0013] A preprocessing module is used to preprocess the three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera to obtain target point cloud data and target image data;

[0014] A construction module, used for constructing a target Voronoi diagram of a monitoring area based on the target point cloud data and the target image data;

[0015] A fusion module, used for performing feature extraction and feature fusion on the target point cloud data and the target image data according to the target Voronoi diagram to obtain a fused feature vector;

[0016] A detection module, used for inputting the fused feature vector into a deep learning model for target detection and classification, and obtaining an initial detection result for each potentially dangerous target;

[0017] An evaluation module, used to perform a danger level evaluation based on the initial detection result to obtain a danger level and a confidence level of each potential danger target;

[0018] The optimization module is used to optimize the monitoring parameters of the laser radar and the security camera according to the danger level and the confidence level, and output a device collaborative monitoring parameter combination.

[0019] The third aspect of the present invention provides a radar-based hazard alarm device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the radar-based hazard alarm device executes the above-mentioned radar-based hazard alarm method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned radar-sensing-based hazard warning method.

[0021] In the technical solution provided by the present invention, by combining the three-dimensional point cloud data of the laser radar and the image data of the security camera, the all-round perception of the monitoring scene is realized, the limitation of a single data source is overcome, and the accuracy of target detection and recognition is improved. The optimized Voronoi diagram is constructed by using the improved Lloyd algorithm and the environmental complexity constraint, and the dynamic division of the monitoring area is realized, so that the monitoring resources can be reasonably allocated according to the environmental complexity. The advanced feature extraction methods such as FPFH and SIFT are adopted, combined with the designed multimodal feature fusion network, the key features of the point cloud and the image are effectively extracted and fused, and the region proposal network and feature pyramid network in the deep learning model are used to achieve high-precision detection and classification of potential dangerous targets, and improve the recognition ability of the system. Combined with the contextual reasoning of the multi-level hazard assessment rule base and the graph neural network, a comprehensive assessment of potential dangerous targets is achieved, and the accuracy and reliability of the hazard level judgment are improved. According to the hazard level and confidence, the monitoring parameters of the laser radar and the security camera are optimized in real time, the coordinated monitoring of the equipment is realized, and the overall monitoring efficiency of the system is improved. Through iterative optimization and coverage evaluation, it is ensured that the monitoring system can be dynamically adjusted to meet the preset coverage requirements, thereby improving the flexibility and adaptability of the system.

[0022] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of an embodiment of a danger alarm method based on radar sensing in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of an embodiment of a danger alarm device based on radar sensing in an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of an embodiment of a hazard warning device based on radar sensing in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0029] To facilitate understanding of this embodiment, a radar sensing-based danger alarm method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0030] 101. Preprocess the three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera to obtain target point cloud data and target image data;

[0031] It is understandable that the execution subject of the present invention may be a danger alarm device based on radar sensing, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0032] Specifically, three-dimensional point cloud data is collected by laser radar. In order to eliminate the influence of noise points and isolated points, the average distance between each point in the three-dimensional point cloud data and the surrounding points is calculated. Based on the preset standard deviation multiple threshold, the points exceeding the average distance are eliminated, so as to remove abnormal discrete points and obtain more accurate initial point cloud data. The initial point cloud data is spatially divided, and the entire three-dimensional space is divided into fixed-size cube grids, each of which contains a different number of points. The centroid coordinates of all points in each cube grid are calculated to obtain the representative position of each cube, and the target point cloud data is obtained to better reflect the distribution of objects in the monitoring area. At the same time, the original image data collected by the security camera. The original image data is subjected to global histogram analysis, and the overall brightness and contrast information of the image is quantitatively described by calculating the cumulative distribution function of the image. Based on the cumulative distribution function, the pixel values ​​of the image are nonlinearly mapped to obtain the first image data, which effectively improves the contrast of the image and makes the details clearer. Multi-scale feature extraction is performed on the first image data, and the image is decomposed and reconstructed through filters of different scales to obtain the second image data. The second image data is subjected to adaptive block processing, the image is divided into several blocks of appropriate size, and a local histogram is calculated for each block. Based on the information of the local histogram, the pixel values ​​of each block are subjected to local equalization processing to generate the third image data, so as to avoid the over-enhancement phenomenon caused by global processing. The third image data is subjected to residual feature extraction, and the high-frequency information in the image is compared with the original image to extract the residual feature information. The residual feature is added to the third image data to obtain the fourth image data with more prominent details. In order to remove the noise in the image while retaining the edge details, the fourth image data is subjected to edge-preserving filtering processing. A bilateral filtering algorithm is adopted, and the spatial weight and gray value similarity weight between pixels are calculated, and each pixel value is weighted averaged to obtain the fifth image data. The processed fifth image data is subjected to spatiotemporal registration with the target point cloud data. The registration process includes converting the point cloud coordinate system into the image coordinate system by calculating the rigid body transformation matrix, establishing an accurate correspondence between the point cloud data and the image pixels, and obtaining the target image data.

[0033] 102. Construct a target Voronoi diagram of the monitoring area based on the target point cloud data and the target image data;

[0034] Specifically, the entire monitoring area is discretized into grid cells, and the size of the grid is set to 0.5 meters × 0.5 meters, so as to better distinguish and capture the detailed information in the area. The specific location coordinates of the laser radar and security camera are determined by using the target point cloud data and the target image data to form a set of equipment locations. For each position in the set of equipment locations, the corresponding Voronoi unit is calculated. The initial Voronoi diagram is constructed by calculating the Euclidean distance from each grid point in the monitoring area to each equipment location and dividing each grid point into the Voronoi unit corresponding to the nearest equipment location. This process effectively divides the monitoring area into multiple sub-areas, each of which is mainly monitored by a device. To ensure the rationality of the coverage and boundary of each Voronoi unit, the centroid coordinates of each Voronoi unit are calculated based on the set of equipment locations, and a 0.3-meter safety buffer is extended outside the boundary of each Voronoi unit to form a set of Voronoi units with a buffer. The setting of the extended buffer is to provide a wider monitoring margin in practical applications to avoid potential safety risks caused by monitoring blind spots. The target point cloud data and target image data are fused and analyzed, and the environmental complexity index of each grid point in the monitoring area is calculated using the Gaussian process regression method to generate an environmental complexity distribution map, which can reflect the complexity of each location in the monitoring area, such as the density of obstacles and the number of dynamic targets. Based on the environmental complexity distribution map, the average environmental complexity of each Voronoi unit is calculated, and a weight factor is assigned to each device according to the average environmental complexity, where the weight factor of the laser radar is set to 1.5 and the weight factor of the security camera is set to 1.0 to form a device weight set. In this process, the laser radar has a higher weight because of its advantages in three-dimensional environment modeling and distance measurement, while the weight of the security camera is relatively low and is more used to capture visual features. The initial Voronoi diagram is optimized using the improved Lloyd algorithm. The total energy function of the current Voronoi diagram is calculated, where the environmental complexity and device weight are included in the calculation of the energy function, and the total energy function is minimized by the gradient descent method, thereby iteratively updating the shape and size of each Voronoi unit. This ensures that the monitoring area of ​​each device is more balanced and reasonable by comprehensively considering the environmental complexity and device monitoring capabilities. The optimized Voronoi cell boundary is adjusted to better match the actual monitoring needs, reducing the waste of device resources and monitoring blind spots.When smoothing the optimized Voronoi unit boundary, the cubic spline interpolation algorithm is used to fit the boundary points. Under the premise of keeping the topological relationship of the Voronoi unit unchanged, the boundary is locally adjusted to obtain a smoothed Voronoi diagram, eliminating the boundary irregularities caused by the initial optimization, making each unit boundary smoother and more natural, and more in line with the actual physical situation of equipment monitoring. The smoothed Voronoi diagram is strictly inspected to ensure that the area of ​​each Voronoi unit is not less than the set minimum threshold, and the overlapping area between adjacent Voronoi units does not exceed the set maximum threshold. The inspection process is to ensure that each unit has sufficient coverage area, avoid omission of the monitoring area due to too small a unit, and prevent excessive overlap of adjacent units from causing waste of resources or monitoring redundancy. After the above steps, the target Voronoi diagram that meets the expected requirements is finally obtained.

[0035] 103. According to the target Voronoi diagram, feature extraction and feature fusion are performed on the target point cloud data and the target image data to obtain a fused feature vector;

[0036] Specifically, based on the target Voronoi diagram, the target point cloud data is segmented according to the Voronoi unit, and the original three-dimensional point cloud data is divided into multiple sub-point cloud sets corresponding to the Voronoi units. For each sub-point cloud set, the normal vector and curvature of each point are calculated. The normal vector is used to describe the local plane direction of the point, and the curvature is used to quantify the curvature of the point cloud surface. The two together constitute the local geometric features of the point cloud. In each sub-point cloud set, the simplified point feature histogram of each point is calculated, specifically including setting a spherical neighborhood radius of 0.05 meters to ensure that the neighborhood contains enough points for statistical analysis. The k nearest neighbors of each point are calculated, and the angle between the normal vectors, the distance between points, and the angle difference are extracted to generate a 33-dimensional SPFH feature. The SPFH feature can effectively characterize the distribution characteristics of the point cloud in the local neighborhood. The FPFH descriptor is calculated based on the SPFH feature. By weighted accumulation of the SPFH features of each point, the local dependency of the single-point features is eliminated to obtain a 33-dimensional FPFH feature. The FPFH features of all points are combined to form a point cloud feature set. At the same time, the SIFT algorithm is used to extract features for the target image data. The scale space is constructed, the image is Gaussian blurred, and downsampled at different scales to generate 4 groups, each of which contains 5 images of different scales. By calculating the Gaussian difference image, the gradient information and texture features of the image at different scales are obtained, and these features can maintain scale invariance to a large extent. Local extreme points are detected in the DOG image. By comparing the difference between each pixel and its 26 neighboring pixels, the extreme points that meet the contrast threshold and the principal curvature ratio threshold are retained to form a preliminary set of candidate feature points, which effectively represents the key positions in the image, such as corners and edges. In order to improve the positioning accuracy of feature points, the candidate feature point set is accurately positioned at the sub-pixel level. The DOG function is interpolated using Taylor expansion to iteratively optimize the position and scale of feature points, while feature points with unstable positioning are removed to obtain a more accurate set of feature points. The main direction of each feature point is calculated, the gradient direction histogram is statistically calculated in the circular area around the feature point, and the peak of the histogram is selected as the main direction of the feature point. If a feature point has multiple peaks, multiple feature descriptors are created for the feature point to form a 128-dimensional SIFT feature descriptor. The SIFT descriptors of all feature points are combined to form an image feature set. SIFT features can maintain good stability under different scales, rotations, and lighting conditions, and are a commonly used feature representation method in image matching and target recognition. A multimodal feature fusion network is designed. The network adopts a dual-stream architecture, in which the point cloud stream uses the PointNet++ model to process the point cloud feature set, and the image stream uses the ResNet-50 model to process the image feature set.PointNet++ is a point cloud processing network based on hierarchical feature extraction. It extracts richer local geometric features by aggregating the local neighborhood of the point cloud layer by layer. ResNet-50 is a deep convolutional neural network with good image feature extraction capabilities. The point cloud feature set and image feature set are input into the corresponding point cloud stream and image stream respectively, and the high-level features of each are extracted after being processed by the multi-layer perceptron and convolution layer. In the feature fusion layer, the attention mechanism is introduced. By calculating the correlation matrix between the modalities, the relationship between the point cloud features and the image features can be automatically learned, so as to perform adaptive weighted fusion of the features. The attention mechanism can assign appropriate weights between different modalities, improving the expression ability and classification performance of the fused features. The fused feature vector obtained by the feature fusion layer is mapped to the 256-dimensional feature space through the fully connected layer to form the final fused feature vector.

[0037] 104. Input the fused feature vector into the deep learning model for target detection and classification to obtain the initial detection result of each potential dangerous target;

[0038] Specifically, the fused feature vector is input into the backbone network of the deep learning model, which consists of multiple convolutional layers and pooling layers, and performs multi-scale feature extraction on the input feature vector. The convolutional layer locally perceives the input features through filters and extracts low-level features such as edges and textures. As the depth of the network increases, the features extracted layer by layer gradually become more advanced semantic information. The pooling layer reduces the dimension of the feature map through downsampling operations, while retaining key feature information, reducing the amount of calculation and the parameter complexity of the model. After multiple layers of convolution and pooling operations, the model can generate a set of multi-scale feature map sets to represent target information at different resolutions, so as to better meet the needs of target detection at different scales and forms. The feature map set is processed by the feature pyramid network. The feature pyramid network fuses the deep semantic information-rich feature map with the shallow spatial resolution high feature map through top-down paths and lateral connections to generate a set of enhanced multi-scale feature maps. The top-down path introduces the deep high semantic information feature map at the top layer of the feature pyramid, and passes it down layer by layer, and performs lateral connections and fusion with the feature map of the layer at each level. In this way, target information of different scales is integrated into the feature maps of each level, ensuring that deep semantic features can be retained at higher resolutions and spatial detail information can be retained at lower resolutions. The enhanced multi-scale feature map is input into the region proposal network, which generates multiple candidate regions with different scales and aspect ratios on the feature map by sliding windows. The region proposal network generates a set of anchor boxes at each position in the feature map, each anchor box represents a candidate region. The network learns the position information and target category of these anchor boxes, and classifies and regresses them to obtain the initial candidate region set. Non-maximum suppression is performed on the initial candidate region set to eliminate redundant candidate regions with high overlap. For candidate regions with overlap exceeding the set threshold, only the one with the highest score is retained, and other candidate regions with lower scores are eliminated to obtain the target candidate region set. The target candidate region set is mapped to the enhanced multi-scale feature map, and fixed-size region features are extracted through the RoI Align operation. RoI Align is a method for extracting candidate region features from feature maps, which can eliminate the feature offset problem caused by quantization, ensure that the extracted features can accurately correspond to the candidate regions in the original image, and obtain a more accurate candidate region feature set. The candidate region feature set is input into the fully connected layer for target classification and bounding box regression. The classification branch is used to predict the probability distribution of each candidate region belonging to each category, while the regression branch is used to predict the bounding box correction parameters of each candidate region. The category probability distribution represents the confidence that the target belongs to a certain category, such as pedestrians, vehicles, or animals, while the bounding box correction parameters are used to accurately adjust the boundaries of the candidate region to make it more closely fit the actual location of the target. Based on these output information, the category probability and bounding box correction parameters of each candidate region are obtained.Based on the category probability distribution and bounding box correction parameters, the candidate regions are screened and refined. For regions with category probabilities higher than the preset threshold and not belonging to the background category, they are retained, and the corresponding bounding box correction parameters are applied to adjust the boundaries of the candidate regions to obtain more accurate bounding box detection results. The screened and refined bounding box detection results are subjected to cross-scale non-maximum suppression and target tracking association processing. Cross-scale non-maximum suppression is to further eliminate redundant overlapping boxes that may be generated on feature maps of different scales, while target tracking association is to maintain the consistency of the target in multiple consecutive frames of images to avoid target drift or loss due to unstable detection. After the above steps, the initial detection results of each potentially dangerous target are finally obtained, including the target category, confidence, and precise location information.

[0039] 105. Perform a hazard level assessment based on the initial detection results to obtain the hazard level and confidence level of each potential hazard target;

[0040] Specifically, feature extraction is performed on the initial detection results of each potential dangerous target. The content of feature extraction includes the position, size, speed and direction features of the target, which together constitute the feature vector of the target. Position features are expressed in the form of three-dimensional coordinates, describing the specific position of the target in space; size features reflect the physical size of the target, such as height, width and depth; speed features include information such as the moving speed and acceleration of the target, which can capture the motion state of the target; direction features describe the movement direction of the target, such as forward, backward or rotation. A preliminary hazard level assessment is performed on the target feature vector based on a predefined multi-level hazard assessment rule base. The rule base contains a variety of assessment rules, including assessment rules based on target category, behavior characteristics, location information and time information. Rules based on target category distinguish the danger of different types of targets, such as pedestrians, vehicles or other objects; rules based on behavior characteristics consider dynamic information such as the speed, direction and acceleration of the target to judge the abnormal behavior of the target; rules based on location information are combined with the location of the target, such as whether it is close to roads, buildings or other dangerous areas; and rules based on time information consider the difference in danger in different time periods, such as special situations at night or during peak hours. By matching the target feature vector and multi-level rules, the preliminary danger level of the target is obtained, which preliminarily reflects the potential danger of the target. The scene graph is constructed, and the detected targets are used as nodes, and the spatial relationships and interactions between targets are used as edges to form an initial scene graph. The scene graph can represent the various targets and their relationships in the monitoring area in the form of a graph structure. The initial scene graph is input into the designed graph neural network, and the node features and edge features are extracted through multi-layer graph convolution operations, and the message passing mechanism is used to capture the complex interactive relationship between the target and the environment. The graph convolution operation can aggregate the features of each node in the graph structure data, integrate the information of its neighboring nodes into its own node, and thus extract the global features of each target. The message passing mechanism captures the mutual influence and relationship between targets by passing information between nodes, so as to better understand the behavior patterns and potential threats of the targets. After multi-layer graph convolution operations and message passing, the updated scene graph is obtained. Based on the updated scene graph, the relevance score of each target node and its neighboring nodes is calculated. The attention mechanism is used to perform weighted aggregation of the information of neighboring nodes. The attention mechanism can automatically assign different weights according to the degree of influence of different neighboring nodes on the target, so that the node information with greater influence can be more integrated into the feature representation of the target. The obtained target context features can fully express the complex interactive relationship between the target and the environment. The context features of the target are integrated with the preliminary danger level to obtain the final danger level of the target. Based on the Dempster-Shafer evidence theory framework, the danger level of the target is analyzed by applying evidence theory. The quality function of each target at the three danger levels of low, medium and high is calculated.The quality function represents the basic probability distribution of a target belonging to a certain danger level, which can reflect the confidence of different information sources. By integrating multi-source information through evidence combination rules, the multiple danger level information of the target is integrated to obtain a more consistent danger level probability distribution. The advantage of evidence theory is that it can effectively handle uncertainty and conflicting information, so that reasonable evaluation results can still be given when the information is incomplete or the multi-source information is inconsistent. Based on the obtained danger level probability distribution, the comprehensive danger index and confidence of each target are calculated. The comprehensive danger index is the weighted sum of the probabilities of each danger level, which quantitatively represents the overall danger level of the target, while the confidence refers to the value of the highest danger level probability, which is used to evaluate the reliability and certainty of the danger level assessment. The danger level and confidence of each potential dangerous target are obtained.

[0041] 106. According to the danger level and confidence level, optimize the monitoring parameters of LiDAR and security cameras, and output the coordinated monitoring parameter combination of the equipment.

[0042] Specifically, based on the danger level and confidence information of each potential dangerous target, the danger distribution heat map in the monitoring area is calculated. The danger distribution heat map can intuitively reflect the degree of danger at different locations in the area, and the hot spot area represents the high-risk area. By analyzing the hot spot area, a set of coordinates of a high-risk area is obtained, including the location of the potential dangerous target and the risk intensity of the location. Cluster analysis is performed on the coordinate set of the high-risk area. Adjacent or similar high-risk areas are merged into multiple target areas that need to be monitored. Through the clustering algorithm, discrete high-risk points are aggregated together, and the scattered high-risk areas are converted into several relatively concentrated monitoring target areas. According to the spatial distribution information of the target area, the scanning angle and resolution of the laser radar are adjusted. For areas containing multiple high-risk targets, the scanning resolution is appropriately increased and the scanning angle is adjusted to capture the dynamic information of the target more clearly and in more detail, thereby improving the monitoring accuracy. By analyzing the location information of the target area, the optimal scanning strategy of the laser radar is determined, and the optimization parameters of the laser radar are obtained. At the same time, based on the location information of the target area, the distance and viewing angle of each security camera to each target area are calculated to obtain the camera coverage matrix. The coverage matrix contains the horizontal angle, pitch angle and focal length information of each security camera relative to the target area. The camera coverage matrix is ​​used to optimize the monitoring parameters of each security camera, including the horizontal rotation angle, pitch angle and focal length. For cameras closer to the target area, the pitch angle and focal length are appropriately adjusted to capture the detailed information of the target more clearly, while for cameras farther away, the focal length and horizontal angle are adjusted to ensure that there is no blind spot at the boundary of the target area covered by them. The optimized parameters of the lidar and the optimized parameters of the security camera are combined to form an initial device collaborative monitoring parameter set. Conflict detection is performed on the initial device collaborative monitoring parameter set to eliminate the monitoring overlap and blind spot problems between devices. Monitoring overlap will lead to waste of device resources and mutual interference between multiple devices, while monitoring blind spots mean that some high-risk areas are not fully covered by monitoring. Through conflict detection, the overlap of the monitoring range of each device is identified and optimized accordingly, such as reallocating the scanning angle of the lidar or adjusting the pitch angle and focal length of the camera. After conflict detection and optimization adjustment, a coordinated monitoring parameter combination is obtained. The coordinated monitoring parameter combination is input into the preset coverage evaluation model to calculate the monitoring coverage of the entire monitoring area. The coverage evaluation model is used to evaluate whether each target area is adequately covered, whether there are monitoring blind spots, or whether the monitoring accuracy of certain high-risk areas meets the standards under the current equipment configuration. If the monitoring coverage fails to reach the preset threshold, the monitoring parameters are re-optimized. For example, based on the calculation results of the coverage, the scanning range and resolution of the lidar are fine-tuned, or the viewing angle and focal length of the camera are fine-tuned until the monitoring coverage meets the predetermined requirements.Through repeated parameter adjustments and coverage evaluation, the collaborative monitoring parameter combination of the equipment is gradually optimized to obtain a collaborative monitoring parameter combination of the equipment that meets all requirements.

[0043] In the embodiment of the present invention, by combining the three-dimensional point cloud data of the laser radar and the image data of the security camera, the all-round perception of the monitoring scene is realized, the limitation of a single data source is overcome, and the accuracy of target detection and recognition is improved. The optimized Voronoi diagram is constructed by using the improved Lloyd algorithm and the environmental complexity constraint, and the dynamic division of the monitoring area is realized, so that the monitoring resources can be reasonably allocated according to the environmental complexity. By using advanced feature extraction methods such as FPFH and SIFT, combined with the designed multimodal feature fusion network, the key features of the point cloud and the image are effectively extracted and fused. Through the region proposal network and feature pyramid network in the deep learning model, high-precision detection and classification of potential dangerous targets are achieved, and the recognition ability of the system is improved. Combining the multi-level hazard assessment rule base and the context reasoning of the graph neural network, a comprehensive assessment of potential dangerous targets is achieved, and the accuracy and reliability of the hazard level judgment are improved. According to the hazard level and confidence, the monitoring parameters of the laser radar and the security camera are optimized in real time, the coordinated monitoring of the equipment is realized, and the overall monitoring efficiency of the system is improved. Through iterative optimization and coverage evaluation, it is ensured that the monitoring system can be dynamically adjusted to meet the preset coverage requirements, and the flexibility and adaptability of the system are improved.

[0044] In a specific embodiment, the process of executing step 101 may specifically include the following steps:

[0045] The three-dimensional point cloud data is collected by laser radar, and the average distance between each point in the three-dimensional point cloud data and the surrounding points is calculated. Points exceeding the average distance are eliminated based on a preset standard deviation multiple threshold to obtain initial point cloud data;

[0046] Perform spatial division on the initial point cloud data, divide the space into cubic grids of fixed size, and calculate the centroid coordinates of all points in each cubic grid to obtain the target point cloud data;

[0047] Collecting raw image data through a security camera, performing global histogram analysis on the raw image data, calculating a cumulative distribution function, and performing nonlinear mapping on image pixel values ​​based on the cumulative distribution function to obtain first image data;

[0048] Performing multi-scale feature extraction on the first image data and reconstructing the image to obtain second image data, adaptively dividing the second image data into blocks, calculating a local histogram of each block, and locally equalizing the pixel values ​​based on the block histogram to obtain third image data;

[0049] Extracting residual features from the third image data, and adding the residual features to the third image data to obtain fourth image data;

[0050] Performing edge-preserving filtering on the fourth image data, calculating the spatial weight and the gray value similarity weight using a bilateral filtering algorithm, and performing weighted averaging on the pixel values ​​based on the spatial weight and the gray value similarity weight to obtain fifth image data;

[0051] The fifth image data is spatially and temporally registered with the target point cloud data, the point cloud coordinate system is converted to the image coordinate system by calculating the rigid body transformation matrix, and an accurate correspondence between the point cloud data and the image pixels is established to obtain the target image data.

[0052] Specifically, three-dimensional point cloud data is collected by laser radar. Laser radar captures spatial information in the environment with pulses emitted hundreds of thousands to millions of times per second, and generates a set of three-dimensional coordinate points (i.e., point cloud data). Each point cloud data point is represented as ,in , and Represent the three-dimensional coordinates of the point in space. Perform denoising on the point cloud data. For each point in the point cloud data , calculate the average distance between it and its surrounding points .definition For point The neighborhood point set includes points within a certain radius All points within, that is:

[0053] ;

[0054] in is a preset neighborhood radius, Indicates at point All other points in the neighborhood. For each point , the average distance It is expressed as:

[0055] ;

[0056] In the above formula, Represents the number of neighborhood points, that is, point The size of the surrounding neighborhood point set. Based on the preset standard deviation multiple threshold Remove discrete points that are far away from their neighborhood. Calculate the average distance of all points and standard deviation , and then compare each point with the average distance of its neighbors. If the average distance of a point exceeds the global average distance times the standard deviation, it is considered as a noise point and removed. The condition for removing noise points is expressed as:

[0057] ;

[0058] in, Indicate point The average distance and Represent the average distance and standard deviation of the global point cloud respectively. After this step, the point cloud data with noise points removed is the initial point cloud data. The initial point cloud data is spatially divided into fixed-size cubic grids, and the size of each grid is set to of cubes, where is the side length of the grid. In each cubic grid, calculate the centroid coordinates of all points , the centroid coordinates are expressed as the geometric mean of all points in the grid:

[0059] ;

[0060] in, represents the number of points in the current cube grid, are the coordinate values ​​of these points respectively. Through this process, the centroid set of all grids is the target point cloud data. By reducing the dimensionality of the original point cloud data, the redundancy of the data is significantly reduced, and the main characteristics of the spatial distribution are retained. At the same time, the original image data is collected by the security camera, which is usually represented by pixel values ​​in the form of a two-dimensional matrix, that is, each pixel point is represented as ,in are the horizontal and vertical coordinates of the pixel in the image, is the pixel value at that coordinate. In order to improve the contrast and detail of the image, a global histogram analysis is performed on the original image data. The cumulative distribution function (CDF) of the image is calculated. The cumulative distribution function describes the cumulative probability distribution of pixel values ​​and is used for global brightness equalization. Cumulative distribution function It is expressed as:

[0061] ;

[0062] in, is the pixel value range, Indicates that the pixel value is The number of pixels, is the total number of pixels in the image. Based on the cumulative distribution function, the pixel values ​​of the image are nonlinearly mapped, and each pixel value in the original image is Through the mapping function Convert to new pixel value ,in , and obtain the first image data. This mapping method can even out the brightness distribution of the image and improve the contrast of the image. Multi-scale feature extraction is performed on the first image data, and the image is represented in a pyramid-like multi-scale manner through Gaussian filtering and downsampling operations at different scales. More texture and structural information is extracted from images of different scales, and this information is comprehensively utilized when reconstructing the image to obtain the second image data. The second image data is adaptively partitioned to divide the image into several image blocks of appropriate sizes, and a local histogram is calculated for each image block. Based on the local histogram, the pixel values ​​are locally equalized to obtain the third image data. Residual feature extraction is performed on the third image data. The residual feature refers to the difference between the original image and its low-frequency component, which can capture the high-frequency information in the image. The residual feature is added to the third image data to obtain the fourth image data, thereby enhancing the detail expression of the image. The fourth image data is edge-preserving filtered, and the spatial weight and gray value similarity weight are calculated using the bilateral filtering algorithm. Spatial weight and gray value similarity weight The calculation formula is as follows:

[0063] ;

[0064] ;

[0065] in, Represents the spatial coordinates of the pixel point, Represents the pixel value at the corresponding position, and are the parameters that control the spatial distance and grayscale difference respectively. The product of the two weights gives the final bilateral filter weight:

[0066] ;

[0067] The fifth image data is obtained by weighted averaging all neighboring pixels. Bilateral filtering can effectively remove noise while retaining image edge details, making the image smoother. The processed fifth image data is spatially and temporally aligned with the target point cloud data, and the rigid body transformation matrix is ​​calculated. Convert the point cloud coordinate system to the image coordinate system. The rigid body transformation matrix contains the rotation matrix and translation vectors , which is expressed as follows:

[0068] ;

[0069] in, Represents the rotation matrix, which is used to describe the rotation relationship from the point cloud coordinate system to the image coordinate system; Represents the translation vector, which is used to describe the translation relationship between the two. Through the transformation matrix Mapped into the image coordinate system, we get . Accurately find the pixel position corresponding to each point cloud point in the image, so as to establish the precise correspondence between the point cloud data and the image pixels, and finally obtain the target image data.

[0070] In a specific embodiment, the process of executing step 102 may specifically include the following steps:

[0071] Discretize the monitoring area into grids with a grid size of 0.5 m × 0.5 m, and determine the position coordinates of the laser radar and security camera based on the target point cloud data and target image data to obtain the device position set;

[0072] For each location in the device location set, calculate the corresponding Voronoi cell. Specifically, calculate the Euclidean distance from each grid point in the monitoring area to each device location, and divide the grid point into the Voronoi cell corresponding to the nearest device to obtain the initial Voronoi diagram.

[0073] Based on the device position set, the centroid coordinates of each Voronoi cell are calculated, and a 0.3-meter safety buffer is extended outside the boundary of each Voronoi cell to obtain a Voronoi cell set with a buffer zone.

[0074] The target point cloud data and target image data are fused and analyzed, and the environmental complexity index of each grid point in the monitoring area is calculated using the Gaussian process regression method to obtain the environmental complexity distribution map;

[0075] Based on the environmental complexity distribution map, the average environmental complexity of each Voronoi cell is calculated, and a weight factor is assigned to each device according to the average environmental complexity. The weight factor of the lidar is set to 1.5, and the weight factor of the security camera is set to 1.0, and the device weight set is obtained;

[0076] The improved Lloyd algorithm is used to optimize the initial Voronoi diagram, and the shape and size of each Voronoi unit are iteratively updated. The specific process is as follows: the total energy function of the current Voronoi diagram is calculated, the environmental complexity and device weight are included in the calculation of the energy function, and the total energy function is minimized by the gradient descent method to obtain the optimized Voronoi unit boundary;

[0077] The optimized Voronoi unit boundary is smoothed, the boundary points are fitted using the cubic spline interpolation algorithm, and the boundary is locally adjusted while keeping the topological relationship unchanged to obtain a smoothed Voronoi diagram.

[0078] The smoothed Voronoi diagram is tested to ensure that the area of ​​each Voronoi unit is not less than the set minimum threshold and the overlapping area between adjacent Voronoi units does not exceed the set maximum threshold, so as to obtain the target Voronoi diagram.

[0079] Specifically, the monitoring area is discretized into a grid of 0.5 m × 0.5 m, and the entire monitoring area is divided into regular rectangular units. The size of the grid is selected to be 0.5 m × 0.5 m, which can more finely describe the spatial characteristics of the monitoring area while ensuring the balance between spatial resolution and computational efficiency. On this basis, the target point cloud data and target image data are used to determine the specific position coordinates of the lidar and security camera in the monitoring area, and generate a set of device positions. The position coordinates of the device are expressed as ,in and Respectively represent The coordinates of each device on a two-dimensional plane. For each position in the device position set, the corresponding Voronoi cell is calculated. The principle of dividing the Voronoi cell is to compare the distance between each grid point in the monitoring area and the position of each device, and divide the grid point into the Voronoi cell corresponding to the device closest to it. For any grid point , calculate its position to each device Euclidean distance , and its calculation formula is:

[0080] ;

[0081] By comparing the distances from the grid point to all device locations, the grid point Divide to the nearest device The entire monitoring area is divided into multiple Voronoi cells. All points in each cell have the shortest distance to the device at the center of the cell, forming the initial Voronoi diagram. Based on the device position set, the centroid coordinates of each Voronoi cell are calculated. The centroid coordinates are expressed as the average value of the coordinates of all grid points in the cell, that is:

[0082] ;

[0083] in, represents the number of grid points in the Voronoi cell, and are the horizontal and vertical coordinates of these grid points respectively. The centroid coordinates reflect the geometric center position of the Voronoi cell. After calculating the centroid of each Voronoi cell, a 0.3-meter safety buffer zone is extended outside the boundary of each cell to obtain a set of Voronoi cells with a buffer zone. The purpose of extending the buffer zone is to provide a certain amount of redundant space for monitoring equipment to ensure that the target can be detected in time even if it moves out of the cell boundary. The target point cloud data and target image data are fused and analyzed. The Gaussian process regression method is used to calculate the environmental complexity index of each grid point in the monitoring area. Gaussian process regression is a non-parametric Bayesian method for regression analysis that describes the uncertainty of each input point by assigning a Gaussian distribution. Assume that the environmental complexity index of each grid point is , which can be expressed as a continuous function of spatial position:

[0084] ;

[0085] in, represents a Gaussian process, is the mean function, usually set to zero function, is a kernel function, which represents the covariance between any two grid points. The environmental complexity index of each grid point is predicted by the Gaussian process regression method, and the environmental complexity distribution map of the entire monitoring area is obtained, which reflects the complexity of different locations in the monitoring area, such as obstacle density, number of dynamic targets, etc. Based on the environmental complexity distribution map, the average environmental complexity of each Voronoi cell is calculated. , which is the average value of the environmental complexity of all grid points in the unit:

[0086] ;

[0087] in, Indicates The set of all grid points within a Voronoi cell, is the number of grid points in the set, is the environmental complexity index of a grid point. According to the average environmental complexity, a weight factor is assigned to each device. The weight factor of the laser radar is set to 1.5, and the weight factor of the security camera is set to 1.0. The device weight set is obtained. The initial Voronoi diagram is optimized using the improved Lloyd algorithm. The Lloyd algorithm is a Voronoi diagram generation and optimization algorithm that iteratively updates the centroid position of the Voronoi unit and continuously reduces the energy function value of the unit. The total energy function of the current Voronoi diagram is defined as:

[0088] ;

[0089] in, For the The weight factor of each device, For the The area of ​​a Voronoi cell, The location within the unit The environmental complexity index, Indicate point To the centroid The energy function is minimized using the gradient descent method by iteratively calculating the centroid of each Voronoi cell and updating the border of the Voronoi cell. , and get the optimized Voronoi unit boundary. The optimized Voronoi unit boundary is smoothed, and the boundary points are fitted using the cubic spline interpolation algorithm. Cubic spline interpolation can make local adjustments to the boundary while keeping the overall shape and topological relationship of the boundary unchanged, making the boundary smoother and more natural. Assume that the original boundary point is , get new boundary points through cubic spline interpolation , generate a smoothed Voronoi diagram. Check the smoothed Voronoi diagram. Make sure the area of ​​each Voronoi cell is not less than the minimum threshold set by hand To ensure that each unit has sufficient monitoring coverage. At the same time, detect the overlapping area between adjacent Voronoi units to ensure that it does not exceed the set maximum threshold If the area of ​​any cell is less than the minimum threshold or the overlapping area of ​​adjacent cells exceeds the maximum threshold, the boundary of the Voronoi cell is re-optimized and adjusted. Through repeated iteration and optimization of the above steps, the target Voronoi diagram that meets the requirements is finally obtained.

[0090] In a specific embodiment, the process of executing step 103 may specifically include the following steps:

[0091] Based on the target Voronoi diagram, the target point cloud data is segmented into sub-point cloud sets corresponding to the Voronoi units, and the normal vector and curvature are calculated for each sub-point cloud set to obtain local geometric features;

[0092] Calculate the FPFH descriptor for each point in the sub-point cloud set, specifically including: calculating the simplified point feature histogram, setting the spherical neighborhood radius to 0.05 meters, calculating the normal vector angle, the distance between point pairs and the angle difference for each point's k nearest neighbors, and obtaining a 33-dimensional SPFH feature;

[0093] The FPFH descriptor is calculated based on the SPFH feature, and the SPFH features of each point are weighted and accumulated to obtain a 33-dimensional FPFH feature. The FPFH features of all points are combined to form a point cloud feature set.

[0094] SIFT feature extraction is performed on the target image data. First, the scale space is constructed, and the image is Gaussian blurred and downsampled to generate 4 groups, each of which contains 5 scales. The Gaussian difference image is calculated to obtain the scale space representation;

[0095] Detect local extreme points in the DOG image, compare each pixel with its 26 neighboring pixels, retain feature points that meet the contrast threshold and principal curvature ratio threshold, and obtain a set of candidate feature points;

[0096] The candidate feature point set is accurately located at the sub-pixel level, the DOG function is interpolated using Taylor expansion, the position and scale of the feature points are iteratively optimized, and unstable points are eliminated to obtain an accurate feature point set;

[0097] Calculate the main direction of the feature point, count the gradient direction histogram in the circular area around the feature point, select the peak value as the main direction, create multiple feature descriptors for feature points with multiple peak values, and obtain a 128-dimensional SIFT feature descriptor to form an image feature set;

[0098] A multimodal feature fusion network is designed with a dual-stream architecture. PointNet++ is used for the point cloud stream and ResNet-50 is used for the image stream. The point cloud feature set and the image feature set are input into the corresponding point cloud stream and image stream respectively. High-level features are extracted through multi-layer perceptron and convolutional layers. The attention mechanism is introduced in the feature fusion layer, and the correlation matrix between modalities is calculated. The features are adaptively weighted and fused. Finally, the fused features are mapped to a 256-dimensional feature space through a fully connected layer to obtain a fused feature vector.

[0099] Specifically, the target point cloud data is segmented based on the target Voronoi diagram. The Voronoi diagram divides the entire space into multiple regions, and the points in each region are closest to its center point. Based on the Voronoi diagram, the point cloud data is divided into sub-point cloud sets corresponding to the Voronoi units. Assume that the entire point cloud data set is , where each point With three-dimensional coordinates . Divide these points according to the Voronoi unit, and each sub-point cloud set Corresponding to a Voronoi cell ,in In each sub-point cloud set In the process, the normal vector and curvature of each point are calculated to obtain local geometric features. The normal vector is an important geometric feature that describes the surface shape of the point cloud and is calculated by principal component analysis. , select all points in its neighborhood , expressed as ,in is a neighborhood radius. Before calculating the normal vector, calculate the covariance matrix of these neighborhood points:

[0100] ;

[0101] in, Indicate point The covariance matrix of For point The centroid coordinates of all points in the neighborhood, is the number of neighborhood points. By performing eigenvalue decomposition on the covariance matrix, the eigenvector corresponding to the minimum eigenvalue is the point The normal vector of . The curvature is calculated by the minimum eigenvalue and the sum of the eigenvalues:

[0102] ;

[0103] in, is the minimum eigenvalue, and are the other two eigenvalues. Curvature can reflect the change of local curvature of the point cloud surface at that point, and is an important indicator to characterize local geometric features. Calculate the FPFH descriptor for each point in the sub-point cloud set. FPFH is an effective method to describe the local geometric features of point clouds, and can effectively capture the structural information of point clouds in the neighborhood. Calculate the simplified point feature histogram of each point. SPFH describes the local features of points by analyzing the normal vector angle of points in the spherical neighborhood of each point, the distance between point pairs, and the angle difference between normal vectors. Set the spherical neighborhood radius to 0.05 meters, for each point , calculate all points in its neighborhood The normal vector angle , distance between point pairs and angle difference , and obtain the 33-dimensional SPFH feature .

[0104] ;

[0105] in, represents the angle between the normal vectors, Indicate point and The distance between Indicates the angle difference between normal vectors. When calculating the FPFH descriptor based on the SPFH feature, the SPFH features of each point are weighted and accumulated to obtain the final 33-dimensional FPFH feature. The FPFH feature set is expressed as:

[0106] ;

[0107] in, For point The number of neighboring points. Combine the FPFH features of all points to form the feature set of the entire point cloud. Perform SIFT feature extraction on the target image data. Construct the scale space by Gaussian blurring and downsampling the image. Assume that the original image is , the Gaussian blurred image is ,in is the standard deviation of the Gaussian kernel. The formula for Gaussian blur is:

[0108] ;

[0109] in, is the Gaussian kernel function. Subtract the Gaussian blurred images of different scales to get the Gaussian difference image:

[0110] ;

[0111] in, and Represents the standard deviation of Gaussian blur at two different scales. The DOG image is used to detect local extreme points. By comparing each pixel with its 26 neighboring pixels, the feature points that meet the contrast threshold and the principal curvature ratio threshold are retained to form a set of candidate feature points. The candidate feature point set is accurately located at the sub-pixel level, and the DOG function is interpolated using Taylor expansion. The position and scale of the feature points are iteratively optimized, and unstable points are removed to obtain a set of accurate feature points. For each accurate feature point, its main direction is calculated, and the gradient direction histogram is statistically calculated in the circular area around the feature point, and the peak value is selected as the main direction. If a feature point has multiple peaks, multiple feature descriptors are created for the feature point. A 128-dimensional SIFT feature descriptor is obtained to form an image feature set. A multimodal feature fusion network is designed for feature fusion. The multimodal feature fusion network adopts a two-stream architecture, in which the point cloud stream uses the PointNet++ model. PointNet++ is a point-based hierarchical feature extraction network that can extract local and global features of point clouds at different scales. The image stream uses the ResNet-50 model. ResNet-50 is a deep convolutional neural network with excellent image feature extraction capabilities. The point cloud feature set is input into the point cloud stream of PointNet++, and the image feature set is input into the image stream of ResNet-50. High-level features are extracted through the multi-layer perceptron and convolutional layer respectively. In the feature fusion layer, the attention mechanism is introduced to calculate the correlation matrix between modalities. The attention mechanism can assign different weights according to the feature importance of different modalities, thereby adaptively weighted fusion of features. For point cloud features and image features , calculate its correlation matrix

[0112] ;

[0113] in, and is a learnable weight matrix, and the softmax function is used to map each element of the correlation matrix to between 0 and 1. The point cloud and image features are weighted fused through the correlation matrix to obtain the final fusion feature .

[0114] ;

[0115] The fused features are mapped to a 256-dimensional feature space through a fully connected layer to obtain the final fused feature vector.

[0116] Among them, the point cloud feature set is sampled and grouped, 1024 center points are selected using the farthest point sampling algorithm, and a spherical neighborhood with a radius of 0.2 meters is constructed based on each center point to obtain a point cloud local feature group; the point cloud local feature group is input into the first layer Set Abstraction module of the PointNet++ network, and feature extraction and maximum pooling operations are performed on each local feature group through a multi-layer perceptron to obtain a feature representation of 512 points; the output of the first layer Set Abstraction module is sampled and grouped for the second time, 256 center points are selected, and a spherical neighborhood with a radius of 0.4 meters is constructed, and the features are input into the second layer Set Abstraction module to obtain a feature representation of 256 points; the output of the second layer Set Abstraction module is input into the third layer Set Abstraction module, perform global feature extraction, and obtain 1024-dimensional global point cloud features; preprocess the image feature set, scale the image to 224×224 pixels, and perform normalization to obtain standardized image features; input the standardized image features into the ResNet-50 network, and extract features through 5 residual blocks. Each residual block contains multiple convolutional layers, batch normalization layers, and ReLU activation functions to obtain 2048-dimensional image features; input the 1024-dimensional global point cloud features and 2048-dimensional image features into two independent fully connected layers, respectively, and adjust the feature dimensions to 512 dimensions to obtain point cloud features and image features of unified dimensions; calculate the attention weights between the point cloud features and image features of unified dimensions, specifically by calculating the point cloud features and image features. The inner product is taken and normalized by the Softmax function to obtain the attention weight matrix; the point cloud features and image features are weightedly fused based on the attention weight matrix to obtain a 1024-dimensional preliminary fusion feature; the preliminary fusion feature is input into the multi-head self-attention module, 8 attention heads are set, and the dimension of each head is 64. The fused context features are obtained through linear projection, scaled dot product attention and splicing operations; the fused context features are residually connected and layer normalized, and the features are transformed through a feedforward neural network to obtain the final 1024-dimensional fusion feature; the 1024-dimensional fusion feature is input into the fully connected layer, through the ReLU activation function and Dropout regularization (dropout rate is 0.5), and finally mapped to the 256-dimensional feature space to obtain the final fusion feature vector.

[0117] In a specific embodiment, the process of executing step 104 may specifically include the following steps:

[0118] The fused feature vector is input into the backbone network of the deep learning model, and multi-scale feature maps are extracted through multi-layer convolution and pooling operations to obtain a feature map set;

[0119] The feature map set is processed by a feature pyramid network, and features of different scales are fused through top-down paths and lateral connections to obtain enhanced multi-scale feature maps;

[0120] The enhanced multi-scale feature map is input into the region proposal network, and candidate regions of multiple scales and aspect ratios are generated on the feature map by sliding windows to obtain an initial candidate region set.

[0121] Perform non-maximum suppression on the initial candidate region set to remove redundant candidate regions with high overlap and obtain the target candidate region set;

[0122] The target candidate region set is mapped to the enhanced multi-scale feature map, and the fixed-size region features are extracted through the RoI Align operation to obtain the candidate region feature set;

[0123] The feature set of the candidate regions is input into the fully connected layer for target classification and bounding box regression to obtain the category probability distribution and bounding box correction parameters of each candidate region;

[0124] Based on the category probability distribution and bounding box correction parameters, the candidate regions are screened and refined, and the regions with category probabilities higher than the preset threshold and non-background categories are retained. The bounding box correction parameters are applied to obtain the bounding box detection results.

[0125] The bounding box detection results are subjected to cross-scale non-maximum suppression and target tracking association to obtain the initial detection result of each potentially dangerous target. The initial detection result includes the target's category, confidence and precise location information.

[0126] Specifically, the fused feature vector is , which means it is composed of 256-dimensional features. The fused feature vector is input into the backbone network of the deep learning model. The backbone network usually consists of multiple convolutional layers and pooling layers to extract multi-scale feature maps. Perform multi-layer convolution operations. The convolution operation slides the convolution kernel on the input features and performs matrix multiplication and addition operations to extract local spatial features. The input feature map of the layer is , the convolution kernel is , then the convolution output is:

[0127] ;

[0128] Among them, * represents the convolution operation, is the activation function (such as ReLU), is the bias term, It is The output feature map of the layer. Through multiple layers of convolution and pooling operations, features of different levels are gradually extracted, from low-level edge and texture information to high-level semantic information. Pooling operations are used to downsample feature maps, using maximum pooling or average pooling. Suppose the maximum pooling operation is MaxPool(), then The layer pooling output is:

[0129]

[0130] After multiple layers of convolution and pooling, a set of multi-scale feature maps is obtained. , these feature maps capture the multi-level information of the input fusion features. The multi-scale feature map set is processed by the feature pyramid network, and the features of different scales are fused through top-down paths and lateral connections to enhance the expressive power of the feature map. Suppose the top feature map is , the top-down path propagates the top-level semantic information to the low-level feature maps through layer-by-layer upsampling and additive fusion operations. The upsampling operation is expressed as:

[0131] ;

[0132] Among them, Upsample() represents the upsampling operation. It is In this way, a set of enhanced multi-scale feature maps are obtained. The enhanced multi-scale feature map is input into the region proposal network, and multiple candidate regions of different scales and aspect ratios are generated on the feature map by sliding the window. Suppose the size of the sliding window is , for each sliding window position ,generate There are candidate region anchor boxes, and the parameters of each anchor box are expressed as ,in and are the width and height of the anchor box respectively. The region proposal network performs binary classification (foreground or background) and bounding box regression on the anchor box through a classification branch and a regression branch. For each anchor box, the regression parameter , adjust the parameters of the candidate region by the following formula:

[0133] ;

[0134] ;

[0135] ;

[0136] ;

[0137] in, is the center coordinate of the adjusted anchor box, is the adjusted width and height of the anchor box. All candidate regions constitute the initial candidate region set. For the initial candidate region set, the region proposal network performs non-maximum suppression to remove redundant candidate regions with high overlap. Non-maximum suppression measures the overlap of two candidate regions by calculating the intersection-over-union ratio between them. Assume that the areas of the two candidate regions are and , the intersection area is , then IoU is expressed as:

[0138] ;

[0139] For regions where the IoU exceeds the set threshold, only the candidate regions with the highest confidence are retained, and the rest are eliminated to obtain a set of target candidate regions. The target candidate region set is mapped to the enhanced multi-scale feature map, and the fixed-size regional features are extracted through the Rol Align operation. Rol Align solves the feature misalignment problem caused by quantization error in Rol Pooling by bilinear interpolation. Suppose the feature map mapping coordinates of a target candidate region are , transform it to a fixed-size output feature map , and its interpolation formula is:

[0140] ;

[0141] in, represents the aligned feature map, is the bilinear interpolation weight, is the coordinate index of the fixed-size feature map. Through Rol Align, target candidate regions of different sizes are mapped to feature sets of the same size. The candidate region feature set is input into the fully connected layer for object classification and bounding box regression. The classification branch of the fully connected layer outputs the category probability distribution of each candidate region. ,in is the number of target categories, Indicates At the same time, the regression branch outputs the bounding box correction parameters of each candidate region Based on the category probability distribution and bounding box correction parameters, the candidate regions are screened and refined. The category probability is higher than the preset threshold. The non-background targets are selected and the correction parameters are applied to adjust the bounding box, and finally the bounding box detection results of each candidate area are obtained. The bounding box detection results are subjected to cross-scale non-maximum suppression (NMS) and target tracking association. Cross-scale NMS can eliminate redundant overlapping boxes on feature maps of different scales, while target tracking association can maintain the consistency and continuity of the target in multiple frames. After the above processing, the initial detection results of each potentially dangerous target are obtained, including the target category, confidence and precise location information.

[0142] In a specific embodiment, the process of executing step 105 may specifically include the following steps:

[0143] Perform feature extraction on the initial detection results of each potentially dangerous target, extract the location, size, speed, and direction features of the target, and obtain the target feature vector;

[0144] Based on the predefined multi-level hazard assessment rule base, a preliminary hazard level assessment is performed on the target feature vector. The rule base contains assessment rules based on target category, behavior characteristics, location information and time information to obtain a preliminary hazard level.

[0145] Construct a scene graph, use the detected targets as nodes, and the spatial relationships and interactions between targets as edges to obtain the initial scene graph, and input the initial scene graph into the designed graph neural network. Through multi-layer graph convolution operations, node features and edge features are extracted, and the message passing mechanism is used to capture the complex interaction relationship between the target and the environment to obtain an updated scene graph.

[0146] Based on the updated scene graph, the relevance score of each target node and its neighboring nodes is calculated, and the attention mechanism is used to perform weighted aggregation on the neighboring node information to obtain the contextual features of the target;

[0147] The context features of the target are integrated with the preliminary danger level to obtain the target danger level;

[0148] The Dempster-Shafer evidence theory framework is applied to the target hazard level, the quality function of each target at the low, medium and high hazard levels is calculated, and the multi-source information is integrated through the evidence combination rule to obtain the probability distribution of the hazard level;

[0149] Based on the probability distribution of danger levels, the comprehensive danger index and confidence level of each target are calculated. The danger index is the weighted sum of the probabilities of each level, and the confidence level is the highest probability value. The danger level and confidence level of each potential dangerous target are obtained.

[0150] Specifically, the core feature information of the target is extracted from the initial detection results. The initial detection results of each potentially dangerous target include information such as the target's position, size, speed, and direction. This information is calculated by the bounding box output by the deep learning model and its changes over time. Assume that the position of the target is expressed in two-dimensional coordinates Indicates that the size is the width of the target and height To represent the speed v and direction The target position is calculated by the change of the target position in multiple frames. The position at time is , at time The position at time is , then the speed of the target and direction It is expressed as:

[0151] ;

[0152] ;

[0153] in, is the time interval between adjacent frames, is the velocity vector, which represents the displacement of the target per unit time, and is a direction vector, indicating the direction of the target's movement. After extracting these features, the feature vector of the target is formed. , describing the spatial position and dynamic behavior of the target. Based on the predefined multi-level risk assessment rule base, the target feature vector Conduct a preliminary hazard level assessment. The rule base contains multiple assessment rules based on target category, behavior characteristics, location information, and time information. Assume that the hazard level assessment rule base consists of a set of rule functions Composition, each rule function Act on the target's feature vector and output the corresponding risk score The output values ​​of all rule functions are weighted and summed to obtain the preliminary hazard level score. Construct a scene graph, use the detected targets as nodes, and the spatial relationships and interactions between targets as edges to obtain the initial scene graph. Scene graph Each node in Represents a target, whose node features are the feature vectors of the target ; Each edge Indicates the target and The initial scene graph is input into the designed graph neural network, and node features and edge features are extracted through multi-layer graph convolution operations. The message passing mechanism is used to capture the complex interaction between the target and the environment to obtain an updated scene graph. In the graph convolution, each node Features In the The update formula of the layer is:

[0154] ;

[0155] in, is a nonlinear activation function, is the learnable weight matrix, For Node The set of neighbor nodes of Representation Node The number of neighbors. Through multi-layer graph convolution operations, nodes gradually aggregate the information of their neighboring nodes to obtain richer contextual features. Based on the updated scene graph, the relevance score of each target node and its neighboring nodes is calculated, and the attention mechanism is used to perform weighted aggregation of neighboring node information. Suppose a target node The context feature is , neighboring nodes The characteristics are , the relevance score is calculated through the attention mechanism:

[0156] ;

[0157] in, Representation Node With neighbor nodes The attention weight, is a learnable parameter in the attention mechanism, and is the weight matrix used to transform node features, Represents the feature concatenation operation. The context features of the target are obtained through weighted aggregation:

[0158] ;

[0159] Contextual features Preliminary hazard rating Fusion is performed to obtain the final danger level of the target , expressed as:

[0160] ;

[0161] in, is a learnable parameter in the fusion process. The Dempster-Shafer evidence theory framework is applied to the danger level of the target to calculate the quality function of each target at the three danger levels of low, medium and high. Based on the obtained probability distribution of danger level, the comprehensive danger index of each target is calculated. and confidence Danger Index is the weighted sum of the probabilities of each level, and the confidence is the highest probability value:

[0162] ;

[0163] ;

[0164] Get the danger level and confidence level of each potential dangerous target.

[0165] In a specific embodiment, the process of executing step 106 may specifically include the following steps:

[0166] Based on the danger level and confidence level of each potential danger target, the danger distribution heat map in the monitoring area is calculated to obtain the coordinate set of high-risk areas, and cluster analysis is performed on the coordinate set of high-risk areas to obtain multiple target areas that need to be monitored;

[0167] According to the spatial distribution of the target area, the scanning angle and resolution of the laser radar are adjusted to obtain the optimization parameters of the laser radar. Based on the location information of the target area, the distance and viewing angle from each security camera to each target area are calculated to obtain the camera coverage matrix.

[0168] Using the camera coverage matrix, the horizontal rotation angle, pitch angle and focal length of each security camera are optimized to obtain the optimized parameters of the camera. The optimized parameters of the lidar and the camera are combined to form the initial equipment collaborative monitoring parameter set.

[0169] Conduct conflict detection on the initial equipment collaborative monitoring parameter set to eliminate monitoring overlaps and blind spots between devices and obtain a coordinated monitoring parameter combination;

[0170] The coordinated monitoring parameter combination is input into the preset coverage evaluation model to calculate the monitoring coverage. If the preset threshold is not reached, the parameters are optimized again until a device collaborative monitoring parameter combination that meets the requirements is obtained.

[0171] Specifically, the danger level and confidence level are mapped to the space of the monitoring area. Suppose the danger level of a potential dangerous target is (between 0 and 1), and the confidence level is (represents the prediction reliability), and its risk impact weight is expressed as L The monitoring area is discretized into uniform grid cells, each grid cell The risk weight It is expressed as:

[0172] ;

[0173] in, is the set of all targets in the unit, and Respectively The danger level and confidence of each target are calculated. The danger distribution heat map of the monitored area is obtained. By setting the danger threshold Filter out high-risk grid cells and obtain the coordinate set of high-risk areas The DBSCAN clustering algorithm is used to cluster high-risk points into multiple key monitoring target areas, each of which has its center coordinates. According to the spatial distribution information of the target area, adjust the scanning angle and resolution of the laser radar. The position of the laser radar is , its scanning angle The calculation formula is:

[0174] ;

[0175] At the same time, adjust the resolution according to the high-risk areas To improve monitoring accuracy. Based on the location information of the target area, calculate the distance from each security camera to each target area and perspective The camera position is ,but:

[0176] ;

[0177] ;

[0178] Get the camera coverage matrix , and optimize the camera's horizontal rotation angle, pitch angle and focal length parameters based on the matrix. The optimized lidar and camera parameters are combined to form the initial equipment collaborative monitoring parameter set .right Conflict detection is performed to eliminate monitoring overlaps and blind spots between devices. If the intersection over union (IoU) of the coverage areas of two devices exceeds a preset threshold, the device parameters are adjusted to reduce overlapping monitoring. The calculation formula for IoU is:

[0179] ;

[0180] in, and Equipment and monitoring area, is the intersection area. After adjustment, the coordinated monitoring parameter combination is obtained .Will Input the coverage evaluation model and calculate the monitoring coverage:

[0181] ;

[0182] in, is the total area of ​​the monitoring area. If the coverage rate does not reach the preset threshold, the device parameters are optimized until the coverage rate meets the requirements.

[0183] Optionally, the coordinated monitoring parameter combination is converted into a device state vector, including the position coordinates, orientation angle and field of view parameters of each device, to obtain an initial device state set; based on the initial device state set, a three-dimensional monitoring space model is constructed, and the monitoring area is discretized into a three-dimensional grid, and the size of each grid is set to 0.1 meter × 0.1 meter × 0.1 meter, to obtain a monitoring space grid set; for each grid point in the monitoring space grid set, its visibility relationship with each monitoring device is calculated, and the field of view angle, distance attenuation and occlusion effect of the device are considered to obtain a grid point visibility matrix; based on the grid point visibility matrix, the coverage of each grid point is calculated, and the grid points covered by at least one device are marked as covered to obtain a covered grid point set; statistics are performed on the covered grid point set to calculate the coverage rate, that is, the ratio of the number of covered grid points to the total number of grid points, to obtain an initial coverage rate; the initial coverage rate is compared with a preset threshold, and if the threshold is reached or exceeded, the current monitoring parameter combination is used as the final output; if the threshold is not reached, the parameter optimization process is entered; the parameters are initialized based on the particle swarm optimization algorithm In the optimization process, the adjustable parameters of each device are encoded into the position vector of the particle, including the horizontal rotation angle, pitch angle and focal length, etc., to obtain the initial state of the particle swarm; for each particle in the particle swarm, the device state is updated according to its position vector, and the coverage is recalculated as the fitness function value to obtain the particle fitness set; according to the particle fitness set, the individual optimal position and the global optimal position of each particle are updated, and the particle position is adjusted based on the predefined speed update formula and position update formula to obtain a new generation of particle swarm; the particle swarm optimization iteration is repeated until the maximum number of iterations is reached or the coverage improvement is less than the set threshold, and the optimal particle position is obtained; the optimal particle position is decoded into device monitoring parameters, including the optimal horizontal rotation angle, pitch angle and focal length of each device, etc., to obtain the optimized device collaborative monitoring parameter combination; the final coverage of the optimized device collaborative monitoring parameter combination is verified. If the preset threshold is still not reached, the parameter settings of the particle swarm optimization algorithm, such as inertia weight, learning factor, etc., are adjusted, and the optimization process is re-executed until the final device collaborative monitoring parameter combination that meets the coverage requirements is obtained.

[0184] The above describes the danger alarm method based on radar sensing in the embodiment of the present invention. The following describes the danger alarm device based on radar sensing in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a danger alarm device based on radar sensing includes:

[0185] The preprocessing module 201 is used to preprocess the three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera to obtain target point cloud data and target image data;

[0186] A construction module 202 is used to construct a target Voronoi diagram of the monitoring area based on the target point cloud data and the target image data;

[0187] A fusion module 203 is used to extract and fuse the target point cloud data and the target image data according to the target Voronoi diagram to obtain a fused feature vector;

[0188] The detection module 204 is used to input the fused feature vector into the deep learning model for target detection and classification to obtain the initial detection result of each potential dangerous target;

[0189] An evaluation module 205 is used to perform a danger level evaluation based on the initial detection result to obtain the danger level and confidence level of each potential danger target;

[0190] The optimization module 206 is used to optimize the monitoring parameters of the laser radar and the security camera according to the danger level and confidence level, and output the device collaborative monitoring parameter combination.

[0191] Through the collaboration of the above components, by combining the three-dimensional point cloud data of the LiDAR and the image data of the security camera, the all-round perception of the monitoring scene is achieved, the limitations of a single data source are overcome, and the accuracy of target detection and recognition is improved. The optimized Voronoi diagram is constructed by using the improved Lloyd algorithm and environmental complexity constraints to realize the dynamic division of the monitoring area, so that the monitoring resources can be reasonably allocated according to the complexity of the environment. The advanced feature extraction methods such as FPFH and SIFT are used, combined with the designed multimodal feature fusion network, to effectively extract and fuse the key features of the point cloud and image. Through the region proposal network and feature pyramid network in the deep learning model, high-precision detection and classification of potential dangerous targets are achieved, and the recognition ability of the system is improved. Combined with the multi-level hazard assessment rule base and the contextual reasoning of the graph neural network, a comprehensive assessment of potential dangerous targets is achieved, and the accuracy and reliability of the hazard level judgment are improved. According to the hazard level and confidence, the monitoring parameters of the LiDAR and security cameras are optimized in real time, the coordinated monitoring of the equipment is realized, and the overall monitoring efficiency of the system is improved. Through iterative optimization and coverage evaluation, it is ensured that the monitoring system can be dynamically adjusted to meet the preset coverage requirements, thereby improving the flexibility and adaptability of the system.

[0192] above Figure 2 The radar-based hazard alarm device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The radar-based hazard alarm device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0193] Figure 33 is a schematic diagram of the structure of a radar-based hazard alarm device provided by an embodiment of the present invention. The radar-based hazard alarm device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage device terminals) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 may be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the radar-based hazard alarm device 300. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the radar-based hazard alarm device 300 to implement the steps of the above-mentioned radar-based hazard alarm method.

[0194] The radar-based hazard warning device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 3 The structure of the radar-sensing-based hazard warning device shown does not constitute a limitation on the radar-sensing-based hazard warning device provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0195] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the radar-sensing-based danger alarm method.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0198] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A danger alarm method based on radar sensing, characterized in that: The method comprises: Preprocess the three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera to obtain target point cloud data and target image data; Constructing a target Voronoi diagram of the monitoring area based on the target point cloud data and the target image data; specifically comprising: discretizing the monitoring area into a grid, with the grid size set to 0.5 m×0.5 m, and determining the position coordinates of the laser radar and the security camera based on the target point cloud data and the target image data to obtain a device position set; for each position in the device position set, calculating its corresponding Voronoi unit, specifically by calculating the Euclidean distance from each grid point in the monitoring area to the position of each device, and dividing the grid point into the Voronoi unit corresponding to the nearest device, to obtain an initial Voronoi diagram; based on the device position set, calculating the centroid coordinates of each Voronoi unit, and extending a 0.3-meter safety buffer zone outside the boundary of each Voronoi unit to obtain a Voronoi unit set with a buffer zone; performing a fusion analysis on the target point cloud data and the target image data, and using the Gaussian process regression method to calculate the environmental complexity index of each grid point in the monitoring area to obtain an environmental complexity distribution map; based on the environmental complexity distribution map, for each Voronoi unit, ronoi unit calculates its average environmental complexity, and assigns a weight factor to each device according to the average environmental complexity. The weight factor of the laser radar is set to 1.5, and the weight factor of the security camera is set to 1.0, so as to obtain a set of device weights; the initial Voronoi diagram is optimized by using the improved Lloyd algorithm, and the shape and size of each Voronoi unit are iteratively updated. The specific process is as follows: the total energy function of the current Voronoi diagram is calculated, the environmental complexity and the device weight are included in the calculation of the energy function, and the total energy function is minimized by the gradient descent method to obtain the optimized Voronoi unit boundary; the optimized Voronoi unit boundary is smoothed, the boundary points are fitted by the cubic spline interpolation algorithm, and the boundary is locally adjusted while keeping the topological relationship unchanged to obtain a smoothed Voronoi diagram; the smoothed Voronoi diagram is tested to ensure that the area of ​​each Voronoi unit is not less than the set minimum threshold, and the overlapping area between adjacent Voronoi units does not exceed the set maximum threshold, so as to obtain the target Voronoi diagram; According to the target Voronoi diagram, feature extraction and feature fusion are performed on the target point cloud data and the target image data to obtain a fused feature vector; Inputting the fused feature vector into a deep learning model for target detection and classification to obtain an initial detection result for each potentially dangerous target; Based on the initial detection results, a danger level assessment is performed to obtain the danger level and confidence of each potentially dangerous target; specifically, the following steps are performed: feature extraction is performed on the initial detection results of each potentially dangerous target, and the position, size, speed, and direction features of the target are extracted to obtain a target feature vector; based on a predefined multi-level danger assessment rule base, a preliminary danger level assessment is performed on the target feature vector, and the rule base contains assessment rules based on target category, behavior characteristics, location information, and time information to obtain a preliminary danger level; a scene graph is constructed, and the detected targets are used as nodes, and the spatial relationships and interactions between targets are used as edges to obtain an initial scene graph, and the initial scene graph is input into the designed graph neural network, and node features and edge features are extracted through multi-layer graph convolution operations, and the message passing mechanism is used to capture the complexity of the target and the environment. interactive relationship to obtain an updated scene graph; based on the updated scene graph, calculate the correlation score of each target node and its neighboring nodes, use the attention mechanism to perform weighted aggregation on the neighboring node information to obtain the contextual features of the target; fuse the contextual features of the target with the preliminary danger level to obtain the target danger level; apply the Dempster-Shafer evidence theory framework to the target danger level, calculate the quality function of each target at the three danger levels of low, medium and high, and fuse multi-source information through evidence combination rules to obtain the danger level probability distribution; based on the danger level probability distribution, calculate the comprehensive danger index and confidence of each target, the danger index is the weighted sum of the probabilities of each level, and the confidence is the highest probability value, and obtain the danger level and confidence of each potential dangerous target; According to the danger level and the confidence level, the monitoring parameters of the laser radar and the security camera are optimized, and a combination of equipment collaborative monitoring parameters is output; specifically, the following steps are performed: based on the danger level and confidence level of each potential danger target, a heat map of danger distribution in the monitoring area is calculated to obtain a set of high-risk area coordinates, and a cluster analysis is performed on the set of high-risk area coordinates to obtain multiple target areas that need to be monitored; according to the spatial distribution of the target area, the scanning angle and resolution of the laser radar are adjusted to obtain the optimization parameters of the laser radar, and based on the location information of the target area, the distance and field of view of each security camera to each target area are calculated; angle to obtain a camera coverage matrix; using the camera coverage matrix, optimize the horizontal rotation angle, pitch angle and focal length of each security camera to obtain the optimized parameters of the camera, and combine the optimized parameters of the lidar with the optimized parameters of the camera to form an initial device collaborative monitoring parameter set; perform conflict detection on the initial device collaborative monitoring parameter set to eliminate monitoring overlaps and blind spots between devices to obtain a coordinated monitoring parameter combination; input the coordinated monitoring parameter combination into a preset coverage evaluation model to calculate the monitoring coverage, and if the preset threshold is not reached, perform parameter re-optimization until a device collaborative monitoring parameter combination that meets the requirements is obtained.

2. The radar-based danger alarm method according to claim 1, characterized in that: The three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera are preprocessed to obtain target point cloud data and target image data, including: Collecting three-dimensional point cloud data by laser radar, calculating the average distance between each point in the three-dimensional point cloud data and surrounding points, and removing points exceeding the average distance based on a preset standard deviation multiple threshold to obtain initial point cloud data; Performing spatial division on the initial point cloud data, dividing the space into cubic grids of a fixed size, and calculating the centroid coordinates of all points in each cubic grid to obtain target point cloud data; Collecting raw image data through a security camera, performing global histogram analysis on the raw image data, calculating a cumulative distribution function, and performing nonlinear mapping on image pixel values ​​based on the cumulative distribution function to obtain first image data; Performing multi-scale feature extraction on the first image data and reconstructing an image to obtain second image data, adaptively dividing the second image data into blocks, calculating a local histogram of each block, and locally equalizing pixel values ​​based on the block histogram to obtain third image data; Extracting residual features from the third image data, and adding the residual features to the third image data to obtain fourth image data; Performing edge-preserving filtering on the fourth image data, calculating the spatial weight and the gray value similarity weight using a bilateral filtering algorithm, and performing weighted averaging on the pixel values ​​based on the spatial weight and the gray value similarity weight to obtain fifth image data; The fifth image data is spatially and temporally registered with the target point cloud data, the point cloud coordinate system is converted to the image coordinate system by calculating the rigid body transformation matrix, and an accurate correspondence between the point cloud data and the image pixels is established to obtain the target image data.

3. The radar-based danger alarm method according to claim 1, characterized in that: The step of extracting and fusing the target point cloud data and the target image data according to the target Voronoi diagram to obtain a fused feature vector includes: Based on the target Voronoi diagram, the target point cloud data is segmented into sub-point cloud sets corresponding to the Voronoi units, and a normal vector and a curvature are calculated for each sub-point cloud set to obtain local geometric features; Calculate the FPFH descriptor for each point in the sub-point cloud set, specifically including: calculating the simplified point feature histogram, setting the spherical neighborhood radius to 0.05 meters, calculating the normal vector angle, the distance between point pairs and the angle difference for each point's k nearest neighbors, and obtaining a 33-dimensional SPFH feature; The FPFH descriptor is calculated based on the SPFH feature, and the SPFH features of each point are weighted and accumulated to obtain a 33-dimensional FPFH feature. The FPFH features of all points are combined to form a point cloud feature set. Perform SIFT feature extraction on the target image data, first construct a scale space, perform Gaussian blur and downsampling on the image, generate 4 groups, each group contains 5 scales, calculate Gaussian difference image, and obtain scale space representation; Detect local extreme points in the DOG image, compare each pixel with its 26 neighboring pixels, retain feature points that meet the contrast threshold and principal curvature ratio threshold, and obtain a set of candidate feature points; The candidate feature point set is accurately located at the sub-pixel level, the DOG function is interpolated using Taylor expansion, the position and scale of the feature points are iteratively optimized, and unstable points are eliminated to obtain an accurate feature point set; Calculate the main direction of the feature point, count the gradient direction histogram in the circular area around the feature point, select the peak value as the main direction, create multiple feature descriptors for feature points with multiple peak values, and obtain a 128-dimensional SIFT feature descriptor to form an image feature set; A multimodal feature fusion network is designed with a dual-stream architecture. PointNet++ is used for the point cloud stream and ResNet-50 is used for the image stream. The point cloud feature set and the image feature set are input into the corresponding point cloud stream and image stream respectively. High-level features are extracted through multi-layer perceptron and convolutional layers. The attention mechanism is introduced in the feature fusion layer, and the correlation matrix between modalities is calculated. The features are adaptively weighted and fused. Finally, the fused features are mapped to a 256-dimensional feature space through a fully connected layer to obtain a fused feature vector.

4. The radar-based danger alarm method according to claim 1, characterized in that: The fused feature vector is input into a deep learning model for target detection and classification to obtain an initial detection result for each potentially dangerous target, including: Inputting the fused feature vector into the backbone network of the deep learning model, extracting multi-scale feature maps through multi-layer convolution and pooling operations, and obtaining a feature map set; Performing feature pyramid network processing on the feature map set, fusing features of different scales through top-down paths and lateral connections, and obtaining an enhanced multi-scale feature map; Inputting the enhanced multi-scale feature map into a region proposal network, generating candidate regions of multiple scales and aspect ratios on the feature map by means of a sliding window, and obtaining an initial candidate region set; Performing non-maximum suppression processing on the initial candidate region set to eliminate redundant candidate regions with high overlap, thereby obtaining a target candidate region set; Mapping the target candidate region set to the enhanced multi-scale feature map, extracting fixed-size region features through a RoI Align operation, and obtaining a candidate region feature set; Input the candidate region feature set into the fully connected layer, perform target classification and bounding box regression, and obtain the category probability distribution and bounding box correction parameters of each candidate region; Based on the category probability distribution and the bounding box correction parameters, the candidate regions are screened and refined, the regions whose category probabilities are higher than a preset threshold and are not background regions are retained, and the bounding box correction parameters are applied to obtain a bounding box detection result; The bounding box detection result is subjected to cross-scale non-maximum suppression and target tracking association to obtain an initial detection result of each potentially dangerous target, wherein the initial detection result includes the category, confidence and precise location information of the target.

5. A danger alarm device based on radar sensing, characterized in that: The device is used to execute the danger warning method based on radar sensing as described in any one of claims 1 to 4, comprising: A preprocessing module is used to preprocess the three-dimensional point cloud data collected by the laser radar and the original image data collected by the security camera to obtain target point cloud data and target image data; A construction module, used for constructing a target Voronoi diagram of a monitoring area based on the target point cloud data and the target image data; A fusion module, used for performing feature extraction and feature fusion on the target point cloud data and the target image data according to the target Voronoi diagram to obtain a fused feature vector; A detection module, used for inputting the fused feature vector into a deep learning model for target detection and classification, and obtaining an initial detection result for each potentially dangerous target; An evaluation module, used to perform a danger level evaluation based on the initial detection result to obtain a danger level and a confidence level of each potential danger target; The optimization module is used to optimize the monitoring parameters of the laser radar and the security camera according to the danger level and the confidence level, and output a device collaborative monitoring parameter combination.

6. A danger alarm device based on radar sensing, characterized in that: The radar-sensing-based danger warning device comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the radar sensing-based hazard warning device to execute the radar sensing-based hazard warning method according to any one of claims 1 to 4.

7. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the radar sensing-based hazard warning method according to any one of claims 1 to 4 is implemented.

Images