Intrusion judgment method and device based on radar track and visual track, computer equipment and medium

Abstract

The invention relates to an intrusion judgment method and device based on a radar track and a visual track, computer equipment and a medium. The intrusion judgment method comprises the following steps: encoding radar point cloud data into a radar feature map aligned with a camera image through voxel reconstruction; performing fusion visual analysis based on the radar feature map and the camera image to obtain a visual track of the target; performing clustering analysis on the radar point cloud data to obtain a radar track of the target; cross-modal matching is carried out on the radar track and the visual track to obtain a matching point pair, and the matching point pair comprises the radar track and the visual track of the to-be-detected target; based on the radar trajectory, obtaining a first intrusion result of the to-be-detected target in a three-dimensional space of the preset area, and based on the visual trajectory, obtaining a second intrusion result of the to-be-detected target in a two-dimensional space of the preset area; and when both the first intrusion result and the second intrusion result represent that the to-be-detected target has the intrusion behavior, determining that the to-be-detected target has the intrusion behavior. The method can accurately judge whether the target to be detected has the intrusion behavior or not.

Description

Technical Field

[0001] This application relates to the field of security monitoring technology, and in particular to an intrusion detection method, device, computer equipment, and medium based on radar trajectory and visual trajectory. Background Technology

[0002] Perimeter security systems play a crucial role in the security of key areas such as industrial parks, substations, oil and gas storage and transportation bases, airports, rail transit, and warehousing and logistics facilities. With the continuous expansion of facilities and the increasing diversification of intrusion methods, the perimeter has become a critical link in preventing external intrusions and avoiding security incidents. Common perimeter intrusions include people climbing over fences, vehicles breaking in, animals disturbing the perimeter, drones approaching at low altitudes, and tools damaging the perimeter. These intrusions can lead to security incidents, equipment damage, production interruptions, and even serious security accidents; therefore, real-time monitoring and rapid early warning of the perimeter are of paramount importance.

[0003] Currently, the main methods used in perimeter security monitoring can be categorized into visual detection, millimeter-wave detection, and fiber optic vibration detection. Visual detection typically uses visible light cameras, and in special cases, infrared or thermal imaging cameras can be used for image acquisition. Algorithms analyze the acquired images to identify intrusion behavior. However, visual detection suffers from light sensitivity; its detection and recognition capabilities decrease in conditions such as nighttime, backlighting, and strong light. It is also affected by weather conditions, with poor image quality and significant image degradation in rain, fog, and snow, resulting in low accuracy in intrusion detection. Millimeter-wave detection typically uses traditional millimeter-wave radar for data acquisition. It analyzes radar echoes to determine the presence of intruders within the area, thus achieving security detection. However, millimeter-wave detection is limited by radar resolution, resulting in weak recognition capabilities and poor visualization. Furthermore, traditional millimeter-wave radar lacks height information, making it unable to effectively judge height-related target behavior, also leading to low accuracy in intrusion detection. Fiber optic vibration detection uses optical fiber as the sensing medium. When the surrounding environment experiences vibrations, impacts, climbing, or other disturbances, the scattered light in the fiber changes. By analyzing the scattered signal, the type, location, and intensity of the event can be determined. It offers ultra-long-distance monitoring capabilities and is less affected by lighting conditions. However, its installation is labor-intensive, costly, and susceptible to environmental noise. Furthermore, it can only identify events, not specific objects. Therefore, there is an urgent need for a method that can accurately determine whether a target is engaging in intrusion. Summary of the Invention

[0004] Therefore, it is necessary to provide an intrusion detection method, device, computer equipment, and medium based on radar trajectory and visual trajectory that can accurately determine whether a target under test has intrusion behavior, in order to address the above-mentioned technical problems.

[0005] An intrusion detection method based on radar trajectory and visual trajectory, the method comprising:

[0006] S1. Encode the radar point cloud data into a radar feature map aligned with the camera image through voxel reconstruction.

[0007] S2. Based on the radar feature maps and camera images of multiple frames, perform fusion visual analysis to obtain the visual trajectory of each target, and perform cluster analysis on the radar point cloud data of multiple frames to obtain the radar trajectory of each target.

[0008] S3. Perform cross-modal matching on the radar trajectory and the visual trajectory to obtain matching point pairs. The matching point pairs include the radar trajectory and visual trajectory of the target to be tested, and the target to be tested is one of the targets.

[0009] S4. Based on the radar trajectory, obtain the first intrusion result of the target in the three-dimensional space of the preset area, and based on the visual trajectory, obtain the second intrusion result of the target in the two-dimensional space of the preset area.

[0010] S5. When both the first intrusion result and the second intrusion result indicate that the target under test has intrusion behavior, it is determined that the target under test has intrusion behavior.

[0011] This application employs cross-modal matching of visual and radar trajectories. This allows for simultaneous target matching utilizing both visual appearance information and radar's 3D positioning capabilities, thereby constructing more reliable matching pairs. This reduces recognition instability caused by occlusion, lighting fluctuations, or sparse radar point clouds, enabling accurate determination of whether a target is engaging in intrusion behavior based on both its radar and visual trajectories. Furthermore, this application can be rapidly deployed without additional hardware or dedicated targets, adapting to perimeter scenarios of varying scales and environments. It achieves all-weather, all-space, and multi-dimensional monitoring of intrusion targets, significantly improving the intelligence and practical applicability of perimeter security.

[0012] In one embodiment, step S1 includes:

[0013] Acquire radar point cloud data and camera images collected from the same viewpoint, and preset the grid range and resolution in the radar coordinate system where the radar point cloud data is located;

[0014] Based on the grid range and the resolution, calculate the voxel index of each point cloud in the radar point cloud data, and divide the point clouds with the same voxel index into the same voxel.

[0015] The voxel features of each voxel are statistically analyzed, and the voxel features are dimensionality reduced to obtain the dimensionality-reduced features of each voxel.

[0016] The voxel center of each voxel is projected onto the pixel coordinate system of the camera image to obtain the pixel position of each voxel center in the pixel coordinate system, and the dimensionality reduction feature of each voxel is written into the RGB channel of the corresponding pixel position to obtain a radar feature map aligned with the camera image.

[0017] In this application, the voxel features of each voxel are statistically analyzed and the voxel features are dimensionality reduced to obtain the dimensionality-reduced features of each voxel. This removes redundant features from the voxel features, retains the main information, and reduces computational complexity and storage overhead.

[0018] In one embodiment, the voxel centers of each voxel are projected onto the pixel coordinate system using a projection matrix, wherein the projection matrix is ​​obtained in the following ways:

[0019] Obtain the intrinsic parameter matrix of the first device that acquires the camera images;

[0020] The first device and the second device that collects the radar point cloud data are used to identify the same target point to obtain a target point pair.

[0021] Based on the target point pair, calculate the rotation matrix and translation vector of the second device relative to the first device;

[0022] The projection matrix is ​​obtained based on the intrinsic parameter matrix, the rotation matrix, and the translation vector.

[0023] In this application, by obtaining the intrinsic parameter matrix of the first device for acquiring camera images, the first device and the second device for acquiring radar point cloud data are used to identify the same target point and obtain a target point pair. Based on the target point pair, the rotation matrix and translation vector of the second device relative to the first device are calculated. Based on the intrinsic parameter matrix, rotation matrix and translation vector, the projection matrix is ​​obtained. In this way, the voxel center of each voxel can be projected to the pixel coordinate system using the projection matrix.

[0024] In one embodiment, step S2 includes:

[0025] Noise information is removed from each frame of radar point cloud data to obtain processed point cloud data;

[0026] The processed point cloud data is clustered using a density-based clustering algorithm to obtain target clusters for each frame of the processed point cloud data, and the target clusters are accompanied by attribute information.

[0027] Based on the attribute information of each target cluster in each frame of processed point cloud data, inter-frame association is performed to obtain the initial radar trajectory of each target.

[0028] The initial radar trajectory is interpolated to obtain the radar trajectory of each target.

[0029] In this application, noise information is removed from each frame of radar point cloud data to obtain processed point cloud data. A density-based clustering algorithm is used to perform cluster analysis on the processed point cloud data to obtain target clusters for each frame of processed point cloud data. The target clusters are accompanied by attribute information. Based on the attribute information of each target cluster in each frame of processed point cloud data, inter-frame correlation is performed to obtain the initial radar trajectory of each target. The initial radar trajectory is interpolated to obtain the radar trajectory of each target. This makes the final radar trajectory more accurate and smoother.

[0030] In one embodiment, step S5 is followed by:

[0031] Based on the planar coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data, the horizontal displacement of the target under test is calculated, and based on the height coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data, the height change of the target under test is calculated.

[0032] The instantaneous velocity of the target is obtained based on the first displacement of the target in adjacent frames and the first time difference between adjacent frames. The average velocity of the target is obtained based on the second displacement of the target in multiple frames and the second time difference between multiple frames.

[0033] The motion direction vector of the target under test is determined based on the radar trajectory, and the angle between the motion direction vector and the reference direction vector of the preset area is calculated.

[0034] The duration of the target's stay in the preset area is statistically analyzed, and it is determined whether the target exhibits delayed behavior in the preset area, thus obtaining the determination result.

[0035] Based on the horizontal displacement, the height change, the instantaneous velocity, the average velocity, the included angle, the dwell time, and the judgment result, the intrusion behavior of the target under test is classified.

[0036] In this application, the horizontal displacement of the target under test is calculated based on the planar coordinates of the radar trajectory in continuous multi-frame radar point cloud data. The height change of the target under test is calculated based on the height coordinates of the radar trajectory in continuous multi-frame radar point cloud data. The instantaneous velocity of the target under test is obtained based on the first displacement of the target under test in adjacent frames and the first time difference between adjacent frames. The average velocity of the target under test is obtained based on the second displacement of the target under test in multiple frames and the second time difference between multiple frames. The motion direction vector of the target under test is determined based on the radar trajectory, and the angle between the motion direction vector and the reference direction vector of the preset area is calculated. The dwell time of the target under test in the preset area is counted, and it is determined whether there is a delay behavior of the target under test in the preset area, and the judgment result is obtained. In this way, the intrusion behavior of the target under test can be accurately classified based on the horizontal displacement, height change, instantaneous velocity, average velocity, angle, dwell time, and judgment result, which makes up for the deficiency of visual recognition in low light or complex posture scenes.

[0037] In one embodiment, step S2 includes:

[0038] Based on the radar feature map and the camera image, a radar-visual feature-level fusion network is used to output the detection information of each target. The radar-visual feature-level fusion network includes a radar feature branch and an image feature branch. The radar feature branch runs in parallel with the image feature branch. The radar feature branch is used to extract the spatial structure features, reflection intensity features, and multi-scale information of each target from the radar feature map layer by layer. The image feature branch is used to extract the image texture and target contour from the camera image. The detection information includes target category, target bounding box, and confidence level.

[0039] Based on the detection information from multiple frames of radar feature maps and camera images, a tracking algorithm is used to generate the visual trajectory of each target.

[0040] In this application, by relying on the radar-visual feature-level fusion network, the image texture and contour intensity of the camera image can be fused with the spatial structure and reflection characteristics of the radar feature map at the feature level, thereby improving the overall detection accuracy and robustness in complex environments.

[0041] In one embodiment, step S3 includes:

[0042] The radar trajectory is projected onto the pixel coordinate system of the camera image to obtain the radar projection point;

[0043] Calculate the target distance between the radar projection point and the image detection box, and obtain a pixel distance score based on the target distance and a preset width. The image detection box is the detection box associated with the visual trajectory.

[0044] The pixel coverage score is determined based on the overlap between the image detection box and the radar detection box. The radar detection box is obtained based on the bounding box of the radar projection point or the target cluster to which the radar projection point belongs.

[0045] The pixel distance score and the pixel coverage score are weighted and summed to obtain the matching consistency score between the radar trajectory and the visual trajectory;

[0046] An average score is obtained based on the matching consistency score between the radar trajectory and the visual trajectory of the valid frames within the sliding window.

[0047] A cost matrix is ​​constructed based on the average score, and the minimum total cost allocation is solved using the Hungarian algorithm to obtain the initial matching point pairs.

[0048] Obtain the matching status of each initial matching point pair in multiple consecutive sliding windows;

[0049] When the initial matching point pair is determined to be a valid match in multiple sliding windows, the initial matching point pair is determined to be a matching point pair.

[0050] In this application, radar projection points are obtained by projecting the radar trajectory onto the pixel coordinate system of the camera image. The target distance between the radar projection point and the image detection box is calculated, and a pixel distance score is obtained based on the target distance and a preset width. The image detection box is a detection box associated with the visual trajectory. A pixel coverage score is determined based on the overlap between the image detection box and the radar detection box. The radar detection box is obtained based on the radar projection point or the bounding box of the target cluster to which the radar projection point belongs. The pixel distance score and the pixel coverage score are weighted and summed to obtain the matching consistency score between the radar trajectory and the visual trajectory. An average score is obtained based on the matching consistency score between the radar trajectory and the visual trajectory of the effective frames within the sliding window. A cost matrix is ​​constructed based on the average score, and the minimum total cost allocation is solved using the Hungarian algorithm to obtain initial matching point pairs. The matching status of each initial matching point pair in multiple consecutive sliding windows is obtained. When the matching status of the initial matching point pair in multiple sliding windows is determined to be a valid match, the initial matching point pair is determined to be a matching point pair. This can solve the problem of unstable matching between radar and vision in the temporal / spatial domains.

[0051] An intrusion detection device based on radar trajectory and visual trajectory, the device comprising:

[0052] The image alignment module is used to encode radar point cloud data into radar feature maps aligned with camera images through voxel reconstruction.

[0053] The analysis module is used to perform visual analysis on multiple frames of radar feature maps and multiple frames of camera images to obtain the visual trajectory of each target, and to perform cluster analysis on multiple frames of radar point cloud data to obtain the radar trajectory of each target.

[0054] The matching module is used to perform cross-modal matching on the radar trajectory and the visual trajectory to obtain matching point pairs. The matching point pairs include the radar trajectory and visual trajectory of the target to be tested, and the target to be tested is one of the targets.

[0055] An intrusion detection module is used to obtain a first intrusion result of the target in the three-dimensional space of a preset area based on the radar trajectory, and to obtain a second intrusion result of the target in the two-dimensional space of the preset area based on the visual trajectory.

[0056] The early warning module is used to determine that the target under test has intrusion behavior when both the first intrusion result and the second intrusion result indicate that the target under test has intrusion behavior.

[0057] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0058] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0059] The aforementioned intrusion detection method, device, computer equipment, and medium based on radar trajectory and visual trajectory encode radar point cloud data into radar feature maps aligned with camera images through voxel reconstruction. Visual analysis is performed on multiple frames of radar feature maps and multiple frames of camera images to obtain the visual trajectory of each target. Cluster analysis is performed on multiple frames of radar point cloud data to obtain the radar trajectory of each target. Cross-modal matching is performed on the radar trajectory and visual trajectory to obtain matching point pairs, which include the radar trajectory and visual trajectory of the target to be tested, where the target to be tested is one of the targets. Based on the radar trajectory, a first intrusion result is obtained in the three-dimensional space of the target to be tested within a preset area. Based on the visual trajectory, a second intrusion result is obtained in the two-dimensional space of the target to be tested within the preset area. When the first and second intrusion results indicate that the target to be tested has intrusion behavior, a warning prompt is output. The beneficial effect of this application is that by performing cross-modal matching of visual trajectory and radar trajectory, the appearance information of vision and the three-dimensional positioning capability of radar can be used simultaneously for target matching, thereby constructing a more reliable matching pair and reducing the recognition instability caused by occlusion, illumination fluctuation or sparse radar point cloud. Thus, based on the radar trajectory and visual trajectory of the target, it is possible to accurately determine whether the target has intrusion behavior. Attached Figure Description

[0060] Figure 1 This is an application environment diagram of an intrusion detection method based on radar trajectory and visual trajectory in one embodiment;

[0061] Figure 2 This is a flowchart illustrating an intrusion detection method based on radar and visual trajectories in one embodiment.

[0062] Figure 3 This is a flowchart illustrating the classification of intrusion behaviors in one embodiment;

[0063] Figure 4 This is a schematic diagram illustrating the output of detection information for each target using a Ravec feature-level fusion network in one embodiment.

[0064] Figure 5 This is a flowchart of the overall detection process in one embodiment;

[0065] Figure 6 This is a structural block diagram of an intrusion detection device based on radar trajectory and visual trajectory in one embodiment;

[0066] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] The intrusion detection method based on radar trajectory and visual trajectory provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 interacts with server 104 via a wired / wireless channel. A data storage system stores the data that server 104 needs to process. The server controls the radar to encode radar point cloud data into radar feature maps aligned with camera images through voxel reconstruction; the server performs visual analysis on multiple frames of radar feature maps and multiple frames of camera images to obtain the visual trajectories of each target; the server controls the radar to perform cluster analysis on multiple frames of radar point cloud data to obtain the radar trajectories of each target; the server performs cross-modal matching on the radar trajectories and visual trajectories to obtain matching point pairs, which include the radar trajectory and visual trajectory of the target to be tested, where the target to be tested is one of the targets; based on the radar trajectory, the server controls the radar to determine the first intrusion result of the target to be tested in the three-dimensional space of a preset area, and based on the visual trajectory, the server obtains the second intrusion result of the target to be tested in the two-dimensional space of the preset area; when the first and second intrusion results indicate that the target to be tested has intrusion behavior, the server outputs a warning prompt. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. Server 104 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers.

[0069] In one embodiment, such as Figure 2 As shown, an intrusion detection method based on radar trajectory and visual trajectory is provided, and this method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0070] S1. Encode the radar point cloud data into a radar feature map aligned with the camera image through voxel reconstruction.

[0071] Encoding radar point cloud data into radar feature maps aligned with camera images via voxel reconstruction is a crucial preliminary step for achieving radar-visual feature-level fusion. The goal is to transform sparse, unstructured radar point cloud data into regular two-dimensional feature representations aligned with camera images, facilitating subsequent pixel-level or region-level fusion with camera images.

[0072] S2. Based on the fusion visual analysis of multi-frame radar feature maps and multi-frame camera images, the visual trajectory of each target is obtained. Cluster analysis is performed on multi-frame radar point cloud data to obtain the radar trajectory of each target.

[0073] The fused visual analysis utilizes a radar-visual feature-level fusion network and tracking algorithms for fused visual analysis. The visual trajectory is a two-dimensional trajectory, while the radar trajectory is a three-dimensional trajectory.

[0074] The target is a target that appears in radar point cloud data and camera images. For example, vehicles or people.

[0075] S3. Perform cross-modal matching on the radar trajectory and visual trajectory to obtain matching point pairs. The matching point pairs include the radar trajectory and visual trajectory of the target to be tested, and the target to be tested is one of the targets.

[0076] Cross-modal matching refers to the process of associating, aligning, and fusing radar trajectories and visual trajectories in multimodal data processing.

[0077] In some embodiments, a radar-vision cross-modal encoder, a transformer cross-modal matcher, or a statistical model can also be used to perform cross-modal matching of radar and visual trajectories. The statistical model can be a maximum likelihood statistical model.

[0078] S4. Based on the radar trajectory, obtain the first intrusion result of the target in the three-dimensional space of the preset area, and based on the visual trajectory, obtain the second intrusion result of the target in the two-dimensional space of the preset area.

[0079] The preset area is the area where intrusion behavior needs to be detected.

[0080] The first intrusion result can be determined based on whether the radar trajectory of the target falls within the three-dimensional space of the preset area. For example, if a certain trajectory point of the target's radar trajectory is located within the three-dimensional space of the preset area, then the first intrusion result can be determined as the target engaging in intrusion behavior.

[0081] The second intrusion result can be determined based on whether the visual trajectory of the target falls within the two-dimensional space of the preset area. For example, if a certain trajectory point in the visual trajectory of the target falls within the two-dimensional space of the preset area, then the second intrusion result can be determined as the target exhibiting intrusion behavior.

[0082] S5. When both the first intrusion result and the second intrusion result indicate that the target under test has intrusion behavior, it is determined that the target under test has intrusion behavior.

[0083] Furthermore, when the first and second intrusion results indicate that the target under test has engaged in intrusion behavior, the presence of intrusion behavior is confirmed, and a warning alert is output. The output of the warning alert may take the form of, but is not limited to, sound alerts, light alerts, and text alerts.

[0084] In some embodiments, when the first intrusion result indicates that the target under test has intrusion behavior, it is determined that the target under test has intrusion behavior, and an early warning prompt is output.

[0085] In some embodiments, when the second intrusion result indicates that the target under test has intrusion behavior, it is determined that the target under test has intrusion behavior, and an early warning prompt is output.

[0086] The aforementioned intrusion detection method based on radar and visual trajectories performs cross-modal matching between visual and radar trajectories. This allows for target matching by simultaneously utilizing visual appearance information and radar's 3D positioning capabilities, thereby constructing more reliable matching pairs and reducing recognition instability caused by occlusion, lighting fluctuations, or sparse radar point clouds. This enables accurate determination of whether a target is engaging in intrusion behavior based on both its radar and visual trajectories. Furthermore, this application allows for rapid deployment without additional hardware or dedicated targets, adapting to perimeter scenarios of varying sizes and environments. It achieves all-weather, all-space, and multi-dimensional monitoring of intrusion targets, significantly improving the intelligence and practical applicability of perimeter security.

[0087] In one embodiment, step S1 includes:

[0088] Acquire radar point cloud data and camera images collected from the same viewpoint, and preset the grid range and resolution in the radar coordinate system where the radar point cloud data is located;

[0089] Based on the grid range and resolution, the voxel index of each point cloud in the radar point cloud data is calculated, and point clouds with the same voxel index are divided into the same voxel.

[0090] The voxel features of each voxel are statistically analyzed, and the voxel features are dimensionality reduced to obtain the dimensionality-reduced features of each voxel.

[0091] The voxel center of each voxel is projected onto the pixel coordinate system of the camera image to obtain the pixel position of each voxel center in the pixel coordinate system. The dimensionality reduction features of each voxel are written into the RGB channel of the corresponding pixel position to obtain a radar feature map aligned with the camera image.

[0092] In this system, a first device for acquiring camera images and a second device for acquiring radar point cloud data are rigidly mounted on the same base. The optical axes of the first and second devices are collinear or nearly collinear in space, resulting in highly overlapping observation fields of view. Furthermore, the angle between the central axis of the main beam of the second device and the optical axis of the first device is less than 5 degrees to achieve near-coaxial observation. The second device can be either a 4D millimeter-wave radar or a lidar, while the first device is a camera. The 4D millimeter-wave radar possesses all-weather imaging capabilities, maintaining continuous target detection and tracking even in complex weather conditions such as nighttime, backlighting, and rain / fog, avoiding the missed detections and false detections that occur with traditional vision systems in low-light, occluded, and image-degraded environments. Simultaneously, the three-dimensional coordinates and height information output by the 4D millimeter-wave radar effectively compensate for the inherent lack of depth in two-dimensional images, enabling accurate identification of intrusion behaviors closely related to height, such as climbing, vaulting, and jumping, achieving fine-grained action recognition and spatial behavior analysis that traditional millimeter-wave radar cannot accomplish.

[0093] Both radar point cloud data and camera images consist of multiple frames, so the radar feature map also consists of multiple frames.

[0094] The grid range includes the maximum value x in the x-direction. max and minimum value x min The maximum value of y in the y direction max and minimum value y min The maximum value z in the z-direction max and minimum value z min Resolution is the size of the grid, and it includes resolution in the x, y, and z directions. Further, based on the resolution and grid extent, the following formula can be used... Calculate the number of grid cells in the x, y, and z directions, v x Let v be the resolution in the x-direction. y Let v be the resolution in the y-direction. z This represents the resolution in the z-direction.

[0095] The formula for calculating the voxel index (i,j,k) of each point cloud (x,y,z) is as follows: , , .

[0096] The voxel characteristics include, but are not limited to, the number of points in the voxel, the presence of a moving target within the voxel, and the average RCS (Radar Cross Section) of the voxel. The number of points indicates the density of the point cloud within the voxel. Each point cloud also has a velocity value, so the presence of a moving target within the voxel can be determined based on the average velocity of each point cloud; a non-zero average velocity indicates the potential presence of a moving target. The RCS can be used to determine the surface reflectivity of each target.

[0097] Dimensionality reduction of voxel features f can be achieved using the dimensionality reduction matrix W. PCA Implementation. Specifically, using the dimensionality reduction matrix W. PCA The expression for dimensionality reduction of the voxel feature f is: , g is the dimensionality-reduced feature f after dimensionality reduction, b is the bias term, C1 is the dimension of the dimensionality-reduced feature, and C2 is the dimension of the voxel feature.

[0098] Dimensionality reduction features are obtained using principal component analysis (PCA). The process of obtaining dimensionality reduction features using PCA includes:

[0099] Step 1: Obtain multiple training voxel features from the training set / validation set, and the multiple training voxel features constitute a voxel feature matrix;

[0100] Step 2: Calculate the column mean of the voxel feature matrix. Subtract the corresponding column mean from the training voxel features of each dimension to obtain the mean-centered matrix. The calculation formula is: , To train the voxel feature matrix, This is the column mean;

[0101] Step 3: Calculate the covariance matrix or perform singular value decomposition on the mean-centered matrix. That is, calculate the covariance matrix for the already centered mean-centered matrix. The formula for calculating the covariance matrix is: ,in Mean-centered matrix The transpose of , where N represents the number of training voxel features and S is the covariance matrix;

[0102] Step 4: Take the vectors of the first three principal components as row vectors to form a dimension reduction matrix.

[0103] Principal component analysis can retain the most variance information and provides better interpretability.

[0104] Voxel centers of each voxel (x c ,y c ,z c The formula for calculating ) is, , , Because voxels correspond to "grid numbers", and the center of a voxel is in the middle of a grid, we use +0.5.

[0105] In some embodiments, if multiple voxel centers are projected to the same pixel location in the pixel coordinate system, a depth-first strategy is used to retain the projection result of the nearest voxel center. Distance refers to the distance between the voxel center and the second device. The distance between the voxel center and the second device can be obtained by solving for Euclidean distance.

[0106] The corresponding pixel position refers to the pixel position of the voxel center projected onto the pixel coordinate system. For example, if the voxel center of voxel A is projected onto point B in the pixel coordinate system, then the dimensionality reduction feature of voxel A is written into the RGB (Red-Green-Blue) channel of point B, thus obtaining a color radar feature map.

[0107] In some embodiments, the intrusion detection method based on radar trajectory and visual trajectory further includes: normalizing the dimensionality reduction features, writing the normalized dimensionality reduction features into the RGB channel of the corresponding pixel position, and obtaining a radar feature map aligned with the camera image.

[0108] In this embodiment, by statistically analyzing the voxel features of each voxel and performing dimensionality reduction processing on the voxel features, the dimensionality-reduced features of each voxel are obtained. This removes redundant features from the voxel features, retains the main information, and reduces computational complexity and storage overhead.

[0109] In one embodiment, the voxel center of each voxel is projected onto the pixel coordinate system using a projection matrix. The projection matrix is ​​obtained in the following ways:

[0110] Obtain the intrinsic parameter matrix of the first device used to acquire camera images;

[0111] The first device and the second device that collects radar point cloud data are used to identify the same target point to obtain a target point pair.

[0112] Based on the target point pair, calculate the rotation matrix and translation vector of the second device relative to the first device;

[0113] The projection matrix is ​​obtained based on the intrinsic parameter matrix, rotation matrix, and translation vector.

[0114] The voxel center is multiplied by the projection matrix to obtain the pixel position of each voxel center in the pixel coordinate system.

[0115] The intrinsic parameter matrix is f x and f y c is the focal length pixel value of the first device. x and c y Let s be the principal point and s be the shear coefficient. The intrinsic parameter matrix can be obtained through checkerboard calibration. Furthermore, if the distortion coefficient of the first device is greater than the preset distortion value, the distortion coefficient is also obtained so that the pixels in the camera image can be mapped back to the ideal surface using the distortion coefficient.

[0116] In some embodiments, the number of target point pairs is at least one. Multiple target point pairs can be obtained by simultaneously identifying target points using the first device and the second device multiple times. The target points identified each time may be the same or different. For example, in the first identification, the first device and the second device simultaneously identify target point A, and in the second identification, the first device and the second device simultaneously identify target point B.

[0117] The target point pair includes the two-dimensional coordinates identified by the first device and the three-dimensional coordinates identified by the second device.

[0118] The rotation matrix and translation vector can be obtained using nonlinear optimization methods or the PnP algorithm.

[0119] The expression for calculating the projection matrix P is as follows: K is the intrinsic parameter matrix, R is the rotation matrix, and t is the translation vector.

[0120] In this embodiment, by acquiring the intrinsic parameter matrix of the first device that acquires camera images, the first device and the second device that acquires radar point cloud data are used to identify the same target point to obtain a target point pair. Based on the target point pair, the rotation matrix and translation vector of the second device relative to the first device are calculated. Based on the intrinsic parameter matrix, rotation matrix and translation vector, the projection matrix is ​​obtained. In this way, the voxel center of each voxel can be projected to the pixel coordinate system using the projection matrix.

[0121] In one embodiment, step S2 includes:

[0122] Noise information is removed from each frame of radar point cloud data to obtain processed point cloud data;

[0123] Density-based clustering algorithm is used to perform cluster analysis on the processed point cloud data to obtain the target clusters of each frame of processed point cloud data. The target clusters are accompanied by attribute information.

[0124] Based on the attribute information of each target cluster in each frame of processed point cloud data, inter-frame correlation is performed to obtain the initial radar trajectory of each target.

[0125] The initial radar trajectory is interpolated to obtain the radar trajectory of each target.

[0126] The process of removing noise from each frame of radar point cloud data to obtain processed point cloud data includes: preset RCS threshold and velocity threshold, removing point clouds with RSC less than the RCS threshold, removing point clouds with velocity less than the velocity threshold, and removing ground point clouds, thus obtaining processed point cloud data.

[0127] Density-based clustering algorithms were used to perform cluster analysis on the processed point cloud data, and the target clusters of each frame of processed point cloud data were obtained as follows:

[0128] Step 1: For point p in the processed point cloud data, calculate the number of neighboring points within the preset neighborhood radius of point p. If the number of neighboring points is greater than the preset minimum clustering value, then mark point p as a core point; otherwise, mark point p as a boundary point or a candidate noise point, until all points in the processed point cloud data have been marked.

[0129] Step 2: Randomly select an unvisited core point as the seed point, initialize a new target cluster, and add the selected core point to the target cluster; initialize an expansion queue, and add all unvisited points within the preset neighborhood radius of the core point to the expansion queue; sequentially take point q from the expansion queue and mark point q as visited; if point q has not yet belonged to any cluster, add it to the current target cluster; determine whether point q is a core point. If point q is a core point, add all unvisited points within the neighborhood of point q that do not belong to any cluster to the expansion queue for subsequent expansion; repeat the above process until the expansion queue is empty, completing the construction of the current target cluster;

[0130] Step 3: Iterate through Step 2 until all point clouds in the processed point cloud data have been accessed, and output multiple target clusters.

[0131] Attribute information includes, but is not limited to, position, velocity, size, and confidence level. Inter-frame association connects these target clusters in time into an initial radar trajectory. The process is divided into: state modeling, prediction, similarity calculation, association solution, and trajectory update.

[0132] In some embodiments, to address the point cloud flickering problem in radar measurements, a trajectory preservation strategy is adopted for targets that are obscured or temporarily disappear. That is, if a target cluster of a certain trajectory in a certain frame disappears, the trajectory should not be deleted immediately, but should be retained for a period of time, and then restored after being re-associated with the target cluster, so as to obtain the initial radar trajectory of each target.

[0133] Interpolating the initial radar trajectory can make the final radar trajectory smoother.

[0134] In this embodiment, noise information is removed from each frame of radar point cloud data to obtain processed point cloud data. Density-based clustering algorithm is used to perform cluster analysis on the processed point cloud data to obtain target clusters for each frame of processed point cloud data. The target clusters are accompanied by attribute information. Based on the attribute information of each target cluster in each frame of processed point cloud data, inter-frame correlation is performed to obtain the initial radar trajectory of each target. The initial radar trajectory is interpolated to obtain the radar trajectory of each target. This makes the final radar trajectory more accurate and smoother.

[0135] In one embodiment, step S5 is followed by:

[0136] Based on the planar coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data, the horizontal displacement of the target under test is calculated, and based on the height coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data, the height change of the target under test is calculated.

[0137] The instantaneous velocity of the target is obtained based on the first displacement of the target in adjacent frames and the first time difference between adjacent frames. The average velocity of the target is obtained based on the second displacement of the target in multiple frames and the second time difference between multiple frames.

[0138] The motion direction vector of the target under test is determined based on the radar trajectory, and the angle between the motion direction vector and the reference direction vector of the preset area is calculated.

[0139] The duration of the target's stay in the preset area is recorded, and it is determined whether the target exhibits delayed behavior in the preset area, thus obtaining the judgment result.

[0140] Based on horizontal displacement, height change, instantaneous velocity, average velocity, included angle, dwell time, and judgment results, the intrusion behavior of the target under test is classified.

[0141] Among them, continuous multi-frame radar point cloud data refers to multiple frames of radar point cloud data acquired continuously. The radar trajectory has three-dimensional coordinates. Based on the two-dimensional plane coordinates of the radar trajectory of the target in continuous multi-frame radar point cloud data, the horizontal displacement of the target can be calculated; based on the height coordinates of the radar trajectory in continuous multi-frame radar point cloud data, the height change of the target can be calculated.

[0142] Adjacent frames refer to two consecutively acquired frames of radar point cloud data. The first displacement is calculated based on the three-dimensional coordinates of the radar trajectory of the target in the adjacent frames. The first time difference is the difference in timestamps between adjacent frames, where the timestamp is the acquisition time of the radar point cloud data. Specifically, if the timestamp of the preceding frame in an adjacent frame is A, and the timestamp of the following frame in an adjacent frame is B, then the first time difference is BA. The instantaneous velocity is obtained by dividing the first displacement by the first time difference.

[0143] The second displacement is calculated based on the three-dimensional coordinates of the radar trajectory of the target in multi-frame radar point cloud data. Specifically, the second displacement is calculated based on the three-dimensional coordinates of the radar trajectory in the first frame of radar point cloud data and the three-dimensional coordinates of the radar trajectory in the last frame of radar point cloud data. Multi-frame radar point cloud data refers to point cloud data containing the target. The first frame of radar point cloud data is the first frame in the multi-frame radar point cloud data, and the last frame of radar point cloud data is the last frame in the multi-frame radar point cloud data. The second time difference is the difference between the timestamp of the first frame of radar point cloud data and the timestamp of the last frame of radar point cloud data. The average velocity is obtained by dividing the second displacement by the second time difference.

[0144] Both the first and second displacements can be obtained by solving for the Euclidean distance.

[0145] By fitting the trajectory points in the radar trajectory of the target under test, the motion direction vector of the target can be obtained. Since the preset area is a pre-defined area, the reference direction vector of the preset area can be directly obtained.

[0146] included angle The calculation expression is as follows , Let the direction of motion be the vector. This is the reference direction vector.

[0147] The dwell time can be determined based on the duration of the radar trajectory of the target being measured within a preset area. The duration of the radar trajectory within the preset area can be determined based on the number of frames of radar point cloud data associated with the radar trajectory. For example, if the radar trajectory is within the preset area from the first frame of radar point cloud data to the Nth frame of radar point cloud data, then the dwell time is the difference between the timestamp of the first frame of radar point cloud data and the timestamp of the Nth frame of radar point cloud data.

[0148] The determination of whether the target exhibits delayed behavior within a preset area is based on its average speed, location coverage area, and path length. Location coverage area refers to the minimum spatial range covered by all trajectory points of the target within the preset area, and path length refers to the total length of the trajectory actually traversed by the target within the preset area.

[0149] Based on horizontal displacement, height change, instantaneous velocity, average velocity, included angle, dwell time, and judgment results, the classification of intrusion behavior of the target under test includes: inputting horizontal displacement, height change, instantaneous velocity, average velocity, included angle, dwell time, and judgment results into a pre-trained behavior model to classify the intrusion behavior of the target under test, thus obtaining the classification result of the intrusion behavior of the target under test. The specific flowchart of intrusion behavior classification is as follows... Figure 3 As shown. Intrusion behaviors include, but are not limited to, vaulting, climbing, jumping, and crawling.

[0150] In a specific application, during a circumvention intrusion, the following conditions and scoring function apply: .

[0151] in For the change in height, The fence height for the preset area. This indicates the preset height threshold. This represents the instantaneous velocity of the target being measured. This represents the preset speed threshold, and t represents the time spent during the movement. th This indicates a preset time threshold. These represent the weights of each feature in the scoring function. This is the scoring function. This scoring function scores intrusion behavior from three aspects: the change in height must be at least greater than the fence height. The instantaneous maximum velocity must be greater than a velocity threshold to ensure the target has sufficient kinetic energy to overcome the fence. Finally, the time taken for the target to cross the fence should be as short as possible to exclude slow passage through doors or stairs. All calculated features need to be normalized using the sigmoid activation function to ensure consistent scaling before weighted calculation.

[0152] In this embodiment, the horizontal displacement of the target is calculated based on the planar coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data. The height change of the target is calculated based on the height coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data. The instantaneous velocity of the target is obtained based on the first displacement of the target in adjacent frames and the first time difference between adjacent frames. The average velocity of the target is obtained based on the second displacement of the target in multiple frames and the second time difference between multiple frames. The motion direction vector of the target is determined based on the radar trajectory, and the angle between the motion direction vector and the reference direction vector of the preset area is calculated. The dwell time of the target in the preset area is counted, and it is determined whether there is a delay behavior of the target in the preset area, and the judgment result is obtained. In this way, the intrusion behavior of the target can be accurately classified based on the horizontal displacement, height change, instantaneous velocity, average velocity, angle, dwell time, and judgment result, which makes up for the deficiency of visual recognition in low light or complex posture scenes.

[0153] In some embodiments, a voxelized point cloud temporal classifier can also be used to classify intrusion behaviors. Voxelized point cloud temporal classifiers include, but are not limited to, PointRNN (Point Recurrent Neural Network) and Point Recurrent Neural Network (Convolutional Neural Network – Long Short-Term Memory Network for Radar Feature Extraction).

[0154] In one embodiment, step S2 includes:

[0155] Based on radar feature maps and camera images, a radar-visual feature-level fusion network is used to output the detection information of each target. The radar-visual feature-level fusion network includes a radar feature branch and an image feature branch. The radar feature branch and the image feature branch run in parallel. The radar feature branch is used to extract the spatial structure features, reflection intensity features and multi-scale information of each target from the radar feature map layer by layer. The image feature branch is used to extract the image texture and target contour from the camera image. The detection information includes target category, target box and confidence score.

[0156] Based on the detection information from multi-frame radar feature maps and camera images, a tracking algorithm is used to generate the visual trajectory of each target.

[0157] The radar-visual feature-level fusion network is a model obtained by adding a radar feature branch to the existing YOLOv5 image backbone network. This network is used to analyze radar feature maps and camera images. The radar feature branch and image feature branch operate in parallel within the network. During the feature convergence stage, the outputs of the two branches are fused using a concat method, enabling the detector to simultaneously perceive image texture features and radar spatial structure features, and adaptively learn the correlation between the two during training.

[0158] During the training phase of the radar-visual feature-level fusion network, the training radar feature maps, after cross-modal calibration and pixel-level alignment, and the corresponding training image frames are synchronously input into the network. This allows the network to gradually learn and exploit the coupling characteristics between the training radar feature maps and the training image frames. Benefiting from the reliable detection capabilities of radar in low-light, obstructed, rainy, and foggy environments, traditional visual models struggle to extract effective information from degraded images when image quality significantly degrades due to environmental fluctuations. However, the radar-visual feature-level fusion network of this application can compensate for visual features using the three-dimensional spatial structure features output from the radar feature branches, thereby maintaining the accuracy and continuity of detection.

[0159] Through multimodal deep fusion, the radar-visual feature-level fusion network constructed in this application not only retains the advantages of image feature branches in appearance recognition and detail analysis, but also significantly improves the detection robustness in low light and harsh environments, so that the target recall rate remains stable in complex scenes.

[0160] The radar-visual feature-level fusion network performs joint inference on camera images and radar feature maps, outputting detection information, namely target category, target bounding box, and confidence score. Frame-by-frame, the detection information output by the radar-visual feature-level fusion network is used as an observation set. A tracking algorithm associates the detection information of consecutive frames to achieve continuous tracking of each target and generate visual trajectories for each target. Specifically, the tracking algorithm associates the independent target bounding boxes in each frame, assigns them unique identifiers, and forms continuous and stable visual trajectories, thereby achieving long-term, robust tracking of multiple targets.

[0161] A schematic diagram illustrating the output of detection information for each target using the Ravis feature-level fusion network is shown below. Figure 4 As shown in the figure. The image captured by the camera is the camera image.

[0162] In this embodiment, by relying on the radar-visual feature-level fusion network, the image texture and contour intensity of the camera image can be fused with the spatial structure and reflection characteristics of the radar feature map at the feature level, thereby improving the overall detection accuracy and robustness in complex environments.

[0163] In one embodiment, step S3 includes:

[0164] The radar trajectory is projected onto the pixel coordinate system of the camera image to obtain the radar projection point;

[0165] Calculate the target distance between the radar projection point and the image detection box, and obtain the pixel distance score based on the target distance and the preset width. The image detection box is a detection box associated with the visual trajectory.

[0166] The pixel coverage score is determined based on the overlap between the image detection box and the radar detection box. The radar detection box is obtained based on the bounding box of the radar projection point or the target cluster to which the radar projection point belongs.

[0167] The pixel distance score and pixel coverage score are weighted and summed to obtain the matching consistency score between the radar trajectory and the visual trajectory.

[0168] The average score is obtained based on the matching consistency score between radar trajectory and visual trajectory of effective frames within the sliding window.

[0169] A cost matrix is ​​constructed based on the average score, and the minimum total cost allocation is solved using the Hungarian algorithm to obtain the initial matching point pairs.

[0170] Obtain the matching status of each initial matching point pair in multiple consecutive sliding windows;

[0171] When the initial matching point pair is determined to be a valid match in multiple sliding windows, the initial matching point pair is identified as a matching point pair.

[0172] The radar trajectory includes multiple trajectory points, which are projected onto the pixel coordinate system of the camera image using a projection matrix. The image detection boxes associated with the visual trajectory are obtained by optimizing the target boxes output by the radar-visual feature-level fusion network using a tracking algorithm. The radar point cloud data corresponding to the radar projection points and the image detection boxes are from the same frame.

[0173] The formula for calculating pixel distance score is as follows: , For the target distance, The preset width, This is the pixel distance score for the k-th frame image. The k-th frame image includes both simultaneously acquired camera images and radar point cloud data. The preset width is an empirical value, generally set to 20 pixels. A smaller kernel results in a sharper Gaussian curve and is more stringent about pixel offsets; even a small pixel offset can cause the score to drop rapidly. A larger kernel is more lenient, allowing for greater positional deviations, and the score drops more slowly. The target distance is the shortest Euclidean distance between the radar projection point and the image detection box.

[0174] The formula for calculating pixel coverage score is as follows: , For image detection bounding boxes, For radar detection frames, The pixel coverage score for the k-th frame image.

[0175] If a bounding box exists for the target cluster to which the radar projection point belongs, the 2D bounding box in the pixel coordinate system is directly used as the radar detection box. If no bounding box exists, the radar detection box is obtained based on the radar projection point. The process of obtaining the radar detection box based on the radar projection point is as follows: with the radar projection point as the center, the length and width are each expanded by a preset length to obtain the radar detection box. The bounding box of the target cluster to which the radar projection point belongs is obtained by cluster analysis of the radar point cloud data.

[0176] A frame is considered a "valid frame" only if both the radar trajectory and the visual trajectory are valid observations.

[0177] The formula for calculating the consistency score between radar trajectory and visual trajectory is as follows: , The score represents the consistency score between the radar trajectory and the visual trajectory in the k-th frame image. For weights.

[0178] In some embodiments, if the vast majority of frames within a time window lack visual trajectories or there is no target to be detected in the radar point cloud data, the average score should be set to unmatchable.

[0179] In some embodiments, noise information in each frame of radar point cloud data is removed to obtain processed point cloud data; radar trajectory is obtained through processed point cloud data, and radar trajectory is projected onto the pixel coordinate system where the camera image is located to obtain radar projection points.

[0180] In this embodiment, the radar projection point is obtained by projecting the radar trajectory onto the pixel coordinate system of the camera image. The target distance between the radar projection point and the image detection box is calculated, and a pixel distance score is obtained based on the target distance and a preset width. The image detection box is a detection box associated with the visual trajectory. The pixel coverage score is determined based on the overlap between the image detection box and the radar detection box. The radar detection box is obtained based on the radar projection point or the bounding box of the target cluster to which the radar projection point belongs. The pixel distance score and the pixel coverage score are weighted and summed to obtain the matching consistency score between the radar trajectory and the visual trajectory. The average score is obtained based on the matching consistency score between the radar trajectory and the visual trajectory of the effective frames within the sliding window. A cost matrix is ​​constructed based on the average score, and the minimum total cost allocation is solved using the Hungarian algorithm to obtain the initial matching point pairs. The matching status of each initial matching point pair in multiple consecutive sliding windows is obtained. When the matching status of the initial matching point pair in multiple sliding windows is determined to be a valid match, the initial matching point pair is determined as a matching point pair. This can solve the problem of unstable matching between radar and vision in the temporal / spatial domains.

[0181] This application also provides an application scenario in which the aforementioned intrusion detection method based on radar trajectory and visual trajectory is applied. Specifically, the application of the intrusion detection method based on radar trajectory and visual trajectory in this scenario is as follows:

[0182] The server obtains the projection matrix based on the intrinsic parameter matrix, rotation matrix, and translation vector. The server reconstructs and encodes the radar point cloud data into radar feature maps aligned with the camera images using voxel reconstruction. Based on the radar feature maps and camera images, the server outputs the detection information for each target using a radar-visual feature-level fusion network. Based on the detection information from multiple frames of radar feature maps and camera images, the server generates the visual trajectory of each target using a tracking algorithm. The server performs cluster analysis on the multi-frame radar point cloud data to obtain the radar trajectory of each target. The server performs cross-modal matching on the radar trajectory and visual trajectory to obtain matching point pairs, which include the radar trajectory and visual trajectory of the target being tested, where the target being tested is one of the targets. The server extracts multiple features from the radar trajectory of the target being tested and classifies the intrusion behavior of the target being tested based on the extracted features. Based on the radar trajectory of the target being tested, the server obtains the first intrusion result of the target being tested in the three-dimensional space of the preset area; based on the visual trajectory of the target being tested, the server obtains the second intrusion result of the target being tested in the two-dimensional space of the preset area. When the first and second intrusion results indicate that the target being tested has intrusion behavior, an early warning is output. The overall detection flowchart is as follows: Figure 5 As shown.

[0183] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0184] Based on the same inventive concept, this application also provides an intrusion detection device based on radar trajectory and visual trajectory for implementing the intrusion detection method based on radar trajectory and visual trajectory described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the intrusion detection device based on radar trajectory and visual trajectory provided below can be found in the limitations of the intrusion detection method based on radar trajectory and visual trajectory described above, and will not be repeated here.

[0185] In one embodiment, such as Figure 6 As shown, an intrusion detection device based on radar trajectory and visual trajectory is provided, comprising:

[0186] The image alignment module is used to encode radar point cloud data into radar feature maps aligned with camera images through voxel reconstruction.

[0187] The analysis module is used to perform visual analysis on multiple frames of radar feature maps and multiple frames of camera images to obtain the visual trajectory of each target, and to perform cluster analysis on multiple frames of radar point cloud data to obtain the radar trajectory of each target.

[0188] The matching module is used to perform cross-modal matching on the radar trajectory and the visual trajectory to obtain matching point pairs. The matching point pairs include the radar trajectory and visual trajectory of the target to be tested, and the target to be tested is one of the targets.

[0189] An intrusion detection module is used to obtain a first intrusion result of the target in the three-dimensional space of a preset area based on the radar trajectory, and to obtain a second intrusion result of the target in the two-dimensional space of the preset area based on the visual trajectory.

[0190] The early warning module is used to determine that the target under test has intrusion behavior when both the first intrusion result and the second intrusion result indicate that the target under test has intrusion behavior.

[0191] The modules in the aforementioned intrusion detection device based on radar and visual trajectories can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0192] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores radar point cloud data, radar feature maps, camera images, visual trajectories, radar trajectories, first intrusion results, and second intrusion results. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an intrusion detection method based on radar and visual trajectories.

[0193] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0194] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0195] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0196] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0197] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An intrusion detection method based on radar trajectory and visual trajectory, characterized in that, The method includes: S1. Encode the radar point cloud data into a radar feature map aligned with the camera image through voxel reconstruction. S2. Based on the radar feature maps and camera images of multiple frames, perform fusion visual analysis to obtain the visual trajectory of each target, and perform cluster analysis on the radar point cloud data of multiple frames to obtain the radar trajectory of each target. S3. Perform cross-modal matching on the radar trajectory and the visual trajectory to obtain matching point pairs. The matching point pairs include the radar trajectory and visual trajectory of the target to be tested, and the target to be tested is one of the targets. S4. Based on the radar trajectory, obtain the first intrusion result of the target in the three-dimensional space of the preset area, and based on the visual trajectory, obtain the second intrusion result of the target in the two-dimensional space of the preset area. S5. When both the first intrusion result and the second intrusion result indicate that the target under test has intrusion behavior, it is determined that the target under test has intrusion behavior.

2. The method according to claim 1, characterized in that, Step S1 includes: Acquire radar point cloud data and camera images collected from the same viewpoint, and preset the grid range and resolution in the radar coordinate system where the radar point cloud data is located; Based on the grid range and the resolution, calculate the voxel index of each point cloud in the radar point cloud data, and divide the point clouds with the same voxel index into the same voxel. The voxel features of each voxel are statistically analyzed, and the voxel features are dimensionality reduced to obtain the dimensionality-reduced features of each voxel. The voxel center of each voxel is projected onto the pixel coordinate system of the camera image to obtain the pixel position of each voxel center in the pixel coordinate system, and the dimensionality reduction feature of each voxel is written into the RGB channel of the corresponding pixel position to obtain a radar feature map aligned with the camera image.

3. The method according to claim 2, characterized in that, The voxel centers of each voxel are projected onto the pixel coordinate system using a projection matrix, wherein the projection matrix is ​​obtained in the following ways: Obtain the intrinsic parameter matrix of the first device that acquires the camera images; The first device and the second device that collects the radar point cloud data are used to identify the same target point to obtain a target point pair. Based on the target point pair, calculate the rotation matrix and translation vector of the second device relative to the first device; The projection matrix is ​​obtained based on the intrinsic parameter matrix, the rotation matrix, and the translation vector.

4. The method according to claim 1, characterized in that, Step S2 includes: Noise information is removed from each frame of radar point cloud data to obtain processed point cloud data; The processed point cloud data is clustered using a density-based clustering algorithm to obtain target clusters for each frame of the processed point cloud data, and the target clusters are accompanied by attribute information. Based on the attribute information of each target cluster in each frame of processed point cloud data, inter-frame association is performed to obtain the initial radar trajectory of each target. The initial radar trajectory is interpolated to obtain the radar trajectory of each target.

5. The method according to claim 1, characterized in that, Step S5 is followed by: Based on the planar coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data, the horizontal displacement of the target under test is calculated, and based on the height coordinates of the radar trajectory in multiple consecutive frames of radar point cloud data, the height change of the target under test is calculated. The instantaneous velocity of the target is obtained based on the first displacement of the target in adjacent frames and the first time difference between adjacent frames. The average velocity of the target is obtained based on the second displacement of the target in multiple frames and the second time difference between multiple frames. The motion direction vector of the target under test is determined based on the radar trajectory, and the angle between the motion direction vector and the reference direction vector of the preset area is calculated. The duration of the target's stay in the preset area is statistically analyzed, and it is determined whether the target exhibits delayed behavior in the preset area, thus obtaining the determination result. Based on the horizontal displacement, the height change, the instantaneous velocity, the average velocity, the included angle, the dwell time, and the judgment result, the intrusion behavior of the target under test is classified.

6. The method according to claim 1, characterized in that, Step S2 includes: Based on the radar feature map and the camera image, a radar-visual feature-level fusion network is used to output the detection information of each target. The radar-visual feature-level fusion network includes a radar feature branch and an image feature branch. The radar feature branch runs in parallel with the image feature branch. The radar feature branch is used to extract the spatial structure features, reflection intensity features, and multi-scale information of each target from the radar feature map layer by layer. The image feature branch is used to extract the image texture and target contour from the camera image. The detection information includes target category, target bounding box, and confidence level. Based on the detection information from multiple frames of radar feature maps and camera images, a tracking algorithm is used to generate the visual trajectory of each target.

7. The method according to claim 6, characterized in that, Step S3 includes: The radar trajectory is projected onto the pixel coordinate system of the camera image to obtain the radar projection point; Calculate the target distance between the radar projection point and the image detection box, and obtain a pixel distance score based on the target distance and a preset width. The image detection box is the detection box associated with the visual trajectory. The pixel coverage score is determined based on the overlap between the image detection box and the radar detection box. The radar detection box is obtained based on the bounding box of the radar projection point or the target cluster to which the radar projection point belongs. The pixel distance score and the pixel coverage score are weighted and summed to obtain the matching consistency score between the radar trajectory and the visual trajectory; An average score is obtained based on the matching consistency score between the radar trajectory and the visual trajectory of the valid frames within the sliding window. A cost matrix is ​​constructed based on the average score, and the minimum total cost allocation is solved using the Hungarian algorithm to obtain the initial matching point pairs. Obtain the matching status of each initial matching point pair in multiple consecutive sliding windows; When the initial matching point pair is determined to be a valid match in multiple sliding windows, the initial matching point pair is determined to be a matching point pair.

8. An intrusion detection device based on radar trajectory and visual trajectory, characterized in that, The device includes: The image alignment module is used to encode radar point cloud data into radar feature maps aligned with camera images through voxel reconstruction. The analysis module is used to perform visual analysis on multiple frames of radar feature maps and multiple frames of camera images to obtain the visual trajectory of each target, and to perform cluster analysis on multiple frames of radar point cloud data to obtain the radar trajectory of each target. The matching module is used to perform cross-modal matching on the radar trajectory and the visual trajectory to obtain matching point pairs. The matching point pairs include the radar trajectory and visual trajectory of the target to be tested, and the target to be tested is one of the targets. An intrusion detection module is used to obtain a first intrusion result of the target in the three-dimensional space of a preset area based on the radar trajectory, and to obtain a second intrusion result of the target in the two-dimensional space of the preset area based on the visual trajectory. The early warning module is used to determine that the target under test has intrusion behavior when both the first intrusion result and the second intrusion result indicate that the target under test has intrusion behavior.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images