Security monitoring method, electronic device, and medium

Abstract

The application provides a security monitoring method, an electronic device and a medium. The method comprises the following steps: collecting image data and three-dimensional point cloud data of a monitoring area; performing target identification on the image data to obtain at least one candidate image region; performing target identification on the three-dimensional point cloud data to obtain at least one candidate point cloud set; matching the at least one candidate image region with the at least one candidate point cloud set to obtain a matching result; and executing an alarm strategy based on the matching result. The technical solution of the application can reduce the false alarm rate of the alarm.

Description

Technical Field

[0001] This application relates to the field of security monitoring technology, specifically to a security monitoring method, electronic device, and medium. Background Technology

[0002] Security monitoring technology, as a core means of protecting the security of areas such as substations and key cultural relics protection units, can be used to identify intrusion targets and prevent security threats. However, existing security monitoring technologies struggle to effectively distinguish between genuine intrusion targets and environmental interference, resulting in a high false alarm rate. Summary of the Invention

[0003] In view of this, embodiments of this application provide a security monitoring method, electronic device, and medium that can reduce the false alarm rate.

[0004] In a first aspect, embodiments of this application provide a security monitoring method, comprising: collecting image data and three-dimensional point cloud data of a monitored area; performing target recognition on the image data to obtain at least one candidate image region; performing target recognition on the three-dimensional point cloud data to obtain at least one candidate point cloud; matching the at least one candidate image region with the at least one candidate point cloud to obtain a matching result; and executing an alarm strategy based on the matching result.

[0005] Secondly, embodiments of this application provide a security monitoring device, comprising: an acquisition module for acquiring image data and three-dimensional point cloud data of a monitored area; a first target recognition module for performing target recognition on the image data to obtain at least one candidate image region; a second target recognition module for performing target recognition on the three-dimensional point cloud data to obtain at least one candidate point cloud set; a matching module for matching the at least one candidate image region with the at least one candidate point cloud set to obtain a matching result; and an alarm module for executing an alarm strategy based on the matching result.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the security monitoring method of the first aspect described above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for performing the security monitoring method of the first aspect described above.

[0008] Fifthly, embodiments of this application provide a computer program product, which includes a computer program. When the computer program is executed by the processor of a computer device, it enables the computer device to perform the security monitoring method described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a chip, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the security monitoring method of the first aspect described above.

[0010] This application provides a security monitoring method, electronic device, and medium. It collects image data and 3D point cloud data of a monitored area, performs target recognition on the image data to obtain at least one candidate image region, performs target recognition on the 3D point cloud data to obtain at least one candidate point cloud, matches the at least one candidate image region and the at least one candidate point cloud to obtain a matching result, and executes an alarm strategy based on the matching result. By employing sensing channels based on different physical principles (such as optical sensing and radar / laser sensing), a matching relationship is established between the visual detection result (candidate image region) of the same potential intrusion target in the image plane and its spatial detection result (candidate point cloud) in 3D space. Only when the visual detection result and the spatial detection result can be successfully matched is the potential intrusion target confirmed as a real intrusion target and an alarm is triggered. This effectively filters out false alarms caused by the limitations of a single channel and significantly reduces the false alarm rate. Attached Figure Description

[0011] Figure 1 The diagram shown is a schematic representation of the system architecture of a security monitoring system provided in an exemplary embodiment of this application.

[0012] Figure 2 The diagram shown is a flowchart of a security monitoring method provided in an exemplary embodiment of this application.

[0013] Figure 3 The diagram shown is a flowchart of a security monitoring method provided in another exemplary embodiment of this application.

[0014] Figure 4 The diagram shown is a structural schematic of a security monitoring device provided in an exemplary embodiment of this application.

[0015] Figure 5 The diagram shown is a block diagram of an electronic device for performing a security monitoring method according to an exemplary embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] Exemplary System Figure 1 The diagram shown is a schematic representation of the system architecture of a security monitoring system provided in an exemplary embodiment of this application. Figure 1 As shown, the security monitoring system 100 may include an optical imaging device 110, a spatial ranging device 120, an infrared sensor 130, a processing unit 140, and an alarm device 150. The optical imaging device 110 can be used to acquire image data of the monitored area; the device type of the optical imaging device 110 may include a surveillance camera, a zoom camera, etc. The spatial ranging device 120 can be used to acquire three-dimensional point cloud data and background point cloud data of the monitored area; the device type of the spatial ranging device 120 may include a mechanical rotating lidar, a solid-state area array lidar, etc. The infrared sensor 130 can be used to acquire infrared thermal imaging images of the monitored area; the device type of the infrared sensor 130 may include an uncooled infrared focal plane array detector, a cooled infrared detector, a microbolometer, etc. The processing unit 140 can be used to perform target recognition on the image data and three-dimensional point cloud data, and to match the identified candidate image regions and candidate point cloud sets; the device type of the processing unit 140 may include a computer, an embedded processor, a microcontroller, etc. The processing unit 140 can execute an alarm strategy based on the matching results of candidate image regions and candidate point clusters, such as generating an alarm command based on the current matching results. The alarm device 150 can be used to execute an alarm based on the alarm command, such as issuing an alarm signal. The device type of the alarm device 150 may include an audible and visual alarm, a voice broadcasting device, a display screen, etc.

[0018] In one example, the optical imaging device 110 can acquire image data of the monitored area and send the image data to the processing unit 140; the spatial ranging device 120 can acquire three-dimensional point cloud data and background point cloud data of the monitored area and send the three-dimensional point cloud data and background point cloud data to the processing unit 140; and the infrared sensor 130 can acquire infrared thermal imaging images of the monitored area and send the infrared thermal imaging images to the processing unit 140.

[0019] Furthermore, the processing unit 140 can receive image data sent by the optical imaging device 110, perform target recognition on the image data, and obtain at least one candidate image region.

[0020] Furthermore, the processing unit 140 can receive three-dimensional point cloud data and background point cloud data sent by the spatial ranging device 120, determine the target point cloud located outside the background point cloud data in the three-dimensional point cloud data, cluster the target point cloud, and obtain at least one candidate point cloud set.

[0021] Furthermore, the processing unit 140 can calculate the first similarity between each candidate image region in at least one candidate image region and each candidate point cloud in at least one candidate point cloud, determine a set of candidate image regions and candidate point clouds whose first similarity satisfies the preset first similarity requirement, and obtain the target image region and target point cloud that are successfully matched.

[0022] Furthermore, the processing unit 140 can receive infrared thermal imaging images sent by the infrared sensor 130, identify thermal contour regions in the infrared thermal imaging images, calculate a second similarity between the thermal contour regions and the target image regions and / or target point clusters, and if the second similarity meets a preset second similarity requirement, verify the accuracy of the matching result and execute the corresponding alarm strategy. For example, the processing unit 140 can generate an alarm command based on the alarm strategy and send the alarm command to the alarm device 150.

[0023] Furthermore, the alarm device 150 can receive and execute alarm commands sent by the processing unit 140 to issue an alarm signal, so that security personnel can respond promptly and take appropriate measures. The alarm signal may include at least one of the following: light alarm signal, audible alarm signal, voice alarm signal, text alarm signal, etc.

[0024] It should be understood that the above application scenario examples are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited thereto. Rather, the embodiments of this application can be applied to any applicable scenario.

[0025] Exemplary methods Figure 2 The diagram shown is a schematic flowchart of a security monitoring method provided in an exemplary embodiment of this application. Figure 2 As shown, this security monitoring method may include the following:

[0026] 210: Collect image data and 3D point cloud data of the monitored area.

[0027] In one embodiment, the monitored area may include the physical space area for security monitoring. For example, the monitored area may include substations, key cultural relics protection units, warehousing and logistics areas, etc., which are within the monitoring range of the security system.

[0028] In one embodiment, image data may include two-dimensional images or video streams captured by optical imaging devices (such as surveillance cameras, zoom cameras, etc.) that reflect the appearance and texture information of objects in the monitored area. For example, image data may include red-green-blue (RGB) color images acquired by a visible light camera, panoramic images acquired by a fisheye lens and corrected for distortion, or wide-field-of-view images stitched together from multiple camera perspectives. As another example, image data may be seamless panoramic images, which can be generated by acquiring raw fisheye images using 360° panoramic cameras deployed in the monitored area, and then performing distortion correction and stitching processing on the acquired raw fisheye images to eliminate lens distortion and generate image data.

[0029] In one embodiment, the 3D point cloud data may include point cloud data collected by a spatial ranging device (e.g., a mechanical rotating lidar, a solid-state area array lidar, etc.) over a monitored area. For example, the 3D point cloud data may include a set of 3D spatial coordinates of points on the surfaces of various objects within the monitored area. The monitored area may include static environmental objects, such as buildings, plants, and other physical entities whose positions remain fixed (or do not undergo significant displacement) over a long period. In some cases, the monitored area may further include moving objects, such as people, animals, or objects suddenly introduced into the area. Exemplarily, the 3D point cloud data may include point cloud data corresponding to static environmental objects in the monitored area; moreover, when moving objects appear in the monitored area, the 3D point cloud data may further include point cloud data corresponding to the moving objects.

[0030] For example, a mechanical rotating lidar can be used to perform a 360° rotating scan of the monitored area to obtain seamless panoramic raw point cloud data, which can then be used as 3D point cloud data.

[0031] In one embodiment, before acquiring image data of the monitored area, the device for acquiring image data (such as optical imaging device 110) can be leveled, such as to ensure that the device is placed horizontally.

[0032] In one embodiment, the 3D point cloud data can be obtained by filtering the raw point cloud data acquired by the spatial ranging device. For example, methods for filtering the raw point cloud data may include voxel mesh downsampling, statistical outlier removal, etc.

[0033] In one embodiment, before collecting 3D point cloud data of the monitored area, the device for collecting 3D point cloud data (such as the spatial ranging device 120) can be leveled.

[0034] 220: Perform target recognition on the image data to obtain at least one candidate image region.

[0035] In one embodiment, target recognition may include a process of detecting and locating potential intrusion targets from image data and / or 3D point cloud data. Further, intrusion targets may include physical entities posing a security threat; for example, the types of intrusion targets may include wild animals, suspicious intruders, suspicious vehicles, etc. Further still, target recognition on image data may be a process of detecting and locating one or more candidate image regions that can characterize potential intrusion targets from image data using image recognition algorithms. For example, image recognition algorithms may include deep learning target detection algorithms (such as Transformer-based detection models), target detection algorithms based on handcrafted features, etc.

[0036] In one embodiment, the 3D point cloud data may contain spatial geometric information such as the absolute position, size, shape, and spatial relationship of the potential intrusion target, and the image data may contain visual appearance information such as the texture, color, and outline of the potential intrusion target.

[0037] In one embodiment, the candidate image region may include a local region in the image data that may contain an intrusion target. Specifically, an image recognition algorithm can be used to perform target recognition on the image data to obtain the candidate image region. Furthermore, the type of image recognition algorithm may include single-stage detection algorithms, two-stage detection algorithms, etc.

[0038] In one embodiment, the candidate image region can be represented by B. i = (x i , y i , w i , h i , conf i cls i ) characterization, where i is the index number of the candidate image region, B i Let x represent the i-th candidate image region. i The x-coordinate of the center point of the i-th candidate image region is y. i w represents the ordinate of the center point of the i-th candidate image region. i h represents the width of the i-th candidate image region. i conf represents the height of the i-th candidate image region. i cls represents the confidence level of the intrusion target represented by the i-th candidate image region. i This represents the appearance type of the intrusion target characterized by the i-th candidate image region. Further, cls i The value can include at least one of a person, a vehicle, or an animal.

[0039] 230: Perform target identification on the 3D point cloud data to obtain at least one candidate point cloud set.

[0040] In one embodiment, the candidate point cloud set may include local point cloud data in the 3D point cloud data that can characterize potential intrusion targets. Specifically, a point cloud recognition algorithm can be used to perform target recognition on the 3D point cloud data to obtain the candidate point cloud set. Furthermore, the type of point cloud recognition algorithm may include shape detection algorithms based on random sampling consistency, point cloud detection algorithms based on deep learning, etc.

[0041] In one embodiment, target identification of three-dimensional point cloud data may include a process of detecting and locating one or more candidate point clouds that can characterize potential intrusion targets from the three-dimensional point cloud data using a point cloud recognition algorithm.

[0042] 240: Match at least one candidate image region with at least one candidate point cloud to obtain the matching result.

[0043] In one embodiment, matching may include the process of establishing a correspondence between candidate image regions and candidate point clouds. For example, a candidate image region may represent the presence of an intrusion target in that region, and a candidate point cloud may also represent the presence of an intrusion target in a local region corresponding to that point cloud. When a candidate image region and a candidate point cloud match in spatial location, such as both representing a local region within a monitored area, then the candidate image region and the candidate point cloud are considered to have matched successfully. Furthermore, when a candidate image region and a candidate point cloud match in both spatial location and in terms of the type of intrusion target, that is, both representing the presence of the same type of intrusion target in a local region within a monitored area, then the candidate image region and the candidate point cloud are considered to have matched successfully.

[0044] In one embodiment, a successfully matched candidate image region and candidate point set can characterize a corresponding local area within the monitored area as having a potential intrusion target, meaning that the probability of an intrusion target existing in that local area is high. For example, an alarm can be triggered when a candidate image region successfully matches a candidate point set.

[0045] In one embodiment, at least one candidate image region can be matched with at least one candidate point cloud in sequence. For example, a candidate image region can be selected from at least one candidate image region, and a candidate point cloud can be selected from at least one candidate point cloud, and the selected candidate image region can be matched with the selected candidate point cloud. As another example, at least two candidate image regions can be selected from at least one candidate image region, and at least two candidate point clouds can be selected from at least one candidate point cloud, and for each selected candidate image region, the candidate image region can be matched with each selected candidate point cloud. Yet another example is that for each candidate image region in at least one candidate image region, the candidate image region can be matched with each candidate point cloud in at least one candidate point cloud.

[0046] In one embodiment, the matching result may include one or more sets of candidate image regions and candidate point clouds that have been successfully / unsuccessfully matched. For example, when all matching attempts between candidate image regions and candidate point clouds fail, the matching result may include one or more sets of candidate image regions and candidate point clouds that have failed to match. As another example, when all matching attempts between candidate image regions and candidate point clouds succeed, the matching result may include one or more sets of candidate image regions (i.e., target image regions) and candidate point clouds (i.e., target point clouds) that have been successfully matched. Yet another example, when both successfully matched candidate image regions and candidate point clouds and unsuccessfully matched candidate image regions and candidate point clouds exist, the matching result includes both the unsuccessfully matched one or more sets of candidate image regions and candidate point clouds and the successfully matched one or more sets of candidate image regions (i.e., target image regions) and candidate point clouds (i.e., target point clouds).

[0047] In one embodiment, for a successfully matched target image region and target point set, a three-dimensional axial bounding box of the target point set can be constructed, and the physical dimensions of the three-dimensional axial bounding box (such as at least one of length, width, height, and volume) can be calculated. If the physical dimensions of the three-dimensional axial bounding box do not meet a preset physical dimension requirement, the target image region and target point set are discarded. Furthermore, the preset physical dimension requirement can be determined based on actual conditions. For example, if the target image region represents a potential intrusion target with a humanoid or large animal appearance, the preset physical dimension requirement could be that the height of the three-dimensional axial bounding box is between 0.5 meters and 2.5 meters. As another example, regardless of the appearance type of the intrusion target represented by the target image region, the preset physical dimension requirement is that the volume of the three-dimensional axial bounding box is between 0.05 cubic meters and 10 cubic meters.

[0048] In one embodiment, the matching result may include successfully matched target image regions and target point clusters. For a successfully matched target image region and target point cluster, if the distance between a preset point in the target point cluster and a reference point in the monitoring area is greater than or equal to a preset distance, then the target image region and target point cluster are discarded. Further, the reference point in the monitoring area may be an equivalent point determined based on actual conditions to represent a key protected area within the monitoring area. Further still, the preset point may be determined based on actual conditions, representing the spatial location of the target point cluster. The distance between the preset point and the reference point in the monitoring area characterizes the distance between the intrusion targets contained in the target point cluster and the key protected area. For example, the preset point may be an equivalent center point determined based on the average coordinates of all points in the target point cluster, or the point in the target point cluster closest to the reference point, etc.

[0049] In one embodiment, the type of potential intrusion target can be determined by combining the appearance type information of the potential intrusion target provided by the target image region and the size information of the three-dimensional axial bounding box of the target point cluster. For example, if the appearance type of the potential intrusion target is humanoid and the height of the three-dimensional axial bounding box is within the range of human height, then the type of potential intrusion target is determined to be a person. As another example, if the appearance type of the potential intrusion target is a regular geometric shape and the length, width, and height of the three-dimensional axial bounding box are within the range of the length, width, and height of a vehicle, then the type of potential intrusion target is determined to be a vehicle.

[0050] 250: Execute alarm policies based on matching results.

[0051] In one embodiment, an alarm policy may include logical rules determined based on matching results for triggering alarm patterns of different levels and / or types.

[0052] In one embodiment, an alarm strategy can be executed based on the matching results using the following method: Level 1 Alarm: If the type of the potential intrusion target is a type requiring attention (such as personnel or vehicles), and the distance between the preset point of the target point cluster and the reference point of the monitoring area is within the first warning range (such as 30-50 meters), the system sends a text reminder message to the mobile terminal carried by the security personnel through the communication module; Level 2 Alarm: If the type of the potential intrusion target is a type requiring attention (such as personnel or vehicles), and the distance between the preset point of the target point cluster and the reference point of the monitoring area is within the second warning range (such as 10-30 meters), or the distance between the preset point of the target point cluster and the reference point of the monitoring area is within the first warning range but continues to shorten, triggering a Level 1 alarm of the audible and visual alarm in the monitoring area (such as a warning light flashing once per second and a buzzer sounding in a pattern of two seconds on, one second off), and sending a notification containing a screenshot of the target image area to the mobile terminal carried by the security personnel through the communication module; Level 3 ... the system sends a text reminder message to the mobile terminal carried by the security personnel through the communication module. If the target is a type of object requiring attention (e.g., personnel or vehicles), and the distance between the preset point in the target point cloud and the reference point in the monitoring area is within the third alert range (e.g., 0-10 meters), or the distance between the preset point in the target point cloud and the reference point in the monitoring area is within the second alert range, but the rate of decrease in distance between the preset point in the target point cloud and the reference point in the monitoring area is greater than or equal to a preset rate threshold (e.g., 0.5 m / s, 1 m / s, 2 m / s, etc.), a secondary alarm (e.g., warning lights flashing at a frequency of 10 times per second, and a continuous buzzing sound) is triggered on the audible and visual alarm device in the monitoring area. Furthermore, a notification containing a screenshot of the target image area, the type of potential intrusion target, and a trajectory prediction of the potential intrusion target (e.g., using methods such as Kalman filtering, particle filtering, or trajectory prediction models based on long short-term memory networks to analyze the positional change patterns of potential intrusion targets contained in the target point cloud in continuous multi-frame 3D point cloud data, thereby obtaining a trajectory prediction of the potential intrusion target) is sent to the mobile terminal carried by security personnel via the communication module.

[0053] Furthermore, in one embodiment, if the matching result shows that there is no successfully matched target image region and target point cluster, the alarm strategy may include not generating an alarm instruction, that is, not issuing an alarm.

[0054] This application provides a security monitoring method that collects image data and 3D point cloud data of a monitored area, performs target recognition on the image data to obtain at least one candidate image region, performs target recognition on the 3D point cloud data to obtain at least one candidate point cloud, matches the at least one candidate image region and the at least one candidate point cloud to obtain a matching result, and executes an alarm strategy based on the matching result. By employing sensing channels based on different physical principles (such as optical sensing and radar / laser sensing), a matching relationship is established between the visual detection result (candidate image region) of the same potential intrusion target in the image plane and its spatial detection result (candidate point cloud) in 3D space. Only when the visual detection result and the spatial detection result can be successfully matched is the potential intrusion target confirmed as a real intrusion target and an alarm is triggered, thereby effectively filtering false alarms caused by the limitations of a single channel and significantly reducing the false alarm rate.

[0055] According to one embodiment of this application, the matching result includes a successfully matched target image region and a target point cloud. Matching at least one candidate image region with at least one candidate point cloud to obtain a matching result includes: calculating a first similarity between each candidate image region in the at least one candidate image region and each candidate point cloud in the at least one candidate point cloud; determining a set of candidate image regions and candidate point clouds whose first similarity satisfies a preset first similarity requirement, thereby obtaining a successfully matched target image region and target point cloud.

[0056] In one embodiment, the first similarity may include a metric for quantifying the spatial consistency (i.e., contour overlap) between the candidate image region and the candidate point cloud. Further, the spatial consistency between the candidate image region and the candidate point cloud can be characterized by the intersection-union ratio (IU) and / or the center distance between them. For example, the first similarity may include a composite score calculated based on the IU and center distance between the candidate projected region and the candidate image region, or a metric calculated based on other reasonable geometric relationships between the candidate image region and the candidate point cloud.

[0057] In one embodiment, the preset first similarity requirement may include a pre-set quantization threshold based on actual conditions. This quantization threshold is used to determine whether the match between the candidate image region and the candidate point cloud is successful. For example, the preset first similarity requirement may be a fixed scalar threshold (such as 0.5, 0.6, 0.7, etc.). Alternatively, the preset first similarity requirement may be a threshold function that is dynamically adjusted according to actual conditions.

[0058] In one embodiment, for each candidate image region in at least one candidate image region, a candidate point cloud with the highest first similarity to the candidate image region can be selected, and it can be determined whether the similarity between the candidate image region and the candidate point cloud satisfies a preset first similarity. If the preset first similarity is satisfied, it is determined that the candidate image region and the candidate point cloud are successfully matched.

[0059] In one embodiment, candidate image regions that fail to match any candidate point set can be identified as visual virtual scenes (e.g., visual virtual scenes can characterize erroneous target recognition results caused by environmental interference factors such as changes in light and shadow, specular reflection, and swaying leaves), and such visual virtual scenes can be discarded.

[0060] In one embodiment, a cluster of candidate points that fails to match any candidate image region can be identified as a target to be observed (for example, a target to be observed can represent a potential intrusion target that failed to be confirmed by matching due to insufficient lighting, failure of optical imaging device to detect, etc.), and the target to be observed can be recorded in the monitoring log for further verification of the target to be observed.

[0061] In this embodiment, a first similarity is calculated between each candidate image region and each candidate point cloud. Based on a preset first similarity requirement, a set of candidate image regions and candidate point clouds that meet the conditions are selected to obtain the successfully matched target image region and target point cloud. The solution in this embodiment calculates the first similarity between candidate image regions and candidate point clouds, and uses candidate point clouds whose first similarity meets the preset requirement as successfully matched target image regions and target point clouds. This provides a high-confidence quantitative standard for determining the matching results, improves the accuracy of the matching results, and reduces the false alarm rate.

[0062] According to one embodiment of this application, calculating a first similarity between each candidate image region in at least one candidate image region and each candidate point cloud in at least one candidate point cloud includes: projecting at least one candidate point cloud onto the coordinate system where at least one candidate image region is located to obtain at least one candidate projection region, wherein at least one candidate projection region corresponds one-to-one with at least one candidate point cloud; calculating the intersection-union ratio and / or center distance between each candidate projection region in the candidate image region and at least one candidate projection region; and calculating the first similarity based on the intersection-union ratio and / or center distance.

[0063] In one embodiment, the candidate projection region may include a two-dimensional region determined based on the two-dimensional point set obtained by projecting the candidate point cloud onto the coordinate system of the candidate image region through coordinate transformation. For example, the candidate projection region may include: the minimum bounding rectangle of the two-dimensional point set, the convex hull of the two-dimensional point set, etc.

[0064] In one embodiment, at least one candidate point set can be projected onto the coordinate system of at least one candidate image region to obtain at least one candidate projection region based on the following method: Using a coordinate transformation algorithm (such as a checkerboard corner point calibration algorithm, a perspective n-point algorithm, etc.), calculate the transformation matrix between the coordinate system of at least one candidate image region and the coordinate system of at least one candidate point set. Using this transformation matrix and the intrinsic parameters of the image acquisition device, project at least one candidate point set onto the coordinate system of at least one candidate image region to obtain a two-dimensional point set in the coordinate system of at least one candidate image region, and determine at least one candidate projection region based on this two-dimensional point set. For example, the minimum bounding rectangle of the two-dimensional point set can be calculated, and the resulting rectangular region can be used as the candidate projection region. Another example is that the convex hull of the two-dimensional point set can be calculated, and the calculated convex hull can be used as the candidate projection region.

[0065] In one embodiment, the intersection-to-union ratio can include the ratio between the overlap area (intersection area) between the candidate image region and the candidate projection region and the union area between the candidate image region and the candidate projection region.

[0066] In one embodiment, the center distance may include the direct distance or normalized distance between the center point of the candidate image region and the center point of the candidate projection region. For example, the type of center distance may include: Euclidean distance, Manhattan distance, Chebyshev distance, or other suitable distance between the center point of the candidate image region and the center point of the candidate projection region. Further, the center distance may be a normalized distance obtained by normalizing the direct distance between the two center points; for example, the Euclidean distance, Manhattan distance, Chebyshev distance, or other suitable distance may be normalized to obtain the normalized distance.

[0067] In one embodiment, the intersection-union ratio (IoU) between the candidate image region and the candidate projection region can be calculated using the following formula: IoU ik = Area(B i ∩ P k ) / Area(B i ∪ P k ), where i represents the index number of the candidate image region, k represents the index number of the candidate projection region, and IoU ik B represents the intersection-union ratio between the i-th candidate image region and the k-th candidate projection region. i P represents the i-th candidate image region. k Area(B) represents the k-th candidate projection region. i ∩ P kArea(B) represents the overlap area between the i-th candidate image region and the k-th candidate projection region. i ∪ P k ) represents the area of ​​the union between the i-th candidate image region and the k-th candidate projection region.

[0068] In one embodiment, the first similarity between the candidate image region and the candidate projection region can be calculated using the following first similarity calculation formula: S ik = IoU ik × (1 - D ik ), where i represents the index number of the candidate image region, k represents the index number of the candidate projection region, and IoU ik D represents the intersection-union ratio between the i-th candidate image region and the k-th candidate projection region. ik The center distance between the i-th candidate image region and the k-th candidate projection region.

[0069] In this embodiment, the candidate point cloud is projected onto the coordinate system of the candidate image region to obtain the candidate projection region. The intersection-union ratio (IU) and / or center distance between the candidate projection region and the candidate image region are calculated. Based on the IU and / or center distance, the first similarity is calculated. This embodiment of the application effectively overcomes the obstacle that two-dimensional image data and three-dimensional point cloud data cannot be directly compared due to their different modes by projecting the candidate point cloud onto the coordinate system of the candidate image region and calculating geometric relationship metrics (such as IU and center distance) within a unified coordinate system. This provides the prerequisite for calculating the first similarity. In addition, the intersection-union ratio (IUGR), as a geometric metric for measuring the degree of overlap between regions, can intuitively reflect the degree of coincidence between the candidate image region and the candidate projection region on the image plane. The center distance can measure the degree of positional deviation between the candidate image region and the candidate projection region. Even if the size of the candidate image region and the candidate point set are not completely consistent, as long as the center points of the candidate image region and the candidate point set are close enough, they may still have high spatial consistency. By combining the IUGR and the center distance to calculate the first similarity, the matching degree between the candidate image region and the candidate point set can be comprehensively evaluated from two dimensions: the degree of overlap and the degree of positional deviation. This can improve the accuracy of the matching results and reduce the false alarm rate.

[0070] According to one embodiment of this application, calculating the intersection-union ratio and / or center distance between a candidate image region and each candidate projection region in at least one candidate projection region includes: determining a first center point of the candidate image region and determining a second center point of the candidate projection region; calculating the original distance between the first center point and the second center point; calculating the first diagonal length of the candidate image region and calculating the second diagonal length of the candidate projection region; and normalizing the original distance based on the first diagonal length and the second diagonal length to obtain the center distance.

[0071] In one embodiment, the first center point may include the geometric center point of the candidate image region. For example, if the candidate image region is rectangular, the center point of the candidate image region may be located at the intersection of the two diagonals of the rectangle. As another example, if the candidate image region is a polygon with more than four sides, the center point of the candidate image region may be the centroid of the polygon.

[0072] In one embodiment, the second center point may include the geometric center point of the candidate projection region. For example, if the candidate projection region is rectangular, the center point of the candidate image region may be located at the intersection of the two diagonals of the rectangle. As another example, if the candidate projection region is a polygon with more than four sides, the center point of the candidate projection region may be the centroid of the polygon.

[0073] In one example, the candidate image region itself can be a rectangle. For example, when performing target recognition on image data, a rectangular candidate image region can be obtained directly, or an irregularly shaped image region can be obtained first, and then the minimum bounding rectangle of the image region can be calculated to obtain a rectangular candidate image region.

[0074] In one example, after projecting the candidate point cloud onto the coordinate system of the candidate image region through coordinate transformation, a two-dimensional point set can be obtained. The candidate projection region of the rectangle can be obtained by calculating the minimum bounding rectangle or the convex hull of the two-dimensional point set.

[0075] In one embodiment, the original distance may include the Euclidean straight-line distance between the first center point and the second center point.

[0076] In one embodiment, the first diagonal length may include the diagonal length of the candidate image region.

[0077] In one embodiment, the second diagonal length may include the diagonal length of the candidate projection region.

[0078] In one embodiment, the center distance can be obtained by normalizing the original distance based on the lengths of the first and second diagonals using the following normalization formula: D ik = min(Dorig / (L i1 +L k2 ), 1.0), where i represents the index number of the candidate image region, k represents the index number of the candidate projection region, and D ik D represents the center distance between the first center point of the i-th candidate image region and the second center point of the k-th candidate projection region. orig L represents the original distance between the first center point of the i-th candidate image region and the second center point of the k-th candidate projection region. i1 L represents the length of the first diagonal of the i-th candidate image region. k2 This represents the length of the second diagonal of the k-th candidate projection region. When D... orig / (L i1 +L k2 When ) is less than 1, then D ik = D orig / (L i1 +L k2 ); when D orig / (L i1 +L k2 When D is greater than or equal to 1, ik =1.

[0079] In this embodiment, the first diagonal length and first center point of the candidate image region are determined, and the second diagonal length and second center point of the candidate projection region are determined. The first and second diagonal lengths are used as scale normalization factors to normalize the original distance between the first and second center points, thus obtaining the center distance. In this way, the center distance can represent the degree of deviation between the first and second center points relative to the physical size of the candidate image region and the candidate projection region, ensuring the scale invariance of the center distance, solving the interference of the "near is larger and far is smaller" problem caused by perspective projection, and improving the versatility of the first similarity calculation method.

[0080] According to one embodiment of this application, an alarm strategy is executed based on the matching result, including: acquiring an infrared thermal imaging image of the monitored area; verifying the accuracy of the matching result using the infrared thermal imaging image; and executing the alarm strategy if the matching result is verified to be accurate.

[0081] In one embodiment, the infrared thermal imaging image may include image data representing the distribution of thermal radiation within a monitored area, captured by an infrared sensor. Further, the type of infrared sensor may include an uncooled infrared focal plane array detector, a cooled infrared detector, a microbolometer, etc.

[0082] In one embodiment, infrared radiation data of the monitored area can be collected by an infrared sensor deployed in the monitored area. Further, the infrared sensor may include an optical system, an infrared detector, and signal processing circuitry. The optical system can focus the infrared radiation signal of the monitored area onto the infrared detector, which in turn can convert the infrared radiation signal into an electrical signal, and then amplify and convert the electrical signal into an infrared thermal image.

[0083] In one embodiment, before acquiring infrared thermal imaging images of the monitored area, the device for acquiring infrared thermal imaging images (such as infrared sensor 130) can be leveled.

[0084] In one embodiment, acquiring infrared thermal images of the monitored area and using these images to verify the accuracy of the matching results can be performed when the illuminance of the monitored area is less than or equal to a preset illuminance threshold. Furthermore, the preset illuminance threshold can be determined based on actual conditions. For example, the preset illuminance threshold could be 45 lux, 50 lux, 55 lux, etc.

[0085] In this embodiment, infrared thermal imaging images are used to verify the accuracy of the matching results. If the matching results are verified to be accurate, an alarm strategy is executed. The solution of this application embodiment, by introducing infrared thermal imaging images for auxiliary verification on the basis of dual-channel matching verification, forms a multi-level verification mechanism, which can effectively identify and filter erroneous matches that may be caused by special interference (such as sensor failure, complex camouflage of intrusion targets, etc.), thereby reducing the false alarm rate.

[0086] According to one embodiment of this application, the matching result includes a successfully matched target image region and a target point cloud. The accuracy of the matching result is verified using an infrared thermal imaging image, including: identifying a thermal contour region in the infrared thermal imaging image; calculating a second similarity between the thermal contour region and the target image region and / or the target point cloud; and verifying the accuracy of the matching result based on the second similarity.

[0087] In one embodiment, the thermal profile region may include a thermal signal region in an infrared thermal imaging image that is identified by an image processing algorithm (such as a temperature threshold segmentation algorithm, an edge detection extraction algorithm, etc.) and characterizes a potential intrusion target.

[0088] In one embodiment, the second similarity may include measures of spatial consistency between the thermal contour region and the target image region and / or the target point cloud. For example, the second similarity may include a combined score calculated based on the intersection-union ratio and normalized center distance between the thermal contour region and the target image region and / or the target point cloud, or a metric calculated based on other reasonable geometric relationships between the thermal contour region and the target image region and / or the target point cloud.

[0089] In one embodiment, the second similarity between the thermal contour region and the target image region and / or the target point cloud can be calculated using the following method: projecting the thermal contour region onto the coordinate system of the target image region and / or the target point cloud to obtain the thermal contour projection region; calculating the intersection-union ratio and / or center distance between the thermal contour projection region and the target image region and / or the target point cloud; and calculating the second similarity based on the intersection-union ratio and / or center distance.

[0090] In one embodiment, the accuracy of the matching result can be verified based on a second similarity score: if the second similarity score meets a preset second similarity requirement, the matching result is verified as accurate; if the second similarity score does not meet the preset second similarity requirement, the matching result is verified as inaccurate. Further, the second similarity requirement may include a second similarity score greater than or equal to a preset second similarity threshold. Even further, the second similarity threshold can be determined based on actual conditions, such as 0.7, 0.8, 0.9, etc.

[0091] For example, the matching result may include a successful match. For a successfully matched target image region and target point cluster, if a thermal contour region exists in the infrared thermal imaging image, a second similarity can be calculated between the thermal contour region and the target image region or target point cluster. If the second similarity meets a preset second similarity requirement, the matching result is accurate, indicating the presence of an intrusion target in the monitored area, and an alarm strategy can be executed. If the second similarity does not meet the preset second similarity requirement, the matching result is incorrect, indicating the absence of an intrusion target in the monitored area, or a prompt message can be issued to notify technicians to check the parameters of sensors and other equipment (and / or to investigate the monitored area). For a successfully matched target image region and target point cluster, if a thermal contour region does not exist in the infrared thermal imaging image, an alarm strategy may not be executed, and a prompt message may be issued to notify technicians to check the parameters of sensors and other equipment (and / or to investigate the monitored area).

[0092] For example, the matching result may include a matching failure. If a thermal outline area exists in the infrared thermal imaging image at this time, it indicates that there may be an intrusion target in the monitored area. At this time, a prompt message can be issued to notify technicians to check the parameters of sensors and other equipment and investigate the monitored area.

[0093] In one example, the similarity between the thermal contour region and the target image region can be calculated, and this similarity can be used as a second similarity. Alternatively, the similarity between the thermal contour region and the target point set can be calculated, and this similarity can be used as a second similarity. Or, the similarity between the thermal contour region and the target image region can be calculated, and the similarity between the thermal contour region and the target point set can be calculated. A weighted average of these two similarities can be used to obtain a second similarity, for example, the mean of these two similarities can be used as the second similarity.

[0094] In this embodiment, a thermal contour region is identified in the infrared thermal imaging image, and a second similarity is calculated between the thermal contour region and the target image region and / or the target point cloud. Based on the second similarity, the accuracy of the matching result is verified. The solution of this application embodiment identifies a thermal contour region representing a heat source in the infrared thermal imaging data and calculates the similarity between the thermal contour region and the target image region and / or the target point cloud (such as calculating a second similarity), thereby providing a high-confidence quantitative standard for verifying the accuracy of the matching result, thus improving the accuracy of the verification and reducing the false alarm rate.

[0095] According to one embodiment of this application, target identification is performed on three-dimensional point cloud data to obtain at least one candidate point cloud set, including: acquiring background point cloud data of a monitoring area; determining target point clouds located outside the background point cloud data in the three-dimensional point cloud data; and clustering the target point clouds to obtain at least one candidate point cloud set.

[0096] In one embodiment, background point cloud data may include a set of three-dimensional spatial coordinates of various points on the surface of static environmental objects in the monitored area. For example, background point cloud data can be collected using a spatial ranging device (such as a mechanical rotating lidar, a solid-state area array lidar, etc.). Exemplarily, when it is determined manually that the monitored area is safe or that there is no intruding target, point cloud data of the monitored area can be collected using a spatial ranging device to obtain background point cloud data.

[0097] In one embodiment, data can be collected in real time from the monitored area to obtain 3D point cloud data. For example, data can be collected from the monitored area at a certain sampling frequency. Each collection yields 3D point cloud data for that specific collection.

[0098] In one embodiment, the 3D point cloud data may include background point cloud data. If an intrusion target and / or a newly added non-intrusion target, such as a static object, exist in the monitored area, the 3D point cloud data may further include point cloud data corresponding to the intrusion target and / or point cloud data corresponding to the newly added non-intrusion target, i.e., target point cloud. For example, the target point cloud may be a subset of the 3D point cloud data, and the target point cloud may not exist in the background point cloud data.

[0099] In one embodiment, by identifying target point clouds located outside the background point cloud data within the 3D point cloud data, it can be determined whether changes have occurred in the monitored area. For example, the presence of target point clouds indicates a change in the monitored area, which may be caused by the movement of existing objects in the monitored area (such as swaying leaves), the appearance of an intruding target, or the addition of a static object. Furthermore, the target point clouds can be clustered or analyzed to determine whether potential intruding targets exist in the monitored area.

[0100] In one embodiment, clustering may include a process of using a clustering algorithm to group spatially adjacent point clouds to obtain at least one set of point clouds (i.e., candidate point clouds) that are mutually separated and represent different potential intrusion targets. For example, clustering algorithms may include Euclidean clustering, density-based clustering, or hierarchical clustering, etc.

[0101] In one embodiment, both background point cloud data and 3D point cloud data can be single-frame point cloud data. Background point cloud data can be acquired at a preset first time point (e.g., the 30th, 40th, or 50th second) after the device for acquiring background point cloud data (e.g., spatial ranging device 120) is activated. 3D point cloud data can be acquired at a preset second time point (e.g., the 60th, 70th, or 80th second, where the second time point must be after the first time point) after the device for acquiring 3D point cloud data (e.g., spatial ranging device 120) is activated. The target point cloud located outside the background point cloud data within the 3D point cloud data can then be determined. For example, the target point cloud can be determined by comparing the single-frame background point cloud data and the single-frame 3D point cloud data point by point. For example, the background point cloud data of a single frame can be voxelized to obtain a voxel model of the background point cloud of a single frame, and the 3D point cloud data of a single frame can be voxelized to obtain a voxel model of the 3D point cloud of a single frame. The voxel model of the background point cloud of a single frame and the voxel model of the 3D point cloud of a single frame are compared voxel by voxel to determine the first target voxel that is empty in the background point cloud voxel model but is occupied in the background point cloud voxel model. The point cloud included in the first target voxel is the target point cloud.

[0102] In one embodiment, background point cloud data may include sub-background point cloud data of multiple frames, and 3D point cloud data may include sub-3D point cloud data of multiple frames. After the device for acquiring background point cloud data (such as the spatial ranging device 120) is activated, continuous background point cloud data acquisition is performed on the monitoring area within a preset first time period (e.g., 0-30 seconds, 30-60 seconds, etc.) to obtain sub-background point cloud data of multiple frames. The sub-background point cloud data of multiple frames are merged to obtain an initial background point cloud set (background point cloud data), and the initial background point cloud set is voxelized to obtain an initial background voxel model. After the device for acquiring 3D point cloud data (such as the spatial ranging device 120) is activated, continuous 3D point cloud data acquisition is performed on the monitoring area within a preset second time period (e.g., 60-100 seconds, 90-120 seconds, etc., the second time period must be after the first time period), to obtain sub-3D point cloud data of multiple frames. The sub-3D point cloud data of each frame in the sub-3D point cloud data of multiple frames is voxelized to obtain sub-3D point cloud voxel models of multiple frames. Furthermore, the initial background voxel model can be used as the sub-background voxel model of the first frame (corresponding to the background point cloud data within the first time period), and the sub-background voxel model can be updated based on the adaptive update formula (the adaptive update formula is BGM). t+1 = α BGM t + CVM t -α CVM t t represents the frame index number, CVM t Describes the sub-3D point cloud voxel model of frame t, BGM t Denotes the sub-background voxel model of frame t, BGM t+1 (Represents the sub-background voxel model of frame t+1), obtain the sub-background voxel model corresponding to each frame in the second time period (corresponding to the background point cloud data in the second time period), and compare the sub-background voxel model and the sub-3D point cloud voxel model corresponding to each frame in the second time period voxel by voxel to determine the second target voxel that is empty in the sub-background voxel model but occupied in the sub-3D point cloud voxel model. The point cloud included in the second target voxel is the target point cloud.

[0103] In one embodiment, before collecting background point cloud data of the monitored area, the device for collecting background point cloud data (such as the spatial ranging device 120) can be leveled.

[0104] In this embodiment, background point cloud data of the monitored area is acquired, target point clouds located outside the background point cloud data are identified in the 3D point cloud data, and the target point clouds are clustered to obtain at least one candidate point cloud set. The monitored scene contains static environmental objects such as the ground, buildings, and fixed facilities. These static environmental objects are non-threatening background information for security monitoring. The solution in this embodiment compares the 3D point cloud data with the background point cloud data, excluding point clouds corresponding to static environmental objects from the target point cloud, thereby effectively filtering out environmental background interference, improving the accuracy of target identification from the 3D point cloud data, and reducing the false alarm rate. Furthermore, the solution in this embodiment clusters the target point cloud to obtain at least one candidate point cloud set, integrating discrete points into a point cloud set with a complete geometric contour. This provides a spatially complete 3D target representation for subsequent matching, thereby improving the accuracy of the matching results and reducing the false alarm rate.

[0105] According to one embodiment of this application, devices for acquiring image data (such as optical imaging device 110), devices for acquiring three-dimensional point cloud data and background point cloud data (such as spatial ranging device 120), and devices for acquiring infrared thermal imaging images (such as infrared sensor 130) can be integrated into the same mobile device.

[0106] In one embodiment, the type of mobile device may include drones, vehicle-mounted mobile monitoring stations, portable mobile stands, etc.

[0107] In this embodiment, the device for acquiring image data, the device for acquiring 3D point cloud data and background point cloud data, and the device for acquiring infrared thermal imaging images are integrated into the same mobile device, realizing rapid deployment that is "ready to use as soon as it is deployed". This eliminates the cumbersome wiring and multi-device debugging process, and is suitable for security monitoring scenarios that require rapid response, such as emergency security, field operations, and temporary perimeters, thereby improving the scenario applicability of the security monitoring method.

[0108] According to one embodiment of this application, the image data is a panoramic image, and the three-dimensional point cloud data is panoramic point cloud data.

[0109] In this embodiment, the image data is a panoramic image, and the three-dimensional point cloud data is a panoramic point cloud data, forming a perception field covering the entire monitoring area. This eliminates blind spots in the monitoring from a physical perspective and improves the rigor of security monitoring.

[0110] Figure 3 The diagram shown is a flowchart of a security monitoring method provided in another exemplary embodiment of this application. Figure 3 The example is Figure 2 Examples of the embodiments are provided below. To avoid repetition, similarities can be found in the descriptions of the embodiments, and will not be repeated here. Figure 3As shown, this security monitoring method may include the following:

[0111] 310: Collect image data, 3D point cloud data, background point cloud data, and infrared thermal imaging images of the monitored area.

[0112] Specifically, the relevant content regarding the monitoring area, image data, 3D point cloud data, background point cloud data, and infrared thermal imaging images can be found in the relevant descriptions in the embodiments. To avoid repetition, they will not be repeated here.

[0113] 320: Perform target recognition on image data to obtain at least one candidate image region, and determine the target point cloud located outside the background point cloud data in the 3D point cloud data, and cluster the target point cloud to obtain at least one candidate point cloud set.

[0114] Specifically, the relevant content of target recognition, candidate image region, target point cloud, clustering, and candidate point cloud set can be found in the relevant descriptions in the embodiments. To avoid repetition, it will not be repeated here.

[0115] 330: Match at least one candidate image region with at least one candidate point cloud to obtain a matching result, which includes the successfully matched target image region and the target point cloud.

[0116] Specifically, the relevant content regarding matching, matching results, target image regions, and target point clusters can be found in the relevant descriptions in the embodiments, and will not be repeated here to avoid repetition.

[0117] 340: Verify the accuracy of the matching result using infrared thermal imaging. If the matching result is verified to be accurate, proceed to step 350. If the matching result is verified to be inaccurate, proceed to step 360.

[0118] Specifically, the relevant content on verifying the accuracy of the matching results using infrared thermal imaging images can be found in the relevant descriptions in the embodiments, and will not be repeated here to avoid repetition.

[0119] 350: Execute alarm policies based on matching results.

[0120] Specifically, the relevant details of the alarm strategy can be found in the descriptions in the embodiments, and will not be repeated here to avoid repetition.

[0121] 360: Discard the matching results.

[0122] Exemplary device Figure 4 The diagram shown is a structural schematic of a security monitoring device provided in an exemplary embodiment of this application. This security monitoring device can be applied to electronic devices. Figure 4As shown, the security monitoring device 400 includes: a data acquisition module 410, a first target recognition module 420, a second target recognition module 430, a matching module 440, and an alarm module 450.

[0123] The acquisition module 410 is used to acquire image data and 3D point cloud data of the monitored area. The first target recognition module 420 is used to perform target recognition on the image data to obtain at least one candidate image region. The second target recognition module 430 is used to perform target recognition on the 3D point cloud data to obtain at least one candidate point cloud set. The matching module 440 is used to match the at least one candidate image region with the at least one candidate point cloud set to obtain a matching result. The alarm module 450 is used to execute an alarm strategy based on the matching result.

[0124] This application provides a security monitoring device that collects image data and 3D point cloud data of a monitored area, performs target recognition on the image data to obtain at least one candidate image region, performs target recognition on the 3D point cloud data to obtain at least one candidate point cloud, matches the at least one candidate image region and the at least one candidate point cloud to obtain a matching result, and executes an alarm strategy based on the matching result. By employing sensing channels based on different physical principles (such as optical sensing and radar / laser sensing), a matching relationship is established between the visual detection result (candidate image region) of the same potential intrusion target in the image plane and its spatial detection result (candidate point cloud) in 3D space. Only when the visual detection result and the spatial detection result can be successfully matched is the potential intrusion target confirmed as a real intrusion target and an alarm is triggered, thereby effectively filtering false alarms caused by the limitations of a single channel and significantly reducing the false alarm rate.

[0125] According to an embodiment of this application, the matching result includes a successfully matched target image region and a target point cloud. The matching module 440 is used to: calculate a first similarity between each candidate image region in at least one candidate image region and each candidate point cloud in at least one candidate point cloud; determine a set of candidate image regions and candidate point clouds whose first similarity satisfies a preset first similarity requirement, and obtain a successfully matched target image region and target point cloud.

[0126] According to one embodiment of this application, the matching module 440 is configured to: project at least one candidate point cloud onto the coordinate system of at least one candidate image region to obtain at least one candidate projection region, wherein the at least one candidate projection region corresponds one-to-one with at least one candidate point cloud; calculate the intersection-union ratio and / or center distance between each candidate projection region in the candidate image region and the at least one candidate projection region; and calculate a first similarity based on the intersection-union ratio and / or center distance.

[0127] According to one embodiment of this application, the matching module 440 is used to: determine a first center point of a candidate image region and a second center point of a candidate projection region; calculate the original distance between the first center point and the second center point; calculate the length of the first diagonal of the candidate image region and the length of the second diagonal of the candidate projection region; and normalize the original distance based on the length of the first diagonal and the length of the second diagonal to obtain the center distance.

[0128] According to one embodiment of this application, the alarm module 450 is used to: acquire infrared thermal imaging images of the monitored area; verify the accuracy of the matching results using the infrared thermal imaging images; and execute an alarm strategy if the matching results are verified to be accurate.

[0129] According to one embodiment of this application, the matching result includes a successfully matched target image region and a target point cloud. The alarm module 450 is used to: identify a thermal contour region in an infrared thermal imaging image; calculate a second similarity between the thermal contour region and the target image region and / or the target point cloud; and verify the accuracy of the matching result based on the second similarity.

[0130] According to one embodiment of this application, the second target recognition module 430 is used to: acquire background point cloud data of the monitoring area; determine the target point cloud located outside the background point cloud data in the three-dimensional point cloud data; and cluster the target point cloud to obtain at least one candidate point cloud set.

[0131] It should be understood that the operation and function of the acquisition module 410, the first target recognition module 420, the second target recognition module 430, the matching module 440, and the alarm module 450 in the above embodiments can be referred to the above. Figure 2 or Figure 3 The description of the security monitoring method provided in the embodiments will not be repeated here to avoid repetition.

[0132] Figure 5 The diagram shown is a block diagram of an electronic device 500 for performing a security monitoring method according to an exemplary embodiment of this application. Specifically, the electronic device 500 may be a server, a mobile device, a control device for a mobile device, a server interacting with a mobile device, a controller, or other devices.

[0133] Reference Figure 5 The electronic device 500 includes a processing component 510, which further includes one or more processors, and memory resources represented by memory 520 for storing instructions, such as application programs, that can be executed by the processing component 510. The application programs stored in memory 520 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 510 is configured to execute instructions to perform the aforementioned security monitoring method.

[0134] Electronic device 500 may also include a power supply component configured to perform power management of electronic device 500, a wired or wireless network interface configured to connect electronic device 500 to a network, and an input / output (I / O) interface. Electronic device 500 can be operated based on an operating system stored in memory 520, such as Windows Server. TM Mac OSX TM Unix TM Linux TM FreeBSD TM Or similar.

[0135] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by the processor of the aforementioned electronic device 500, the electronic device 500 is enabled to perform a security monitoring method.

[0136] A computer program product includes a computer program that, when executed by a processor of a computer device, enables the computer device to perform the security monitoring method provided in any of the above embodiments.

[0137] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0143] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0146] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A security monitoring method, characterized in that, The method includes: Collect image data and 3D point cloud data of the monitored area; Target recognition is performed on the image data to obtain at least one candidate image region; Target identification is performed on the three-dimensional point cloud data to obtain at least one candidate point cloud set; The at least one candidate image region is matched with the at least one candidate point cloud to obtain a matching result; Based on the matching results, an alarm policy is executed.

2. The method according to claim 1, characterized in that, The matching result includes successfully matched target image regions and target point clouds. The process of matching the at least one candidate image region with the at least one candidate point cloud to obtain the matching result includes: Calculate the first similarity between each candidate image region in the at least one candidate image region and each candidate point cloud in the at least one candidate point cloud; A set of candidate image regions and candidate point clouds that meet the preset first similarity requirements are identified, and the successfully matched target image regions and target point clouds are obtained.

3. The method according to claim 2, characterized in that, The calculation of the first similarity between each candidate image region in the at least one candidate image region and each candidate point cloud in the at least one candidate point cloud set includes: Project the at least one candidate point cloud onto the coordinate system where the at least one candidate image region is located to obtain at least one candidate projection region, and the at least one candidate projection region corresponds one-to-one with the at least one candidate point cloud. Calculate the intersection-union ratio and / or center distance between the candidate image region and each candidate projection region in the at least one candidate projection region; The first similarity is calculated based on the intersection-union ratio and / or the center distance.

4. The method according to claim 3, characterized in that, The calculation of the intersection-union ratio and / or center distance between the candidate image region and each candidate projection region in the at least one candidate projection region includes: Determine a first center point of the candidate image region and a second center point of the candidate projection region; Calculate the original distance between the first center point and the second center point; Calculate the length of the first diagonal of the candidate image region and the length of the second diagonal of the candidate projection region; The original distance is normalized based on the lengths of the first and second diagonals to obtain the center distance.

5. The method according to claim 1, characterized in that, The step of executing an alarm strategy based on the matching result includes: Acquire infrared thermal imaging images of the monitored area; The accuracy of the matching results is verified using the infrared thermal imaging image. If the matching result is verified to be accurate, the alarm policy is executed.

6. The method according to claim 5, characterized in that, The matching result includes successfully matched target image regions and target point clusters. Verifying the accuracy of the matching result using the infrared thermal imaging image includes: Identify thermal contour regions in the infrared thermal imaging image; Calculate the second similarity between the thermal contour region and the target image region and / or the target point cloud; The accuracy of the matching results is verified based on the second similarity score.

7. The method according to any one of claims 1 to 6, characterized in that, The step of performing target identification on the three-dimensional point cloud data to obtain at least one candidate point cloud set includes: Obtain the background point cloud data of the monitored area; Determine the target point cloud in the three-dimensional point cloud data that is located outside the background point cloud data; Cluster the target point cloud to obtain at least one candidate point cloud set.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the security monitoring method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is used to execute the security monitoring method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by the processor of a computer device, enables the computer device to perform the security monitoring method according to any one of claims 1 to 7.

Images