Machine vision microwave vibration fusion intrusion detection system, method and device

Abstract

This application relates to the field of smart security technology, specifically providing a system, method, and device for machine vision microwave vibration fusion intrusion detection. The system may include: multiple detection devices, a detection host, and a post-processing terminal. Each detection device is used to collect monitoring data of at least the detection area. The detection host is used to acquire point cloud data and detection data of the detection area. The post-processing terminal is used to detect the data to be detected obtained from the screening of the detection data and point cloud data to determine the intrusion detection result. The intrusion detection result includes whether an intrusion behavior and intrusion target exist, and if an intrusion target exists, the type of intrusion target and the intrusion action. The embodiments of this application can improve data processing efficiency, thereby ensuring the real-time performance of intrusion detection.

Description

Technical Field

[0001] This application relates to the field of smart security technology, and more specifically, to a system, method, and device for machine vision microwave vibration fusion intrusion detection. Background Technology

[0002] Perimeter security systems are based on physical security measures, typically steel mesh fences. Intrusion detection devices such as vibrating fiber optic cables, vibrating cables, and vibration detectors are deployed on the fence to collect vibration signals and determine the presence of intrusion; this type of intrusion detection is passive. Intrusion detection devices such as microwave radar and electromagnetic induction cables are deployed on or inside the fence to detect intrusion through changes in electromagnetic fields; this type of intrusion detection is active.

[0003] Currently, after multiple intrusion detection devices in existing technologies collect data, the further processing and analysis of the data are all undertaken by backend servers. This results in a large data processing volume, low processing efficiency, and large latency for backend servers, which cannot meet the real-time requirements of security scenarios. Summary of the Invention

[0004] The purpose of some embodiments of this application is to provide a system, method and device for machine vision microwave vibration fusion intrusion detection. The technical solutions of the embodiments of this application can realize dynamic collaborative processing of multimodal data, reduce the data processing pressure of the post-processing end and ensure the real-time performance of intrusion detection in security scenarios.

[0005] In a first aspect, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection system, comprising: multiple detection devices, a detection host, and a post-processing terminal; each of the multiple detection devices is used to collect monitoring data of at least a detection area; wherein the monitoring data includes raw vibration data, raw spectrum data, and image detection data; the detection host is used to acquire point cloud data and detection data of the detection area; wherein the detection data includes vibration detection data, Doppler spectrum data, or image detection data; the detection data is acquired based on the monitoring data; the post-processing terminal is used to detect the data to be detected obtained by filtering the detection data and the point cloud data, and determine the intrusion detection result; wherein the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

[0006] The intrusion detection system in some embodiments of this application includes multiple detection devices, a detection host, and a post-processing terminal. The detection host acquires detection data and point cloud data obtained from monitoring data and sends them to the post-processing terminal. The post-processing terminal determines the intrusion detection result based on the detection data and point cloud data. In this application, by adding a detection host to undertake some of the data processing work of the post-processing terminal in the prior art, data collaborative processing between the detection host and the post-processing terminal is achieved. This reduces the data transmission pressure and the data processing pressure of the post-processing terminal, improves intrusion detection efficiency, and meets the real-time requirements of intrusion detection in security scenarios.

[0007] In some embodiments, the detection host is configured to: receive the monitoring data sent by each detection device; perform data stitching operations on the original vibration data and the original spectrum data respectively to obtain vibration stitched data and the Doppler spectrum data; and perform feature extraction operations on the vibration stitched data to obtain the vibration detection data.

[0008] In some embodiments of this application, the detection host obtains detection data by performing data splicing and feature extraction operations on the received monitoring data, providing accurate data support for subsequent intrusion detection and analysis.

[0009] In some embodiments, each detection device is configured to: perform a data stitching operation on the original vibration data and the original spectrum data respectively to obtain vibration stitched data and the Doppler spectrum data; perform a feature extraction operation on the vibration stitched data to obtain the vibration detection data; and send the detection data to the detection host.

[0010] In some embodiments of this application, the detection device can process monitoring data to obtain detection data and send it to the detection host, thereby reducing the data processing pressure on the detection host and improving the flexibility of data processing.

[0011] In some embodiments, when the detection data includes the Doppler spectrum data and the image detection data, the post-processing end is used to: acquire vibration stitched data after performing a data stitching operation on the original vibration data; and perform feature extraction operation on the vibration stitched data to obtain the vibration detection data.

[0012] In some embodiments of this application, the post-processing end can perform data feature extraction operations, thereby improving the flexibility of data processing.

[0013] In some embodiments, the method of performing feature extraction on the vibration splicing data to obtain the vibration detection data includes: performing feature extraction on the vibration splicing data from multiple dimensions corresponding to the time domain and frequency domain respectively, to obtain the vibration detection data including feature values ​​of multiple dimensions; wherein, the multiple dimensions in the time domain include maximum and minimum values, mean, standard deviation, variance, kurtosis, skewness, root mean square value or kurtosis factor; the multiple dimensions in the frequency domain include spectrum, dominant frequency, secondary frequency or frequency band energy.

[0014] Some embodiments of this application can achieve multi-feature extraction of vibration splicing data by performing feature extraction operations from multiple dimensions in the time and frequency domains, providing comprehensive data support for subsequent data analysis.

[0015] In some embodiments, each detection device is used to: quantize the initially acquired vibration detection signal and spectrum signal to obtain the original vibration data and the original spectrum data; wherein the quantization process includes signal scaling, signal filtering and signal encoding.

[0016] Some embodiments of this application achieve data standardization by quantifying the initially collected data.

[0017] In some embodiments, the detection host is used to: identify an intrusion target within the detection area by detecting the point cloud data; send the intrusion target to the post-processing end; and the post-processing end is used to determine the intrusion detection result based on the data to be detected and the intrusion target.

[0018] In some embodiments of this application, the detection host detects intrusion targets by analyzing point cloud data and sends them to the post-processing end, so that the post-processing end can combine the data to be detected to determine the intrusion detection results, thereby improving the accuracy of the intrusion detection results.

[0019] In some embodiments, the detection host is used to: determine whether an intrusion target exists by detecting the point cloud data and / or the image detection data; generate an emergency warning signal if an intrusion target is confirmed to exist; send the emergency warning signal to the target detection device corresponding to the image detection data; the target detection device is used to output warning information; wherein the target detection device is one or more of the plurality of detection devices.

[0020] The detection host in some embodiments of this application can detect point cloud data and image detection data, and generate an emergency warning signal when an intrusion target is present so that the target detection device can output warning information, thereby reducing the data processing pressure of the post-processing end while ensuring the real-time detection of intrusion.

[0021] In some embodiments, the post-processing end is used to: align the vibration detection data, the Doppler spectrum data, the image detection data, and the point cloud data to obtain multimodal data; use a dynamic gated fusion network to score various modal data in the multimodal data to obtain modal weights; and filter the detection data and the point cloud data using the modal weights to determine the data to be detected; wherein the data to be detected includes some or all of the data in the detection data and the point cloud data.

[0022] Some embodiments of this application use modal weights obtained by scoring various modal data in multimodal data to filter detection data and point cloud data to obtain data to be detected, thereby selecting usable high-value data to be detected and improving the accuracy of intrusion detection results.

[0023] In some embodiments, the post-processing end is used to: obtain the environmental feature vector of the detection area; input the environmental feature vector and the multimodal data into the dynamic gating fusion network, and output the modal weights.

[0024] Some embodiments of this application determine the modality weights by comprehensively considering environmental feature vectors and multimodal data, which can achieve accurate screening of valuable data.

[0025] In some embodiments, the post-processing end is used to: input the vibration detection data and the Doppler spectrum data into a first detection model to obtain a first detection result; the first detection result characterizes whether an intrusion behavior exists; input the image detection data and the point cloud data into a second detection model to obtain a second detection result; the second detection result includes whether an intrusion object exists, and when the intrusion object exists, the intrusion object category and the intrusion action of the intrusion object; and generate the intrusion detection result based on the first detection result and the second detection result.

[0026] In some embodiments of this application, the post-processing end analyzes different data using a first detection model and a second detection model to determine whether intrusion behavior and intrusion targets exist, so as to efficiently generate more accurate intrusion detection results.

[0027] In some embodiments, the post-processing end is used to: monitor the backend load in real time; when it is confirmed that the backend load exceeds the backend load threshold, switch the first detection model and the second detection model to a backup model; wherein the computational overhead of the backup model is lower than that of the first detection model and the second detection model; when the backend load recovers to a state that does not exceed the backend load threshold, switch the backup model to the first detection model and the second detection model.

[0028] In some embodiments of this application, the post-processor compares the back-end load with the back-end load threshold to determine whether to switch models, thereby dynamically optimizing the computational overhead of the post-processor and ensuring the real-time performance of intrusion detection.

[0029] In some embodiments, there are multiple detection areas, and each detection area corresponds to a set of data to be detected; the post-processing end is used to: detect the corresponding data to be detected according to the priority order of the detection areas, and obtain the intrusion detection result of each detection area in the multiple detection areas.

[0030] Some embodiments of this application can achieve hierarchical detection of detection areas by determining the intrusion detection results of each detection area in order of priority.

[0031] In some embodiments, the post-processing end is used to: generate an alarm control signal if the intrusion behavior is confirmed in the intrusion detection result; and / or generate an early warning control signal if the intrusion target is confirmed in the intrusion detection result; and send the alarm control signal and / or the early warning control signal to the detection device where the intrusion behavior and / or the intrusion target exists, so that the detection device can generate and output alarm signals and / or early warning signals.

[0032] Some embodiments of this application generate alarm control signals and / or warning control signals and send them to the corresponding detection devices so that the detection devices can issue timely alarms or warnings, thereby improving the security effect.

[0033] In some embodiments, the post-processing end is used to: obtain the number of the detection device where the intrusion behavior and / or the intrusion target exists; render the corresponding detection device on an electronic map according to the number to display the intrusion detection result; wherein the electronic map displays the location and number of each detection device among multiple detection devices in the detection area.

[0034] Some embodiments of this application visualize intrusion detection results by combining them with electronic maps to display the location of intrusion detection results.

[0035] In some embodiments, the post-processing end is used to: display video data collected by any detection device in response to a user's operation command on the electronic map for any detection device, or display video image data collected by associated cameras around the any detection device.

[0036] Some embodiments of this application can display corresponding video data by operating on an electronic map, thereby achieving a visual and intuitive display of the video data.

[0037] In some embodiments, the detection host is configured to: send a heartbeat to the post-processing terminal at a preset period; if a response information corresponding to the heartbeat is received from the post-processing terminal within a preset time period, then send the point cloud data and the detection data to the post-processing terminal; if the response information is not received within the preset time period, then detect the point cloud data and the detection data to determine the intrusion detection result.

[0038] In some embodiments of this application, the detection host employs a heartbeat mechanism to monitor the operation of the post-processing end, ensuring that intrusion detection results can be obtained normally and in real time.

[0039] In some embodiments, the detection host is used to: monitor the host load in real time and simultaneously acquire the backend load of the post-processing end; when the host load exceeds the host load threshold, send the spliced ​​vibration data after stitching the original vibration data to the post-processing end so that the post-processing end can extract features from the spliced ​​vibration data to obtain the vibration detection data; when the backend load exceeds the backend load threshold, detect the detection data and the data to be detected obtained by filtering the point cloud data to determine the intrusion detection result.

[0040] In some embodiments of this application, the detection host can dynamically adjust the relevant data processing operations by monitoring the host load and backend load, thereby improving data processing efficiency and meeting the real-time requirements of security scenarios.

[0041] In some embodiments, the detection host is used to: acquire idle hosts among a plurality of detection hosts within a preset range whose load data for each detection host does not exceed the host load threshold; and, if the host load exceeds the host load threshold, send detection tasks to the idle hosts according to task priority; wherein, the task priority of the feature extraction task in the detection task is lower than that of the intrusion detection task.

[0042] Some embodiments of this application obtain idle hosts within a preset range so that when the host load exceeds the host load threshold, the idle hosts can perform the probing task, thereby achieving load balancing among hosts and ensuring data processing efficiency.

[0043] In some embodiments, a host detection model is deployed on the detection host; a backend detection model is deployed on the post-processing end; both the host detection model and the backend detection model are used to detect the data to be detected and output the intrusion detection result; the host detection model is trained using a target training set; the target training set includes intrusion sample data and detection result data; the detection result data is obtained by inputting the intrusion sample data into the backend detection model.

[0044] Some embodiments of this application deploy host detection models and backend detection models on the detection host and backend respectively, enabling intrusion detection at both ends; even if one end fails, intrusion detection can still proceed normally, improving the reliability of the entire system.

[0045] Secondly, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection system, applied in perimeter security scenarios. The system includes multiple detection devices, a detection host, and a post-processing terminal. Each of the multiple detection devices is used to quantize initial data collected from the detection area to obtain monitoring data. The monitoring data includes raw vibration data, raw spectrum data, and image detection data. The detection host is used to perform data stitching and feature extraction on the received monitoring data to obtain detection data, and to collect point cloud data of the detection area. The detection data includes vibration detection data, Doppler spectrum data, and image detection data. The post-processing terminal is used to perform data processing based on the detection data... The modal weights corresponding to the data and the point cloud data are used to detect the data to be detected obtained by filtering the detection data and the point cloud data to determine the intrusion detection result. The intrusion detection result includes whether an intrusion behavior and intrusion object exist, and the intrusion object category and intrusion action when the intrusion object exists. The intrusion detection result is used to determine whether to generate a control signal, which is used to control the multiple detection devices to generate alarm signals and / or warning signals. The detection host is also used to: send heartbeats to the post-processing end according to a preset period; if no response information is received from the post-processing end within a preset time period, the host detection model is used to detect the point cloud data and the detection data to determine the intrusion detection result.

[0046] Thirdly, some embodiments of this application provide a method for machine vision microwave vibration fusion intrusion detection. This method is applied to the detection host of an intrusion detection system, which further includes multiple detection devices and a post-processing terminal. The method includes: acquiring point cloud data and detection data of a detection area; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; sending the point cloud data and the detection data to the post-processing terminal, so that the post-processing terminal can filter out the data to be detected from the point cloud data and the detection data, and then detect the data to be detected to determine the intrusion detection result; the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and the intrusion object category and intrusion action when the intrusion object exists.

[0047] Fourthly, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection method. This method is applied to the post-processing end of an intrusion detection system, which further includes multiple detection devices and a detection host. The method includes: acquiring detection data of a detection area and point cloud data sent by the detection host; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; filtering the detection data and the point cloud data to obtain data to be detected; detecting the data to be detected to determine the intrusion detection result; wherein the intrusion detection result includes whether an intrusion behavior and intrusion object exist, and the intrusion object category and intrusion action when the intrusion object exists.

[0048] Fifthly, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection device. This device is applied to the detection host of an intrusion detection system, which further includes multiple detection devices and a post-processing end. The device includes: a first acquisition module for acquiring point cloud data and detection data of a detection area; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; and a transmission module for transmitting the point cloud data and the detection data to the post-processing end, so that the post-processing end can filter out the data to be detected from the point cloud data and the detection data, detect the data to be detected, and determine the intrusion detection result; the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and the intrusion object category and intrusion action when the intrusion object exists.

[0049] Sixthly, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection device. This device is applied to the post-processing end of an intrusion detection system, which further includes multiple detection devices and a detection host. The device includes: a second acquisition module for acquiring detection data of a detection area and point cloud data sent by the detection host; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; a filtering module for filtering the detection data and the point cloud data to acquire data to be detected; and a detection module for detecting the data to be detected to determine the intrusion detection result; wherein the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and the intrusion object category and intrusion action when the intrusion object exists.

[0050] In a seventh aspect, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the methods described in any of the embodiments of the third or fourth aspect.

[0051] Eighthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the third or fourth aspect.

[0052] Ninthly, some embodiments of this application provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the third or fourth aspect. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 System diagram of machine vision microwave vibration fusion intrusion detection provided for some embodiments of this application; Figure 2 One of the flowcharts for a machine vision microwave vibration fusion intrusion detection method provided for some embodiments of this application; Figure 3The second flowchart of a machine vision microwave vibration fusion intrusion detection method provided for some embodiments of this application; Figure 4 One of the device composition block diagrams for machine vision microwave vibration fusion intrusion detection provided for some embodiments of this application; Figure 5 A second block diagram of a machine vision microwave vibration fusion intrusion detection device provided for some embodiments of this application; Figure 6 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0055] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0056] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0057] In related technologies, after multiple detection devices in a perimeter security system collect data, the further processing and analysis of this data are all handled by the backend server (i.e., the post-processing end, hereinafter referred to as the backend). This results in a large data processing load, low processing efficiency, and significant latency for the backend server, failing to meet the real-time requirements of security scenarios. Furthermore, if the initial data includes a large amount of sensor data collected by cameras, infrared thermal imaging, and laser imaging radar, the limitations of the transmission network bandwidth must also be considered. Problems with the network or the backend server can lead to widespread system failure.

[0058] In view of this, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection system, which includes multiple detection devices, a detection host, and a post-processing terminal. The multiple detection devices are communicatively connected to the detection host, and the detection host is communicatively connected to the post-processing terminal. During intrusion detection analysis, the detection host can share some or all of the data processing and intrusion detection work of the post-processing terminal based on the status of the post-processing terminal, thereby reducing data transmission pressure and data processing pressure on the post-processing terminal, improving the efficiency of intrusion detection, and meeting the real-time requirements of security scenarios.

[0059] The following is in conjunction with the appendix Figure 1 The overall structure of a machine vision microwave vibration fusion intrusion detection system provided by some embodiments of this application is illustrated by way of example.

[0060] like Figure 1As shown, some embodiments of this application provide a machine vision microwave vibration fusion intrusion detection system. This intrusion detection system includes multiple detection devices 100, a detection host 200, and a post-processing terminal 300. Multiple detection devices 100 (e.g., 20 detection devices) are connected to one detection host 200, and multiple detection hosts 200 are connected to the post-processing terminal 300. In a perimeter security system, the detection devices 100 and the detection host 200 can be deployed sequentially at certain intervals. Each detection device can correspond to an independent number and location information. The post-processing terminal 300 pre-stores the number and location information of each detection device and establishes a correspondence between the number and location information of each detection device and the detection host 200. Although... Figure 1 The diagram only shows one probe host 200 connected to the post-processing terminal 300, but in actual operating environments, there may be multiple probe hosts 200 connected to the post-processing terminal 300.

[0061] The functions of each unit in the intrusion detection system are illustrated below.

[0062] Each of the multiple detection devices 100 is used to: collect at least the monitoring data of the detection area.

[0063] The detection device 100, also known as a detector, can be installed on the fence of a perimeter security system. Its maximum detection range is called the detection area. In some implementations, a single detection device 100 integrates multiple sensors, such as a vibration sensor, a wide-angle camera, and a Doppler radar. Vibration detection is a passive detection method, while Doppler microwave radar (Doppler radar for short) and visual sensors (such as a wide-angle camera) form an early warning zone outside the fence, which is an active detection method. When a target enters the early warning zone, a suspicious target can be detected through Doppler data analysis or video image analysis, generating an intrusion warning signal.

[0064] Specifically, vibration sensors can collect vibration detection signals generated by the fence due to external factors; wide-angle cameras can collect image detection data within the detection area; and Doppler radar can collect spectral signals within the detection area. Each data segment collected by different detection devices 100 carries the corresponding device number. Detection device 100 can also preprocess the collected vibration detection signals and spectral signals (e.g., signal scaling, filtering, or encoding) to obtain monitoring data that meets the requirements of subsequent intrusion detection analysis. The monitoring data includes the raw vibration data corresponding to the vibration detection signals, the raw spectral data corresponding to the spectral signals, and the image detection data.

[0065] In some implementations, in addition to data acquisition and data preprocessing functions, the detection device 100 can also perform data splicing and feature extraction operations on monitoring data under special circumstances (e.g., when the detection host 200 is under high load or data processing is abnormal) to obtain detection data.

[0066] Optionally, multiple sensors can be integrated into the detection device 100 in a pluggable manner, allowing for flexible deployment of different sensors as needed.

[0067] Alternatively, while one detection device 100 may be a single sensor, multiple different types of detection devices 100 can be deployed at a single location. For example, a vibration sensor, a wide-angle camera, and a Doppler radar can be deployed at the same location. These three sensors are discrete components and are not integrated together. Of course, the deployment locations of different types of detection devices 100 do not have to be at the same point, but rather at nearby points.

[0068] The detection host 200 is used to: acquire point cloud data and detection data of the detection area; wherein, the detection data includes vibration detection data, Doppler spectrum data or image detection data; the detection data is acquired based on the monitoring data.

[0069] The image detection data can be collected by the detection device 100 or by a visual sensor deployed on the detection host 200. The specific method can be determined based on the actual situation.

[0070] The detection host 200 can connect to multiple detection devices 100 via a bus topology to acquire detection data related to the detection areas of the multiple detection devices 100; the detection data includes vibration detection data, Doppler spectrum data, and image detection data. The detection host 200 acquires the detection data in several ways: First, the detection host 200 directly receives the monitoring data sent by the detection device 100, and processes the received monitoring data (e.g., data splicing and feature extraction operations) to obtain the detection data.

[0071] Secondly, the detection device 100 processes the monitoring data, generates detection data, and sends it to the detection host 200.

[0072] Third, for a portion of the monitoring data, the detection device 100 processes it to generate detection data and sends it to the detection host 200; while for the remaining monitoring data, the detection host 200 processes it to generate detection data.

[0073] For example, a stable and unchanging algorithm is embedded in the detection device 100, while potentially variable algorithms are placed on the detection host 200. Exemplarily, quantization filtering of the vibration sensor is performed by the detection device 100, while feature extraction is performed by the detection host 200.

[0074] For example, preprocessing that can significantly compress the data volume is performed by the probe device 100, while preprocessing with a low compression ratio is performed by the probe host 200. Example: FFT compresses the raw IQ data into a spectrogram (high compression ratio), which is performed by the probe device 100.

[0075] For example, multimodal data from the same physical location that requires strict temporal and spatial alignment can be jointly processed by the same device to avoid asynchrony caused by transmission. For instance, vibration data and Doppler data can be jointly encoded by the detection device 100.

[0076] Fourth, by default, the monitoring data is processed by the detection host 200. However, when the detection host 200 is overloaded, some or all of the data processing will be transferred to the detection device 100 or backend processing.

[0077] In some embodiments of this application, the post-processing end 300 is used to: detect the detection data and the data to be detected obtained by filtering the point cloud data, and determine the intrusion detection result; wherein, the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and the intrusion object category and intrusion action when the intrusion object exists.

[0078] The post-processing unit 300, also known as the backend server (or simply backend), primarily filters and analyzes the detection data and point cloud data to determine the final intrusion detection results. The backend can store the priority order of multiple detection areas. During data filtering and analysis, data can be processed in order according to the priority of the detection area to which it belongs; that is, high-priority detection areas are processed first, and low-priority detection areas are processed later.

[0079] The backend can determine the intrusion detection results in several ways: First, the backend can be equipped with pre-trained detection models; these models include a first detection model and a second detection model. Under normal circumstances (i.e., when the backend load does not exceed the backend load threshold), the detection models can be used to perform intrusion detection on the data to be detected and determine the intrusion detection results.

[0080] Secondly, a backup model is also deployed on the backend; when the backend load exceeds the backend load threshold, it can automatically switch from the detection model to the backup model to perform the task of determining the intrusion detection result.

[0081] Third, when the backend load exceeds the backend load threshold, the backend can assign the data filtering task to the detection host 200. The backend only performs the task of determining the intrusion detection result, thereby enabling the detection host 200 and the backend to work together to determine the intrusion detection result and reduce the backend data processing pressure.

[0082] Fourth, when there is a data processing failure at the backend, the system can automatically switch to the detection host 200 to perform the task of determining the intrusion detection results, in order to meet the real-time requirements of intrusion detection in security scenarios. The detection host 200 is equipped with a lightweight host detection model, which performs the same functions as the backend detection model, but its detection accuracy is slightly lower than that of the backend detection model.

[0083] The method for determining whether there is a processing failure in the backend is as follows: the detection host 200 and the backend adopt a two-way heartbeat mechanism. For example, the detection host 200 can actively send heartbeats to the backend at regular intervals to monitor the backend's operating status; when the backend's operating status is abnormal, the system automatically switches to the detection host 200 to perform the task of determining the intrusion detection result.

[0084] After obtaining the intrusion detection results, the backend can also send control signals to the corresponding detection device 100 based on the intrusion detection results, so that the detection device 100 can output early warning or alarm information.

[0085] In addition, the backend can be integrated with electronic maps to display the number, location information, and whether there are alarms or warnings for each detection device 100 on the electronic map; at the same time, when users interact with the electronic map, they can also view the video or image data collected by each detection device 100 or surrounding detection devices in real time.

[0086] Based on the above embodiments, optionally, the detection host 200 may be equipped with a laser point cloud radar, which can collect point cloud data of the detection area. Alternatively, a visual sensor may also be deployed on the detection host 200 to collect point cloud data of the detection area.

[0087] The difference between the laser point cloud radar or visual sensor on the detection host 200 and the laser point cloud radar or visual sensor (if any) on the detector (detection device 100) is that the former has a longer coverage distance (for example, the detection host can cover 100 meters, while the detector covers 10 meters). Since the detection host 200 is generally installed inside the fence, an emergency alarm zone is formed inside the fence; when a target climbs over the fence and enters this zone, the system generates an emergency alarm and reports the target's specific information; the target's specific information includes the associated detector number, location information (e.g., installation location or latitude and longitude coordinates), and the target's orientation measured by the laser radar on the detection host 200.

[0088] Based on the above embodiments, optionally, a host detection model may be deployed on the detection host 200; a backend detection model may be deployed on the post-processing end 300; both the host detection model and the backend detection model are used to detect the data to be detected and output the intrusion detection result; the host detection model is obtained by training through a target training set; the target training set includes intrusion sample data and detection result data; the detection result data is obtained by inputting the intrusion sample data into the backend detection model.

[0089] For example, under normal circumstances, the backend can be responsible for the configuration management, status monitoring, fault and alarm management of all detection hosts and detectors in the entire system. In special circumstances, if the backend malfunctions or experiences anomalies, the detection host can also operate independently; similarly, if the detection host experiences data processing anomalies and can only perform data transmission, the backend can also take over the data processing work of the detection host. Specifically, the detection host 200 is equipped with a host detection model, which includes a first model for image and point cloud data processing and a second model for vibration and Doppler data processing. The first and second models can be used to detect the data to be detected and output intrusion detection results. Similarly, the backend detection model includes a first detection model and a second detection model, which can also detect the data to be detected and output intrusion detection results.

[0090] To make the accuracy of the host detection model as close as possible to that of the backend detection model, the backend detection model can analyze a large amount of unlabeled data (i.e., intrusion sample data) and output corresponding labeled data (i.e., detection result data). This data can be used to build a target training set to train the host detection model, enabling it to learn the generalization ability of the backend detection model and improve the accuracy of intrusion detection.

[0091] Based on the above embodiments, optionally, each detection device 100 is used to: quantize the initially acquired vibration detection signal and spectrum signal to obtain the original vibration data and the original spectrum data; wherein, the quantization processing includes signal scaling, signal filtering and signal encoding.

[0092] For example, the detection device 100 can collect data at a fixed frequency and quantize the initially collected vibration detection signal and spectrum signal. Specifically, signal scaling, signal filtering, and signal encoding operations are performed on the vibration detection signal and spectrum signal respectively to obtain the original vibration data and original spectrum data. Signal scaling scales the original signal (i.e., the vibration detection signal or spectrum signal) to a specific range (e.g., 0~1) to obtain the scaled signal (i.e., the scaled vibration signal or spectrum signal). Signal filtering applies a moving average filter to the scaled signal to remove random noise and outliers (outliers are signal values ​​that are outside the set range), resulting in the filtered signal (i.e., the filtered vibration data or spectrum data). Signal encoding encodes the filtered signal according to a set frame data encoding format to obtain the original vibration data and original spectrum data. The frame data encoding format includes a frame header, detection device number, packet length, sensed data (i.e., the filtered vibration data and spectrum data), CRC checksum, and frame tail. The CRC verification code is calculated using the CRC-16 algorithm and added to the data frame to enable verification of the original vibration data and original spectrum data. Optionally, the sensing data can simultaneously include the original vibration data and the original spectrum data to achieve joint encoding and spatiotemporal alignment of the dual-modal data.

[0093] Optionally, in one implementation, when the detection device 100 does not receive a high load warning from the detection host 200, or when the detection device 100 does not have pre-set rules for data splicing and feature extraction, the detection device 100 sends the aforementioned monitoring data to the detection host 200.

[0094] When the microprocessor or MCU (Microcontroller Unit) of the detection device 100 has high processing performance, or when the detection device 100 receives a high load warning from the detection host 200, or when the detection device 100 has pre-set rules for data splicing and feature extraction, in another embodiment, each detection device 100 is used to: perform data splicing operations on the original vibration data and the original spectrum data respectively to obtain vibration spliced ​​data and the Doppler spectrum data; perform feature extraction operations on the vibration spliced ​​data to obtain the vibration detection data; and send the detection data to the detection host.

[0095] For example, the detection device 100 can first perform a data stitching operation on the original vibration data and the original spectrum data, and then perform a feature extraction operation on the stitched vibration data to obtain the detection data. Subsequently, the detection device 100 sends the detection data to the detection host 200.

[0096] Based on the above embodiments, optionally, the detection host 200 is configured to: receive the monitoring data sent by each detection device; perform data splicing operations on the original vibration data and the original spectrum data respectively to obtain vibration splicing data and the Doppler spectrum data; and perform feature extraction operations on the vibration splicing data to obtain the vibration detection data.

[0097] For example, after receiving the monitoring data sent by the detection device 100, the detection host 200 first performs a data splicing operation on the original vibration data and the original spectrum data, and then performs a feature extraction operation on the spliced ​​vibration splicing data to obtain the detection data.

[0098] Specifically, the data stitching operation mainly targets the original vibration data and the original spectrum data. The implementation process of the data stitching operation is as follows: multiple consecutive frames of data (original vibration data or original spectrum data) from a single detector are stitched together. During stitching, the data is first decoded according to the frame data encoding format, and the decoded data is subjected to CRC (Cyclic Redundancy Check). After the check passes, the valid data is merged according to the detector number (i.e., data with the same detector number are merged) and saved to the corresponding buffer queue. Specifically, a circular queue can be used to store the data.

[0099] The CRC verification process is as follows: The frame result of the detector (i.e., the original vibration data or original spectrum data encoded according to the frame data encoding format) contains a CRC verification code (2 bytes). When the detector host 200 receives the frame result, it recalculates the CRC to be verified and compares it with the original CRC verification code in the frame. If the two are consistent, the decoded data passes the verification; otherwise, it is erroneous data and fails the verification.

[0100] Specifically, the data feature extraction operation is mainly aimed at vibration splicing data. The implementation process of the data feature extraction operation is as follows: perform feature extraction operation on the vibration splicing data from multiple dimensions corresponding to the time domain and frequency domain respectively to obtain the vibration detection data including feature values ​​of multiple dimensions; wherein, the multiple dimensions in the time domain include extreme values, mean, standard deviation, variance, kurtosis, skewness, root mean square value or kurtosis factor; the multiple dimensions in the frequency domain include spectrum, dominant frequency, secondary frequency or frequency band energy.

[0101] For example, vibration data is typically a one-dimensional or three-dimensional time-series vibration waveform, and its feature extraction employs a joint time-domain and frequency-domain analysis approach. Time-domain analysis extracts features across multiple dimensions, including: maximum value, minimum value, mean, standard deviation, variance, kurtosis, skewness, root mean square value, and kurtosis factor. Frequency-domain analysis uses Fourier transform to convert the time-domain signal of the spliced ​​data to the frequency domain, specifically extracting features across multiple dimensions, such as: spectrum (amplitude or energy of different frequency components), dominant frequency, secondary frequency, and frequency band energy (e.g., low-frequency, mid-frequency, and high-frequency bands). Additionally, it is necessary to extract some features from the previous frame of the spliced ​​vibration data, such as mean, maximum value, and energy. Finally, the extracted feature values ​​from the current and previous frames across multiple dimensions are merged to obtain the vibration detection data.

[0102] As can be seen, the main computational burden of the above data feature extraction comes from the Fast Fourier Transform (FFT) algorithm, which can be calculated using the DSP library (Digital Signal Processing library) that comes with a specific series of processors. This library can be optimized to significantly improve the execution efficiency of the FFT algorithm.

[0103] Optionally, when the detection data includes the Doppler spectrum data and the image detection data, the post-processing end 300 is used to: acquire vibration stitched data after performing a data stitching operation on the original vibration data; and perform feature extraction operation on the vibration stitched data to obtain vibration detection data.

[0104] For example, when the computational load involved in data feature extraction is relatively large, it can be implemented in the backend. When the data feature extraction operation is executed in the backend, the detector only collects and preprocesses the data in the detection area. The detection host 200 performs a data splicing operation and packages and encapsulates the spliced ​​data before sending it to the backend. The backend unpacks the data, performs CRC verification, and then performs feature extraction and subsequent intrusion detection operations.

[0105] As can be seen from the above embodiments, the data feature extraction operation can be implemented on the detector, the detector host, or the backend; the implementation process can be flexibly allocated according to the actual situation. If the feature dimension is relatively fixed and the detector processor performance meets the requirements, implementation on the detector should be given priority. If the feature dimension is expected to change, such as adding several features (e.g., the algorithm is optimized and upgraded), it should be implemented on the detector host 200 and the backend (in this case, the detector firmware does not need to be changed, only the detector host firmware is refreshed or the backend program is upgraded). In particular, the comparison between the number of detectors and the number of detector hosts can also be considered, i.e., the number of detectors that one detector host can aggregate, while also considering the network transmission bandwidth limit from the detector to the detector host. If feature extraction is performed on the detector, only a small amount of feature data needs to be transmitted from the detector to the host compared to the original data, saving network bandwidth, and the detector host can aggregate more detectors. Specifically, it can be dynamically adjusted according to the actual situation, and this application embodiment does not make specific limitations here.

[0106] Based on the above embodiments, optionally, the detection host 200 is used to: identify intrusion targets within the detection area by detecting the point cloud data; send the intrusion targets to the post-processing end; the post-processing end is used to: determine the intrusion detection result based on the data to be detected and the intrusion targets.

[0107] For example, the point cloud data collected by the detection host 200 can contain the three-dimensional coordinates and reflection intensity information of objects. By detecting the point cloud data, the outline of the object can be obtained (as a specific example of an intrusion target). Specifically, a pre-trained neural network model is loaded, and the trained neural network model is used to perform forward inference on the point cloud data to obtain the target type (as a specific example of an intrusion target). The process of acquiring the pre-trained neural network model includes: calibrating the historical point cloud data collected on-site (using software tools to calibrate the spatial location and classification of the target, such as people, vehicles, objects, etc.); then using the calibrated data to train the model, finally obtaining the neural network model. The model can use lightweight algorithms such as PointPillars, which have good real-time performance and accuracy. The above model training process can be completed offline in the backend or using a dedicated training server; after training, the neural network model is deployed to the detection host 200.

[0108] After the detection host 200 completes the initial detection of the intrusion target, it can send the data to the backend. The backend can combine the data to be detected and the intrusion target to comprehensively determine the intrusion detection result.

[0109] Optionally, the detection host 200 is configured to: determine whether an intrusion target exists by detecting the point cloud data and / or the image detection data; generate an emergency warning signal if an intrusion target is confirmed to exist; send the emergency warning signal to the target detection device corresponding to the image detection data; the target detection device is configured to output warning information; wherein the target detection device is one or more of the plurality of detection devices.

[0110] For example, in emergency situations, the detection host 200 can directly interact with the detectors, enabling them to promptly output early warning information. Specifically, the pre-trained network detection model deployed on the detection host 200 can infer from the input point cloud data and / or image detection data to determine the presence of an intrusion target. If an intrusion target is found, the target's category can be automatically identified, and the detector number and location information of the intrusion target can be located, thereby generating an emergency warning signal. The detection host 200 then sends the emergency warning signal to the corresponding target detector, which outputs warning information. The warning information can be a voice announcement, a siren, or a light flashing, etc. This network detection model can be trained using calibrated point cloud data and / or video image data on a ResNet or YOLO (You Only Look Once, real-time target detection) algorithm.

[0111] Optionally, the detection host 200 is configured to: send a heartbeat to the post-processing terminal according to a preset period; if a response information corresponding to the heartbeat is received from the post-processing terminal within a preset time period, then send the point cloud data and the detection data to the post-processing terminal; if the response information is not received within the preset time period, then detect the point cloud data and the detection data to determine the intrusion detection result.

[0112] For example, the detection host 200 and the backend can detect the backend's operating status through a heartbeat mechanism. Specifically, the detection host 200 can send heartbeats to the backend at preset intervals (e.g., 5s or 10s). Upon receiving the heartbeat, the backend sends a response message to the detection host 200. If the detection host 200 receives the response message within the preset time period, it indicates that the backend is operating normally; otherwise, it is abnormal. In this case, the intrusion detection system can automatically transfer the task of determining the intrusion detection result to the detection host 200, which then detects the point cloud data and detection data to obtain the intrusion detection result. Simultaneously, the detection host 200 can send a backend anomaly alert to staff, allowing them to promptly check the backend's specific situation. In other embodiments, the backend can also proactively send heartbeats to the detection host 200 to detect its operating status. When the detection host 200 is abnormal, the backend can take over the data processing work from the detection host 200; the implementation principle is similar to the above embodiments, and will not be repeated here to avoid repetition.

[0113] In actual operation, the load of the intrusion detection system, including the detection host 200 and the backend, fluctuates dynamically over time. For example, during peak periods, the detection host may be overloaded, leading to a significant increase in response latency; during off-peak periods, resources are idle, resulting in waste. To achieve dynamic optimization of the intrusion detection system and ensure data processing efficiency, the detection host 200 may optionally: monitor the host load in real time and simultaneously acquire the backend load of the post-processing end; when the host load exceeds the host load threshold, send the spliced ​​vibration data (composed of the original vibration data) to the post-processing end so that the post-processing end can extract features from the spliced ​​vibration data to obtain the vibration detection data; when the backend load exceeds the backend load threshold, detect the detection data and the data to be detected obtained by filtering the point cloud data to determine the intrusion detection result.

[0114] For example, the detection host 200 can be equipped with a dynamic scheduling module that can monitor the host load in real time and obtain the backend load. The host load and backend load can be characterized using parameters such as CPU utilization, temperature, and data processing efficiency. By comparing the host load with host load thresholds and the backend load with backend load thresholds, the data processing content of the detection host and backend can be determined. For example, when the host load exceeds the host load threshold, the backend can perform data feature extraction operations to reduce the data processing load of the detection host. When the backend load exceeds the backend load threshold, the detection host can perform the task of determining the intrusion detection result.

[0115] In addition, the detection host can also output the confidence level of the intrusion detection results. When the confidence level of the intrusion detection results output by the detection host is lower than the confidence level threshold, the backend can use the detection model to review the intrusion detection results output by the detection host after the backend load returns to normal, so as to obtain more accurate intrusion detection results.

[0116] In other embodiments, the dynamic scheduling module can also be deployed in the backend, whereby the backend monitors and analyzes the host load and backend load to dynamically adjust the data processing content of the probe host and the backend. The implementation principle of backend dynamic adjustment is similar to that of the probe host described above, and will not be repeated here to avoid repetition.

[0117] It is understood that the data processing content of the probe host and backend can be flexibly adjusted according to the actual load to adapt to different load scenarios. The host load threshold and the backend load threshold can be the same or different, and their specific values ​​can be set according to the actual application scenario. This application embodiment does not impose specific limitations here.

[0118] Optionally, the detection host 200 is used to: acquire idle hosts among a plurality of detection hosts within a preset range whose load data for each detection host does not exceed the host load threshold; and, if the host load exceeds the host load threshold, send detection tasks to the idle hosts according to task priority; wherein, the task priority of the feature extraction task in the detection task is lower than that of the intrusion detection task.

[0119] For example, the probe host 200 can also detect the load of other probe hosts in the vicinity (i.e., within a preset range, which can refer to a preset physical distance or network topology hop count) to identify idle hosts in the vicinity that are below the host load threshold. When the probe host 200's own host load exceeds the host load threshold, it can send its data processing tasks (i.e., feature extraction tasks or intrusion detection tasks) to any idle host or the nearest idle host to achieve cross-node load balancing. Different data processing tasks can be assigned corresponding priorities, with lower-priority tasks being preferentially transferred to idle hosts for processing.

[0120] Based on the above embodiments, in order to achieve intrusion detection based on multiple types of data, optionally, the post-processing end 300 is used to: align the vibration detection data, the Doppler spectrum data, the image detection data, and the point cloud data to obtain multimodal data; use a dynamic gating fusion network to score the various modal data in the multimodal data to obtain the modal weights; and filter the detection data and the point cloud data using the modal weights to determine the data to be detected; wherein, the data to be detected includes some or all of the data in the detection data and the point cloud data. The so-called data alignment includes time alignment based on the acquisition timestamps of each data point, and / or spatial alignment based on the installation location of each detection device.

[0121] For example, multiple data types can be processed uniformly, mapping data from different physical domains to a shared semantic embedding space to achieve semantic-level alignment. Specifically, a one-dimensional convolutional network or Transformer is used to encode vibration detection data to obtain vibration modal data. Doppler spectrograms are extracted from Doppler spectrum data and encoded using lightweight ViT to obtain spectral modal data. Spatiotemporal tokens are extracted from image detection data using TimeSformer or VideoMAE to obtain image modal data. PointPillars or VoxelNet are used to output BEV (bird's-eye view) feature maps from point cloud data. The vibration modal data, spectral modal data, image modal data, and BEV feature maps obtained above constitute multimodal data. Since not all modal data are valid at all times, a dynamic gating fusion network is used to filter the multimodal data to determine the data to be detected for subsequent intrusion detection. The dynamically gated fusion network can be a lightweight MLP (Multilayer Perceptron) or a Gating Transformer (a Transformer model with a gating mechanism). This network scores various modalities in the input multimodal data, obtaining modal weights for each. For example, nighttime images (or videos) are invalid, with a modal weight of 0; strong wind interference reduces the weight of the vibration modal data. Modal weights range from 0 to 1. Modal data with weights higher than a threshold (e.g., 0.5) are used as the detection data in subsequent calculations, thereby reducing computational redundancy and improving robustness and energy efficiency.

[0122] Optionally, the post-processing end 300 is used to: acquire the environmental feature vector of the detection area; input the environmental feature vector and the multimodal data into the dynamic gating fusion network, and output the modal weights.

[0123] For example, in real-world operating environments, the reliability of different sensors is greatly affected by the environment. For instance, at night or in rainy or foggy weather, the quality of video images degrades, and their weights should be automatically reduced; in strong winds, vibration sensors are interfered with, and their weights should be reduced. Based on this, the detection host 200 or its backend can be equipped with an environmental perception module. This module can process the environmental data collected by the received environmental sensors to obtain environmental feature vectors. Environmental sensors can be deployed on the detection host 200 or on a fence to communicate with the host 200 or its backend. Alternatively, the environmental perception module can use a deployed environmental classification model (the weather classification model can be a lightweight CNN model to identify various environments such as fog, rain, snow, and nighttime) to extract environmental feature vectors from the image detection data. The environmental feature vectors and multimodal data are simultaneously input into a dynamic gating fusion network. The environmental feature vectors are used to adjust the baseline weights of the multimodal data, and the weights are further optimized based on the effectiveness of each modality, ultimately outputting the weights for each modality. For example, if the environmental perception module detects nighttime, the weight baseline value of the image mode will be automatically reduced; if strong wind is detected, the weight baseline value of the vibration mode will be reduced; the dynamic gating network will then fine-tune the weights based on the actual data of the multimodal data, and finally output the weights of each mode.

[0124] Based on the above embodiments, optionally, the post-processing end 300 is used to: input the vibration detection data and the Doppler spectrum data into a first detection model to obtain a first detection result; the first detection result characterizes whether an intrusion behavior exists; input the image detection data and the point cloud data into a second detection model to obtain a second detection result; the second detection result includes whether an intrusion object exists, and when the intrusion object exists, the intrusion object category and the intrusion action of the intrusion object; and generate the intrusion detection result based on the first detection result and the second detection result.

[0125] For example, the backend deploys a first detection model for vibration and spectrum-based data detection, and a second detection model for image and point cloud-based data detection. These two models perform intrusion detection on data in different modalities to generate the final intrusion detection result. The first detection model analyzes vibration detection data and Doppler spectrum data to determine if intrusion behavior exists. The second detection model analyzes image detection data and point cloud data to determine if an intrusion target exists. If an intrusion target is found, it can output the intrusion target category (e.g., person, vehicle, or object) and intrusion action (e.g., climbing, throwing objects from a height, taking photos, etc.). Finally, the intrusion behavior, intrusion target type, and intrusion action are combined to generate the final intrusion detection result. For example, the intrusion detection result might be: abnormal vibration detected on the west side of the wall (confidence level 85%), point cloud showing a 1.7m high target climbing, and video image confirming a human-shaped target.

[0126] Understandably, the first and second detection models are obtained by training the target detection model and the target classification model, respectively. Specifically, supervised machine learning is used to train the target detection model to identify intrusion behaviors. Training data acquisition and calibration tools are used to collect feature data of various intrusion behaviors in a real-world environment and perform behavior calibration (e.g., no intrusion, knocking, impact, shaking, climbing, etc.). During the training process using the training data, the model is evaluated to confirm whether its recognition accuracy meets the expected value. If it does not, the physical consistency loss function (used to constrain the logical consistency between detection results of different modalities; for example, if an image detects a human target, but the Doppler data does not detect the corresponding speed, the loss value is increased) is used to optimize the model parameters. Training is repeated until the expected value is reached, outputting the first detection model. The target detection model can be a neural network algorithm, decision tree, random forest, or Bayesian algorithm, etc. The training dataset for the second detection model includes images and point cloud data containing various intrusion objects and intrusion behaviors. This training dataset is used to train the target classification model to obtain the second detection model that meets the expected value. In practical applications, after the backend loads the trained first and second detection models, it inputs the corresponding data for inference and obtains the corresponding intrusion classification results.

[0127] Optionally, the post-processing terminal 300 is used to: monitor the backend load in real time; when it is confirmed that the backend load exceeds the backend load threshold, switch the first detection model and the second detection model to a backup model; wherein the computational overhead of the backup model is lower than that of the first detection model and the second detection model; when the backend load recovers to a state that does not exceed the backend load threshold, switch the backup model to the first detection model and the second detection model.

[0128] For example, a backup model with lower computational overhead can be deployed internally in the backend. This backup model can be a simplified version of the first and second detection models, for example, with fewer network layers or channels. When the backend monitors its own load, if it detects that its current backend load exceeds the backend load threshold, it can switch models to use the backup model with lower computational overhead to analyze the data to be detected and determine the intrusion detection results. Once the backend load returns to normal operation, it can switch back to the detection mode of the first and second detection models.

[0129] Optionally, there are multiple detection areas, and one detection area corresponds to a set of data to be detected; the post-processing end 300 is used to: detect the corresponding data to be detected according to the priority order of the detection areas, and obtain the intrusion detection result of each detection area in the multiple detection areas.

[0130] For example, since there are many detection areas, and their importance varies depending on their location, the backend can store the priority order of multiple detection areas. During intrusion detection, important detection areas (i.e., high-priority areas) can be prioritized for intrusion detection to obtain the intrusion detection results for those important areas.

[0131] Optionally, the post-processing end 300 is used to: generate an alarm control signal if the intrusion behavior is confirmed in the intrusion detection result; and / or generate an early warning control signal if the intrusion object is confirmed in the intrusion detection result; and send the alarm control signal and / or the early warning control signal to the detection device where the intrusion behavior and / or the intrusion object exists, so that the detection device can generate and output an alarm signal and / or an early warning signal.

[0132] For example, vibration detection is a passive detection method, and the intrusion behavior detected by it through a first detection model is usually defined as an alarm; video and radar are active detection methods, and the intrusion object and intrusion action detected by them through a second detection model are usually defined as an alarm. That is, alarms and alarms can be distinguished based on the detection method; they can also be defined based on the area where the intrusion occurs, i.e., intrusion behavior or actions occurring outside the fence are alarms, while intrusion behavior or actions occurring on the fence are alarms. Specifically, when there is intrusion behavior, or when there is an intruder and an intrusion action, an alarm control signal can be generated and the detector number that caused the intrusion behavior can be identified and sent to the corresponding detector, which outputs an alarm signal. When there is an intruder but no intrusion action, or when the intruder is a pedestrian, animal, or vehicle, an alarm control signal can be generated and the detector numbers around the intruder can be identified and sent to the corresponding detector, which outputs an alarm signal. The alarm signal and the alarm signal can include at least one of the following: lighting, voice, and horn.

[0133] In other embodiments, if the second detection model detects a suspicious target (i.e., an intruder) outside the fence, the operating parameters of the vibration sensor can be dynamically adjusted to increase the vibration signal acquisition frequency, thereby improving the accuracy of subsequent intrusion detection. If the intruder outside the fence is detected to be approaching the fence further and attempting to climb over or cut through it, the intrusion behavior can be detected through vibration detection, and an intrusion alarm signal can be generated. This constitutes a two-layer protection system consisting of a warning zone outside the fence and an alarm zone within the fence.

[0134] Optionally, the post-processing terminal 300 is used to: obtain the number of the detection device where the intrusion behavior and / or the intrusion target exists; render the corresponding detection device on an electronic map according to the number to display the intrusion detection result; wherein the electronic map displays the location and number of each detection device among multiple detection devices in the detection area.

[0135] For example, the backend can integrate the status and location information of detectors with an electronic map. Each detector has a unique identifier, and based on its specific installation location or latitude and longitude coordinates (i.e., location information), each detector is individually marked on the map according to its actual installation location or latitude and longitude coordinates, recording the mapping relationship between electronic map elements and detectors. The identifier and location information of each detector are displayed on the map elements. When intrusion behavior or an intrusion target is detected on a detector, the corresponding map element can be found based on the detector identifier, and color-rendered to display the intrusion detection result. Detectors exhibiting abnormal behavior can also be displayed on the electronic map.

[0136] Optionally, the post-processing end 300 is used to: in response to a user's operation command on the electronic map for any detection device, display video data collected by the detection device, or display video image data collected by associated cameras around the detection device.

[0137] For example, users can also interact with the electronic map displayed on the front end. When a user clicks on a detector on the electronic map (as a specific example of an operation command), the real-time video or image data collected by that detector can be displayed. Furthermore, the backend records the mapping relationship between detectors and surrounding associated cameras, and can store video data collected by these associated cameras. If a user wants to view video images from associated cameras, they can also instantly retrieve and view them by clicking. These associated cameras can be cameras on other detectors deployed in the same area as the detector, or they can be independently deployed third-party surveillance cameras; the mapping relationship between detectors and associated cameras can be pre-configured based on the relative positions of the detectors and cameras.

[0138] In other embodiments of this application, a machine vision microwave vibration fusion intrusion detection system is applied to perimeter security scenarios. The system includes multiple detection devices, a detection host, and a post-processing terminal. Each detection device is used to quantize initial data from the collected detection area to obtain monitoring data. The monitoring data includes raw vibration data, raw spectrum data, and image detection data. The detection host is used to perform data stitching and feature extraction on the received monitoring data to obtain detection data and collect point cloud data from the detection area. The detection data includes vibration detection data, Doppler spectrum data, and image detection data. The post-processing terminal is used to... The modal weights corresponding to the detection data and the point cloud data are used to detect the data to be detected obtained by filtering the detection data and the point cloud data, and to determine the intrusion detection result. The intrusion detection result includes whether an intrusion behavior and intrusion object exist, and the intrusion object category and intrusion action when the intrusion object exists. The intrusion detection result is used to determine whether to generate a control signal, which is used to control the multiple detection devices to generate alarm signals and / or warning signals. The detection host is also used to: send heartbeats to the post-processing end according to a preset period; if no response information is received from the post-processing end within a preset time period, the intrusion detection result is determined based on the point cloud data and the detection data.

[0139] The following is in conjunction with the appendix Figure 2 The present application provides an exemplary embodiment of the implementation process of machine vision microwave vibration fusion intrusion detection performed by the detection host 200.

[0140] Please see the appendix Figure 2 , Figure 2 This application provides a flowchart of a machine vision microwave vibration fusion intrusion detection method according to some embodiments. The machine vision microwave vibration fusion intrusion detection method may include: S210, acquiring point cloud data and detection data of a detection area; wherein, the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; S220, sending the point cloud data and the detection data to the post-processing end, so that the post-processing end can filter out the data to be detected from the point cloud data and the detection data, detect the data to be detected, and determine the intrusion detection result; the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

[0141] The following is in conjunction with the appendix Figure 3The present application provides an exemplary embodiment of the implementation process of machine vision microwave vibration fusion intrusion detection performed by the post-processing terminal 300.

[0142] Please see the appendix Figure 3 , Figure 3 This application provides a flowchart of a machine vision microwave vibration fusion intrusion detection method according to some embodiments. The machine vision microwave vibration fusion intrusion detection method may include: S310, acquiring detection data of a detection area and point cloud data sent by the detection host; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; S320, filtering the detection data and the point cloud data to obtain data to be detected; S330, detecting the data to be detected to determine the intrusion detection result; wherein the intrusion detection result includes whether an intrusion behavior and intrusion object exist, and when the intrusion object exists, the intrusion object category and intrusion action.

[0143] It should be noted that the specific implementation process of each step in the detection host 200 and the post-processing end 300 can be referred to the method embodiment provided above. To avoid repetition, detailed descriptions are omitted here.

[0144] Please refer to Figure 4 , Figure 4 The diagram illustrates a block diagram of a machine vision microwave vibration fusion intrusion detection apparatus provided in some embodiments of this application. It should be understood that this machine vision microwave vibration fusion intrusion detection apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this machine vision microwave vibration fusion intrusion detection apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0145] Figure 4The machine vision microwave vibration fusion intrusion detection device includes at least one software function module that can be stored in a memory or embedded in the device in the form of software or firmware. This device is applied to the detection host of an intrusion detection system and includes: a first acquisition module 410, used to acquire point cloud data and detection data of the detection area; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; and a sending module 420, used to send the point cloud data and the detection data to the post-processing end, so that the post-processing end can filter out the data to be detected from the point cloud data and the detection data, detect the data to be detected, and determine the intrusion detection result; the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

[0146] Please refer to Figure 5 , Figure 5 The diagram illustrates a block diagram of a machine vision microwave vibration fusion intrusion detection apparatus provided in some embodiments of this application. It should be understood that this machine vision microwave vibration fusion intrusion detection apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this machine vision microwave vibration fusion intrusion detection apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0147] Figure 5 The machine vision microwave vibration fusion intrusion detection device includes at least one software function module that can be stored in a memory or embedded in the device in the form of software or firmware. This device is applied to the post-processing end of an intrusion detection system and includes: a second acquisition module 510, used to acquire detection data of the detection area and point cloud data sent by the detection host; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; a filtering module 520, used to filter the detection data and the point cloud data to obtain data to be detected; and a detection module 530, used to detect the data to be detected and determine the intrusion detection result; wherein the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

[0148] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0149] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0150] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0151] like Figure 6 As shown, some embodiments of this application provide an electronic device 600, which includes a memory 610, a processor 620, and a computer program stored in the memory 610 and executable on the processor 620. When the processor 620 reads the program from the memory 610 via a bus 630 and executes the program, it can implement the methods of any of the above embodiments.

[0152] Processor 620 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 620 can be a microprocessor.

[0153] The memory 610 can be used to store instructions executed by the processor 620 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 620 of this disclosure embodiment can be used to execute the instructions in the memory 610 to implement the methods shown above. The memory 610 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0154] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0156] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A system for machine vision microwave vibration fusion intrusion detection, characterized in that, The system includes multiple detection devices, a detection host, and a post-processing terminal; Each of the plurality of detection devices is used to collect monitoring data of at least the detection area; wherein the monitoring data includes raw vibration data, raw spectrum data and image detection data; The detection host is used to acquire point cloud data and detection data of the detection area; wherein, the detection data includes vibration detection data, Doppler spectrum data, or image detection data; the detection data is acquired based on the monitoring data; The post-processing end is used to detect the detection data and the data to be detected obtained by filtering the point cloud data, and determine the intrusion detection result; wherein, the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

2. The system as described in claim 1, characterized in that, The detection host is used for: Receive the monitoring data sent by each detection device; Perform data stitching operations on the original vibration data and the original spectrum data respectively to obtain vibration stitched data and Doppler spectrum data; The vibration splicing data is subjected to feature extraction to obtain the vibration detection data.

3. The system as described in claim 1, characterized in that, Each detection device is used for: Perform data stitching operations on the original vibration data and the original spectrum data respectively to obtain vibration stitched data and Doppler spectrum data; Perform feature extraction on the vibration splicing data to obtain the vibration detection data; The detection data is sent to the detection host.

4. The system as described in claim 1, characterized in that, When the detection data includes the Doppler spectrum data and the image detection data, the post-processing terminal is used for: Obtain vibration stitched data after performing a data stitching operation on the original vibration data; The vibration splicing data is subjected to feature extraction to obtain the vibration detection data.

5. The system as described in any one of claims 2-4, characterized in that, The method of performing feature extraction on the vibration splicing data to obtain the vibration detection data includes: Feature extraction operations are performed on the vibration splicing data from multiple dimensions corresponding to the time domain and frequency domain respectively to obtain the vibration detection data including feature values ​​of multiple dimensions; wherein, the multiple dimensions in the time domain include maximum and minimum values, mean, standard deviation, variance, kurtosis, skewness, root mean square value or kurtosis factor; the multiple dimensions in the frequency domain include spectrum, dominant frequency, secondary frequency or frequency band energy.

6. The system as described in any one of claims 1-4, characterized in that, Each detection device is used for: The initially acquired vibration detection signal and spectrum signal are quantized to obtain the original vibration data and the original spectrum data; wherein, the quantization process includes signal scaling, signal filtering and signal encoding.

7. The system as described in any one of claims 1-4, characterized in that, The detection host is used for: By detecting the point cloud data, intrusion targets within the detection area are identified; The intrusion target is sent to the post-processing terminal; The post-processing unit is used to determine the intrusion detection result based on the data to be detected and the intrusion target.

8. The system as described in any one of claims 1-4, characterized in that, The detection host is used for: The presence of an intrusion target is determined by detecting the point cloud data and / or the image detection data. Upon confirmation of an intrusion target, an emergency warning signal is generated; Send the emergency warning signal to the target detection device corresponding to the image detection data; The target detection device is used to output early warning information; wherein, the target detection device is one or more of the plurality of detection devices.

9. The system as described in any one of claims 1-4, characterized in that, The post-processing terminal is used for: The vibration detection data, the Doppler spectrum data, the image detection data, and the point cloud data are aligned to obtain multimodal data; A dynamic gated fusion network is used to score various modal data in the multimodal data to obtain the modal weights; The detection data and the point cloud data are filtered by the modal weights to determine the data to be detected; wherein, the data to be detected includes some or all of the data in the detection data and the point cloud data.

10. The system as described in claim 9, characterized in that, The post-processing terminal is used for: Obtain the environmental feature vector of the detection area; The environmental feature vector and the multimodal data are input into the dynamic gating fusion network, and the modal weights are output.

11. The system as described in any one of claims 1-4, characterized in that, The post-processing terminal is used for: The vibration detection data and the Doppler spectrum data are input into the first detection model to obtain the first detection result; the first detection result characterizes whether there is an intrusion behavior. The image detection data and the point cloud data are input into the second detection model to obtain the second detection result; The second detection result includes whether an intrusion object exists, and if the intrusion object exists, the intrusion object category and the intrusion action of the intrusion object; The intrusion detection result is generated based on the first detection result and the second detection result.

12. The system as claimed in claim 11, characterized in that, The post-processing terminal is used for: Real-time monitoring of backend load; If it is confirmed that the backend load exceeds the backend load threshold, the first detection model and the second detection model are switched to a backup model; wherein the computational cost of the backup model is lower than that of the first detection model and the second detection model. When the backend load recovers to a state that does not exceed the backend load threshold, the backup model is switched to the first detection model and the second detection model.

13. The system as described in any one of claims 1-4, characterized in that, The detection area is multiple, and each detection area corresponds to a set of data to be detected; the post-processing end is used for: The corresponding data to be detected is detected according to the priority order of the detection areas to obtain the intrusion detection results of each detection area in multiple detection areas.

14. The system as described in any one of claims 1-4, characterized in that, The post-processing terminal is used for: If the intrusion detection results confirm the existence of the intrusion behavior, an alarm control signal is generated; and / or, If the intrusion target is confirmed to exist in the intrusion detection results, an early warning control signal is generated; Send the alarm control signal and / or the warning control signal to the detection device where the intrusion behavior and / or the intrusion target exists, so that the detection device can generate and output the alarm signal and / or warning signal.

15. The system as described in any one of claims 1-4, characterized in that, The post-processing terminal is used for: Obtain the identification number of the detection device that detects the intrusion behavior and / or the intrusion target; The corresponding detection devices are rendered on the electronic map according to the numbering to display the intrusion detection results; wherein, the electronic map displays the location and number of each detection device among the multiple detection devices in the detection area.

16. The system as described in claim 15, characterized in that, The post-processing terminal is used for: In response to a user's operation command on the electronic map for any detection device, the system displays video data collected by the detection device, or displays video image data collected by associated cameras around the detection device.

17. The system as described in any one of claims 1-4, characterized in that, The detection host is used for: Send heartbeats to the post-processing terminal at a preset period; If a response message corresponding to the heartbeat is received from the post-processing terminal within a preset time period, the point cloud data and the detection data are sent to the post-processing terminal. If the response information is not received within the preset time period, the point cloud data and the detection data are detected to determine the intrusion detection result.

18. The system as described in any one of claims 1-4, characterized in that, The detection host is used for: Real-time monitoring of host load, and simultaneously obtaining the backend load of the post-processing end; When the host load exceeds the host load threshold, the original vibration data spliced ​​together is sent to the post-processing end so that the post-processing end can extract features from the vibration spliced ​​data to obtain the vibration detection data. If the backend load exceeds the backend load threshold, the detection data and the data to be detected obtained by filtering the point cloud data are detected to determine the intrusion detection result.

19. The system as described in claim 18, characterized in that, The detection host is used for: Identify idle hosts among multiple probe hosts within a preset range whose load data for each probe host does not exceed the host load threshold; If the host load exceeds the host load threshold, a probe task is sent to the idle host according to the task priority; wherein, the feature extraction task in the probe task has a lower task priority than the intrusion detection task.

20. The system as described in any one of claims 1-4, characterized in that, The detection host is equipped with a host detection model; the post-processing end is equipped with a back-end detection model; both the host detection model and the back-end detection model are used to detect the data to be detected and output the intrusion detection result; the host detection model is trained using a target training set; the target training set includes intrusion sample data and detection result data; the detection result data is obtained by inputting the intrusion sample data into the back-end detection model.

21. A system for machine vision microwave vibration fusion intrusion detection, characterized in that, The system is applied to perimeter security scenarios; the system includes multiple detection devices, a detection host, and a post-processing terminal. Each of the plurality of detection devices is used to quantize the initial data collected from the detection area to obtain monitoring data; wherein, the monitoring data includes raw vibration data, raw spectrum data and image detection data; The detection host is used to perform data stitching and feature extraction on the received monitoring data to obtain detection data; and to collect point cloud data of the detection area; wherein, the detection data includes vibration detection data, Doppler spectrum data and image detection data; The post-processing end is used to detect the data to be detected obtained by filtering the detection data and the point cloud data based on the modal weights corresponding to the detection data and the point cloud data, respectively, and to determine the intrusion detection result; wherein, the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action; the intrusion detection result is used to determine whether to generate a control signal, and the control signal is used to control the multiple detection devices to generate alarm signals and / or warning signals; The detection host is also used to: send heartbeats to the post-processing terminal at a preset period; if no response information is received from the post-processing terminal within a preset time period, the host detection model is used to detect the point cloud data and the detection data to determine the intrusion detection result.

22. A method for machine vision microwave vibration fusion intrusion detection, characterized in that, The method is applied to the detection host of an intrusion detection system, which further includes multiple detection devices and a post-processing terminal; the method includes: The system acquires point cloud data and detection data of the detection area; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data. The point cloud data and the detection data are sent to the post-processing end so that the post-processing end can filter out the data to be detected from the point cloud data and the detection data, and then detect the data to be detected to determine the intrusion detection result. The intrusion detection result includes whether there is an intrusion behavior and an intrusion object, as well as the intrusion object category and intrusion action when the intrusion object exists.

23. A method for machine vision microwave vibration fusion intrusion detection, characterized in that, The method is applied to the post-processing end of an intrusion detection system, which further includes multiple detection devices and a detection host; the method includes: The system acquires detection data of the detection area and point cloud data sent by the detection host; wherein the detection data includes vibration detection data, Doppler spectrum data, and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data, and image detection data; The detection data and the point cloud data are filtered to obtain the data to be detected; The data to be detected is tested to determine the intrusion detection result; wherein, the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

24. A device for machine vision microwave vibration fusion intrusion detection, characterized in that, The device is used in the detection host of an intrusion detection system, which also includes multiple detection devices and a post-processing terminal; the device includes: The first acquisition module is used to acquire point cloud data and detection data of the detection area; wherein, the detection data includes vibration detection data, Doppler spectrum data and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data and image detection data; The sending module is used to send the point cloud data and the detection data to the post-processing end, so that the post-processing end can filter out the data to be detected from the point cloud data and the detection data, detect the data to be detected, and determine the intrusion detection result; the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, as well as the intrusion object category and intrusion action when the intrusion object exists.

25. A device for machine vision microwave vibration fusion intrusion detection, characterized in that, The device is used in the post-processing stage of an intrusion detection system, which further includes multiple detection devices and a detection host; the device includes: The second acquisition module is used to acquire detection data of the detection area and point cloud data sent by the detection host; wherein, the detection data includes vibration detection data, Doppler spectrum data and image detection data; the detection data is acquired based on monitoring data collected by the multiple detection devices; the monitoring data includes raw vibration data, raw spectrum data and image detection data; The filtering module is used to filter the detection data and the point cloud data to obtain the data to be detected; The detection module is used to detect the data to be detected and determine the intrusion detection result; wherein, the intrusion detection result includes whether there is an intrusion behavior and an intrusion object, and when the intrusion object exists, the intrusion object category and intrusion action.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 22-23.

27. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 22-23.

Images