Intelligent three-dimensional electronic sand table monitoring system and method based on multi-source data fusion

Abstract

The invention discloses an intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion, and the system comprises a multi-source heterogeneous sensor module which is used for accessing a multi-source heterogeneous sensor; the intelligent analysis and fusion module is used for carrying out preliminary classification and recognition on targets captured by different sensors based on a YOLOv11 target classification recognizer, carrying out comprehensive judgment on preliminary classification and recognition results from different sensors by adopting a decision-level fusion algorithm based on confidence weighting, generating a new target recognition result, and sending the new target recognition result to a cloud server; effective targets and invalid alarms are distinguished; the space positioning module is used for measuring the position of a sensor and calibrating the sensor in a three-dimensional electronic sand table, and inferring the position of an effective target provided by the intelligent analysis and fusion module by utilizing an LBS base station positioning technology and combining with the position of the existing sensor; and the three-dimensional electronic sand table visualization module is used for visualizing the effective target.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of security monitoring and spatial information technology, and in particular to an intelligent monitoring system combining three-dimensional GIS, target recognition and digital twin technology for special scenarios such as military areas, prisons and energy facilities, specifically a multi-source data fusion-based intelligent three-dimensional electronic sand table monitoring system and method. BACKGROUND

[0002] Traditional monitoring systems rely on manual identification, with a high comprehensive false alarm rate (> 50%), and cannot realize spatial linkage of multi-source data (video / infrared / sensor, etc.), two-dimensional electronic maps lack three-dimensional spatial positioning capability, it is difficult to dynamically simulate complex scene emergency plans, and heterogeneous sensor data protocols are not compatible, resulting in low data fusion efficiency. SUMMARY

[0003] To overcome the shortcomings of the prior art, the present application provides a multi-source data fusion-based intelligent three-dimensional electronic sand table monitoring system and method, which intelligently analyzes and fuses multi-source data, combines spatial positioning, effectively improves the accuracy of monitoring target recognition, significantly reduces the false alarm rate, and supports rapid locking and processing of alarm targets in a three-dimensional electronic sand table.

[0004] According to an aspect of the present application, a multi-source data fusion-based intelligent three-dimensional electronic sand table monitoring system is provided, comprising: A multi-source heterogeneous sensor module for accessing multi-source heterogeneous sensors; An intelligent analysis and fusion module for performing preliminary classification and identification of targets captured by different sensors based on a YOLOv11 target classification identifier, and using a confidence-weighted decision fusion algorithm to comprehensively judge the preliminary classification and identification results from different sensors, generate new target recognition results, and distinguish between valid targets and invalid alarms; A spatial positioning module for measuring sensor positions and calibrating in a three-dimensional electronic sand table, using LBS base station positioning technology combined with existing sensor positions to infer the positions of valid targets provided by the intelligent analysis and fusion module; A three-dimensional electronic sand table visualization module for visualizing the valid targets.

[0005] As a further technical solution, the multi-source heterogeneous sensors include visible light cameras, infrared sensors, unmanned aerial vehicle sensors, Internet of Things terminals and optical fiber vibration sensors.

[0006] As a further technical solution, the fusion decision process of the intelligent analysis and fusion module is represented as follows: , Wherein, representing the final fusion judgment result, representing the decision-level fusion strategy, representing the target recognition result after Kalman filtering denoising of different sensors, the dashed box in the upper right corner representing the feature symbols of different sensors, V represents the visible light camera feature, IN represents the infrared sensor feature, UA represents the unmanned aerial vehicle sensor feature, and RA represents the optical fiber vibration sensor feature, representing the Internet of Things terminal data feature, wherein the video feature is extracted according to key frames.

[0007] As a further technical solution, the spatial positioning module comprises: a measurement unit for measuring the positions of various sensors by using real-time difference positioning (RTK) technology and calibrating in a three-dimensional electronic sand table; a calculation unit for inferring the position of an effective target provided by the intelligent analysis and fusion module by using LBS base station positioning technology in combination with the known sensor positions; and a conversion unit for converting screen coordinates displayed by the monitoring system into geographic coordinates.

[0008] As a further technical solution, when inferring the position of an effective target provided by the intelligent analysis and fusion module, the calculation unit further executes the following instructions: obtain two independent time differences formed by at least three base stations; based on the LBS base station positioning technology, in combination with the known sensor positions and the time differences, construct two independent positioning solution equations; solve the positioning solution equations to obtain the position of the effective target.

[0009] As a further technical solution, when converting screen coordinates displayed by the monitoring system into geographic coordinates, the conversion unit realizes this by using the following formula: , wherein, represents the geographic coordinates, T represents a projection matrix, represents the screen coordinates, and C represents camera parameters.

[0010] According to an aspect of the present disclosure, an intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion is provided, comprising: obtaining target information collected by multi-source heterogeneous sensors; based on a YOLOv11 target classification identifier, respectively performing preliminary classification and identification on targets captured by different sensors, and using a decision-level fusion algorithm based on confidence weighting to comprehensively judge the preliminary classification and identification results from different sensors, generate new target recognition results, and distinguish effective targets from invalid alarms; Acquire the position of the multi-source heterogeneous sensor and calibrate in the three-dimensional electronic sand table, utilize the LBS base station positioning technology to combine the existing sensor position, deduce the position of the effective target; Visualize the effective target.

[0011] According to an aspect of the specification of the present application, a non-transitory computer readable storage medium is provided, which stores computer instructions, the computer instructions causing the computer to perform the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion.

[0012] According to an aspect of the specification of the present application, an electronic device is provided, comprising: a memory for storing a computer program; a processor for implementing the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion when executing the computer program.

[0013] According to an aspect of the specification of the present application, a computer program product is provided, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion.

[0014] Compared with the prior art, the present application has the beneficial effects that: The present application constructs a full-link intelligent system from "perception-cognition-positioning-visualization" through the organic cooperation of each module, effectively solves the series of problems such as "data island, high false alarm rate, lack of spatial information, non-intuitive situation, and lack of decision support" existing in the traditional security monitoring system, and significantly improves the overall efficiency of global monitoring, intelligent early warning and accurate decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 The architecture schematic diagram of the intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion provided by the embodiments of the present application.

[0017] Figure 2 The data flow conversion schematic diagram of the intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion provided by the embodiments of the present application.

[0018] Figure 3A target space positioning schematic diagram provided by the embodiment of the present application.

[0019] Figure 4 A three-dimensional electronic sand table display interface schematic diagram provided by the embodiment of the present application.

[0020] Figure 5 An architecture schematic diagram of an electronic device provided by the embodiment of the present application.

[0021] Figure 6 A flowchart of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion provided by the embodiment of the present application. DETAILED DESCRIPTION

[0022] The terms "comprising" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to the steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0023] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form new technical schemes, and the combination is not restricted by the order of steps and / or structure composition mode, but must be based on the implementation by those skilled in the art. When the combination of technical schemes contradicts each other or cannot be implemented, it should be considered that the combination of technical schemes does not exist, and is not within the protection scope of the present application.

[0024] The embodiment of the present application provides a kind of intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion, its system architecture diagram as shown in Figure 1 Data flow chart as shown in Figure 2 Specifically as follows:

[0025] Multi-source heterogeneous sensor module is used to access multi-source heterogeneous sensors. The module includes the access of visible light camera, infrared sensor, unmanned aerial vehicle sensor, Internet of Things terminal and fiber optic vibration sensor, realizes the attribute hanging of video target and sensor data.

[0026] The intelligent analysis and fusion module includes a YOLOv11 target classification identifier and a multi-sensor fusion unit. The YOLOv11 target classification identifier is used for preliminary classification and identification of targets captured by different sensors.

[0027] The YOLOv11 target detection model is composed of a Backbone network layer, a Neck network layer and a head network layer.

[0028] The Backbone network layer is the core part of the YOLOv11 target detection model, which is composed of a CBS (Conv-BN-SiLU) basic module, a C3k2 module, a SPPF (Spatial Pyramid Pooling Fast) module and a C2PSA (Cross-Level Pyramid Slice Attention) module, and is mainly responsible for extracting image features. Among them, layers 1, 2, 4, 6 and 8 are CBS basic modules, layers 3, 5, 7 and 9 are C3k2 modules, and layers 10 and 11 are SPPF module and C2PSA module respectively. The CBS basic module is composed of a standard convolution layer (Conv), a batch normalization layer (BN) and a SiLU activation function layer. The C3K2 module, which inputs a part through ordinary convolution and another part through multiple C3k (when C3k parameter is True) or Bottleneck structure, finally fuses the results of the two branches through 1×1 convolution. The SPPF module is composed of a convolution layer and multiple serial 5×5 max pooling layers. The C2PSA module introduces a PSA (Position-Sensitive Attention) attention mechanism in C2f (Concatenation with Fusion).

[0029] The Neck network layer is mainly responsible for fusing and optimizing multi-scale features extracted from the backbone network to provide higher quality feature maps for the detection head. This part adopts an up-sampling fusion + down-sampling enhancement structure. In the fusion process, the deep layer feature size is enlarged by up-sampling to align with the shallow layer feature, and then further processed by the C2PSA module to gradually generate higher resolution feature maps. Starting from the highest resolution feature map generated from the fusion path, the feature size is reduced by convolution and down-sampling (convolution with a stride of 2), and then sequentially spliced with the features of the next level, and then the C3k2 module is used to strengthen the feature expression, and finally three feature maps of different scales are outputted for detecting small, medium and large targets.

[0030] The head network layer is responsible for generating the final prediction of target detection and classification, which is composed of three detection heads, each of which is composed of two CBS blocks + 1×1 convolution and two deep separable convolutions DWConv + 1×1 convolution in parallel.

[0031] The multi-sensor fusion unit receives the preliminary identification results from different sensors (each result contains target type, confidence, etc.), denoises using Kalman filtering, and finally generates a more reliable and accurate target identification result by comprehensively judging the analysis results from different sensors using a decision-level fusion algorithm based on confidence weighting, and divides it into effective targets and invalid alarms, wherein the effective targets include people, vehicles, animals, and the invalid alarms include vegetation shaking and weather influence. This method effectively avoids the problem of high false alarm rate caused by relying on a single sensor in traditional monitoring systems.

[0032] Kalman filtering is a recursive process, each cycle includes prediction and update. The prediction is based on the optimal estimate of the last time to predict the state at the current time, which is expressed as follows: Among them, represents the predicted value, F represents the state transition matrix, represents the posteriori estimate, represents the priori estimate variance, represents the uncertainty of the last time estimate, Q represents the process noise covariance. The update is to use the current actual observation value to correct the predicted value to get a better final estimate, which is expressed as follows: Among them, represents the Kalman gain, H represents the observation matrix, R represents the observation noise covariance, represents the posteriori state estimate at the current time (the optimal result after denoising), the current observation value, represents the updated uncertainty, represents the unit matrix.

[0033] The fusion decision process can be expressed as follows: Among them, represents the final fusion judgment result, represents the decision-level fusion strategy based on confidence weighting, ​​​​​​The target recognition results of different sensors after Kalman filtering denoising are represented, and the dashed box in the upper right corner represents the feature symbols of different sensors, V represents a visible light camera feature, IN represents an infrared sensor feature, UA represents a UAV sensor feature, RA represents a fiber optic vibration sensor feature, and IT represents an Internet of Things terminal data feature, wherein the video features are extracted according to key frames.

[0034] The process of the decision-level fusion strategy based on confidence weighting is as follows: Each sensor i outputs a decision (category) and a confidence . The confidence is a number between 0 and 1, indicating the degree of certainty of the classifier for its decision; For each possible category , a fused score is calculated, that is, the confidences of all classifiers whose decisions are are added up; The final fused decision selects the category with the highest score.

[0035] The example diagram of the spatial positioning of the embodiment of the application is referred to Figure 3 The spatial positioning module measures the positions of devices such as cameras and sensors by using RTK, and accurately calibrates in a three-dimensional electronic sand table. According to the existing positions of sensors such as cameras, the LBS base station positioning technology infers the positions of effective targets provided by the intelligent analysis and fusion module in the screen, and converts the screen coordinates displayed by the monitoring system into geographic coordinates, so as to facilitate users to quickly lock and process alarm targets.

[0036] The principle of LBS base station positioning is to use a base station with a known position to inversely deduce the coordinates of an unknown position. Its mathematical basis is the triangulation method, which can be expressed by the following formula: , wherein, represents the time difference of the device reaching base station 1 and base station 2, and represent the real distances of the device reaching base station 1 and 2, and represent the position coordinates of the effective target, and x and y represent the coordinates of the base station (the dashed box with a subscript in the lower corner represents different subscripts, such as 1 or 2), and x and y need to be solved by two independent equations, that is, at least three base stations are needed to form two independent time differences.

[0037] Converting screen coordinates into geographic coordinates means converting alarm coordinates provided by the monitoring system into geographic coordinates to provide positioning support for subsequent early warning, and the conversion process can be expressed by the following formula: , wherein, represents a geographical coordinate, T represents a projection matrix, represents a screen coordinate, C represents a camera parameter.

[0038] a visualization module visualizes the execution information of the above steps on a three-dimensional electronic sand table, and a three-dimensional electronic sand table display interface of an embodiment of the present application is shown in Figure 4 The three-dimensional electronic sand table is constructed based on a Cesium engine, supports loading of DEM, DOM and other data, can visualize targets acquired by multiple source sensors and subjected to intelligent analysis and positioning, and can realize functions such as two-dimensional and three-dimensional electronic map display, viewpoint navigation, two-dimensional and three-dimensional command plotting, three-dimensional measurement, digital twin deduction and the like.

[0039] In an example embodiment, based on the same inventive concept as the foregoing system embodiment, an intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion is also included, as shown in Figure 6 , comprising:

[0040] S1, acquiring target information collected by multi-source heterogeneous sensors by using a multi-source heterogeneous sensor module, and sending the target information to a data processing end of a monitoring system.

[0041] S2, in the data processing end, by an intelligent analysis and fusion module, based on a YOLOv11 target classification identifier, respectively performing preliminary classification and identification on targets captured by different sensors, and using a decision-level fusion algorithm based on confidence weighting to comprehensively judge preliminary classification and identification results from different sensors, generate new target identification results, and distinguish valid targets from invalid alarms.

[0042] S3, acquiring positions of the multi-source heterogeneous sensors by a spatial positioning module and calibrating in a three-dimensional electronic sand table, and inferring positions of the valid targets by using LBS base station positioning technology in combination with existing sensor positions.

[0043] S4, visualizing and displaying the valid targets by a visualization module.

[0044] In an example embodiment, a computer readable storage medium is also included, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of the foregoing intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion.

[0045] As Figure 5As shown, in an exemplary embodiment, an electronic device is also included, comprising at least one processor, at least one memory and at least one communication bus. The communication bus is used to connect the data acquisition device and the monitoring system, for transmitting the collected data to the monitoring system, and receiving and executing control instructions from the monitoring system.

[0046] The memory stores a computer program, and the computer program includes computer readable instructions. The processor calls the computer readable instructions stored in the memory through the communication bus, and executes the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion.

[0047] In an exemplary embodiment, a computer program product is provided, comprising computer programs / instructions that, when executed by a processor, implement the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion.

[0048] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0049] In summary of the embodiments, the present application discloses an intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion, comprising: a multi-source heterogeneous sensor module for accessing visible light, infrared, unmanned aerial vehicle, radar and Internet of Things terminal and other sensor data; an intelligent analysis and fusion module using a YOLOv11 target recognition classifier for preliminary identification, and through a decision-level fusion algorithm based on confidence weighting, the multi-source recognition results are analyzed and verified in coordination, and the effective target and invalid alarm are accurately distinguished; a spatial positioning module combining RTK and LBS base station positioning technology to realize accurate mapping of the geographical coordinates of the device and the target; a three-dimensional electronic sand table visualization module based on Cesium engine construction, supporting the visualization integration and interactive analysis of multi-source monitoring data and recognition results.

[0050] The present application successfully identifies real threats in harsh environments through multi-source sensor cooperation and decision-level fusion, effectively filters false positives caused by vegetation obstruction and light interference, and improves the target positioning accuracy from the meter level of ordinary GPS to sub-meter level, providing accurate and reliable decision support for commanders, and significantly improving the intelligent level and rapid response capability of security.

[0051] It should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. An intelligent 3D electronic sand table monitoring system based on multi-source data fusion, characterized in that, include: Multi-source heterogeneous sensor module, used to connect multi-source heterogeneous sensors; The intelligent analysis and fusion module is used to perform preliminary classification and recognition of targets captured by different sensors based on the YOLOv11 target classifier, and to use a confidence-weighted decision-level fusion algorithm to comprehensively judge the preliminary classification and recognition results from different sensors, generate new target recognition results, and distinguish between valid targets and invalid alarms. The spatial positioning module is used to measure the sensor position and calibrate it in a three-dimensional electronic sand table. It uses LBS base station positioning technology combined with the existing sensor position to infer the position of the effective target provided by the intelligent analysis and fusion module. The three-dimensional electronic sand table visualization module is used to visualize the effective targets.

2. The intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion according to claim 1, characterized in that, The multi-source heterogeneous sensors include: visible light cameras, infrared sensors, drone sensors, IoT terminals, and fiber optic vibration sensors.

3. The intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion according to claim 1, characterized in that, The fusion decision-making process of the intelligent analysis and fusion module is represented as follows: ， in, This represents the final fusion judgment result. Represents a decision-level integration strategy. The dashed box in the upper right corner represents the target recognition results after Kalman filtering denoising from different sensors. The symbols represent the characteristics of different sensors: V represents visible light camera characteristics, IN represents infrared sensor characteristics, UA represents drone sensor characteristics, and RA represents fiber optic vibration sensor characteristics. This represents the data characteristics of IoT terminals, where video features are extracted based on keyframes.

4. The intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion according to claim 1, characterized in that, The spatial positioning module includes: a measurement unit for measuring the position of each sensor using real-time differential positioning (RTK) technology and calibrating it in a three-dimensional electronic sand table; a calculation unit for inferring the position of the effective target provided by the intelligent analysis and fusion module by using LBS base station positioning technology combined with the existing sensor positions; and a conversion unit for converting the screen coordinates displayed by the monitoring system into geographic coordinates.

5. The intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion according to claim 4, characterized in that, When inferring the location of the valid target provided by the intelligent analysis and fusion module, the computing unit also executes the following instructions: Obtain the two independent time differences formed by at least three base stations; Based on LBS base station positioning technology, and combining the known sensor locations and the time difference, two independent positioning solution equations are constructed. Solve the positioning equation to obtain the position of the effective target.

6. The intelligent three-dimensional electronic sand table monitoring system based on multi-source data fusion according to claim 4, characterized in that, The conversion unit uses the following formula to convert the screen coordinates displayed by the monitoring system into geographic coordinates: ， in, T represents geographic coordinates, and T represents the projection matrix. C represents screen coordinates, and C represents camera parameters.

7. A method for monitoring intelligent 3D electronic sand table based on multi-source data fusion, characterized in that, include: Acquire target information collected by multi-source heterogeneous sensors; Based on the YOLOv11 target classifier, the system performs preliminary classification and recognition of targets captured by different sensors. Then, it uses a confidence-weighted decision-level fusion algorithm to comprehensively judge the preliminary classification and recognition results from different sensors, generate new target recognition results, and distinguish between valid targets and invalid alarms. The positions of multi-source heterogeneous sensors are acquired and calibrated in a three-dimensional electronic sand table. The position of the effective target is inferred by combining LBS base station positioning technology with the existing sensor positions. Visualize the effective targets.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion as described in claim 7.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion as described in claim 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the intelligent three-dimensional electronic sand table monitoring method based on multi-source data fusion as described in claim 7.

Images