A fusion monitoring method, device, system and medium

Abstract

The application discloses a kind of fusion monitoring method, device, system and medium, and through the video stream fusion perception data of acquisition, multi-mode heterogeneous internet of things technology application etc., not only accurate equipment itself abnormal monitoring can be realized, also reliable data is provided for edge computing and local decision, to directly drive local execution equipment, effectively improve decision response speed.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 63 / 436,111, filed on December 30, 2022, the entirety of which is incorporated herein by reference. Technical Field

[0003] This invention relates to the field of video surveillance technology, and in particular to a fusion monitoring method, device, system and medium. Background Technology

[0004] Surveillance equipment is a crucial component of security technology systems, with a wide range of applications. It's used not only for security in industries such as finance, cultural heritage, military, and jewelry stores, but also for safety production and on-site management in sectors like transportation, healthcare, airports, train stations, ports, and factories. Surveillance equipment primarily uses cameras and auxiliary devices to capture real-time video of the monitored scene, recording changes in the scene in real time to facilitate appropriate decision-making.

[0005] As the core of surveillance equipment, cameras are crucial to the functionality and security of the equipment. Currently, cameras in surveillance equipment have limited functions, typically only collecting video data of the scene and sending it to the server for decision-making, making it difficult to reliably monitor their own equipment status, resulting in slow decision response and reduced equipment reliability. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a fusion monitoring method, device, system and medium, which aims to achieve reliable monitoring of the device's own status and fusion monitoring of local multi-sensor data, thereby improving decision response speed.

[0007] The first aspect of this invention provides a fusion monitoring method, comprising:

[0008] Acquire the operational status data and video streams of the target monitoring equipment;

[0009] Receive sensing data and control status data collected by local sensing devices;

[0010] Real-time prediction is performed by fusing the working status data, video stream, perception data, and control status data using a trained deep learning model.

[0011] Based on real-time prediction results and working status data, equipment anomalies are identified, and corresponding early warning actions are taken when equipment anomalies are detected.

[0012] Edge computing and local decision-making are performed on real-time prediction results and perceived data, and the corresponding local execution devices and / or target monitoring devices are driven to perform specified actions based on the results of the decisions.

[0013] In one embodiment, the method further includes:

[0014] The target monitoring device establishes a secure connection channel with the server to securely transmit all communication data.

[0015] In one embodiment, the method further includes:

[0016] Based on the operating status data, predict the normal power consumption range and trend of the target monitoring device;

[0017] The actual power consumption of the target monitoring device is statistically analyzed within a preset time period to obtain the trend of actual power consumption changes.

[0018] When the actual power is not within the normal power consumption range, or when the actual power consumption trend does not match the predicted trend, the target monitoring device is powered according to the preset power supply strategy.

[0019] In one embodiment, controlling the power supply to the target monitoring device according to a preset power supply strategy specifically refers to:

[0020] When power consumption is abnormal, the power supply to the target monitoring device is cut off, and the power is restored after a preset power-off time.

[0021] Continue to observe whether the target monitoring device has abnormal power consumption. If so, cut off the power supply to the target monitoring device again, and restore power by increasing the preset power-off time by a preset step size. Repeat this process until the preset power-off time reaches the maximum power-off time.

[0022] In one embodiment, the method further includes:

[0023] Enable port whitelisting to monitor port connections of the target monitoring device and block port connections that are not in the port whitelist;

[0024] By parsing and tracking the communication protocol of the target monitoring device, the legitimate new ports that the target monitoring device needs to enable are identified, and the ports are added to a temporary whitelist. Further protocol parsing and tracking are performed, and when the port is no longer in use, it is removed from the temporary whitelist.

[0025] In one embodiment, the sensing data and control status data are received through multiple local sensor interfaces.

[0026] In one embodiment, driving instructions are output through multiple local execution interfaces to drive the corresponding local execution device to perform a specified action.

[0027] A second aspect of the present invention provides a fusion monitoring device, comprising:

[0028] The acquisition module is used to acquire the working status data of the target monitoring device and the collected video stream;

[0029] The receiving module is used to receive sensing data and control status data collected by local sensing devices;

[0030] The fusion prediction module is used to make real-time predictions of the working status data, video stream, perception data, and control status data using a trained deep learning model.

[0031] The anomaly identification module is used to identify equipment anomalies based on real-time prediction results and working status data, and to perform corresponding early warning actions when equipment anomalies are found.

[0032] The fusion decision module is used to perform edge computing and local decision-making on real-time prediction results and perception data, and drive the corresponding local execution device and / or target monitoring device to perform specified actions based on the results of the decision.

[0033] A third aspect of the present invention provides a converged monitoring system, the system comprising at least one processor; and,

[0034] A memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the aforementioned fusion monitoring method.

[0036] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the above-described fusion monitoring method.

[0037] This invention discloses a fusion monitoring method, device, system, and medium. Compared with the prior art, the embodiments of this invention use deep learning to predict by fusing sensor data from collected video streams and information such as device operating status. This not only enables accurate monitoring of device anomalies but also provides reliable data for edge computing and local decision-making, directly driving local execution devices and effectively improving decision response speed. Attached Figure Description

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0039] Figure 1 A system framework diagram for integrated monitoring;

[0040] Figure 2 A flowchart of the fusion monitoring method provided in an embodiment of the present invention;

[0041] Figure 3 This is a flowchart of a device status monitoring process based on deep learning.

[0042] Figure 4 Here is a flowchart of a deep learning-based image and fusion sensing algorithm.

[0043] Figure 5 This is a schematic diagram of the functional modules of the fusion monitoring device provided in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the hardware structure of the fusion monitoring system provided in an embodiment of the present invention;

[0045] Figure 7 This is a diagram of the next-generation Internet of Things (IoT) architecture. Detailed Implementation

[0046] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.

[0047] Surveillance equipment is a crucial component of security technology systems, with a wide range of applications. It's used not only for security in industries such as finance, cultural heritage, military, and jewelry stores, but also for safety production and on-site management in sectors like transportation, healthcare, airports, train stations, ports, and factories. Surveillance equipment primarily uses cameras and auxiliary devices to capture real-time video of the monitored scene, recording changes in the scene in real time to facilitate appropriate decision-making.

[0048] As the core of surveillance equipment, cameras are crucial to the functionality and security of the system. Currently, cameras in surveillance equipment are often functionally limited and lack security. For example, existing cameras transmit data without encryption, making them vulnerable to security attacks; they have few interfaces, making it difficult to integrate other sensor data for edge intelligence. Additional sensor devices often require server involvement, resulting in slow response times and inability to function properly when the network is down; cameras typically only collect video data of the scene and send it to the server for decision-making, unable to monitor their own physical state, such as tilt detection, orientation detection, or internal temperature and humidity detection; conventional detection of device operating status is prone to false alarms and missed alarms due to varying operating conditions, such as increased current during focusing, tilting during rotation, increased bandwidth when receiving two simultaneous streams, and decreased bitrate when using certain video compression formats; devices or combinations of devices with edge AI capabilities generally only perform calculations on video content, without integrating data from multiple sensors, and the calculation results must be sent to the server for decision-making, leading to slow response times and inability to make decisions when the network is down.

[0049] To address the problems existing in the cameras of the aforementioned monitoring equipment, this invention proposes a fusion monitoring method, which can be applied to... Figure 1 In the system framework shown, the system framework implements the fusion monitoring method provided in the embodiments of the present invention through processors, network ports, power management, security chips, WiFi modules, backup communication modules, broadcast output modules, input / output interfaces, etc., so that based on the video stream collected by the target monitoring device, information such as the working status of the device and the sensing data collected by external sensing devices are fused to perform deep learning prediction. This not only enables accurate monitoring of the device's own anomalies, but also provides reliable data for edge computing and local decision-making, so as to directly drive the local execution device to perform corresponding actions, effectively improving decision quality and response speed.

[0050] like Figure 2 The diagram shows a flowchart of the fusion monitoring method provided in an embodiment of the present invention. The method specifically includes the following steps:

[0051] S101. Obtain the working status data of the target monitoring equipment and the collected video stream;

[0052] S102, Receive sensing data and control status data collected by local sensing devices;

[0053] S103. Real-time prediction by fusing the working status data, video stream, perception data and control status data using a trained deep learning model;

[0054] S104. Identify equipment anomalies based on real-time prediction results and working status data, and execute corresponding early warning actions when equipment anomalies are present.

[0055] S105. Perform edge computing and local decision-making on the real-time prediction results and perception data, and drive the corresponding local execution device and / or target monitoring device to perform the specified action based on the result of the decision.

[0056] In this embodiment, the target monitoring equipment specifically refers to various monitoring devices with cameras in scenarios requiring video surveillance, such as water accumulation monitoring devices in smart city applications, flame monitoring devices in forest fire prevention applications, license plate recognition devices in smart parks, and so on.

[0057] Once the target monitoring device is activated, it acquires the video stream it collects, as well as the device's operating status data. This operating status data includes, but is not limited to, network IP and port usage, protocol data, data traffic, video stream, device configuration data, real-time device motion information, power consumption, and device physical status. Physical status monitoring further includes tilt detection, orientation detection, and internal temperature and humidity detection. Furthermore, it can detect the camera's cruise zoom status using a compass, gyroscope, and position sensor. In addition, it receives sensing data and control status data collected by local sensing devices. Local sensing devices specifically refer to various types of sensors, including but not limited to temperature sensors, humidity sensors, gas sensors, pressure sensors, vibration sensors, distance sensors, infrared sensors, optical sensors, and displacement sensors. The specific types are not limited here, as long as they can sense the measured information and transform it into electrical signals or other required forms of information output according to a certain rule to meet the requirements of the sensor terminal device for information transmission, processing, storage, display, recording, and control.

[0058] By leveraging the NPU processing and AI algorithms built into the target monitoring device, a pre-trained deep learning model can be run to perform real-time predictions by fusing the operational status data, video stream, perception data, and control status data. During training and deployment, the deep learning model utilizes not only the video stream data captured by the camera but also the perception and control status data from external sensing devices, as well as the operational status data of the target monitoring device itself. This integration of model prediction with local perception and control results in more accurate predictions and superior performance.

[0059] Based on the model's prediction results, on the one hand, it can combine network IP and port usage, protocol data, data traffic, video streams, device configuration, real-time device motion information, power consumption, and device physical status data to identify device anomalies and confirm whether the device's operating status is abnormal. It can predict abnormal power consumption, abnormal traffic, network security threats, and abnormal physical states, enabling corresponding early warning actions to be taken when device anomalies are detected. For example... Figure 3 As shown, during the training phase, data related to the working status of the target monitoring equipment is continuously collected. After data cleaning, the data is saved as a historical dataset. A deep learning model for the working status of the equipment is trained using the historical dataset. By learning from the working status data and validating and optimizing the model locally, the trained deep learning model can generate corresponding prediction results based on the input working status data. Combining the real-time prediction results with the actual working status data, the model can identify equipment anomalies and generate warnings when the equipment is in an abnormal state. The warning data can be handed over to the edge decision program to generate corresponding warning processing actions. For example, the target device is currently pushing a video stream to the server at 1080p resolution and H.265 compression algorithm, and is cruising 20 points every 3 minutes. During the cruising process, horizontal rotation and vertical adjustment need to be controlled. The external ambient temperature is 26 degrees Celsius and the wind speed is 3 meters per second. Under these conditions, the deep learning algorithm can predict that the internal temperature of the device is about 35 degrees Celsius and the tilt angle of the device is between 20 and 70 degrees Celsius. If the actual physical state monitored exceeds this range, it indicates that there is an abnormality in the device, such as the casing being blocked, the device being unstable, or the device malfunctioning. In this case, the corresponding early warning actions will be executed in time, such as issuing audible and visual prompts, cutting off the power supply to the device, etc., to ensure the reliability of the device's own operating status.

[0060] On the other hand, the prediction results of deep learning models can serve as data support for edge computing and edge decision-making, such as... Figure 4 As shown, compared to conventional edge AI which only performs algorithmic predictions on single video data, the fusion monitoring method provided in this embodiment of the invention runs multiple edge AI models. During model training, it uses not only conventional image data but also external perception data and real-time external control status data. This combines model prediction with local perception and control, making the prediction results of fusion monitoring more accurate. For example, in license plate recognition applications, in addition to the video data collected by the camera, perception data such as ambient light intensity and rainfall data collected by sensors, as well as control status data of supplementary lighting equipment, are used as input parameters to the deep learning model for recognition and prediction. If the current external perception environment has a low recognition rate, the supplementary lighting intensity of the supplementary lighting equipment is controlled to achieve the optimal recognition rate.

[0061] After real-time prediction using a deep learning model, the prediction results are further processed by edge computing and local decision-making programs. Combining the results of edge AI, external perception data, cloud strategies, and control, local edge decisions, actions, and warnings are generated in real time. This can directly drive external execution devices and / or target devices to perform corresponding actions, with rapid response, continued operation even when offline, and control of video and image transmission based on algorithm results, reducing server load. For example, in smart city applications, edge AI algorithms from cameras can identify the depth of road water accumulation. When there is no water or minimal water accumulation, the identification result is sent to the server. When the water depth reaches level 1, roadside warning signs are activated and displayed "Road flooded, proceed with caution" to warn pedestrians and drivers to proceed with caution. When the water depth reaches level 2, the warning signs display "Deep water, danger, no passage," and roadblocks in the direction of oncoming traffic are automatically activated, and water pumps are started to drain the water, achieving rapid decision-making response based on accurate identification of the monitoring footage.

[0062] Among them, edge computing and local decision-making and cloud policies are controllable. For example, when the network is good, it can switch between local decision-making and cloud policies at will, or it can be preferred to use cloud policies. This allows the latest deployed policies to be obtained from the cloud server to control the local execution devices and target monitoring devices. At the same time, when the network is good, cloud policies and controls are obtained from the cloud server and stored locally as local decisions. This allows on-site decisions and action outputs to be made directly through local decisions when the network is poor or disconnected. This directly drives the corresponding local execution devices and / or target monitoring devices to perform specified actions, making the monitoring response faster and still working when the network is disconnected, thus improving the reliability of the monitoring system.

[0063] In one embodiment, the sensing data and control status data are received through multiple local sensor interfaces.

[0064] Traditional cameras have limited interfaces and are difficult to integrate with other sensing data to achieve edge intelligence. Using additional sensing devices often requires server involvement, resulting in slow response times and inability to function properly when the network is down. This embodiment, however, supports multiple sensor interfaces locally, directly receiving external sensing data transmitted from multiple sensors through these interfaces to achieve edge computing and edge decision-making functions that integrate external sensing data.

[0065] In one embodiment, driving instructions are output through multiple local execution interfaces to drive the corresponding local execution device to perform a specified action.

[0066] Since existing cameras usually only focus on video content acquisition and processing, other local devices often need to make decisions and control through a server, resulting in untimely responses and inability to act promptly when the network is poor. In this embodiment, multiple execution interfaces are supported locally, and drive instructions are output to corresponding local execution devices through the execution interfaces to drive the local execution devices to perform specified actions, directly controlling the local execution devices to perform actions to achieve local linkage response and improving the response speed.

[0067] In one embodiment, the method further includes:

[0068] The target monitoring device establishes a secure connection channel with the server to securely transmit all communication data.

[0069] In this embodiment, traditional camera communication is not encrypted, or only the video stream is encrypted while other data is not encrypted, resulting in a decrease in the security of the monitoring device and making it vulnerable to security attacks. To improve transmission security in this embodiment, the device has a security chip that can enhance the security of key and certificate storage and use. After the target monitoring device is powered on, it first establishes a secure connection channel with the server, and all communication data of the target monitoring device is encrypted and securely transmitted through this secure connection channel, ensuring the security of communication data. Specific encryption algorithms can be used, such as the Data Encryption Standard (DES for short), the Advanced Encryption Standard (AES for short), ECC, the Tiny Encryption Algorithm (TEA for short), and SM4 (an algorithm of national cryptography) to implement encryption and decryption. This embodiment does not limit this.

[0070] In one embodiment, the method further includes:

[0071] Predict the normal power consumption range interval and change trend of the target monitoring device according to the working state data;

[0072] Statistically calculate the actual power of the target monitoring device within a preset time period to obtain the actual power consumption change trend;

[0073] When the actual power is not within the normal power consumption range interval, or the actual power consumption change trend does not match the predicted change trend, perform power supply control on the target monitoring device according to a preset power supply strategy.

[0074] In this embodiment, based on a deep learning model, the normal power consumption range and trend of the device can be predicted according to the device's current working status data (such as the running protocol, video bitrate, focus adjustment action, cruise action, etc.). Then, the real-time power supply current and voltage are detected, and the real-time power is calculated. The actual power consumption trend is obtained by statistically analyzing the actual power over a period of time. If the actual power is not within the predicted normal power consumption range or the actual power consumption trend does not match the predicted trend, it is determined to be an abnormal power consumption. When the power consumption is abnormal, the device is powered according to a preset power supply strategy to avoid the device being in an abnormal power consumption state for a long time and to ensure the safety of the device power supply.

[0075] Furthermore, the target monitoring equipment is powered according to a preset power supply strategy, specifically:

[0076] When power consumption is abnormal, the power supply to the target monitoring device is cut off, and the power is restored after a preset power-off time.

[0077] Continue to observe whether the target monitoring device has abnormal power consumption. If so, cut off the power supply to the target monitoring device again, and restore power by increasing the preset power-off time by a preset step size. Repeat this process until the preset power-off time reaches the maximum power-off time.

[0078] In this embodiment, when power consumption is abnormal, the power supply can be temporarily cut off to reset the target device. After a certain preset power-off time (e.g., 10 seconds), the power supply is attempted to be restored. Then, the power consumption is observed again through the above process. If the abnormality still exists, the same power-off process is performed again, and the preset power-off time is increased by a preset step size before power is restored. This process is repeated to reset the device several times, with the power-off time gradually increased each time until the maximum power-off time is reached. For example, the first power-off is 10 seconds, the second is 20 seconds, the third is 30 seconds, the fourth is 60 seconds, and the fifth is 180 seconds. Subsequent power-off reset times are all 180 seconds, achieving efficient and reasonable power supply control.

[0079] In one embodiment, the method further includes:

[0080] Enable port whitelisting to monitor port connections of the target monitoring device and block port connections that are not in the port whitelist;

[0081] By parsing and tracking the communication protocol of the target monitoring device, the legitimate new ports that the target monitoring device needs to enable are identified, and the ports are added to a temporary whitelist. Further protocol parsing and tracking are performed, and when the port is no longer in use, it is removed from the temporary whitelist.

[0082] In this embodiment, a port whitelist is enabled to monitor port connections of the target monitoring device, filtering unauthorized connections and protocols to achieve firewall functionality. Specifically, it detects the data content and traffic of authorized ports on the target monitoring device; abnormal content and traffic can generate alarms, and further actions such as temporarily blocking the network or restarting the target device can be performed. Specifically, authorized ports in the whitelist transmit predetermined communication protocols and data; by analyzing the transmitted data, it can be determined whether it conforms to the predetermined communication protocols and data, as well as the predictable traffic. For example, if port 1935 is opened to transmit video streams via the RTMP protocol, all data on this port should conform to the RTMP protocol, and its content should be compressed video data, such as containing I-frames, P-frames, and B-frames. Furthermore, by obtaining the target device's configuration through the API, information such as video resolution, compression format, and compression rate can be obtained. Therefore, the data traffic on this port should match the traffic expected in the video configuration. By enabling a port whitelist, connections to ports not on the whitelist will be blocked. Some functions of the target device will generate new ports and connections during operation. For example, a call function implemented using the SIP protocol might only have one port connected during standby. When a new call or dialing action is initiated, the server will allocate a new (dynamic, non-fixed) port. The target device uses this port to send and receive voice data with the server. In this case, it is necessary to parse the SIP protocol, store the dynamically allocated port in a temporary whitelist, and continuously monitor whether the data on this port is voice data. When this port is no longer in use, it should be removed from the temporary whitelist. This achieves dynamic port connection monitoring, ensuring device security without affecting its normal data transmission function.

[0083] It should be noted that there is no necessary sequential order among the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn. For example, steps S101 and S102 may be executed in parallel, that is, simultaneously acquiring working status data, video stream, sensing data and control status data; or steps S101 may be executed first and then steps S102 may be executed, or steps S102 may be executed first and then steps S101 may be executed. This embodiment does not limit this.

[0084] For example, one application scenario of the fusion monitoring method provided in this embodiment of the invention is a multi-dimensional tree monitoring scenario. A multi-dimensional tree monitoring terminal includes: a tilt sensor, a tree circumference sensor, a soil nutrient sensor, a pyroelectric infrared sensor, a radar sensor, a microphone, a horizontal camera and a sky camera that apply the fusion monitoring method. The tilt sensor is used to monitor the tilt status of the trunk and branches; the tree circumference sensor is used to monitor the diameter of the tree and obtain the tree's growth status; the soil nutrient sensor is used to monitor the tree's growth environment; the pyroelectric infrared sensor and the radar sensor are used to detect whether someone is approaching; the microphone is used to monitor tree theft and identify sawing sounds and tree falling sounds; the horizontal camera is used to identify human activities and capture images of tree damage; and the sky camera is used to identify the development of tree branches and leaves.

[0085] The usage of this multi-dimensional tree monitoring terminal includes: when pyroelectric infrared and / or sensor radar detects someone approaching, it activates a horizontal camera to monitor human activity; simultaneously, a microphone monitors for sawing sounds or a tilt sensor detects tree tilting and sends an alarm to the system; periodically collects data from tree circumference sensors, soil nutrient sensors, and sky camera to analyze tree growth; and inputs the sensor data into a deep learning engine, which, after training with a large amount of data, obtains environmental quality improvement values, NOVI data, forest value data, carbon sequestration data, etc.

[0086] This multi-dimensional tree monitoring terminal can perform real-time prediction by fusing operational status data, acquired video streams, sensor data, and control status data from various sensors using a deep learning model. By combining the real-time prediction results with operational status data, it can identify equipment anomalies and execute corresponding early warning actions when anomalies are detected. Furthermore, the multi-dimensional tree monitoring terminal itself has edge computing capabilities, enabling local decision-making and better supporting the operation of multi-mode heterogeneous IoT. Specifically, it runs multiple edge AI models, using not only conventional image data but also external sensor data and real-time external control status data during model training. This combines model predictions with local perception and control, making the prediction results of the fused monitoring more accurate. The prediction results are then fed to edge computing and local decision-making programs for further processing. By combining the results of edge AI, external sensor data, cloud policies, and control, it generates local edge decisions, actions, and early warnings in real time, directly driving external execution devices and / or target devices to perform corresponding actions, improving the speed of monitoring decision response. For example, pyroelectric infrared and / or inductive radar can be used to detect if someone is approaching; horizontal cameras can capture images of tree damage, such as illegal logging; microphones can monitor tree theft, such as identifying sawing sounds and tree falling sounds; and AI analysis can be performed on sensor data, tree species data, forest grid data, and image information. Based on the AI ​​analysis results and external perception data, it can quickly deduce whether there is illegal logging and control the alarm to sound for rapid response and early warning.

[0087] For example, one application scenario of the fusion monitoring method provided in this embodiment of the invention is a weakly obstructive road gate scenario for flooding. A weakly obstructive water curtain road gate system includes: a submersible pump, a water filtration device, a dye addition device, a water curtain projector, spray pipes, nozzles, a display screen, a loudspeaker, a floodwater sensor, and a smart camera using the fusion monitoring method. The submersible pump pumps floodwater from the road surface to the sprinkler system. Before entering the pump, the floodwater passes through the water filtration device to remove large impurities. During this process, the dye addition device automatically adds dye. The sprinkler system delivers water to various nozzles, each with different inner diameters and angles to ultimately form a complete water curtain. The water curtain projector projects warning signs and text onto the water curtain. To reduce costs, fixed content is preferred, with flashing to enhance the warning effect. The display screen and loudspeaker provide warnings. The floodwater sensor detects the current water depth on the road surface, and the smart camera collects on-site images. The system integrates on-site images, operational status data, sensor data, and control status data for fusion monitoring.

[0088] The operation of this low-obstruction water curtain road gate system is as follows: Water depth sensors and cameras detect road water accumulation in real time. When the water depth is shallow and safe, LED displays and loudspeakers show the current water depth and advise passage slowly. When the water is deep and passage is dangerous, LED displays and loudspeakers warn of water depth danger and prohibit passage. Submersible pumps are activated to form a water curtain on the road surface, preventing vehicles and pedestrians from passing. Water curtain projectors can project warning icons or prohibition text onto the water curtain to enhance the warning effect. Smart terminals send on-site data and images to the service and receive control commands from the service. On-site cameras continuously monitor the road conditions, and the integrated perception data is used to identify vehicle traffic conditions and whether any vehicles or pedestrians are trapped through AI algorithms. In case of a crisis, high-priority alarm information is pushed to the server. AI algorithms can also be used to integrate monitoring of road water depth and water flow direction. Because the water curtain does not have strong obstruction characteristics, it will not damage vehicles if they cannot stop in time. Emergency rescue vehicles and other special vehicles that need to pass can directly pass through the water curtain. When the water depth drops to a safe level, stop the water pump and change the warning messages on the LED screen.

[0089] This low-obstruction water curtain barrier system uses a deep learning model to fuse operational status data, acquired video streams, sensor data, and control status data in real time for prediction. By combining real-time prediction results with operational status data, it can identify equipment anomalies and execute corresponding early warning actions when anomalies are detected. Furthermore, the system possesses edge computing capabilities, enabling local decision-making and better supporting the operation of multi-mode heterogeneous IoT. Specifically, it runs multiple edge AI models, using not only conventional image data but also external sensor data and real-time external control status data during model training. This combines model predictions with local perception and control, resulting in more accurate predictions from the fused monitoring system. The prediction results are then further processed by edge computing and local decision-making programs. By combining the results of edge AI, external sensor data, cloud policies, and control, it generates local edge decisions, actions, and early warnings in real time, directly driving external execution devices and / or target devices to perform corresponding actions, thus improving the speed of monitoring decision response. For example, by using data from water depth sensors and on-site images from cameras to identify the depth of road flooding, the identification results are sent to the server when there is no flooding or only a small amount of flooding, reducing the server's workload. When the flooding reaches depth 1, the roadside warning signs are activated and display "Flooded Road, proceed with caution" to warn pedestrians and drivers to proceed with caution. When the flooding reaches depth 2, the warning signs display "Deep Water, Danger, No Passage," and the roadblocks in the direction of oncoming traffic are automatically activated, and water pumps are started to drain the water, enabling rapid decision-making and response based on accurate identification of the monitoring footage.

[0090] It is understood that the fusion monitoring method provided in this embodiment of the invention can also be applied to scenarios requiring video surveillance, such as smart cities, forest fire prevention, and smart parks, and can also be used to monitor other networked devices, such as IP broadcasting, outdoor LED screens, and IP intercom devices.

[0091] Another embodiment of the present invention provides a fusion monitoring device, such as... Figure 5 As shown, device 1 includes: an acquisition module 11 for acquiring the working status data of the target monitoring device and the collected video stream; a receiving module 12 for receiving the sensing data and control status data collected by the local sensing device; a fusion prediction module 13 for making real-time predictions on the working status data, video stream, sensing data, and control status data using a trained deep learning model; and an anomaly identification module 14 for identifying device anomalies based on the real-time prediction results and working status data, and performing corresponding early warning actions when device anomalies are present.

[0092] The fusion decision module 15 is used to perform edge computing and local decision-making on real-time prediction results and perception data, and drive the corresponding local execution device and / or target monitoring device to perform specified actions based on the decision results.

[0093] The module referred to in this invention is a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the fusion monitoring execution process. For specific implementation methods of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0094] In one embodiment, device 1 further includes:

[0095] The secure connection module is used to establish a secure connection channel between the target monitoring device and the server to securely transmit all communication data.

[0096] In one embodiment, device 1 further includes:

[0097] The power consumption prediction module is used to predict the normal power consumption range and trend of the target monitoring device based on the working status data.

[0098] The power consumption statistics module is used to count the actual power of the target monitoring device within a preset time period and obtain the actual power consumption change trend.

[0099] The power supply control module is used to control the power supply of the target monitoring device according to a preset power supply strategy when the actual power is not within the normal power consumption range, or when the actual power consumption change trend does not match the predicted change trend.

[0100] In one embodiment, controlling the power supply to the target monitoring device according to a preset power supply strategy specifically refers to:

[0101] When power consumption is abnormal, the power supply to the target monitoring device is cut off, and the power is restored after a preset power-off time.

[0102] Continue to observe whether the target monitoring device has abnormal power consumption. If so, cut off the power supply to the target monitoring device again, and restore power by increasing the preset power-off time by a preset step size. Repeat this process until the preset power-off time reaches the maximum power-off time.

[0103] In one embodiment, device 1 further includes:

[0104] The interception module is used to enable a port whitelist to monitor port connections of the target monitoring device and to block port connections that are not in the port whitelist.

[0105] In one embodiment, the sensing data and control status data are received through multiple local sensor interfaces.

[0106] In one embodiment, driving instructions are output through multiple local execution interfaces to drive the corresponding local execution device to perform a specified action.

[0107] Another embodiment of the present invention provides a fusion monitoring system, such as Figure 6 As shown, system 10 includes:

[0108] One or more processors 110 and memory 120, Figure 6 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0109] Processor 110 is used to perform various control logics of system 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0110] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the fusion monitoring method in the embodiments of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the fusion monitoring method in the above method embodiments.

[0111] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the system 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] One or more units are stored in memory 120, and when executed by one or more processors 110, perform the following steps:

[0113] Acquire the operational status data and video streams of the target monitoring equipment;

[0114] Receive sensing data and control status data collected by local sensing devices;

[0115] Real-time prediction is performed by fusing the working status data, video stream, perception data, and control status data using a trained deep learning model.

[0116] Based on real-time prediction results and working status data, equipment anomalies are identified, and corresponding early warning actions are taken when equipment anomalies are detected.

[0117] Edge computing and local decision-making are performed on real-time prediction results and perceived data, and the corresponding local execution devices and / or target monitoring devices are driven to perform specified actions based on the decision results.

[0118] In one embodiment, the method further includes:

[0119] The target monitoring device establishes a secure connection channel with the server to securely transmit all communication data.

[0120] In one embodiment, the method further includes:

[0121] Based on the operating status data, predict the normal power consumption range and trend of the target monitoring device;

[0122] The actual power consumption of the target monitoring device is statistically analyzed within a preset time period to obtain the trend of actual power consumption changes.

[0123] When the actual power is not within the normal power consumption range, or when the actual power consumption trend does not match the predicted trend, the target monitoring device is powered according to the preset power supply strategy.

[0124] In one embodiment, controlling the power supply to the target monitoring device according to a preset power supply strategy specifically refers to:

[0125] When power consumption is abnormal, the power supply to the target monitoring device is cut off, and the power is restored after a preset power-off time.

[0126] Continue to observe whether the target monitoring device has abnormal power consumption. If so, cut off the power supply to the target monitoring device again, and restore power by increasing the preset power-off time by a preset step size. Repeat this process until the preset power-off time reaches the maximum power-off time.

[0127] In one embodiment, the method further includes:

[0128] Enable port whitelisting to monitor port connections of the target monitoring device and block port connections that are not in the port whitelist.

[0129] In one embodiment, the sensing data and control status data are received through multiple local sensor interfaces.

[0130] In one embodiment, driving instructions are output through multiple local execution interfaces to drive the corresponding local execution device to perform a specified action.

[0131] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, perform the following steps:

[0132] Acquire the operational status data and video streams of the target monitoring equipment;

[0133] Receive sensing data and control status data collected by local sensing devices;

[0134] Real-time prediction is performed by fusing the working status data, video stream, perception data, and control status data using a trained deep learning model.

[0135] Based on real-time prediction results and working status data, equipment anomalies are identified, and corresponding early warning actions are taken when equipment anomalies are detected.

[0136] Edge computing and local decision-making are performed on real-time prediction results and perceived data, and the corresponding local execution devices and / or target monitoring devices are driven to perform specified actions based on the decision results.

[0137] In one embodiment, the method further includes:

[0138] The target monitoring device establishes a secure connection channel with the server to securely transmit all communication data.

[0139] In one embodiment, the method further includes:

[0140] Based on the operating status data, predict the normal power consumption range and trend of the target monitoring device;

[0141] The actual power consumption of the target monitoring device is statistically analyzed within a preset time period to obtain the trend of actual power consumption changes.

[0142] When the actual power is not within the normal power consumption range, or when the actual power consumption trend does not match the predicted trend, the target monitoring device is powered according to the preset power supply strategy.

[0143] In one embodiment, controlling the power supply to the target monitoring device according to a preset power supply strategy specifically refers to:

[0144] When power consumption is abnormal, the power supply to the target monitoring device is cut off, and the power is restored after a preset power-off time.

[0145] Continue to observe whether the target monitoring device has abnormal power consumption. If so, cut off the power supply to the target monitoring device again, and restore power by increasing the preset power-off time by a preset step size. Repeat this process until the preset power-off time reaches the maximum power-off time.

[0146] In one embodiment, the method further includes:

[0147] Enable port whitelisting to monitor port connections of the target monitoring device and block port connections that are not in the port whitelist.

[0148] In one embodiment, the sensing data and control status data are received through multiple local sensor interfaces.

[0149] In one embodiment, driving instructions are output through multiple local execution interfaces to drive the corresponding local execution device to perform a specified action.

[0150] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), sync link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0151] In summary, the fusion monitoring method, apparatus, system, and medium disclosed in this invention involve: acquiring the operating status data of the target monitoring device and the collected video stream; receiving sensing data and control status data collected by the local sensing device; performing real-time prediction by fusing the operating status data, video stream, sensing data, and control status data using a trained deep learning model; identifying device anomalies based on the real-time prediction results and operating status data; performing edge computing and local decision-making on the real-time prediction results and sensing data; and driving the corresponding local execution device and / or the target monitoring device to perform specified actions based on the decision results. By fusing the collected video stream, sensing data, and device operating status information for deep learning prediction, accurate monitoring of device anomalies can be achieved, and reliable data can be provided for edge computing and local decision-making to directly drive the local execution device, effectively improving the decision response speed.

[0152] Figures 1-6 Examples of these embodiments can be applied to, for example Figure 7 The next generation of Internet of Things shown, and Figure 7 The technologies within the system support and integrate with each other, effectively solving many bottleneck problems in IoT applications, such as high latency, high power consumption, incomplete network coverage, low data capacity, different communication protocols, data insecurity, and application terminal allocation of communication resources. This invention significantly enhances the application value and user experience of IoT in various environments, improves application efficiency, and truly realizes the effective application of "Internet of Everything," achieving, but is not limited to, the following:

[0153] The next generation of the Internet of Things (IoT) is characterized by weakening the boundaries between sensing, communication, computing, control, and application in the traditional IoT, improving the interoperability between the layers, and promoting mutual promotion between the layers with the guidance of dynamic, on-demand, and rational resource allocation, so as to achieve overall system optimization.

[0154] Among these, the communication link is particularly crucial. Based on the multi-mode heterogeneous network, which is specially designed for smart twins / smart empowerment in various industries, the multi-mode heterogeneous network is an effective improvement and enhancement of existing wireless communication and networks. Through the dynamic coordination and allocation of communication parameters, multiple networking methods and network resources, it realizes ubiquitous, dynamic and real-time effective communication, improves spectrum utilization, network resource utilization, and increases network coverage and performance.

[0155] Multimode heterogeneous networks possess polymorphism, dynamically adjusting communication parameters based on physical location to establish a network. Besides mainstream communication modes, they also include advanced networking methods such as Mesh, relay, and SDN. They support flexible scheduling and scalable wireless link access and management technologies, support link self-healing, and provide highly utilized, highly stable, and easily recoverable professional wireless network services.

[0156] Multimode heterogeneous networks are closely integrated with industry needs, dynamically adjusting communication parameters based on industry requirements and / or physical location. These parameters include source coding, channel coding, signal time slots, transmit power, carrier frequency, carrier bandwidth, modulation scheme, transmit power, and receiver sensitivity. Different communication requirements employ different communication strategies. For example, high-bandwidth communication requirements can utilize data transmission point splitting, multipath concurrency during transmission, and receiver point aggregation, combined with high-quality-of-service (QoS) allocation strategies. Conversely, high-reliability communication requirements can employ multi-path redundant transmission methods, ensuring reliable delivery while reducing latency caused by sequential switching between multiple communication modes.

[0157] The deep integration of multi-mode heterogeneous networks and sensing terminals allows the sensing system to dynamically adjust sampling intervals and accuracy based on its own conditions, such as battery level, perceived data values, rate of change of perceived data, preset thresholds, and network conditions. This further adjusts parameters like transmission frequency, transmission power, and modulation method, thus simultaneously considering response time, overall power consumption, and network bandwidth usage. The sensing devices, combined with edge computing, achieve edge correction and self-correction, and can also generate edge decisions to directly drive the control terminal.

[0158] The multi-mode heterogeneous network possesses autonomous capabilities. The base stations / gateways can each have their own distributed edge core network (or communication server), and can automatically switch to the edge core network when the connection to the server-side core network is interrupted. In the event of a network outage, base stations / gateways can form a network via wireless or wired means, with one base station / gateway serving as the core network. The edge core network provides hierarchical and regional communication in the event of a network outage or weak network, providing necessary data exchange support for domain-specific edge computing.

[0159] With the support of multi-mode heterogeneous networks and artificial intelligence, the core network and base stations can collect link information from base stations, routing nodes, and terminals, including: communication standard, communication path, signal-to-noise ratio, packet loss rate, latency, channel occupancy, etc. Through deep learning, link prediction can be performed to deduce better networking and communication solutions. On demand (data transmission rate, response time, reliability, connection distance, etc.), the connection mode (direct connection to base station, mesh network, point-to-point), transmission path (single path, multi-path), and radio frequency parameters (modulation method, rate, spectrum occupancy, receiving bandwidth) of the devices can be adaptively adjusted.

[0160] Multi-mode heterogeneous networks can enhance cloud-edge collaborative computing capabilities. The AI ​​industry algorithm platform supports unified management and operation and maintenance of computing power and service resources. It can dynamically allocate computing power and algorithm tasks of fog computing, edge computing and the AI ​​industry algorithm platform itself according to industry applications, computing power, network and communication conditions. It can automatically scale up and down according to the actual configuration scenario to improve the utilization of computing resources.

[0161] The next-generation AIoT system comprises multiple layers: from bottom to top, they are the terminal layer, transmission layer, support layer, AI business platform layer, and city operation integrated IOC layer;

[0162] The next-generation AIoT system also includes: a security management platform, a unified operation and maintenance management platform, and IT resource services; among which, the security management platform and the unified operation and maintenance management platform are vertically integrated across all levels, providing end-to-end services across the entire chain.

[0163] IT resource services provide services to the support layer, the artificial intelligence business platform layer, and the comprehensive IOC layer for city operations.

[0164] Data from the surface fire monitoring terminal is transmitted back through a multi-mode heterogeneous network. When a fire occurs, the surface fire terminal transmits a real-time video stream from the scene. Because the network coverage in the forest area is poor, it is very difficult to transmit video. The multi-mode heterogeneous network can ensure that the terminal that detects the fire can transmit video smoothly by allocating communication resources.

[0165] Video fusion monitoring equipment provides a secure channel for all data transmission, while the blockchain security management platform provides the same security services, including security certificate issuance and key management.

[0166] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The computer program can be stored in a non-volatile, computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, floppy disk, flash memory, optical storage, etc.

[0167] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A fusion monitoring method characterized by, The method comprises: acquiring working state data of a target monitoring device and a collected video stream; receiving perception data and control state data collected by a local perception device; fusing and predicting the working state data, the video stream, the perception data, and the control state data through a deep learning model; performing target monitoring device anomaly identification according to a prediction result and the working state data, and performing corresponding early warning processing actions when there is device anomaly; performing edge computing and local decision-making on the prediction result and the perception data, and driving corresponding local execution devices and / or the target monitoring device to perform specified actions according to a result of the decision-making; wherein the fusion monitoring method is used for tree multi-dimensional monitoring; wherein the video stream comes from a horizontal camera for capturing a tree damage behavior and a sky camera for identifying tree branch and leaf development, and wherein the tree damage behavior is captured based on monitoring that someone is approaching through thermal infrared and / or induction radar.

2. The fusion monitoring method of claim 1, wherein The method further comprises: The target monitoring device establishes a secure connection channel with a server to securely transmit all communication data.

3. The fusion monitoring method of claim 1, wherein The method further comprises: predicting a normal power consumption range interval and a variation trend of the target monitoring device according to the working state data; statistically obtaining an actual power consumption variation trend of the target monitoring device within a preset time period; controlling power supply of the target monitoring device according to a preset power supply strategy when the actual power is not within the normal power consumption range interval or the actual power consumption variation trend does not match the predicted variation trend.

4. The fusion monitoring method of claim 3, wherein The power supply control of the target monitoring device according to the preset power supply strategy comprises: cutting off power supply of the target monitoring device when there is power consumption anomaly, and re-supplying power after maintaining a preset power-off time; continuously observing whether there is power consumption anomaly of the target monitoring device, and if there is, cutting off power supply of the target monitoring device again and re-supplying power after increasing the preset power-off time by a preset step, and so on, until the preset power-off time reaches a maximum power-off time.

5. The fusion monitoring method of claim 1, wherein, The method further comprises: monitoring port connections of the target monitoring device through a port whitelist, and intercepting port connections not within the port whitelist; tracking and identifying a legal new port to be enabled by the target monitoring device through protocol analysis, listing the port in a temporary whitelist, and further performing protocol analysis tracking, and removing the port from the temporary whitelist when the port is no longer used.

6. The fusion monitoring method according to any one of claims 1 to 5, characterized by, The perception data and the control state data are received through multiple local sensor interfaces.

7. The fusion monitoring method according to any one of claims 1 to 5, characterized by, A driving instruction is output through multiple local execution interfaces to drive corresponding local execution devices to perform specified actions.

8. The fusion monitoring method of claim 1, wherein, The method further comprises: switching local decision-making to cloud strategy when the network is good, obtaining the latest deployed strategy from the cloud server, and controlling the local execution devices and the target monitoring device.

9. The fusion monitoring method of claim 1, wherein, The method further comprises: obtaining the latest deployed strategy from the cloud server, controlling the local execution devices and the target monitoring device, and storing the strategy locally.

10. The fusion monitoring method according to any one of claims 1 to 5, characterized by, The method further comprises: transmitting the working state data, the video stream, the sensing data and the control state data through a multi-mode heterogeneous network; and adjusting one or more of the following communication parameters of the multi-mode heterogeneous network: source coding, channel coding, signal time slot, transmission power, carrier frequency, carrier bandwidth, modulation mode, transmission power, and receiving sensitivity.

11. The fusion monitoring method according to any one of claims 1 to 5, characterized by, The method further comprises: adjusting the sampling interval, sampling precision, transmission frequency, transmission power, and modulation mode of the network condition change sensing data according to the power, sensing data value, sensing data change rate, and preset threshold.

12. The fusion monitoring method according to any one of claims 1 to 5, characterized by, The method further comprises: under the condition of insufficient network coverage, weak network, or network disconnection, ensuring real-time return of the video stream and the sensing data of the monitoring terminal during fire, abnormal event or emergency instruction transmission through dynamic allocation of communication resources and / or multi-path parallel transmission mode.

13. The fusion monitoring method according to any one of claims 1 to 5, characterized by, The method further comprises: monitoring tree felling through a microphone, and identifying sawing sound and / or tree falling sound.

14. A fusion monitoring apparatus characterized by comprising: The method comprises: an acquisition module configured to acquire working state data, configured and collected video stream of a target monitoring device; a receiving module configured to receive sensing data and control state data collected by a sensing device; a fusion prediction module configured to predict the working state data, the video stream, the sensing data and the control state data through a deep learning model; an abnormality identification module configured to identify the prediction result and the working state data, and perform corresponding warning processing actions when there is device abnormality; a fusion decision module configured to perform edge computing and local decision on the prediction result and the sensing data, and drive corresponding execution devices and / or the target monitoring device to perform specified actions according to the result; wherein the fusion monitoring device is used for tree multi-dimensional monitoring; wherein the video stream used for the tree multi-dimensional monitoring comes from a horizontal camera for capturing tree destruction behavior and a sky camera for identifying tree branch and leaf development; and wherein the capturing of the tree destruction behavior is based on monitoring of a person approaching through thermal infrared and / or induction radar.

15. The fusion monitoring apparatus of claim 14, wherein, The device further comprises: a secure connection module configured to enable the target monitoring device to establish a secure connection channel with a server to securely transmit all communication data.

16. The fusion monitoring apparatus of claim 14, wherein The device further comprises: a power consumption prediction module configured to predict a normal power consumption range interval and a change trend of the target monitoring device according to the working state data; a power consumption statistical module configured to statistically obtain an actual power consumption change trend of the target monitoring device within a preset time period; a power supply control module configured to perform power supply control on the target monitoring device according to a preset power supply strategy when the actual power is not within the normal power consumption range interval or the actual power consumption change trend does not match the predicted change trend.

17. The fusion monitoring apparatus of claim 16, wherein The power supply control on the target monitoring device according to the preset power supply strategy specifically refers to: cutting off the power supply of the target monitoring device when there is power consumption abnormality, and re-supplying power after maintaining a preset power-off time. Continuously observing whether the target monitoring device has power consumption anomaly, if yes, cutting off the power supply of the target monitoring device again, increasing the preset power-off time by a preset step, and then re-supplying power, and so on, until the preset power-off time reaches the maximum power-off time.

18. The fusion monitoring device of claim 14, wherein, The device further comprises: An intercepting module for starting a port whitelist to monitor the port connection of the target monitoring device, and intercepting the port connection not in the port whitelist.

19. The fusion monitoring apparatus of claim 14, wherein, The device further comprises: A switching module for switching the local decision to a cloud strategy, and obtaining the latest deployed strategy from the cloud server to control the local execution device and the target monitoring device.

20. The fusion monitoring device of claim 14, wherein, The device further comprises: An obtaining module for obtaining the latest deployed strategy from the cloud server to control the local execution device and the target monitoring device when the network is good, and storing it locally.

21. The fusion monitoring apparatus of claim 14, wherein, The device further comprises: A tree multi-dimensional monitoring terminal, characterized in that it comprises: The horizontal camera and / or the sky camera apply the fusion monitoring method according to any one of claims 1-9.

22. The fusion monitoring device according to claim 21, further comprising one or more of the following: An inclination sensor for monitoring the inclination state of the main stem and the branch; A tree circumference sensor for monitoring the diameter of the tree to obtain the growth of the tree; A soil nutrition sensor for monitoring the growth environment of the tree; A thermal infrared and induction radar for detecting whether someone is approaching; A microphone for monitoring tree logging and identifying sawing sound and tree falling sound.

23. The fusion monitoring apparatus of claim 21, wherein The tree multi-dimensional monitoring terminal further performs real-time prediction on the working state data, the collected video stream, the sensing data of the sensor, and the control state data through a deep learning model, realizes equipment anomaly identification by combining the real-time prediction result with the working state data, and performs corresponding warning processing actions when there is equipment anomaly. The tree multi-dimensional monitoring terminal further processes the prediction result through edge computing and a local decision program to generate local edge decisions, actions, and warnings to drive external execution devices and / or target devices to perform corresponding actions.

24. The fusion monitoring apparatus of claim 14, wherein, The device further comprises a sensing terminal cooperation module for adjusting the sampling interval, sampling precision, sending frequency, transmission power, and modulation mode of the sensing terminal based on the power level, sensing data value, sensing data change rate, preset threshold, and network state of the sensing terminal.

25. A fusion monitoring system characterized by, The system comprises at least one processor; and A memory in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fusion monitoring method according to any one of claims 1-9.

26. A non-transitory computer readable storage medium, comprising: The non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors to enable the one or more processors to perform the fusion monitoring method according to any one of claims 1-9.

27. A method of fusion monitoring, comprising: comprising: Obtaining working state data of the target monitoring device and collected video stream; Receiving sensing data and control state data collected by the sensing device; transmitting working state data, video stream, sensing data and control state data through a multi-mode heterogeneous network; adjusting one or more of the following communication parameters of the multi-mode heterogeneous network: source coding, channel coding, signal time slot, transmission power, carrier frequency, carrier bandwidth, modulation mode, transmission power, receiving sensitivity; adjusting the sampling interval, sampling accuracy, sending frequency, transmission power, modulation mode of the sensing data according to the power, sensing data value, sensing data change rate, preset threshold, network status; fusing and predicting the working state data, video stream, sensing data and control state data through a deep learning model; performing target monitoring equipment anomaly identification according to the prediction result and the working state data, and performing corresponding early warning processing action when there is equipment anomaly; performing edge computing and local decision on the prediction result and the sensing data, and driving the corresponding local execution equipment and / or target monitoring equipment to perform specified actions according to the result of the decision.

28. The fusion monitoring method of claim 27, wherein, The fusion monitoring method is used for multi-dimensional monitoring of trees, and the snapshot of the behavior of damaging trees is based on monitoring that someone is approaching through thermal infrared and / or inductive radar.

29. The fusion monitoring method of any one of claims 27-28, wherein, The fusion monitoring method further includes collecting and learning the link information of base stations, routing nodes and terminals, including communication system, signal-to-noise ratio, packet loss rate, delay, channel occupancy rate, and making link prediction through deep learning to adaptively adjust the connection mode, transmission path and / or radio frequency parameters of the equipment based on transmission rate, response time, reliability and / or connection distance, thereby generating the optimal connection network and / or communication scheme.

30. The fusion monitoring method of any one of claims 27-28, wherein, The fusion monitoring method further includes unified scheduling and operation and maintenance of fog computing, edge computing and cloud computing resources, dynamic allocation and automatic scaling of computing power and algorithm tasks according to industry application scenarios, computing power distribution, network and communication status, to improve computing resource utilization and task response speed.

31. The fusion monitoring method of any one of claims 27-28, wherein, The fusion monitoring method further includes: under the condition of insufficient network coverage, weak network or network interruption, through dynamic allocation of communication resources and / or multi-path parallel transmission mode, the real-time backhaul of video stream and sensing data of the monitoring terminal is guaranteed when transmitting fire, abnormal events or emergency instructions.

32. A fusion monitoring device, characterized by comprises: an acquisition module configured to acquire working state data of a target monitoring device and collected video stream; a receiving module configured to receive sensing data and control state data collected by a sensing device; a communication module configured to transmit the working state data, video stream, sensing data and control state data through a multi-mode heterogeneous network; the communication module is further configured to adjust one or more of the following communication parameters: source coding, channel coding, signal time slot, transmission power, carrier frequency, carrier bandwidth, modulation mode, transmission power, receiving sensitivity, according to system operation requirements or network status; a sensing and control terminal cooperation module configured to adjust the sampling interval, sampling accuracy, sending frequency, transmission power, modulation mode of the sensing terminal based on the power level of the sensing terminal, the sensing data value, the sensing data change rate, the preset threshold, the network status; a fusion prediction module configured to fuse and predict the working status data, the video stream, the perception data, and the control status data by using a deep learning model; an anomaly identification module configured to identify anomalies of the target monitoring device based on the prediction result and the working status data, and perform corresponding early warning processing actions when there are device anomalies; a fusion decision module configured to perform edge computing and local decision based on the prediction result and the perception data, and drive corresponding execution devices and / or the target monitoring device to execute instructions or actions according to the result.

33. The fusion monitoring apparatus of claim 32, wherein, The communication module of the device communicates with multiple base stations, gateways, and edge computing nodes, and dynamically adjusts the communication parameters to realize multi-path concurrency and / or link self-healing.

34. The fusion monitoring device of any one of claims 32, further comprising an artificial intelligence module configured to collect and learn link information of base stations, routing nodes, and terminals, including communication standards, signal-to-noise ratio, packet loss rate, delay, channel occupancy rate, and make link prediction through deep learning to adaptively adjust the connection mode, transmission path, and / or radio frequency parameters of the device based on transmission rate, response time, reliability, and / or connection distance, thereby generating an optimal connection network and / or communication scheme.

35. The fusion monitoring apparatus of any of claims 32-34, wherein, The device is connected to an algorithm center in the artificial intelligence industry, and the algorithm center uniformly schedules and operates fog computing, edge computing, and cloud computing resources, dynamically allocates and automatically scales the computing power and algorithm tasks according to the industry application scenarios, computing power distribution, network and communication states, to improve the utilization rate of computing resources and the response speed of tasks.

36. The fusion monitoring apparatus of any of claims 32-34, wherein, The system further comprises a blockchain security management unit configured to provide end-to-end encryption and identity authentication for the video stream, the perception data, and / or the control data.

37. The fusion monitoring apparatus of any of claims 32-34, wherein, The multi-mode heterogeneous network is applied to forest fire prevention monitoring, industrial control, intelligent transportation, or urban security, and under the condition of insufficient network coverage, weak network, or network interruption, the communication resources are dynamically allocated and / or multi-path parallel transmission is used to ensure the real-time return of video stream and perception data of the monitoring terminal during fire, abnormal events, or emergency instruction transmission.

38. The fusion monitoring device of any one of claims 32-34, wherein the perception device is configured to perform local data rectification and self-correction operations in combination with the edge computing unit.

39. The fusion monitoring device of any one of claims 32-34, wherein the perception device, in combination with the edge computing unit, is configured to generate an edge decision signal for driving the execution of a response action.

40. The fusion monitoring device of claim 32, wherein the fusion monitoring device is used for tree multi-dimensional monitoring.

41. The fusion monitoring device of claim 40, wherein the video stream for tree multi-dimensional monitoring comes from a horizontal camera for capturing tree damage behavior and a sky camera for identifying tree branch and leaf growth conditions.

42. The fusion monitoring device of claim 41, wherein the capturing of tree damage behavior is based on the detection of a person approaching by thermal infrared and / or induction radar monitoring.

Images