A method, system, electronic device and storage medium for preventing external force damage monitoring
By using multi-model fusion and cross-frame target tracking technology, the problems of low efficiency and uneven resource utilization in traditional monitoring methods have been solved, achieving efficient and accurate detection of external force damage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN DINGYIYUAN TECH DEV CO LTD
- Filing Date
- 2025-10-14
- Publication Date
- 2026-07-21
Smart Images
Figure CN121191091B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a monitoring method, system, electronic device, and storage medium for preventing damage from external forces. Background Technology
[0002] In critical areas such as power transmission lines and oil and gas pipelines, incidents of external damage occur frequently. Construction machinery, vehicles, kites, and bird nests are common sources of damage, seriously threatening the safety and normal operation of these facilities. Traditional monitoring methods, such as manual inspections and ordinary cameras, have significant shortcomings: manual inspections are limited by time and manpower, making 24 / 7 monitoring impossible; ordinary cameras lack intelligent analysis capabilities, making it difficult to automatically identify acts of external damage, and are prone to false alarms and missed alarms in complex environments, resulting in poor system stability and an inability to meet actual monitoring needs.
[0003] With the development of artificial intelligence technology, deep learning-based detection algorithms have been introduced into the field of security monitoring. However, existing AI solutions face challenges when deploying multiple models. Different models have significantly different requirements for computing resources and inference latency. Simple serial pipelines or uniform scheduling strategies lead to unbalanced resource utilization, with high-latency models becoming system bottlenecks and low-latency models leaving computing power idle, affecting real-time performance and wasting computing power. Furthermore, the fusion of multi-model results is limited to simple weighting or logical AND / OR operations, lacking deep coupling with business scenarios and an adaptive fusion mechanism, thus restricting the improvement of detection accuracy.
[0004] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a monitoring method, system, electronic device, and storage medium for preventing damage from external forces.
[0006] In a first aspect, the present invention provides a monitoring method for preventing damage from external forces, the technical solution of which is as follows: Real-time acquisition of image data of the target monitoring area; The image data is preprocessed to extract the region of interest and perform image enhancement. Based on multiple heterogeneous target models, reasoning is performed on the enhanced region of interest image to obtain multidimensional recognition results, which are then fused to generate a fused recognition result. Based on the fusion recognition results, cross-frame target tracking is performed. When any target is detected in multiple consecutive frames and is located in a dangerous area within the target monitoring area, and the fusion confidence is greater than a preset value, an effective alarm event for the target is generated and an alarm is triggered.
[0007] The beneficial effects of the monitoring method for preventing damage from external forces according to the present invention are as follows: The method of this invention can solve the problems of low efficiency, false alarms and missed alarms, and uneven resource utilization of multiple models in traditional monitoring methods, significantly improving the real-time performance, accuracy and resource utilization of monitoring, and effectively reducing the risk of external damage.
[0008] Based on the above solution, the monitoring method for preventing external damage according to the present invention can be further improved as follows.
[0009] In one alternative approach, the steps of preprocessing the image data, extracting the region of interest image, and performing image enhancement include: The region of interest (ROI) image is extracted from the image data using an attention-based image cropping algorithm. Adaptive histogram equalization is performed on the region of interest to obtain the enhanced region of interest image.
[0010] The advantages of adopting the above-mentioned optional methods are: by further using image cropping and adaptive equalization based on attention mechanism, the target region in the monitoring image can be accurately extracted and enhanced, effectively improving the detection accuracy and efficiency of the model.
[0011] In one alternative approach, it also includes: Based on the preset task type, select and call the corresponding multiple target heterogeneous models from the model repository.
[0012] The advantages of adopting the above optional approach are: to further dynamically call heterogeneous models in the model repository according to the task type, realize on-demand allocation of resources, and improve resource utilization and task execution flexibility in multi-model scenarios.
[0013] In one optional approach, the plurality of heterogeneous target models include: an object detection model, a semantic segmentation model, and an action recognition model; the multidimensional recognition result includes: object detection boxes and corresponding initial confidence scores, danger zone masks, and action classification results; the step of inferring from the enhanced region of interest image based on the plurality of heterogeneous target models to obtain the multidimensional recognition result includes: Using the target detection model, obtain the target detection box and the corresponding initial confidence score corresponding to the enhanced region of interest image; Using the semantic segmentation model, the danger region mask corresponding to the enhanced region of interest image is obtained; Using the behavior recognition model, obtain the behavior classification result corresponding to the enhanced region of interest image; Specifically, the target detection model is configured with a high-priority instance group and a small batch size for real-time inference; the semantic segmentation model is configured with dynamic batching and a large batch size to improve the utilization of computing resources; and the behavior recognition model and the target detection model are executed concurrently.
[0014] The advantages of adopting the above-mentioned optional methods are: further combining object detection, semantic segmentation and behavior recognition models to form a multi-dimensional collaborative reasoning mechanism, coupled with differentiated model configuration strategies, to optimize the allocation of computing resources and ensure the real-time performance and accuracy of the monitoring method.
[0015] In one optional approach, the step of fusing the multidimensional recognition results to generate the fused recognition result includes: Calculate the intersection-union ratio (IoU) between the target detection bounding box and the danger zone mask; When the intersection-union ratio is greater than a set threshold, the initial confidence is weighted and amplified to obtain the fusion confidence, so as to generate the fusion recognition result containing the target detection box, the fusion confidence, the danger area mask and the behavior classification result.
[0016] The beneficial effect of adopting the above optional method is that by further calculating the intersection-union ratio of the target detection box and the dangerous area mask and weighting the initial confidence, the deep fusion of multi-dimensional recognition results can be achieved, which significantly improves the accuracy of dangerous behavior recognition.
[0017] In one alternative approach, the step of performing cross-frame target tracking based on the fused recognition result includes: The predicted position of the target detection box in the next frame image is predicted based on Kalman filtering. The Hungarian algorithm is used to associate the actual detected target bounding boxes in the next frame image with the predicted locations, and a unique target ID is assigned to the same target. Based on the target ID, cross-frame target tracking is performed.
[0018] The advantages of using the above-mentioned optional methods are: further application of Kalman filtering and Hungarian algorithm to achieve cross-frame target tracking, ensuring stable identification and association of targets in continuous frames, and effectively enhancing the real-time monitoring capability of dynamic scenes.
[0019] In one alternative approach, it also includes: When any of the targets triggers an alarm, an alarm cooldown timer is started for that target to suppress repeated alarms during the cooldown period.
[0020] The advantages of adopting the above optional methods are: to further set a cooling timer for the alarm target, avoid repeated alarm interference, improve alarm effectiveness and user experience, and ensure timely response to critical alarm information.
[0021] Secondly, the present invention provides a monitoring system for preventing damage from external forces, the technical solution of which is as follows: It includes: a data acquisition module, a preprocessing module, an inference and fusion module, and a monitoring module; The acquisition module is used to: acquire image data of the target monitoring area in real time; The preprocessing module is used to: preprocess the image data, extract the region of interest image, and perform image enhancement; The inference fusion module is used to: infer the enhanced region of interest image based on multiple target heterogeneous models, obtain multi-dimensional recognition results and fuse them to generate a fused recognition result; The monitoring module is used to: perform cross-frame target tracking based on the fusion recognition results; when any target is detected in multiple consecutive frames as being located in a dangerous area within the target monitoring area and the fusion confidence level is greater than a preset value, generate a valid alarm event for the target and trigger an alarm.
[0022] The beneficial effects of the monitoring system for preventing damage from external forces according to the present invention are as follows: The system of this invention can solve the problems of low efficiency, false alarms and missed alarms, and uneven resource utilization of multiple models in traditional monitoring methods, significantly improving the real-time performance, accuracy and resource utilization of monitoring, and effectively reducing the risk of external damage.
[0023] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the monitoring method for preventing external damage as described in this invention.
[0024] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the monitoring method for preventing external damage as described in this invention.
[0025] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0026] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of a monitoring method for preventing damage from external forces according to the present invention. Figure 2 This is a schematic diagram of the overall process; Figure 3 This is a schematic diagram of an embodiment of the monitoring system for preventing damage from external forces according to the present invention; Figure 4 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation
[0027] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0028] Figure 1 This diagram illustrates a flowchart of an embodiment of a method for monitoring and preventing external damage provided by the present invention. This method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the method for monitoring and preventing external damage by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps: S1. Acquire image data of the target monitoring area in real time.
[0029] The target monitoring area refers to the physical space that needs to be monitored to prevent damage from external forces; for example, the corridor and surrounding area under a section of high-voltage transmission line. Image data refers to digital data representing visual information of the target monitoring area acquired by image acquisition equipment; for example, video streams or image frames of the environment surrounding the transmission line captured by a camera.
[0030] It should be noted that in this embodiment, image data is acquired through a camera array deployed in the target monitoring area. The camera array supports two triggering modes: timed acquisition and dynamic triggering. Timed acquisition acquires images at preset time intervals, such as acquiring one frame every 10 minutes in a power line inspection scenario. Dynamic triggering, on the other hand, analyzes motion changes or edge features in the image in real time and immediately triggers acquisition when abnormal dynamics are detected. This is suitable for scenarios requiring rapid response, such as protection of power transmission lines from external damage and perimeter security of industrial parks. The acquired image data is output in RGB color space and NHWC format (i.e., dimensions are height, width, and channels).
[0031] S2. Preprocess the image data to extract the region of interest and perform image enhancement.
[0032] The region of interest (ROI) refers to a sub-image area cropped from the original image data that contains potential risk targets or critical facilities; for example, an image patch cropped from a panoramic view of a power transmission line near the tower base or directly below the conductor.
[0033] S3. Based on multiple heterogeneous target models, reasoning is performed on the enhanced region of interest image to obtain multidimensional recognition results, which are then fused to generate a fused recognition result.
[0034] Here, "target heterogeneous model" refers to a combination of multiple machine learning models that differ in structure, function, or computational characteristics; for example, a model set consisting of a YOLO object detection model, a semantic segmentation model, and a behavior recognition model. "Enhanced region of interest image" refers to a region of interest image optimized by image processing algorithms to improve feature visibility; for example, an image of the base of a tower after adaptive histogram equalization, enhancing the contours and textures of the machinery. "Fusion recognition result" refers to structured recognition information resulting from the comprehensive processing of the outputs of multiple heterogeneous models; for example, a set including object detection boxes, weighted confidence scores, hazard area masks, and behavior classification results.
[0035] S4. Perform cross-frame target tracking based on the fusion recognition results. When any target is detected in multiple consecutive frames and is located in a dangerous area of the target monitoring area and the fusion confidence is greater than a preset value, generate an effective alarm event for the target and trigger an alarm.
[0036] Cross-frame target tracking refers to the technical process of continuously associating the same target and maintaining its motion trajectory in consecutive image frames; for example, continuously tracking the movement path of an excavator in a video using Kalman filtering and the Hungarian algorithm.
[0037] Here, "any target" refers to an object or entity identified in the image that may pose a risk of external force damage; for example, an excavator working under a power transmission line. "Hazardous area" refers to a sub-area within the target monitoring area that has been identified as susceptible to external force damage; for example, the area within 5 meters directly beneath a power transmission line. "Fusion confidence" refers to the comprehensive confidence value regarding the existence and risk level of the target obtained after fusing results from multiple models; for example, the confidence of the excavator target increases from 0.7 to 0.84 after being weighted by a hazardous area mask. "Preset value" refers to a pre-set threshold parameter used to determine whether an alarm is triggered; for example, the alarm threshold for fusion confidence is set to 0.8. "Valid alarm event" refers to an alarm triggering instance that meets all alarm conditions and is confirmed by the system; for example, an alarm event generated when the excavator is located in the hazardous area for 5 consecutive frames and the fusion confidence exceeds 0.8.
[0038] Specifically, after obtaining the fusion recognition result corresponding to any target, the position and state of the target are continuously tracked in multiple consecutive frames of images based on the unique target ID assigned to the target. For each frame, it is determined whether the target is located within the danger area defined by the danger area mask output by the semantic segmentation model, and at the same time, it is determined whether its fusion confidence exceeds a preset value. A time sliding window counter is maintained for the target. Only when the target is detected in multiple consecutive frames and simultaneously meets the conditions of being located within the danger area and having a fusion confidence exceeding the preset value, the count reaches the preset frame requirement, and a valid alarm event for the target is generated. Subsequently, the audible and visual alarm device is immediately triggered and the alarm information is pushed to the monitoring center. At the same time, an alarm cooling timer is started for the target ID. During the cooling period, repeated alarms from the same target ID will be suppressed to ensure the accuracy and effectiveness of the alarm.
[0039] It should be noted that the methods for triggering alarms include, but are not limited to: pushing audible and visual alarm commands to field terminals (such as tower warning lights and audible and visual alarms) via message buses such as MQTT / Kafka / RabbitMQ, and sending messages to the duty personnel's terminals (SMS, APP push).
[0040] The technical solution of this embodiment can solve the problems of low efficiency, false alarms and missed alarms, and uneven resource utilization of multiple models in traditional monitoring methods, significantly improving the real-time performance, accuracy and resource utilization of monitoring, and effectively reducing the risk of external damage.
[0041] In one alternative approach, the steps of preprocessing the image data, extracting the region of interest image, and performing image enhancement include: The region of interest (ROI) image is extracted from the image data using an attention-based image cropping algorithm.
[0042] Among them, the image cropping algorithm based on the attention mechanism refers to an algorithm that prioritizes the extraction of key regions in an image by simulating the visual attention mechanism; for example, prioritizing the cropping of the area below the tower base and conductor in a power transmission line image while ignoring the sky and distant mountains.
[0043] Adaptive histogram equalization is performed on the region of interest to obtain the enhanced region of interest image.
[0044] Adaptive histogram equalization refers to an image enhancement method that dynamically adjusts contrast based on local image features; for example, enhancing the local contrast of the tower base area under backlight conditions to improve the visibility of mechanical equipment.
[0045] It should be noted that the enhanced region of interest image obtained after adaptive histogram equalization is further normalized by performing the following operations: the image is resized to the input size required by the target heterogeneous model (e.g., 640×640 pixels required by the object detection model YOLOv11), and the image pixel values are normalized to map them to the numerical range of [0,1] or [-1,1]. At the same time, the image format is converted from RGB color space and NHWC layout (dimensions are [height, width, channels]) to NCHW layout (dimensions become [channels, height, width]) to conform to the input specifications of deep learning models.
[0046] Among the above-mentioned optional methods, further attention-based image cropping and adaptive equalization processing are used to accurately extract and enhance the target region in the monitoring image, effectively improving the model's detection accuracy and efficiency.
[0047] In one alternative approach, it also includes: Based on the preset task type, select and call the corresponding multiple target heterogeneous models from the model repository.
[0048] The preset task type refers to the category of detection tasks predefined according to the needs of the monitoring scenario; for example, the task type of "protecting power transmission lines from external damage" requires the identification of targets such as construction machinery and kites. The model repository refers to a centralized resource library that stores and manages multiple available machine learning models; for example, a model repository containing the yolov11_projectcar model, a semantic segmentation model, and a behavior recognition model.
[0049] Among the above optional methods, heterogeneous models in the model repository can be dynamically called according to the task type to realize on-demand allocation of resources and improve resource utilization and task execution flexibility in multi-model scenarios.
[0050] In one optional approach, the plurality of heterogeneous target models include: an object detection model, a semantic segmentation model, and an action recognition model; the multidimensional recognition result includes: object detection boxes and corresponding initial confidence scores, danger zone masks, and action classification results; the step of inferring from the enhanced region of interest image based on the plurality of heterogeneous target models to obtain the multidimensional recognition result includes: Using the target detection model, the target detection box and the corresponding initial confidence score are obtained for the enhanced region of interest image.
[0051] In this context, "object detection model" refers to a machine learning model used to detect the location and category of objects in an image; for example, the YOLO model detects objects such as excavators and vehicles in an image and outputs their bounding boxes. "Object detection box" refers to a rectangular bounding box that identifies the location of an object in an image; for example, a rectangle that frames the location of an excavator in an image. "Initial confidence" refers to the initial confidence value of the object detection model's direct output regarding the existence of the object; for example, the YOLO model outputs a confidence value of 0.7 for the excavator object.
[0052] Using the semantic segmentation model, the danger region mask corresponding to the enhanced region of interest image is obtained.
[0053] Semantic segmentation models refer to machine learning models used to classify each pixel in an image to identify specific regions; for example, models used to identify dangerous areas in a power transmission line corridor and generate corresponding masks. A dangerous area mask is a binary image output by the semantic segmentation model that identifies dangerous areas; for example, an image of the same size as the input image where pixels in dangerous areas have a value of 1, and the rest have a value of 0.
[0054] Using the behavior recognition model, the behavior classification result corresponding to the enhanced region of interest image is obtained.
[0055] Here, a behavior recognition model refers to a machine learning model used to identify the type of target behavior or action; for example, a behavior classification model used to determine whether an excavator is in a digging state. The behavior classification result refers to the judgment result of the behavior recognition model regarding the category of the target behavior; for example, outputting "digging behavior" as the excavator's behavior classification result.
[0056] Specifically, the target detection model is configured with a high-priority instance group and a small batch size for real-time inference; the semantic segmentation model is configured with dynamic batching and a large batch size to improve the utilization of computing resources; and the behavior recognition model and the target detection model are executed concurrently.
[0057] It should be noted that the default batch size is 1 to 4 for smaller batches and 8 to 32 for larger batches. These can be adjusted according to the actual situation, and no limit is set here.
[0058] Among the above-mentioned optional methods, further combining object detection, semantic segmentation and behavior recognition models can form a multi-dimensional collaborative reasoning mechanism. With the help of differentiated model configuration strategies, the allocation of computing resources can be optimized to ensure the real-time performance and accuracy of the monitoring method.
[0059] In one optional approach, the step of fusing the multidimensional recognition results to generate the fused recognition result includes: Calculate the cross-union ratio (CUI) between the target detection bounding box and the danger zone mask.
[0060] The intersection-union ratio (IUGR) is the ratio of the overlapping area of two regions to their union area, used to measure the degree of overlap; for example, it is used to calculate the overlap ratio between the excavator target detection box and the mask of the danger zone.
[0061] When the intersection-union ratio is greater than a set threshold, the initial confidence is weighted and amplified to obtain the fusion confidence, so as to generate the fusion recognition result containing the target detection box, the fusion confidence, the danger area mask and the behavior classification result.
[0062] The threshold setting refers to a pre-defined critical value used to determine whether the crossover ratio (CVR) is valid. For example, the default CVR threshold is 0.4, but it can be adjusted according to actual conditions without restriction. Weighted amplification refers to the process of multiplicatively increasing the confidence level to improve its value. For example, when the CVR exceeds 0.4, the initial confidence level of 0.7 is multiplied by 1.2 to obtain 0.84.
[0063] In the above-mentioned optional methods, the cross-union ratio of the target detection box and the dangerous area mask is further calculated and the initial confidence is weighted to achieve deep fusion of multi-dimensional recognition results, which significantly improves the accuracy of dangerous behavior recognition.
[0064] In one alternative approach, the step of performing cross-frame target tracking based on the fused recognition result includes: The predicted location of the target detection box in the next frame image is predicted based on Kalman filtering.
[0065] Kalman filtering refers to an optimized estimation algorithm for predicting the position of a target in the next frame; for example, predicting the position of an excavator in the next frame based on its position and speed in the current frame. Predicted position refers to the position information of the target in a future frame estimated by the prediction algorithm; for example, the bounding box coordinates of the excavator in the next frame predicted by Kalman filtering.
[0066] The Hungarian algorithm is used to associate the actual detected target bounding boxes in the next frame with the predicted locations, and a unique target ID is assigned to the same target.
[0067] The Hungarian algorithm refers to an optimal allocation algorithm for solving the problem of target correlation between frames; for example, matching the actual detected target in the next frame with the predicted location and assigning the same ID. The target ID refers to a unique identifier assigned to each tracked target; for example, assigning ID=001 to a continuously appearing excavator.
[0068] Based on the target ID, cross-frame target tracking is performed.
[0069] Among the above optional methods, Kalman filtering and Hungarian algorithm are further used to achieve cross-frame target tracking, ensuring stable identification and association of targets in continuous frames, and effectively enhancing the real-time monitoring capability of dynamic scenes.
[0070] In one alternative approach, it also includes: When any of the targets triggers an alarm, an alarm cooldown timer is started for that target to suppress repeated alarms during the cooldown period.
[0071] The alarm cooldown timer refers to a timing device used to control the time interval between repeated alarms for the same target; for example, a 30-second cooldown timer is started for target ID=001. The cooldown period refers to the time period during which the alarm cooldown timer is effective, during which repeated alarms for the same target are suppressed; for example, during the 30-second cooldown period, the system will not issue new alarms for the excavator with ID=001.
[0072] Among the above optional methods, a cooling timer can be further set for the alarm target to avoid repeated alarm interference, improve alarm effectiveness and user experience, and ensure timely response to critical alarm information.
[0073] Figure 2 A schematic diagram of the overall process of the monitoring scheme to prevent damage from external forces is provided. Figure 2 middle: 1) The camera array serves as the image acquisition source and supports two working modes: timed triggering and intrusion triggering. Timed triggering acquires images according to a preset period, while intrusion triggering analyzes motion or edge changes in the image in real time and immediately starts capturing images when abnormal dynamics are detected, thus adapting to the real-time requirements of different monitoring scenarios.
[0074] 2) The acquired raw image data is sent to the Triton preprocessing module, which is responsible for preprocessing the image. This mainly includes scaling the image to the model input size, normalizing the pixel values to map them to a specified numerical range, and converting the image data from the original RGB color space and NHWC layout to the NCHW layout to adapt to the input specifications of the deep learning model.
[0075] 3) The preprocessing module also handles task routing, selecting and calling the appropriate target heterogeneous model from the model repository based on the actual detection task type, such as the yolov11_projectcar model, to clarify the current required inference computation task. The preprocessed image data and related parameters are then pushed to the inference queue.
[0076] 4) The inference queue module supports a dynamic batch processing mechanism, which can aggregate multi-frame image tasks and automatically merge adaptation tasks to improve the utilization of hardware computing resources. This module also supports a hybrid scheduling strategy for GPU and CPU.
[0077] 5) Subsequently, the Triton execution scheduling module parses and applies inference parameters according to the preset configuration file, such as batch size, latency and instance group configuration items. Triton automatically selects and allocates them to idle inference replicas, calls ONNXRuntime or LibTorch as the inference backend to perform calculations, and finally outputs a multidimensional array containing information such as target detection boxes, categories and confidence scores.
[0078] 6) The inference results then enter the Triton post-processing module for parsing and filtering. This module first performs non-maximum suppression to filter redundant detection boxes and retain the true targets with the highest confidence. Subsequently, it performs clustering and event judgment on the detected targets in conjunction with business rules, and outputs structured event results, including event type, risk level, and target coordinate information.
[0079] 7) Two types of response logic are triggered based on the event judgment result. If the identified event meets the preset alarm conditions, an alarm command is pushed to the field terminal through a message bus such as MQTT, Kafka or Redis to trigger the local audible and visual alarm device; at the same time, the event details, including image frames and recognition results, are uploaded to the platform API or database to support remote monitoring and big data analysis, and finally complete the complete closed loop from image acquisition to intelligent alarm.
[0080] Figure 3 A schematic diagram of an embodiment of a monitoring system 200 for preventing damage from external forces provided by the present invention is shown. Figure 3 As shown, the system 200 includes: a data acquisition module 210, a preprocessing module 220, a reasoning and fusion module 230, and a monitoring module 240; The acquisition module 210 is used to: acquire image data of the target monitoring area in real time; The preprocessing module 220 is used to: preprocess the image data, extract the region of interest image, and perform image enhancement; The inference fusion module 230 is used to: infer the enhanced region of interest image based on multiple target heterogeneous models, obtain multi-dimensional recognition results and fuse them to generate a fused recognition result; The monitoring module 240 is used to: perform cross-frame target tracking based on the fusion identification results; when any target is detected in multiple consecutive frames as being located in a dangerous area within the target monitoring area and the fusion confidence level is greater than a preset value, generate an effective alarm event for the target and trigger an alarm.
[0081] In an alternative embodiment, the preprocessing module 220 is specifically used for: The region of interest (ROI) image is extracted from the image data using an attention-based image cropping algorithm. Adaptive histogram equalization is performed on the region of interest to obtain the enhanced region of interest image.
[0082] In an alternative approach, it further includes: a calling module; the calling module is used for: Based on the preset task type, select and call the corresponding multiple target heterogeneous models from the model repository.
[0083] In one optional approach, the plurality of heterogeneous target models include: an object detection model, a semantic segmentation model, and a behavior recognition model; the multidimensional recognition result includes: object detection boxes and corresponding initial confidence scores, danger zone masks, and behavior classification results; the inference fusion module 230 is specifically used for: Using the target detection model, obtain the target detection box and the corresponding initial confidence score corresponding to the enhanced region of interest image; Using the semantic segmentation model, the danger region mask corresponding to the enhanced region of interest image is obtained; Using the behavior recognition model, obtain the behavior classification result corresponding to the enhanced region of interest image; Specifically, the target detection model is configured with a high-priority instance group and a small batch size for real-time inference; the semantic segmentation model is configured with dynamic batching and a large batch size to improve the utilization of computing resources; and the behavior recognition model and the target detection model are executed concurrently.
[0084] In one alternative embodiment, the inference fusion module 230 is specifically used for: Calculate the intersection-union ratio (IoU) between the target detection bounding box and the danger zone mask; When the intersection-union ratio is greater than a set threshold, the initial confidence is weighted and amplified to obtain the fusion confidence, so as to generate the fusion recognition result containing the target detection box, the fusion confidence, the danger area mask and the behavior classification result.
[0085] In an alternative embodiment, the monitoring module 240 is specifically used for: The predicted position of the target detection box in the next frame image is predicted based on Kalman filtering. The Hungarian algorithm is used to associate the actual detected target bounding boxes in the next frame image with the predicted locations, and a unique target ID is assigned to the same target. Based on the target ID, cross-frame target tracking is performed.
[0086] In one optional embodiment, it further includes: an alarm cooling module; the alarm cooling module is used for: When any of the targets triggers an alarm, an alarm cooldown timer is started for that target to suppress repeated alarms during the cooldown period.
[0087] It should be noted that the beneficial effects of the external force damage prevention monitoring system 200 provided in the above embodiments are the same as those of the external force damage prevention monitoring method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.
[0088] The external force damage prevention monitoring system 200 of the present invention can be a computer program (including program code) running on a computer device. For example, the external force damage prevention monitoring system 200 of the present invention is an application software that can be used to execute the corresponding steps in the external force damage prevention monitoring method of the present invention.
[0089] In some embodiments, the external force damage prevention monitoring system 200 of the present invention can be implemented in a combination of hardware and software. As an example, the external force damage prevention monitoring system 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the external force damage prevention monitoring method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.
[0090] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.
[0091] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for preventing external damage monitoring. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the method for preventing external damage monitoring shown in any embodiment of the present invention by calling the computer program.
[0092] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.
[0093] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0094] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.
[0095] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0096] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.
[0097] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.
[0098] It should be noted that, Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.
[0099] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for preventing external damage during monitoring.
[0100] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.
[0101] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned method for preventing external damage monitoring.
[0102] Computer program code for performing the operations of this invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0103] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0104] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0105] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.
[0106] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.
[0107] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.
[0108] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.
[0109] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A monitoring method for preventing damage from external forces, characterized in that, include: Real-time acquisition of image data of the target monitoring area; The image data is preprocessed to extract the region of interest and perform image enhancement. Based on multiple heterogeneous target models, reasoning is performed on the enhanced region of interest image to obtain multidimensional recognition results, which are then fused to generate a fused recognition result. Based on the fusion recognition results, cross-frame target tracking is performed. When any target is detected in multiple consecutive frames and is located in a dangerous area of the target monitoring area and the fusion confidence is greater than a preset value, an effective alarm event for the target is generated and an alarm is triggered. It also includes: selecting and calling the corresponding multiple heterogeneous target models from the model repository according to the preset task type; the multiple heterogeneous target models include: target detection model, semantic segmentation model and behavior recognition model; the multidimensional recognition result includes: target detection box and corresponding initial confidence, danger area mask, behavior classification result; The steps for obtaining multidimensional recognition results by inferring from the enhanced region of interest image based on multiple heterogeneous target models include: Using the target detection model, the target detection box and the corresponding initial confidence score are obtained for the enhanced region of interest image; using the semantic segmentation model, the danger region mask is obtained for the enhanced region of interest image; using the behavior recognition model, the behavior classification result is obtained for the enhanced region of interest image. Specifically, the target detection model is configured with a high-priority instance group and a small batch size for real-time inference; the semantic segmentation model is configured with dynamic batching and a large batch size to improve the utilization of computing resources; the behavior recognition model and the target detection model are executed concurrently; the small batch size is 1 to 4, and the large batch size is 8 to 32 by default; The steps for cross-frame target tracking based on the fusion recognition results include: The predicted position of the target detection box in the next frame image is predicted based on Kalman filtering; the actual detected target detection box in the next frame image is associated with the predicted position using the Hungarian algorithm, and a unique target ID is assigned to the same target; cross-frame target tracking is performed based on the target ID; After obtaining the fusion recognition result for any target, the position and state of the target are continuously tracked in multiple consecutive frames based on the unique target ID assigned to that target. For each frame, it is determined whether the target is located within the danger zone defined by the danger zone mask output by the semantic segmentation model, and at the same time, it is determined whether its fusion confidence exceeds a preset value. A time sliding window counter is maintained for the target. Only when the target is detected in multiple consecutive frames and simultaneously meets the conditions of being located within the danger zone and having a fusion confidence exceeding the preset value, the count reaches the preset frame requirement, and a valid alarm event for the target is generated. Subsequently, the audible and visual alarm device is immediately triggered and the alarm information is pushed to the monitoring center. At the same time, an alarm cooling timer is started for the target ID. During the cooling period, repeated alarms from the same target ID are suppressed.
2. The monitoring method for preventing external damage according to claim 1, characterized in that, The steps of preprocessing the image data, extracting the region of interest, and performing image enhancement include: The region of interest (ROI) image is extracted from the image data using an attention-based image cropping algorithm. Adaptive histogram equalization is performed on the region of interest to obtain the enhanced region of interest image.
3. The monitoring method for preventing external damage according to claim 1, characterized in that, The step of fusing the multidimensional recognition results to generate the fused recognition result includes: Calculate the intersection-union ratio (IoU) between the target detection bounding box and the danger zone mask; When the intersection-union ratio is greater than a set threshold, the initial confidence is weighted and amplified to obtain the fusion confidence, so as to generate the fusion recognition result containing the target detection box, the fusion confidence, the danger area mask and the behavior classification result.
4. A monitoring system for preventing damage from external forces, characterized in that, include: The module consists of an acquisition module, a preprocessing module, an inference and fusion module, and a monitoring module. The acquisition module is used to: acquire image data of the target monitoring area in real time; The preprocessing module is used to: preprocess the image data, extract the region of interest image, and perform image enhancement; The inference fusion module is used to: infer the enhanced region of interest image based on multiple target heterogeneous models, obtain multi-dimensional recognition results and fuse them to generate a fused recognition result; The monitoring module is used to: perform cross-frame target tracking based on the fusion identification result; when any target is detected in a dangerous area of the target monitoring area in multiple consecutive frames and the fusion confidence is greater than a preset value, generate an effective alarm event for the target and trigger an alarm. It also includes: a calling module; the calling module is used to: select and call the corresponding multiple heterogeneous target models from the model repository according to the preset task type; the multiple heterogeneous target models include: target detection model, semantic segmentation model and behavior recognition model; the multidimensional recognition result includes: target detection box and corresponding initial confidence, danger area mask, behavior classification result; The inference fusion module is specifically used for: using the target detection model to obtain the target detection box and the corresponding initial confidence score corresponding to the enhanced region of interest image; using the semantic segmentation model to obtain the danger region mask corresponding to the enhanced region of interest image; and using the behavior recognition model to obtain the behavior classification result corresponding to the enhanced region of interest image. Specifically, the target detection model is configured with a high-priority instance group and a small batch size for real-time inference; the semantic segmentation model is configured with dynamic batching and a large batch size to improve the utilization of computing resources; the behavior recognition model and the target detection model are executed concurrently; the small batch size is 1 to 4, and the large batch size is 8 to 32 by default; The monitoring module is specifically used for: The predicted position of the target detection box in the next frame image is predicted based on Kalman filtering; the actual detected target detection box in the next frame image is associated with the predicted position using the Hungarian algorithm, and a unique target ID is assigned to the same target; cross-frame target tracking is performed based on the target ID; After obtaining the fusion recognition result for any target, the position and state of the target are continuously tracked in multiple consecutive frames based on the unique target ID assigned to that target. For each frame, it is determined whether the target is located within the danger zone defined by the danger zone mask output by the semantic segmentation model, and at the same time, it is determined whether its fusion confidence exceeds a preset value. A time sliding window counter is maintained for the target. Only when the target is detected in multiple consecutive frames and simultaneously meets the conditions of being located within the danger zone and having a fusion confidence exceeding the preset value, the count reaches the preset frame requirement, and a valid alarm event for the target is generated. Subsequently, the audible and visual alarm device is immediately triggered and the alarm information is pushed to the monitoring center. At the same time, an alarm cooling timer is started for the target ID. During the cooling period, repeated alarms from the same target ID are suppressed.
5. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the monitoring method for preventing external damage as described in any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer-readable storage medium to implement the monitoring method for preventing external damage as described in any one of claims 1 to 3.