Monitoring video analysis method and system

Abstract

The invention discloses a monitoring video analysis method and system. The monitoring video analysis method comprises the following steps: performing region-of-interest extraction on a video stream at an edge gateway to obtain a first frame sequence; performing inter-frame similarity filtering on the first frame sequence at the edge gateway to obtain a second frame sequence; identifying a preset target in the second frame sequence at the edge gateway by using a lightweight target detection model; the edge gateway sends the third frame sequence of the preset target identified by the lightweight target detection model to the central server; and identifying whether the third frame sequence contains the target alarm information or not by using a multi-modal large model in the central server. According to the monitoring video analysis method and the monitoring video analysis system, the balance among the computing power, the bandwidth cost and the video analysis performance can be realized.

Description

Technical Field

[0001] This invention relates to the field of video data processing, and in particular to methods and systems for analyzing surveillance videos. Background Technology

[0002] Currently, in the field of large enterprise and public safety video surveillance, the latest architecture for video surveillance systems is "distributed acquisition of video streams by edge gateways + centralized management of video data by central servers." With the development of artificial intelligence technology, enterprises need to use multimodal large-scale models to intelligently analyze surveillance videos in order to identify key information such as violations and security incidents.

[0003] In existing technologies, in order to perform intelligent analysis of the content of surveillance videos, multimodal large models require high computing power and transmission bandwidth to process the corresponding video data due to the limitations of the structure and parameter scale of the multimodal large models. Therefore, multimodal large models can only be deployed in a central server.

[0004] However, the deployment methods of existing technologies have several drawbacks: First, the computing power and bandwidth costs are too high. A single camera outputs a frame rate of approximately 25fps with a resolution of 1080p. If full frame rate analysis is performed on N cameras, the central server needs to process 25×N frames of image data per second. Since the inference time for a single frame of a multimodal large model is often in the thousands, full analysis requires massive computing power. At this point, the bandwidth demand increases linearly with the number of cameras, leading to a surge in operating costs. Second, the risk of missed detections is significant. Existing technologies analyze videos using a fixed-frequency frame extraction method, such as extracting one frame per second. While this can reduce costs, it will miss high-speed moving targets and instantaneous events, such as abnormal equipment movement or personnel violations, failing to meet the reliability requirements of security monitoring. Third, it is difficult to balance operating costs and reliability. If computing power and bandwidth costs are to be reduced, the frame extraction frequency must be decreased, leading to a decrease in performance. If performance is to be improved, computing power and bandwidth costs must be increased. Summary of the Invention

[0005] To overcome one or more technical problems existing in the prior art, the embodiments of the present invention provide a surveillance video analysis method and a surveillance video analysis system, which can achieve a balance between computing power, bandwidth cost and video analysis performance in a video surveillance architecture of edge gateway plus central server.

[0006] The technical solution of the present invention is as follows.

[0007] In a first aspect, the present invention provides a method for analyzing surveillance video, comprising the following steps: The edge gateway extracts the region of interest from the video stream and selects the first frame sequence in the video stream that contains the region of interest. The first frame sequence is filtered for inter-frame similarity at the edge gateway to obtain the filtered second frame sequence. A lightweight target detection model is used at the edge gateway to identify preset targets in the second frame sequence, and then the frames that have identified preset targets are retained to form the third frame sequence. The edge gateway will send the third frame sequence of the target identified by the lightweight target detection model to the central server; A multimodal large model is used on the central server to identify whether the third frame sequence contains target alarm information.

[0008] Preferably, when extracting the region of interest, an image segmentation model is used, and the region of interest is dynamically adjusted across frames through prompts and a dynamic memory mechanism.

[0009] Preferably, the inter-frame similarity filtering includes the following steps: Extract ORB feature points from each frame in the first frame sequence; Calculate the feature similarity of the ORB feature points between each frame in the first frame sequence; If the feature similarity is higher than the dynamic threshold, retain the first frame of the first frame sequence, delete the remaining frames of the first frame sequence, and then use it as the second frame sequence. If the feature similarity is all lower than or equal to the dynamic threshold, the entire first frame sequence is used as the second frame sequence.

[0010] Preferably, the dynamic threshold is obtained using a dynamic region recognition algorithm, which includes the following steps: Each frame is sequentially converted to grayscale and de-noiseed; Create and / or update the background model for each frame; Calculate the difference between the current frame and the background model, and generate a binary foreground mask image for the current frame; The binary foreground mask image is subjected to morphological filtering and connected component screening in sequence; Calculate the dynamic pixel count ratio of the current binary foreground mask image, and use the dynamic pixel count ratio as a dynamic threshold.

[0011] Furthermore, the morphological filtering includes the following steps: Use rectangular structuring elements to remove small-area noise from the binary foreground mask image; Use another rectangular structuring element to fill the holes in the foreground region of the binary foreground mask image.

[0012] Furthermore, the connected component filtering includes the following steps: Calculate the area of ​​all connected regions in the binary foreground mask image; Remove portions of the connected regions in the binary foreground mask image whose area is less than a set number of pixels.

[0013] Preferably, before the edge gateway sends out the third frame sequence, the step includes: performing time-series encoding on the third frame sequence.

[0014] Preferably, before the central server uses the multimodal large model for recognition, it includes the following steps: optimizing the prompt words used to input the multimodal large model for target recognition alarm information by performing weight iteration and generating new prompt words based on the region of interest, preset target, and target alarm information.

[0015] Furthermore, the generation and optimization of prompts for target alarm information are carried out, and the specific steps include: Scene pattern recognition is performed based on the region of interest, preset target, and target alarm information as prompt words, and a prompt word template library is generated. The original input prompt words are semantically segmented to form multiple sub-prompt words; The multiple sub-prompt words are extended and adapted with the prompt word template library to generate multiple new prompt words; Based on the feedback from the recognition results of the multimodal large model, the weights of the prompt words in the prompt word template library are adjusted and new prompt words are added.

[0016] In a second aspect, the present invention provides a surveillance video analysis system, including an edge gateway and a central server; The edge gateway connects to the central server to transmit data. The edge gateway includes a region of interest module, a similarity filtering module, and a lightweight target detection module. The central server is equipped with a multimodal large model module; The region of interest module, similarity filtering module, lightweight object detection module, and multimodal large model module are connected sequentially for data transmission; The Region of Interest (ROI) module is used to extract the ROI from the video stream and filter out the first frame sequence in the video stream that contains the ROI. The similarity filtering module is used to perform inter-frame similarity filtering on the first frame sequence to obtain the filtered second frame sequence; The lightweight target detection module is used to identify preset targets in the second frame sequence, and then retains the frames that have identified preset targets to form the third frame sequence; The edge gateway is used to send the third frame sequence, which is identified as a preset target by the lightweight target detection model, to the central server; The multimodal large model module is used to identify whether the third frame sequence contains target alarm information.

[0017] Compared to existing technologies that are limited by the structure and parameter scale of large multimodal models and can only deploy them on a central server, the monitoring video analysis method and system of this invention have the following advantages: By extracting regions of interest (ROIs), a large number of pixels in the frame that are useless for identifying target alarm information can be filtered out, significantly reducing the amount of data that needs to be analyzed. By filtering based on inter-frame similarity, statically redundant frames can be removed based on the extracted ROIs, further reducing the amount of data to be analyzed. By filtering with a lightweight target detection model, only frames related to the preset alarm information are retained, which greatly reduces the amount of frame data that the edge gateway needs to transmit for detection. When faced with the greatly reduced amount of frame data, the multimodal large model of the central server does not need to invest massive computing power for analysis. In this way, the cost requirements for computing power and bandwidth are no longer high, and more surveillance cameras can be connected to the analysis network without causing a surge in operating costs. It also avoids the use of fixed-frequency frame extraction, ensuring that high-speed moving targets and instantaneous events are not missed, thus meeting the reliability requirements of security monitoring and balancing the contradiction between operating costs and reliability. Attached Figure Description

[0018] Figure 1 This is a flowchart of one embodiment of the surveillance video analysis system of the present invention.

[0019] Figure 2 for Figure 1 A flowchart for inter-frame similarity filtering.

[0020] Figure 3 for Figure 2 A flowchart of the dynamic region recognition algorithm.

[0021] Figure 4 for Figure 1 The flowchart for generating and optimizing prompt words.

[0022] Figure 5 This is a structural framework diagram of one embodiment of the video surveillance analysis method of the present invention. Detailed Implementation

[0023] Please refer to the diagrams, where the same component symbols represent the same components. The principles of the invention are illustrated by way of example implemented in a suitable operating environment. The following description is based on the illustrative specific embodiments of the invention and should not be construed as limiting the invention to other specific embodiments not detailed herein.

[0024] In the following description, specific embodiments of the invention will be illustrated with reference to the steps and symbols of operations performed by one or more computers, unless otherwise stated. Thus, it will be understood that these steps and operations, which are repeatedly mentioned as being performed by a computer, include manipulation by a computer processing unit representing electronic signals of data in a structured format. This manipulation transforms the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise alter the operation of the computer in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the invention are described above and are not intended to be limiting; those skilled in the art will understand that many of the steps and operations described below can also be implemented in hardware.

[0025] As used herein, the terms “component,” “module,” “system,” “interface,” “process,” etc., generally refer to computer-related entities: hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable application, an executing thread, a program, and / or a computer. As illustrated, both an application running on a controller and the controller itself can be components. One or more components may reside within an executing process and / or thread, and components may be located on a single computer and / or distributed across two or more computers.

[0026] Furthermore, the claimed subject matter can be implemented as a method, apparatus, or article of manufacture that uses standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. As used herein, the term "article of manufacture" is intended to include computer programs accessible from any computer-readable device, carrier, or medium. Of course, those skilled in the art will recognize that many modifications can be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0027] Although not required, embodiments are described within the general context of "computer-readable instructions" being executed by one or more electronic devices. Computer-readable instructions can be distributed via computer-readable media. Computer-readable instructions can be implemented as program modules, such as functions, objects, application programming interfaces (APIs), data structures, etc., that perform specific tasks or implement specific abstract data types. Typically, the functionality of such computer-readable instructions can be freely combined or distributed in various environments.

[0028] This document provides various operations of the embodiments. In one and / or more embodiments, one or more of the operations may constitute computer-readable instructions stored on one or more computer-readable media, which, when executed by an electronic device, will cause a computing device to perform the operations. The order in which some or all of the operations are described should not be construed as implying that these operations must be sequentially related. Those skilled in the art will understand alternative orderings that have the benefits of this specification. Moreover, it should be understood that not all operations are required to be present in every embodiment provided herein.

[0029] Furthermore, the term "preferred" as used herein is intended to refer to an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a concrete manner. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. Specifically, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0030] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, resources, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure that performs the function in the exemplary implementation of this disclosure shown herein. Moreover, although a particular feature of this disclosure has been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0031] The electronic hardware architecture of the surveillance video analysis method and system of the present invention comprises: multiple surveillance cameras that acquire video streams, each camera being directly connected and / or connected via a hub and / or connected via a bridge to an edge gateway; an edge gateway receiving the video streams, equipped with a graphics processing card capable of running lightweight target detection models, the specific product of which may be a server equipped with an NVIDIA T4 graphics card; a central server capable of running multimodal large models, the central server may include multiple databases, multiple CPUs, multiple GPU graphics processors, multiple NPU processors, and corresponding hardware information processing programs, the specific product of which may be one or more multi-graphics card servers equipped with NVIDIA TA100, A800, H20, etc., deploying multimodal large models such as Qianwen, Doubao, Chat-Gpt, etc., and a monitoring platform interface for human-computer interaction; data communication between the edge network and the central server can be based on a local network or a remote network. However, the specific product form of the edge gateway and central server where this surveillance video analysis method and system are located is not limited to personal computers, multiprocessor systems, consumer electronic devices, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0032] Please refer to Figure 1 , Figure 2 , Figure 3 , Figure 4 . Figure 1 This is a flowchart illustrating an embodiment of the surveillance video analysis method of the present invention. The surveillance video analysis method of this embodiment includes the following steps: S1: Multiple surveillance cameras capture images of the target location and send the resulting video streams to the edge gateway.

[0033] S2. At the edge gateway, the Region of Interest (ROI) is extracted from the video stream, and the first frame sequence containing the ROI is selected.

[0034] In step S2, it is preferable to use an image segmentation model to extract the region of interest; in other embodiments, a manual setting method or a combination of manual setting and an image segmentation model can also be used to extract the region of interest.

[0035] In step S2, when using an image segmentation model to extract the region of interest, the preferred image segmentation model is the SAM model, specifically the SAM2.0 model.

[0036] In step S2, the ROI is pre-defined according to business requirements. On the SAM2.0 model, the ROI is defined through cue engineering and dynamic memory mechanisms. Users only need to define the ROI in the first frame using point, box, or text cues. Then, the SAM2.0 model uses a memory attention module and a FIFO memory library to dynamically adjust the ROI across frames in the video stream. The quantification criteria for ROI setting include: (1) Intersection over Union (IoU): measures the degree of overlap between the predicted ROI and the actual region; (2) Global Accuracy: The percentage of effective ROIs in the video stream; (3) Inference speed (FPS): The number of frames processed per unit of time.

[0037] In step S2, the first frame sequence after region of interest extraction only contains the content of the region of interest. In practical applications, since the target area to be identified is often much smaller than the background area, the amount of data in the first frame sequence is significantly reduced compared to the amount of data in the original video stream.

[0038] S3. Perform inter-frame similarity filtering on the first frame sequence at the edge gateway to obtain the filtered second frame sequence.

[0039] When performing inter-frame similarity filtering in step S3, the following steps are included: S31. Extract ORB (Oriented Fast and Rotated BRIEF) feature points from each frame in the first frame sequence.

[0040] In this step S31, since the first frame sequence is extracted by ROI, the ORB feature points at this time are obviously the feature points of ROI.

[0041] S32. Calculate the feature similarity of ORB feature points between each frame in the first frame sequence.

[0042] S33. If there is a feature similarity higher than the dynamic threshold, retain the first frame of the first frame sequence, delete the remaining frames of the first frame sequence, and then convert the first frame sequence with the remaining frames deleted and only the first frame retained into the second frame sequence.

[0043] In step S33, the dynamic threshold is obtained using a dynamic region recognition algorithm; the dynamic region recognition algorithm includes the following steps: S331. Perform grayscale conversion and noise reduction on each frame in the first frame sequence in sequence; S332. Create and / or update the background model for each frame; S333. Calculate the difference between the current frame and the background model, and generate a binary foreground mask for the current frame; S334. Perform morphological filtering and connected component filtering on the binary foreground mask image in sequence. In step S334, morphological filtering includes the following steps: removing small-area noise from the binary foreground mask using a 3×3 rectangular structuring element; and filling holes in the foreground region of the binary foreground mask using a 5×5 rectangular structuring element. In step S334, the connected region filtering includes the following steps: calculating the area of ​​all connected regions in the binary foreground mask image; removing the portions of connected regions in the binary foreground mask image whose area is less than 50 pixels. S335. Calculate the proportion of dynamic pixels in the current binary foreground mask and use the proportion of dynamic pixels as the dynamic threshold.

[0044] In other embodiments of step S33, the dynamic region identification algorithm can also be a traditional Gaussian mixture model (GMM).

[0045] S34. If the feature similarity of ORB feature points between each frame in the first frame sequence is lower than or equal to the dynamic threshold, the entire first frame sequence is taken as the second frame sequence.

[0046] In step S3, after the feature similarity of ORB feature points is processed by a dynamic threshold, obvious static scene frames can be further removed, thereby continuing to compress the amount of video data that needs to be processed.

[0047] S4. Use a lightweight target detection model at the edge gateway to identify the preset target in the second frame sequence, and then retain the frames that have identified the preset target to form the third frame sequence.

[0048] In step S4, the preset targets to be identified by the lightweight target detection model include, but are not limited to, "people", "vehicles", "flames", "safety signs", etc. The lightweight target detection model can adopt a lightweight image detection model, including, but not limited to, YOLOv5-Lite, YOLO-Fastest, PicoDet, NanoDet, etc.

[0049] In step S4, the corresponding frame is retained for subsequent processing only when a preset target, such as "person", is detected. Frames without obvious targets are ignored, thus further reducing the amount of data to be processed in subsequent steps.

[0050] S5. The edge gateway will identify the third frame sequence of the preset target by the lightweight target detection model, perform time-series encoding, and send it to the central server.

[0051] In step S5, the third frame sequence is temporally encoded to quantify the temporal relationship between frames, which is used to detect the temporal correlation of subsequent target alarm information identification.

[0052] In step S5, based on the processing of the video stream in the previous steps, the amount of data in the third frame sequence is greatly reduced, and the detection target is not lost. Therefore, the bandwidth required when the edge gateway transmits the third frame sequence to the central server is also greatly reduced, and the computing power requirement of the central server is also greatly reduced.

[0053] S6. Before using the multimodal large model, the central server generates and optimizes the prompt words used to input the multimodal large model to identify target alarm information.

[0054] In step S6, the target alarm information refers to the image information content of the preset target-triggered related events in the third frame sequence.

[0055] In step S6, the optimization of prompt word generation includes the following steps: S61. Based on the region of interest, preset target, and target alarm information as prompt words, perform scene pattern recognition and generate a prompt word template library; S62. Semantically segment the original prompt words input by the user to form multiple sub-prompt words; S63. Extend and adapt multiple sub-prompt words with the prompt word template library to generate multiple structured new prompt words; S64. Iterate based on the recognition results of the multimodal large model, adjust the weights of various prompt words in the prompt word template library, and supplement new prompt words.

[0056] S7. Use a multimodal large model on the central server to identify whether the third frame sequence contains target alarm information based on the prompt words.

[0057] In step S7, the output of the multimodal large model identification includes confidence data.

[0058] S8. If the target alarm information is identified in step S7, the central server issues a visual alarm based on the identification results of the multimodal large model.

[0059] Compared to existing technologies, the surveillance video analysis method in this embodiment filters out a large number of pixels in the frame that are useless for identifying target alarm information by extracting regions of interest, significantly reducing the amount of data to be analyzed. By filtering based on inter-frame similarity, static redundant frames can be removed based on the extracted regions of interest, further reducing the amount of data to be analyzed. By filtering with a lightweight target detection model, only frames related to preset alarm information are retained, which greatly reduces the amount of frame data to be detected transmitted by the edge gateway. The multimodal large model of the central server does not need to invest massive computing power for analysis when faced with the greatly reduced amount of frame data. In this way, the cost requirements for computing power and bandwidth are no longer high, and more surveillance cameras can be connected to the analysis network without causing a surge in operating costs. It also avoids the use of fixed-frequency frame extraction, ensuring that high-speed moving targets and instantaneous events are not missed, thus meeting the reliability requirements of security monitoring and balancing the contradiction between operating costs and reliability.

[0060] Please refer to Figure 5 . Figure 5 This is a structural block diagram of one embodiment of a surveillance video analysis system according to the invention. The surveillance video analysis system of this embodiment includes an edge gateway and a central server; multiple surveillance cameras are respectively connected to the edge gateway to transmit video streams, and the edge gateway is connected to the central server for data transmission.

[0061] The edge gateway is equipped with a region of interest module, a similarity filtering module, and a lightweight object detection module; the central server is equipped with a multimodal large model module; the region of interest module, the similarity filtering module, the lightweight object detection module, and the multimodal large model module are connected in sequence for data transmission.

[0062] The Region of Interest (ROI) module is used to extract the ROI from the video stream and select the first frame sequence in the video stream that contains the ROI.

[0063] In the region of interest module, an image segmentation model is used to extract the region of interest; in other implementations, a manual setting method or a combination of manual setting and an image segmentation model can also be used to extract the region of interest.

[0064] In the Region of Interest (ROI) module, when using an image segmentation model for ROI extraction, the preferred image segmentation model is the SAM model, specifically version SAM2.0. ROIs are pre-defined according to business requirements. On the SAM2.0 model, ROIs are defined through cue engineering and dynamic memory mechanisms. Users only need to define the ROI in the first frame using point, box, or text cues. Then, the SAM2.0 model uses a memory attention module and a FIFO memory library to dynamically adjust the ROI across frames in the video stream.

[0065] In the region of interest module, the first frame sequence after region of interest extraction only contains the content of the region of interest. In practical applications, since the target area to be identified is often significantly smaller than the background area, the amount of data in the first frame sequence is significantly reduced compared to the amount of data in the original video stream.

[0066] The similarity filtering module is used to perform inter-frame similarity filtering on the first frame sequence to obtain the filtered second frame sequence.

[0067] The similarity filtering module includes an ORB feature similarity unit, a dynamic threshold unit, and a comparison unit. The ORB feature similarity unit and the dynamic threshold unit are connected to the SAM2.0 model, and the comparison unit is connected to both the ORB feature similarity unit and the dynamic threshold unit.

[0068] The ORB feature similarity unit is used to calculate the feature similarity of ORB feature points between each frame in the first frame sequence. The processing includes: extracting ORB feature points from each frame in the first frame sequence; and calculating the feature similarity of ORB feature points between each frame in the first frame sequence.

[0069] The dynamic threshold unit is used to calculate the dynamic threshold of the first frame sequence. The dynamic threshold is obtained using a dynamic region recognition algorithm.

[0070] The comparison unit is used to compare the feature similarity with the dynamic threshold and output the second frame sequence. The processing of the comparison unit includes: if there is a feature similarity higher than the dynamic threshold, the first frame of the first frame sequence is retained, the remaining frames of the first frame sequence are deleted, and then the first frame sequence with the remaining frames deleted and only the first frame retained is converted into the second frame sequence; if the feature similarity of ORB feature points between each frame in the first frame sequence is lower than or equal to the dynamic threshold, the entire first frame sequence is used as the second frame sequence.

[0071] The dynamic region recognition algorithm processes as follows: Each frame in the first frame sequence is sequentially grayscaled and de-noiseed; a background model is created and / or updated for each frame; the difference between the current frame and the background model is calculated to generate a binary foreground mask for the current frame; morphological filtering and connected component filtering are sequentially applied to the binary foreground mask; the dynamic pixel count ratio of the current binary foreground mask is calculated and used as a dynamic threshold. Specifically, the morphological filtering process involves: using 3×3 rectangular structuring elements to remove small-area noise from the binary foreground mask, and using 5×5 rectangular structuring elements to fill holes in the foreground region of the binary foreground mask; the connected component filtering process involves: calculating the area of ​​all connected components in the binary foreground mask, and removing components whose connected component area is less than 50 pixels.

[0072] The lightweight object detection module is used to identify a preset target in the second frame sequence, and then retains the frames in which the preset target is identified to form the third frame sequence. The lightweight object detection module is connected to the comparison unit and contains a lightweight object detection model.

[0073] In the lightweight target detection module, the preset targets to be identified by the lightweight target detection model include "people", "vehicles", "flames", "safety signs", etc. The lightweight target detection model can adopt a lightweight image detection model, including but not limited to YOLOv5-Lite, YOLO-Fastest, PicoDet, NanoDet, etc.

[0074] The edge gateway also includes a timing encoding module, which is connected to the lightweight target detection module. Before the edge gateway sends out the third frame sequence, the timing encoding module performs timing encoding on the third frame sequence, that is, adds timestamp information indicating the chronological order to each frame in the third frame sequence. The timing encoding module is also used to send the timing-encoded third frame sequence to the central server. Timing encoding of the third frame sequence can quantify the temporal relationship between frames, which is used to detect the temporal correlation of subsequent target alarm information identification.

[0075] The central server also includes a prompt word optimization module, which is connected to both the temporal coding module and the multimodal large model module. The prompt word optimization module generates and optimizes the prompt words used to input target recognition alarm information into the multimodal large model.

[0076] In the prompt word optimization module, the process of generating and optimizing prompt words is as follows: based on the region of interest, preset target, and target alarm information as prompt words, scene pattern recognition is performed to generate a prompt word template library; the prompt words input by the user are semantically segmented to form multiple sub-prompt words; the multiple sub-prompt words are extended and adapted with the prompt word template library to generate multiple structured prompt words; based on the feedback of the recognition results of the multimodal large model, the weights of various prompt words in the prompt word template library are adjusted and new prompt words are added.

[0077] The multimodal large model module is used to identify whether the third frame sequence contains target alarm information based on prompt words. Target alarm information refers to the image information content of the third frame sequence that triggers related events for a preset target. The output of the multimodal large model recognition includes confidence data. If the multimodal large model module recognizes target alarm information, the central server issues a visual alarm based on the recognition result of the multimodal large model.

[0078] Compared to existing technologies, the surveillance video analysis system in this embodiment can filter out a large number of pixels in the frame that are useless for identifying target alarm information by extracting regions of interest, which greatly reduces the amount of data that needs to be analyzed. By filtering through inter-frame similarity, static redundant frames can be removed based on the extracted regions of interest, further reducing the amount of data to be analyzed. By filtering through a lightweight target detection model, only frames related to preset alarm information are retained, which greatly reduces the amount of frame data that the edge gateway needs to transmit for detection. When faced with the greatly reduced amount of frame data, the multimodal large model of the central server does not need to invest massive computing power for analysis. In this way, the cost requirements for computing power and bandwidth are no longer high, and more surveillance cameras can be connected to the analysis network without causing a surge in operating costs. It also avoids the use of fixed-frequency frame extraction, so that high-speed moving targets and instantaneous events are not missed, thus meeting the reliability requirements of security monitoring and balancing the contradiction between operating costs and reliability.

[0079] Although the present invention has been disclosed above with reference to preferred embodiments, the above preferred embodiments are not intended to limit the present invention. Those skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the claims.

Claims

1. A method for analyzing surveillance video, characterized in that, Including the following steps: The edge gateway extracts the region of interest from the video stream and filters out the first frame sequence in the video stream that contains the region of interest. The edge gateway performs inter-frame similarity filtering on the first frame sequence to obtain a filtered second frame sequence; A lightweight target detection model is used at the edge gateway to identify a preset target in the second frame sequence, and then the frames that identify the preset target are retained to form a third frame sequence; The edge gateway sends the third frame sequence of the target identified by the lightweight target detection model to the central server; The central server uses a multimodal large model to identify whether the third frame sequence contains target alarm information.

2. The surveillance video analysis method according to claim 1, characterized in that, When extracting the region of interest, an image segmentation model is used, and the region of interest is dynamically adjusted across frames through prompts and a dynamic memory mechanism.

3. The surveillance video analysis method according to claim 1, characterized in that, The process of performing inter-frame similarity filtering includes the following steps: Extract ORB feature points from each frame in the first frame sequence; Calculate the feature similarity of the ORB feature points between each frame in the first frame sequence; If the feature similarity is higher than the dynamic threshold, retain the first frame of the first frame sequence, delete the remaining frames of the first frame sequence, and then use it as the second frame sequence. If the feature similarity is all lower than or equal to the dynamic threshold, the entire first frame sequence is used as the second frame sequence.

4. The surveillance video analysis method according to claim 1, characterized in that, The dynamic threshold is obtained using a dynamic region recognition algorithm, which includes the following steps: Each frame is sequentially converted to grayscale and de-noiseed; Create and / or update the background model for each frame; Calculate the difference between the current frame and the background model, and generate a binary foreground mask image for the current frame; The binary foreground mask image is subjected to morphological filtering and connected component screening in sequence; Calculate the dynamic pixel count ratio of the current binary foreground mask image, and use the dynamic pixel count ratio as a dynamic threshold.

5. The surveillance video analysis method according to claim 4, characterized in that, The morphological filtering includes the following steps: Use rectangular structuring elements to remove small-area noise from the binary foreground mask image; Use another rectangular structuring element to fill the holes in the foreground region of the binary foreground mask image.

6. The surveillance video analysis method according to claim 4, characterized in that, The connected component filtering includes the following steps: Calculate the area of ​​all connected regions in the binary foreground mask image; Remove portions of the connected regions in the binary foreground mask image whose area is less than a set number of pixels.

7. The surveillance video analysis method according to claim 1, characterized in that, Before the edge gateway sends out the third frame sequence, the steps include: timing encoding the third frame sequence.

8. The surveillance video analysis method according to claim 1, characterized in that, Before the central server uses the multimodal large model for recognition, the following steps are included: for the prompt words used to input the multimodal large model to identify target alarm information, the weight iteration and new prompt word generation optimization are performed on the prompt words, based on the region of interest, the preset target, and the target alarm information.

9. The surveillance video analysis method according to claim 8, characterized in that, The steps for generating and optimizing the prompts for target alarm messages include: Scene pattern recognition is performed based on the region of interest, preset target, and target alarm information as prompt words, and a prompt word template library is generated. The original input prompt words are semantically segmented to form multiple sub-prompt words; The multiple sub-prompt words are extended and adapted with the prompt word template library to generate multiple new prompt words; Based on the feedback from the recognition results of the multimodal large model, the weights of the prompt words in the prompt word template library are adjusted and new prompt words are added.

10. A surveillance video analysis system, characterized in that, Including edge gateways and central servers; The edge gateway is connected to the central server for data transmission; The edge gateway includes a region of interest module, a similarity filtering module, and a lightweight target detection module. The central server is equipped with a multimodal large model module; The region of interest module, the similarity filtering module, the lightweight target detection module, and the multimodal large model module are sequentially connected for data transmission; The Region of Interest (ROI) module is used to extract the ROI from the video stream and filter out the first frame sequence in the video stream that contains the ROI. The similarity filtering module is used to perform inter-frame similarity filtering on the first frame sequence to obtain the filtered second frame sequence; The lightweight target detection module is used to identify a preset target in the second frame sequence, and then retain the frames that have identified the preset target to form a third frame sequence; The edge gateway is used to send the third frame sequence, which is identified as a preset target by the lightweight target detection model, to the central server; The multimodal large model module is used to identify whether the third frame sequence contains target alarm information.

Images