Monitoring alarm rechecking method and device based on multi-modal large model, and terminal equipment

Abstract

The invention belongs to the field of intelligent security and protection, and particularly relates to a monitoring alarm re-checking method and device based on a multi-modal large model and terminal equipment, the monitoring alarm re-checking method obtains multiple frames of images from an alarm video and inputs the multiple frames of images into the multi-modal large model for analysis, the multiple frames of images in the alarm video provide rich image context information, and the image quality is improved. The model analysis capability is remarkably improved, and adverse effects caused by a fixed shooting angle of a camera, image dithering and the like are reduced; a'sample 'image and a to-be-detected image (namely an alarm image) jointly participate in image analysis, so that the image analysis capability of the model under the condition of few samples is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent security and protection, and particularly relates to a monitoring alarm review method and device based on a multi-modal large model and a terminal device. BACKGROUND

[0002] In the existing security and protection monitoring system, there are many false detections leading to false alarms. The main reasons are as follows:

[0003] 1. On the hardware level, the security and protection system is affected by the camera shooting angle, unstable lighting conditions and image clarity, resulting in poor image imaging effect;

[0004] 2. On the algorithm level, there is analysis of single-frame images, resulting in loss of context information; it is difficult to collect abnormal data; existing target detection models, image segmentation models and image processing algorithms are constrained by model size and computing power, resulting in frequent false detections. SUMMARY

[0005] The embodiments of the present application provide a monitoring alarm review method and device based on a multi-modal large model and a terminal device, aiming to improve the intelligent level of the existing security and protection monitoring system.

[0006] To achieve the above-mentioned purpose, the present application provides a monitoring alarm review method based on a multi-modal large model, comprising the following steps:

[0007] S1. receiving alarm information sent by a monitoring system, wherein the alarm information comprises an alarm category and corresponding alarm images and alarm videos;

[0008] S2. matching a corresponding preset prompt word template according to the alarm category;

[0009] S3. performing time sequence frame extraction on the alarm videos to generate multiple images, or combining the alarm images with sample images;

[0010] S4. inputting the processed image data and prompt words into a multi-modal large model for review reasoning;

[0011] S5. comparing the output results of the multi-modal large model with the original alarm results of the monitoring system, and retaining the alarm when the two are consistent, and deleting the alarm when the two are inconsistent.

[0012] Optionally, the time sequence frame extraction in step S3 is specifically: extracting N frames of images of a continuous time period before and after the alarm time from the alarm video to form an image sequence containing time context information, wherein N is greater than or equal to 3.

[0013] Optionally, when the alarm category in step S1 is not wearing a safety helmet or smoking, the alarm video corresponding to the alarm category is time-series frame-extracted in step S3 to generate multiple images.

[0014] When the alarm category in step S1 is not wearing work clothes, the alarm image corresponding to the alarm category is combined with a work clothes sample image in step S3 to form an input data package, and a few-sample comparison analysis is realized.

[0015] Optionally, the prompt word template in step S2 contains dynamic guidance logic:

[0016] For the prompt word templates of not wearing a safety helmet and smoking, the motion trajectory of the person between multiple frames is required to be analyzed;

[0017] For the prompt word template of not wearing work clothes, the visual feature difference between the sample and the image to be detected is required to be compared.

[0018] Optionally, when the alarm category in step S1 is not wearing a safety helmet, the prompt word template guidance model in step S2 comprehensively analyzes the visible state of the safety helmet during the motion of the person;

[0019] When the alarm category in step S1 is smoking, the prompt word template guidance model in step S2 comprehensively analyzes the action of the person's hand;

[0020] When the alarm category in step S1 is not wearing work clothes, the prompt word template guidance model in step S2 identifies the matching degree of the work clothes features.

[0021] Optionally, the multi-modal large model is obtained by using a self-built dataset to cooperate with the LLaMA-Factory framework and the QLoRA method for large model fine-tuning training on the basis of an open-source multi-modal large model, and the self-built dataset contains the following contents: at least 5000 labeled security scene images; negative samples covering illumination changes, occlusions, and multiple angles; target positions and text descriptions labeled in COCO format.

[0022] Optionally, the fine-tuning adopts the QLoRA method: freeze the original multi-modal large model weight, and realize domain adaptation by training a new adaptive matrix.

[0023] Optionally, step S5 further comprises:

[0024] The multi-modal large model outputs a structured result, including whether the rule is violated and the basis for the explanation;

[0025] When the results are inconsistent, an artificial review process is triggered.

[0026] The application also provides a monitoring alarm review device based on a multi-modal large model, comprising:

[0027] The communication module is configured to receive alarm information of a monitoring system.

[0028] The prompt word engine is configured to call a corresponding prompt word template according to the alarm category.

[0029] The image processor is configured to perform a video frame extraction or image combination operation.

[0030] The multi-modal large model inference module is configured to perform a review analysis and output a structured result.

[0031] The judgment module is configured to compare the original alarm result with the review result and perform a corresponding operation.

[0032] The present application also provides a terminal device, which comprises a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the steps of the monitoring alarm review method according to any one of the above when executing the computer program.

[0033] The monitoring alarm review method, device and terminal device based on a multi-modal large model provided by the present application have the following advantages:

[0034] The monitoring alarm review method obtains multiple frames of images from an alarm video and inputs the images into a multi-modal large model for analysis. The multiple frames of images in the alarm video provide rich image context information, significantly improve the model analysis capability, and reduce the adverse effects caused by a fixed camera shooting angle, image jitter and the like. The “sample” image and the to-be-detected image (i.e., the alarm image) are used together for image analysis, which improves the image analysis capability of the model in the case of few samples. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Among them:

[0037] Figure 1 is a flowchart of the monitoring alarm review method based on a multi-modal large model according to an embodiment of the present application;

[0038] Figure 2 is a structural principle diagram of the monitoring alarm review system based on a multi-modal large model;

[0039] Figure 3 is a schematic diagram of an alarm management interface of a monitoring system;

[0040] Figure 4 is a smoking alarm image detected in a monitoring system;

[0041] Figure 5 is an undressing alarm image detected in a monitoring system;

[0042] Figure 6 is an alarm image of not wearing a safety helmet detected in a monitoring system;

[0043] Figure 7 is a plurality of images obtained by multi-time period frame extraction on an alarm video of not wearing a safety helmet detected in a monitoring system;

[0044] Figure 8 is a schematic diagram of a sample image and an undressing alarm image when detecting a workwear, wherein the upper three images are workwear sample images, and the lower part is a false detection image of the monitoring system, which is a to-be-detected review image;

[0045] Figure 9 is a multi-modal large model workwear detection result schematic diagram shown in an embodiment of the present application;

[0046] Figure 10 is a schematic diagram of a monitoring alarm review device based on a multi-modal large model shown in an embodiment of the present application;

[0047] Figure 11 is a schematic diagram of a terminal device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be implemented in many other different forms, and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0050] The existing intelligent security monitoring system generally uses target detection, image segmentation, key point detection and other algorithms. On the hardware level, it is affected by the camera shooting angle, the complexity of the camera shooting environment, etc. Using a single frame of image, the context information of the image cannot be obtained, and frequent false alarm phenomena are prone to occur. In addition, on the algorithm level, the visual algorithm model in the existing intelligent security monitoring system has poor detection effect in the case of few samples and zero samples.

[0051] Taking the existing security intelligent alarm system as an example, there are frequent false alarm phenomena in multiple detection projects, which reduces the overall intelligent level of the system. For example, in the safety helmet wearing detection project, when the personnel correctly wear the safety helmet, the intelligent security monitoring system will frequently output the result of not wearing the safety helmet when the personnel make various actions such as walking, looking up and turning the head; in the personnel smoking detection project, when the personnel make actions similar to smoking such as touching the nose and wiping the mouth, the intelligent security monitoring system will frequently output the result of detecting smoking action; in the non-wearing of work clothes detection project, the detection effect is very poor for the untrained work clothes category, and the model needs to collect new work clothes images again, and after re-training the model, a better recognition effect can be obtained.

[0052] As shown in Figure 3 , the interface shows the alarm management interface information of the intelligent monitoring system, from which it can be seen that the alarm information includes alarm category, alarm picture and alarm video, etc., and the alarm video contains 5S video stream information before and after the alarm image moment, and the time length before and after here is a configurable parameter, which can be adjusted according to needs.

[0053] In the existing intelligent monitoring alarm system, a rectangular frame or other marking means is usually used in the alarm image to mark the personnel and the area that violate the rules. As shown in Figure 4 , it indicates that the intelligent monitoring system detects that the personnel (left one) marked by the red box exists smoking action, but in fact the personnel does not smoke, but makes a phone call, which exists false detection.

[0054] As shown in Figure 5 , the laboratory requires white gowns as work clothes, but in the red mark box in the above figure, due to the personnel wearing a backpack on the back, the existing intelligent monitoring alarm system produces false detection.

[0055] In Figure 6 , due to the fixed camera shooting angle, during the movement of the personnel, the personnel actually wear safety helmets, but the detection is not accurate, and there are many false detection situations.

[0056] To solve the above problems and improve the intelligence level of the existing intelligent security monitoring system, an embodiment of the present application proposes a monitoring alarm review method based on a multi-modal large model, which further improves the recognition accuracy of the monitoring system by inputting the alarm image and the alarm video into the multi-modal large model for secondary review.

[0057] In one embodiment, as shown in the figure, the monitoring alarm review method based on the multi-modal large model includes the following steps: Figure 1

[0058] S1. Receive the alarm information sent by the monitoring system, which contains the alarm category and the corresponding alarm image and alarm video;

[0059] S2. Match the corresponding preset prompt word template according to the alarm category;

[0060] S3. Time sequence frame extraction is performed on the alarm video to generate multiple images, specifically, N frames of images of continuous time periods before and after the alarm time are extracted from the alarm video to form an image sequence containing time context information, where N≥3. Or the alarm image and the sample image are combined for processing;

[0061] S4. Input the processed image data and the prompt word into the multi-modal large model for review reasoning;

[0062] S5. Compare the output results of the multi-modal large model with the original alarm results of the monitoring system, and keep the alarm if they are consistent, or delete the alarm if they are inconsistent.

[0063] The multi-modal large model used in the embodiments of the present application is obtained by fine-tuning training using a self-built dataset in combination with the LLaMA-Factory framework and the QLoRA method based on an open-source multi-modal large model, wherein more than 5000 images are collected and screened in the self-built dataset, and the images mainly come from alarm images of intelligent security monitoring systems, covering multiple scenes such as daytime / nighttime, strong light / low illumination, different angles, and occlusion; the images are labeled with position classes and image content descriptions in the COCO data labeling format.

[0064] In the model fine-tuning method, the corresponding open-source multi-modal large model weight is loaded in the LLaMA-Factory framework, the original large model weight is frozen using the QLoRA method, only the newly added small-scale adaptive matrix is trained, and the model is fine-tuned. Obtain the trained special multi-modal large model. Experiments prove that the fine-tuned model has further enhanced inference ability in the professional field.

[0065] In combination with Figure 2 ​As shown, after the camera collects video or image data, it is first input into the existing intelligent monitoring system. In the intelligent monitoring system, after detection and processing, the first alarm category, alarm image and alarm video are obtained. Then the intelligent monitoring system sends the alarm category, alarm image and alarm video to the server deploying the multi-modal large model through HTTP, Socket and other communication means. Then the server matches the corresponding prompt word according to the alarm category, and processes the alarm video or alarm image, and inputs them into the multi-modal large model for review and reasoning analysis. Finally, the output result of the multi-modal large model is returned to the intelligent monitoring system, and compared with the first result of the intelligent monitoring system. If the two results are inconsistent, the alarm is deleted and then reviewed by artificial; if the two results are consistent, the alarm information is retained and the final alarm result is determined.

[0066] It should be noted that in step S1, the monitoring system itself has certain task detection capability, but in the hardware layer, it is affected by many factors such as fixed position shooting of the camera, personnel motion change and image jitter. In the algorithm power layer, since the current intelligent security system mainly detects and identifies single images, it cannot obtain the context information of the image; and due to the limitation of model size and algorithm, there are many false detection situations, which need to be reviewed and determined by personnel in the later stage, increasing the workload.

[0067] As shown in the alarm management interface of the current intelligent security monitoring system, the alarm category, alarm picture and alarm video and other key information can be obtained in the management interface. Figure 3

[0068] In the embodiment of the present application, the alarm category, alarm picture and alarm video are packaged and sent to the server deploying the special multi-modal large model through Socket, HTTP and other data transmission means, and then unpacked and image processed in the server end.

[0069] In step S2, after the data packet sent by the monitoring system is obtained and unpacked, the alarm category judged by the monitoring system in the data packet is extracted, such as not wearing a safety helmet, not wearing work clothes, smoking, etc., and then the large model prompt word designed in advance is matched according to the alarm category.

[0070] Matching method: a mapping table is established. In the current monitoring system, different alarm situations are assigned categories. According to the assigned alarm category, the corresponding large model prompt word is mapped. The prompt word of the large model can exist in tables, world, txt files, etc.

[0071] In this step of matching the large model prompt word, multiple sets of prompt words are designed according to different categories of alarm information to improve the analysis and reasoning ability of the model. ​

[0072] The prompt word template contains dynamic guidance logic:

[0073] For the prompt word template of not wearing a safety helmet and smoking, the motion trajectory of the personnel between multiple frames is required to be analyzed;

[0074] For the prompt word template of not wearing work clothes, the visual feature difference between the comparison sample and the image to be detected is required.

[0075] Specifically, when the alarm category in step S1 is not wearing a safety helmet, the prompt word template guidance model comprehensively analyzes the visible state of the safety helmet during the motion of the personnel in step S2; when the alarm category in step S1 is smoking, the prompt word template guidance model comprehensively analyzes the action of the personnel's hands in step S2; when the alarm category in step S1 is not wearing work clothes, the prompt word template guidance model identifies the matching degree of work clothes features in step S2.

[0076] In an implementation manner, for example, in the detection of not wearing a safety helmet, since multiple images in different time periods are input, the prompt word focuses on prompting the large model to pay attention to the motion of the personnel, comprehensively analyze multiple images, and finally generate a final determination result according to the analysis of multiple images.

[0077] Prompt word example for not wearing a safety helmet:

[0078] You are processing multiple images in a video sequence. These images are taken from different time points and show the activities of personnel in a certain scene. Your task is:

[0079] Observe and analyze these images frame by frame to identify whether there are workers in the images and whether they are wearing safety helmets.

[0080] Before making a final judgment, please pay attention to the following points:

[0081] The same personnel may appear in multiple images, and their motion trajectories need to be tracked to avoid repeated judgments;

[0082] Some images may have personnel blocked or poor angles, and other frame information should be used to assist in judgment;

[0083] If someone does not wear a safety helmet in some images, but appears in other images and is confirmed not to wear a safety helmet, it can be determined as a violation;

[0084] The final output should be based on the comprehensive reasoning of all image information to give whether there is the behavior of "not wearing a safety helmet".

[0085] The output format is as follows:

[0086] Whether there is the behavior of not wearing a safety helmet:

Yes / No

[0087] Judgment basis: Brief description:

What images, human actions, or features are used to make the conclusion?

[0088] In the task of tool detection, the corresponding prompt words mainly focus on the comparison between the image to be detected and the sample image.

[0089] Task prompt word example:

[0090] You will receive a set of image inputs, including:

[0091] Multiple sample images of tools: Show the standard tool's wearing style, color, style, and other features;

[0092] One image to be detected: Need to determine whether the staff in it is wearing a tool consistent with or similar to the sample.

[0093] Please follow the steps below for analysis:

[0094] Observe the sample image and summarize the key features of the tool (such as color, style, logo, and wearing method);

[0095] Analyze the image to be detected and identify whether there is a staff in the image;

[0096] If there is a staff, please determine whether he is wearing a tool based on the tool features in the sample image;

[0097] Please note the following factors that may affect the judgment:

[0098] The staff in the image may be partially obscured (such as facing away from the camera, obscured by equipment, etc.);

[0099] Poor light conditions or reflections may make it difficult to identify;

[0100] The tool may have slight differences (such as color depth, accessories), which need to be judged comprehensively whether it belongs to the same type of tool.

[0101] Finally, output a clear judgment conclusion and briefly explain the basis.

[0102] Output format:

[0103] Whether to wear a tool:

Yes / No

[0104] Judgment basis: Brief description:

For example: "The tool style and color in sample images 1 and 3 are highly consistent", "Although there is slight obstruction, but the obvious logo can be seen" etc.

[0105] In step S3, different tasks need to be made according to the alarm categories of the monitoring system to develop corresponding processing methods.

[0106] For example, for safety helmet wearing detection, the main factors affecting the false detection of the intelligent security monitoring system are camera fixed angle shooting and personnel movement, etc. Therefore, this part performs frame extraction processing on the video, extracts multiple images at different time periods from the alarm video, and inputs the corresponding prompt words into the special multi-modal large model for analysis. Specifically, as shown in Figure 7 the alarm video is automatically processed by video frame extraction according to the time sequence, multiple images at different time periods are obtained, and the context information of the images is obtained. At the same time, the multiple images are input into the multi-modal large model, and a comprehensive judgment is made on whether the personnel wear safety helmets. Reduce the false detection caused by the fixed camera shooting angle and personnel movement.

[0107] For tooling detection, the main factors affecting the false detection of the intelligent security monitoring system are that the model detection ability is poor due to the small number of training samples, and the new sample model needs to be retrained. Therefore, this part needs to input the tooling sample image and the image to be detected into the multi-modal large model, and infer and analyze with the corresponding prompt words. As shown in Figure 8 the upper three images in the figure are tooling sample images, and the lower part is a misdetected image of the monitoring system, which is a to-be-detected review image. By inputting the tooling sample image and the to-be-detected review image into the multi-modal large model at the same time, a comparative analysis is performed to obtain the final result. Figure 9 For Figure 8 the large model recognition result of the to-be-detected image, it can be seen that after the monitoring system misdetected, the multi-modal large model performed review detection and obtained the correct review result.

[0108] In summary, when the alarm category in step S1 is not wearing a safety helmet or smoking, multiple images are generated by time sequence frame extraction on the alarm video corresponding to the alarm category in step S3; when the alarm category in step S1 is not wearing a safety helmet, the alarm image and the tooling sample image corresponding to the alarm category are combined into an input data packet in step S3, and a small sample comparison analysis is realized.

[0109] In Step 4, analysis inference is performed using a special-purpose multimodal large model to generate inference results. The core architecture of the multimodal large model is to fuse a visual encoder and a language encoder, which can realize the understanding and joint modeling of image and text information. In the inference process, in the visual feature extraction stage, a visual encoder such as Vision Transformer is used to extract features from the image, and the features are projected into high-dimensional visual embedding vectors to obtain information such as objects, scenes, and spatial relationships in the image; in the text feature encoding stage, the prompt words are mapped into continuous vector representations by a tokenizer and an embedding layer; in the cross-modal feature fusion stage, a cross-attention mechanism is used in the middle layer of the model to interact, so that the language part can dynamically focus on the key areas in the image, and the prompt image features enhance the accuracy of language understanding context; in the language decoding and generation stage, the decoder uses a self-recursive method to gradually generate output text based on the existing image and text; finally, after post-processing, the judgment results are extracted and structured output.

[0110] In Step 5, the monitoring system obtains the inference results of the multimodal large model. If the detection results of the monitoring system are consistent with the analysis results of the multimodal large model, the monitoring system continues to retain the alarm image and the alarm video.

[0111] If the results are inconsistent, the monitoring system will delete the corresponding alarm picture and alarm video, and then it can be reviewed by artificial.

[0112] In summary, the monitoring alarm review method provided by the embodiment has at least the following beneficial technical effects:

[0113] 1. Multiple images are automatically obtained from the alarm video and input into the multimodal large model for analysis, providing rich image context information, significantly improving the model analysis capability, and reducing the adverse effects of fixed camera shooting angle, image jitter, etc.

[0114] 2. The "example" image is used together with the to-be-detected image to participate in image analysis, improving the image analysis capability of the model in the case of few samples;

[0115] 3. A security monitoring dataset is constructed, and the dataset is input into the multimodal large model for model fine-tuning, improving the analysis and inference capability of the multimodal large model in the special field;

[0116] 4. A security monitoring special-purpose large model prompt word template is constructed to obtain efficient and accurate model output results.

[0117] In addition, the embodiment of the present application also provides a monitoring alarm review device based on a multimodal large model, as shown in Figure 10 The monitoring alarm review device comprises:

[0118] A communication module configured to receive alarm information of a monitoring system;

[0119] The prompt word engine is configured to call a corresponding prompt word template according to the alarm category;

[0120] The image processor is configured to perform a video frame extraction or image combination operation;

[0121] The multi-modal large model inference module is configured to perform a review analysis and output a structured result;

[0122] The judgment module is configured to compare the original alarm result with the review result and perform a corresponding operation.

[0123] In addition, the embodiment of the present application further provides a terminal device, as shown in the figure, Figure 11 The terminal device 11 includes a memory 1102 and a processor 1101, the memory 1102 stores a computer program 1103 capable of running on the processor, and the processor 1101 implements the steps of the pathological digital section making method according to any one of the above when executing the computer program 1103.

[0124] The terminal device 11 can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The terminal device 11 can include but is not limited to a processor 1101 and a memory 1102. Those skilled in the art can understand that Figure 11 The terminal device 11 is only an example and does not constitute a limitation on the terminal device 11, and can include more or fewer components than the figure, or combine certain components, or different components, for example, can also include an input / output device, a network access device and the like.

[0125] The processor 1101 can be a central processing unit (CPU), and can also be other general-purpose processors 1101, digital signal processors 1101 (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor 1101 can be a microprocessor 1101 or can also be any conventional processor 1101 or the like.

[0126] The memory 1102 may, in some embodiments, be an internal storage unit of the terminal device 11, such as a hard disk or a memory of the terminal device 11. The memory 1102 may, in other embodiments, also be an external storage device of the terminal device 11, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, and the like equipped on the terminal device 11. Further, the memory 1102 may also include both an internal storage unit and an external storage device of the terminal device 11. The memory 1102 is used to store an operating system, application programs, a Boot Loader, data, and other programs, such as program codes of the computer program 1103 and the like. The memory 1102 may also be used to temporarily store data that has been output or is to be output.

[0127] It should be understood that the above memory is an example but not a limiting description, for example, the memory in the embodiments of the present application can also be a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synch link DRAM (SLDRAM), and a direct memory bus RAM (DRAM), and the like. That is, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable type of memory.

[0128] Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0129] Those of ordinary skill in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and module can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0130] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiment is merely an example, and there can be other division manners. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or modules, and can be in electrical, mechanical or other forms.

[0131] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0132] In addition, each functional module in the embodiments of the present application can be integrated in one processing module, or each module can exist physically separately, or two or more modules can be integrated in one module.

[0133] If the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a cloud server 20) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various program codes that can be stored in the medium.

[0134] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A monitoring alarm review method based on a multimodal large model, characterized in that: The following steps are involved: S1 receives the alarm information sent by the monitoring system, the alarm information includes the alarm category and its corresponding alarm image and alarm video; S2. Match the corresponding preset prompt word template according to the alarm category; S3. Perform time-sequential frame extraction on the alarm video to generate multiple frames, or combine the alarm image with the sample image; S4. Input the processed image data and prompt words into the multimodal large model for verification and reasoning; S5. Compare the output results of the multimodal large model with the original alarm results of the monitoring system. If the two are consistent, retain the alarm; if they are inconsistent, delete the alarm.

2. The monitoring alarm review method according to claim 1, characterized in that: The time-sequential frame extraction in step S3 is specifically as follows: N frames of images in a continuous period before and after the alarm moment are extracted from the alarm video to form an image sequence containing temporal context information, where N≥3.

3. The monitoring alarm review method according to claim 1, characterized in that: When the alarm category in step S1 is not wearing a helmet or smoking, in step S3, the alarm video corresponding to the alarm category is time-sequentially framed to generate multiple frames of images; When the alarm category in step S1 is not wearing work clothes, in step S3, the alarm image corresponding to the alarm category and the work clothes sample image are combined into an input data packet to achieve a small number of sample comparison analysis.

4. The monitoring alarm review method according to claim 3, characterized in that: The prompt word template in step S2 includes dynamic guidance logic: For the prompt word templates of not wearing a helmet and smoking, it is required to analyze the movement trajectory of the person between multiple frames; For the prompt word template of not wearing work clothes, it is required to compare the visual feature differences between the sample and the image to be inspected.

5. The monitoring alarm review method according to claim 4, characterized in that: When the alarm category in step S1 is not wearing a helmet, the prompt word template in step S2 guides the model to comprehensively analyze the visible status of the helmet during the person's movement; When the alarm category in step S1 is smoking, the prompt word template in step S2 guides the model to comprehensively analyze the person's hand movements; When the alarm category in step S1 is not wearing work clothes, the prompt word template in step S2 guides the model to identify the matching degree of work clothes features.

6. The monitoring alarm review method according to any one of claims 1 to 5, characterized in that: The multimodal large model is obtained by fine-tuning the large model based on the open source multimodal large model using a self-built dataset in conjunction with the LLaMA-Factory framework and the QLoRA method. The self-built dataset contains the following contents: at least 5,000 annotated security scene images; negative samples covering lighting changes, occlusions, and multiple angles; and target positions and text descriptions annotated in COCO format.

7. The monitoring alarm review method according to claim 6, characterized in that: The fine-tuning adopts the QLoRA method: freezing the weights of the original multimodal large model and achieving domain adaptation by training a new adaptation matrix.

8. The monitoring alarm review method according to claim 1, characterized in that: Step S5 further comprises: The multimodal large model outputs structured results, including judgments on violations and explanations of the basis; Inconsistent results trigger a manual review process.

9. A monitoring alarm review device based on a multimodal large model, characterized in that: include: Communication module, used for receiving alarm information from the monitoring system; Prompt word engine, used to call the corresponding prompt word template according to the alarm category; An image processor, used to perform video frame extraction or image combination operations; Multimodal large model reasoning module, used to perform review analysis and output structured results; The judgment module is used to compare the original alarm result with the review result and perform corresponding operations.

10. A terminal device, characterized in that: The terminal device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the monitoring alarm review method according to any one of claims 1 to 8 are implemented.

Images