Anomaly Analysis Method and Device Based on Real-time Target Detection and Multimodal Large Model

Abstract

This application provides an anomaly analysis method and apparatus based on real-time object detection and a multimodal large model. The method includes: inputting video frame images into an object detection model to obtain predicted categories and predicted locations; constructing prompt words corresponding to the video frame images, the prompt words including predicted categories and predicted locations; inputting the prompt words and video frame images into a multimodal large model; determining a first image feature corresponding to the predicted category through the multimodal large model; obtaining a region image of the predicted location from the video frame images; determining a second image feature of the region image; if the similarity between the first image feature and the second image feature is greater than a similarity threshold, then determining that the object to be detected is a risky object; analyzing whether the target object has abnormal actions based on the video frame images using the multimodal large model; and outputting the model inference result. This application's solution can effectively reduce the false alarm rate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an anomaly analysis method and apparatus based on real-time target detection and multimodal large model. Background Technology

[0002] Object detection is a computer vision technique used to identify objects appearing in images or videos and to determine their location and category within the image. Anomaly analysis is a computer vision technique used to identify whether target objects (such as pedestrians) in images or videos exhibit abnormal behavior.

[0003] To achieve object detection, a video stream can be captured using a video capture device. The video frame images within the stream are then input into a deep learning model. The deep learning model performs object detection based on these video frame images, determining the object's location and category within the image. Similarly, for anomaly analysis, a video stream can be captured using a video capture device. The video frame images within the stream are then input into a deep learning model. The deep learning model performs anomaly analysis based on these video frame images to determine if any abnormal behavior exists within the target object.

[0004] However, when using deep learning models for object detection, the results exhibit a high false positive rate, indicating a significant possibility of incorrect detection outcomes. Furthermore, when using deep learning models for anomaly analysis, the results also show a high false positive rate, suggesting a significant possibility of incorrect anomaly analysis results. Summary of the Invention

[0005] This application provides an anomaly analysis method based on real-time target detection and multimodal large model. The method includes: inputting a video frame image into a target detection model to obtain a predicted category and a predicted location; wherein, the video frame image includes a target object, the predicted category represents the category of the object to be detected carried by the target object, and the predicted location represents the location of the object to be detected.

[0006] Construct prompt words corresponding to the video frame image, the prompt words including the predicted category and the predicted position, and input the prompt words and the video frame image into the multimodal large model;

[0007] The first image feature corresponding to the predicted category is determined by the multimodal large model, the region image of the predicted location is obtained from the video frame image, and the second image feature of the region image is determined; if the similarity between the first image feature and the second image feature is greater than the similarity threshold, the object to be detected is determined to be a risky object, and the multimodal large model is used to analyze whether the target object has abnormal actions based on the video frame image, and the model inference result is output.

[0008] The model inference results include risk object inference results. If the risk object inference results indicate that the object to be detected is a risk object, the model inference results also include abnormal action inference results, which indicate whether the target object has any abnormal actions.

[0009] This application provides an anomaly analysis device based on real-time target detection and multimodal large model. The device includes: an acquisition module, used to input video frame images into a target detection model to obtain a predicted category and a predicted location; wherein, the video frame image includes a target object, the predicted category represents the category of the object to be detected carried by the target object, and the predicted location represents the location of the object to be detected.

[0010] A construction module is used to construct prompt words corresponding to the video frame image, the prompt words including the predicted category and the predicted position, and input the prompt words and video frame image into the multimodal large model;

[0011] The processing module is used to determine the first image feature corresponding to the predicted category through the multimodal large model, obtain the region image of the predicted position from the video frame image, and determine the second image feature of the region image; if the similarity between the first image feature and the second image feature is greater than the similarity threshold, then the object to be detected is determined to be a risky object, and the multimodal large model is used to analyze whether the target object has abnormal actions based on the video frame image, and output the model inference result;

[0012] The model inference results include risk object inference results. If the risk object inference results indicate that the object to be detected is a risk object, the model inference results also include abnormal action inference results, which indicate whether the target object has any abnormal actions.

[0013] This application provides an electronic device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement an anomaly analysis method based on real-time target detection and multimodal large model.

[0014] This application provides a computer program product, including a computer program that, when executed by a processor, implements an anomaly analysis method based on real-time target detection and multimodal large model.

[0015] This application provides a machine-readable storage medium storing machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement an anomaly analysis method based on real-time target detection and multimodal large model.

[0016] As can be seen from the above technical solutions, in this embodiment, after inputting video frame images into the target detection model to obtain the predicted category and predicted location, a prompt word is constructed based on the predicted category and predicted location. The prompt word and video frame images are then input into a multimodal large model. The multimodal large model determines whether the object to be detected is a risky object based on the predicted category and predicted location, thereby reducing the false alarm rate of the target detection results and minimizing the possibility of errors in the target detection results. Only when the object to be detected is a risky object is the multimodal large model used to analyze whether the target object has abnormal actions based on the video frame images, rather than directly analyzing whether the target object has abnormal actions based on the video frame images, thereby reducing the false alarm rate of the anomaly analysis results and minimizing the possibility of errors in the anomaly analysis results. By constructing a hierarchical collaborative architecture of real-time initial screening and semantic deep analysis, high-efficiency target detection is combined with a multimodal large model with strong generalization capabilities to achieve efficient and accurate anomaly action recognition. This not only significantly improves the cognitive understanding of complex scenes and effectively reduces the false alarm rate, but also greatly saves computational resources by performing deep processing only on key frame images.

[0017] For example, by introducing a real-time initial screening function, a highly real-time target detection model can be used to quickly scan the video stream, analyze the abnormal situation of the current scene, and obtain as much information as possible. Only key video frames with potential risk targets are processed, which effectively solves the problems of resource waste and high latency caused by full-scale in-depth analysis of all video frames and can significantly reduce computational overhead.

[0018] For example, by introducing deep semantic analysis capabilities, and constructing prompt words, four independent reasoning dimensions—target properties, behavioral intentions, scene context, and counterfactual reasoning—are introduced into the multimodal large-scale model. This guides the model to comprehensively analyze target properties, behavioral intentions, scene context, and counterfactual reasoning, achieving high-precision, low-false-positive, and interpretable intelligent early warnings. This overcomes the limitation of prompt words merely serving as text guidance, formalizing the logical reasoning patterns of human experts into a computable semantic reasoning structure. This enables the multimodal large-scale model to possess multidimensional analysis and causal explanation capabilities, representing causal semantic relationships in complex scenarios. Attached Figure Description

[0019] Figure 1 This is the first flowchart of the anomaly analysis method based on real-time target detection and multimodal large model proposed in this application;

[0020] Figure 2 This is the second flowchart of the anomaly analysis method based on real-time target detection and multimodal large model in this application;

[0021] Figure 3This is the first structural diagram of the anomaly analysis device based on real-time target detection and multimodal large model of this application;

[0022] Figure 4 This is the second structural diagram of the anomaly analysis device based on real-time target detection and multimodal large model of this application;

[0023] Figure 5 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0024] This application proposes an anomaly analysis method based on real-time target detection and a multimodal large model. This method can be applied to electronic devices such as edge computing nodes, routers, and edge gateways. See also... Figure 1 The diagram shown is a flowchart of the method, which may include:

[0025] Step 101: Input the video frame image into the object detection model to obtain the predicted category and predicted location; wherein, the video frame image includes the target object (such as the target user, etc.), the predicted category can represent the category of the object to be detected carried by the target object, and the predicted location can represent the location of the object to be detected.

[0026] Step 102: Construct cue words corresponding to the video frame image. The cue words may include the predicted category and the predicted location. Input the cue words and the video frame image into the multimodal large model.

[0027] Step 103: Determine the first image feature corresponding to the predicted category using a multimodal large model. Obtain the region image of the predicted location from the video frame image and determine the second image feature of the region image. If the similarity between the first image feature and the second image feature is greater than a similarity threshold, then the object to be detected is determined to be a risky object. Analyze whether the target object exhibits abnormal behavior based on the video frame image using the multimodal large model and output the model inference result. The model inference result includes a risky object inference result. If the risky object inference result indicates that the object to be detected is a risky object, the model inference result may also include an abnormal behavior inference result, indicating whether the target object exhibits abnormal behavior.

[0028] For example, the prompt may include scene description information and deep inference information. The scene description information is used to indicate that the object to be detected is a potential risk target, and the potential risk target may include a predicted category and a predicted location. In addition, the deep inference information may include target property inference information and behavioral intention inference information. The target property inference information is used to instruct the multimodal large model to determine whether the object to be detected is a risk object. The behavioral intention inference information is used to instruct the multimodal large model to analyze whether the target object has abnormal actions based on video frame images when the object to be detected is a risk object.

[0029] For example, the deep inference information also includes counterfactual inference information, which instructs the multimodal large model to analyze the explanatory information of the target object carrying the object to be detected based on video frame image analysis when the target object does not exhibit abnormal behavior. Therefore, after analyzing whether the target object exhibits abnormal behavior based on video frame images using the multimodal large model, if the target object does not exhibit abnormal behavior, the multimodal large model analyzes the explanatory information of the target object carrying the object to be detected based on video frame images; wherein, the model inference result also includes the explanatory information of the target object carrying the object to be detected.

[0030] For example, the scene description information also includes the scene type associated with the video acquisition device, which is used to acquire video frame images; wherein, after analyzing whether the target object has abnormal actions based on the video frame images using a multimodal large model, if the target object has abnormal actions, the scene risk level of the environment in which the target object is located is analyzed based on the scene type using a multimodal large model; wherein, the model inference result also includes the scene risk level; the scene risk level is used to determine the initial alarm threshold for abnormal actions.

[0031] For example, after analyzing whether a target object exhibits abnormal behavior based on video frame images using a multimodal large model and outputting the model inference result, a decision score output by the multimodal large model can also be obtained. The decision score represents the confidence level of the multimodal large model in determining that the target object exhibits abnormal behavior. If the decision score is greater than a target alarm threshold, alarm data is sent to the user device. The alarm data includes video frame images and the model inference result. The target alarm threshold is either an initial alarm threshold or an adjusted initial alarm threshold. Specifically, the reference false alarm rate corresponding to the prediction category and scene type within a historical time period is statistically analyzed. The reference false alarm rate represents the probability that the abnormal behavior inference result is a false alarm. If the reference false alarm rate is greater than a first tolerance threshold, the initial alarm threshold is increased to obtain the target alarm threshold. If the reference false alarm rate is less than a second tolerance threshold, the initial alarm threshold is decreased to obtain the target alarm threshold. The second tolerance threshold is less than the first tolerance threshold. If the reference false alarm rate is not less than the second tolerance threshold and not greater than the first tolerance threshold, the initial alarm threshold is determined as the target alarm threshold.

[0032] For example, the prompt words also include analysis result response information; wherein the analysis result response information is used to instruct the multimodal large model to output model inference results; and the analysis result response information is used to instruct the multimodal large model to output the comprehensive judgment reasoning corresponding to the model inference results and the output format corresponding to the comprehensive judgment reasoning; and the analysis result response information is used to instruct the multimodal large model to output decision scores.

[0033] For example, after inputting a video frame image into the object detection model and obtaining the predicted category and predicted location, a confidence score corresponding to the predicted category can also be obtained. If the confidence score is greater than the preliminary screening threshold, the video frame image is determined to be a potentially risky video frame, and the operation of constructing prompt words corresponding to the video frame image is performed. If the confidence score is not greater than the preliminary screening threshold, the video frame image is discarded.

[0034] As can be seen from the above technical solutions, in this embodiment, after inputting video frame images into the target detection model to obtain the predicted category and predicted location, a prompt word is constructed based on the predicted category and predicted location. The prompt word and video frame images are then input into a multimodal large model. The multimodal large model determines whether the object to be detected is a risky object based on the predicted category and predicted location, thereby reducing the false alarm rate of the target detection results and minimizing the possibility of errors in the target detection results. Only when the object to be detected is a risky object is the multimodal large model used to analyze whether the target object has abnormal actions based on the video frame images, rather than directly analyzing whether the target object has abnormal actions based on the video frame images, thereby reducing the false alarm rate of the anomaly analysis results and minimizing the possibility of errors in the anomaly analysis results. By constructing a hierarchical collaborative architecture of real-time initial screening and semantic deep analysis, high-efficiency target detection is combined with a multimodal large model with strong generalization capabilities to achieve efficient and accurate anomaly action recognition. This not only significantly improves the cognitive understanding of complex scenes and effectively reduces the false alarm rate, but also greatly saves computational resources by performing deep processing only on key frame images.

[0035] For example, by introducing a real-time initial screening function, a highly real-time target detection model can be used to quickly scan the video stream, analyze the abnormal situation of the current scene, and obtain as much information as possible. Only key video frames with potential risk targets are processed, which effectively solves the problems of resource waste and high latency caused by full-scale in-depth analysis of all video frames and can significantly reduce computational overhead.

[0036] For example, by introducing deep semantic analysis capabilities, and constructing prompt words, four independent reasoning dimensions—target properties, behavioral intentions, scene context, and counterfactual reasoning—are introduced into the multimodal large-scale model. This guides the model to comprehensively analyze target properties, behavioral intentions, scene context, and counterfactual reasoning, achieving high-precision, low-false-positive, and interpretable intelligent early warnings. This overcomes the limitation of prompt words merely serving as text guidance, formalizing the logical reasoning patterns of human experts into a computable semantic reasoning structure. This enables the multimodal large-scale model to possess multidimensional analysis and causal explanation capabilities, representing causal semantic relationships in complex scenarios.

[0037] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0038] To achieve object detection, a deep learning model is deployed on video capture equipment (such as cameras). The equipment captures video streams, and the video frames within the stream are input into the deep learning model. The model then performs object detection based on these frames to determine the object's location and category within the image. Similarly, for anomaly analysis, a deep learning model is deployed on the video capture equipment. The equipment captures video streams, and the model inputs the video frames. The model then performs anomaly analysis based on these frames to determine if any abnormal behavior exists within the target object.

[0039] However, due to the limited computing resources of video capture devices, it is impossible to deploy deep learning models that consume a lot of computing resources when deploying deep learning models on video capture devices. This results in poor performance of deep learning models, a lack of semantic understanding of the scene, and poor generalization ability for newly emerging scenes.

[0040] Therefore, when using deep learning models for object detection, the results exhibit a high false positive rate, indicating a significant possibility of errors. Similarly, when using deep learning models for anomaly analysis, the results also show a high false positive rate, suggesting a significant possibility of errors.

[0041] To address the aforementioned findings, this application proposes an anomaly analysis method based on real-time target detection and a multimodal large model. By constructing a hierarchical collaborative architecture of real-time initial screening and semantic deep analysis, it combines high-efficiency target detection with a multimodal large model possessing strong generalization capabilities, achieving efficient and accurate anomaly action recognition. This not only significantly improves the cognitive understanding of complex scenes and effectively reduces the false alarm rate, but also greatly saves computational resources by performing deep processing only on keyframe images. The method introduces semantic deep analysis functionality, combined with a dynamically generated structured prompting mechanism. By constructing prompt words, it introduces four independent reasoning dimensions—target properties, behavioral intent, scene context, and counterfactual reasoning—into the multimodal large model. This guides the multimodal large model to complete a comprehensive analysis of target properties, behavioral intent, scene context, and counterfactual reasoning, achieving high-precision, low-false-alarm, and interpretable intelligent early warning. It balances real-time performance, privacy protection, and system efficiency, effectively overcoming deficiencies in semantic understanding, generalization ability, and resource consumption.

[0042] In this embodiment, a video stream is acquired by a video acquisition device, and a target detection model with high real-time performance is used to quickly analyze the video stream. Specifically, the video frame images in the video stream are input into the target detection model to obtain the predicted category and predicted position. Then, potentially risky video frames are screened out from the video stream. Finally, the potentially risky video frames are enhanced and denoised to improve the visual quality.

[0043] Then, a structured semantic prompt is dynamically generated based on the predicted category, predicted location, confidence score, and scene type. This structured semantic prompt is called a prompt word. The video frame image and the prompt word are input into a multimodal large model, which performs deep analysis on the image content.

[0044] The multimodal large model can comprehensively consider four dimensions: target nature, behavioral intent, scene context, and counterfactual interpretation, and output structured analysis results including judgment reasons and decision scores. Finally, based on the judgment reasons and decision scores, if the decision score is greater than the target alarm threshold, alarm data can be sent to the user device. This alarm data may include video frame images and judgment reasons.

[0045] This application proposes an anomaly analysis method based on real-time target detection and a multimodal large model. See [link to relevant documentation]. Figure 2 The diagram shown is a flowchart of the method, which may include:

[0046] Step 201: The edge computing node acquires the video stream from the video acquisition device.

[0047] For example, video capture devices can include, but are not limited to, network cameras, high-definition analog cameras, and dome cameras; any device that supports image capture functionality can be used. Video capture devices can be deployed in specific areas, such as public squares, transportation hubs, subway station entrances and exits, school corridors, bank lobbies, and shopping malls. Edge computing nodes can be routers, edge gateways, edge servers, firewalls, switches, etc.; any device located at the edge of a local area network (LAN).

[0048] For example, a target detection model and a multimodal large model can be deployed on a video acquisition device. The video acquisition device acquires the video stream, and an anomaly analysis method based on real-time target detection and the multimodal large model is executed based on the video stream. Alternatively, a target detection model and a multimodal large model can be deployed on an edge computing node. The video acquisition device acquires the video stream and sends it to the edge computing node, which then executes an anomaly analysis method based on real-time target detection and the multimodal large model based on the video stream.

[0049] In this embodiment, the edge computing node performs an anomaly analysis method based on real-time target detection and multimodal large model based on the video stream. That is, the edge computing node needs to obtain the video stream from the video acquisition device.

[0050] For example, a video capture device can capture a video stream, such as continuously capturing scene images at 60 frames per second, and these scene images form a video stream. Then, the video capture device can compress the video stream (e.g., using a video encoding format such as H.264), and send the compressed video stream to the edge computing node via RTSP (Real Time Streaming Protocol) messages. In this way, the edge computing node can obtain the video stream from the video capture device.

[0051] After receiving the video stream, the edge computing node can decode it, thereby restoring the compressed video stream into a series of consecutive video frames, each frame denoted as . t is the frame number.

[0052] Step 202: The edge computing node inputs the video frame image into the object detection model to obtain the predicted category and predicted location. The video frame image includes the target object (e.g., the target user), the predicted category represents the category of the object to be detected carried by the target object, and the predicted location represents the position of the object to be detected.

[0053] For example, an edge computing node can acquire all video frame images from a video stream, or it can acquire only a portion of the video frame images. For any acquired video frame image, the edge computing node inputs the video frame image into an object detection model to obtain a confidence score, predicted category, and predicted location. Then, based on the confidence score, predicted category, and predicted location, it determines whether the video frame image is a potentially risky video frame. This allows the object detection model to quickly analyze the video stream and filter out potentially risky video frames.

[0054] For example, for any given video frame, the edge computing node inputs the video frame into the object detection model to obtain the predicted category, predicted location, and confidence score. For instance, the predicted category represents the category of the object being detected, such as "knife." The predicted location represents the position of the object being detected, such as the bounding box (i.e., the smallest bounding rectangle of the object). The predicted location can be the coordinates of the top-left vertex, the length, and the height of the bounding box; or the coordinates of the top-left, top-right, bottom-left, and bottom-right vertexes; or the coordinates of the center point, length, and height of the bounding box. Furthermore, the confidence score is the confidence score corresponding to the predicted category, representing the degree of confidence the object detection model has in identifying the object as belonging to that predicted category. For example, if the confidence score is 0.8 and the predicted category is "knife," then the confidence level that the object being detected is a "knife" is 80%.

[0055] In summary, it can be seen that after the edge computing node inputs the video frame image into the object detection model and obtains the predicted category and predicted location, the edge computing node can obtain the confidence score corresponding to the predicted category.

[0056] Based on this, if the confidence score is greater than the configured preliminary screening threshold (the preliminary screening threshold can be configured according to actual needs, such as 0.5, 0.6, 0.7, etc.), the edge computing node determines the video frame image as a potentially risky video frame and performs the operation of constructing the corresponding prompt word for the video frame image, that is, it performs subsequent steps for the video frame image. If the confidence score is not greater than the preliminary screening threshold, the edge computing node discards the video frame image, that is, it does not perform subsequent steps for the video frame image.

[0057] For example, the object detection model is a pre-trained object detection model optimized for high real-time performance. The object detection model can employ RT-DETR (Real-Time Detection Transformer) models, etc., for each video frame image within the video stream. It can display video frame images The input is fed into the object detection model for inference, resulting in video frame images. Test results .

[0058] For example, video frame images Test results It is a structured data object containing the following information: (CLASS): The predicted category of the object to be detected, such as "knife" etc.; (SCORE): This test result The confidence score is a floating-point number between 0 and 1, used to represent the degree of confidence that the object detection model determines that the object to be detected belongs to the predicted category. (BBOX): The predicted location of the object to be detected. The predicted location can be the bounding box of the object to be detected, such as the minimum bounding rectangle.

[0059] For example, when obtaining video frame images Test results After that, the test results can be confirmed. The confidence score, if the test result If the confidence score is greater than the initial screening threshold, then the video frame image will be... If the detection result is determined to be a potentially risky video frame, If the confidence score is not greater than the initial screening threshold, then the video frame image will be... Video frames deemed non-potentially risky are discarded. In summary, all potentially risky video frames can be filtered out from the video stream, and subsequent steps can be performed on these potentially risky video frames.

[0060] For example, considering that the target object may carry multiple objects to be detected, therefore, for each video frame image within the video stream... In the video frame image When the input is fed into the object detection model for inference, video frame images can be obtained. Multiple test results, such as test results ={ , ,…, }, This indicates the detection result of the first object to be detected, and so on. This represents the detection result of the nth object to be detected.

[0061] For the detection result of the i-th object to be detected, the detection result is a structured data object containing the following information: (CLASS): The predicted category of the object to be detected; (SCORE): The confidence score of the test result; (BBOX): The predicted position of the object to be detected. Based on this, if the video frame image... If the confidence score of any detection result is greater than the initial screening threshold, then the video frame image can be... Video frames identified as potentially risky, if the video frame image If the confidence scores of all detection results are not greater than the preliminary screening threshold, then the video frame images can be... Video frames deemed non-potentially risky are discarded. .

[0062] In video frame images When a video frame is identified as potentially risky, subsequent steps can be performed based on detection results that meet certain conditions, while detection results that do not meet the conditions can be discarded. For example, if the detection result... If the confidence score is greater than the initial screening threshold, then the detection result is... The test result meets the conditions. If the confidence score is not greater than the initial screening threshold, then the detection result is... If the test result does not meet the criteria, discard the test result. Similarly, this can be applied to video frame images. All test results that meet the criteria will proceed to the next steps.

[0063] Step 203: The edge computing node performs a specified type of processing on the selected video frame images. The specified type of processing may include, but is not limited to, enhancement processing and / or noise reduction processing. There are no restrictions on this processing method.

[0064] For example, for the screened potentially risky video frames, edge computing nodes perform enhancement and denoising processes to obtain optimized video frame images. Denoising is used to eliminate noise generated by the video acquisition device under low-light conditions, and / or to eliminate artifacts caused by video stream compression. By denoising potentially risky video frames, the impact of noise can be reduced, improving image quality. Furthermore, enhancement processes improve the contrast and sharpness of potentially risky video frames, significantly improving image quality in nighttime or low-light scenes.

[0065] Step 204: The edge computing node constructs a prompt corresponding to the video frame image (i.e., the potential risk video frame). The prompt may include, but is not limited to, the prediction category and the prediction location.

[0066] For example, if video frame images Test results If the conditions are met, the prompt word will include the detection result. The predicted category and predicted location in the video frame image Test results If the condition is not met, the prompt word will not include the detection results. The prediction category and prediction location are listed in the table, and so on.

[0067] For example, the cue words (semantic cue words) are cue words in natural language text used to guide the multimodal large model to understand the image context. In this embodiment, the cue words may include scene description information, which indicates that the object to be detected is a potential risk target, and the potential risk target may include the predicted category and predicted location of the object to be detected. In addition, the scene description information may also include the scene type associated with the video acquisition device, and the video acquisition device is used to acquire the aforementioned video frame images.

[0068] For example, since the location of the video capture equipment is known, edge computing nodes can obtain the scene type associated with the video capture equipment. This scene type is pre-defined through manual configuration during the system deployment phase. For instance, scene types could include public squares, transportation hubs, subway station entrances / exits, school corridors, bank lobbies, and shopping malls. This scene type serves as prior contextual knowledge, guiding the multimodal large model to perform semantic reasoning by combining behavioral norms and safety common sense within a specific environment.

[0069] For example, an edge computing node can generate a list of object detection results [object_list], which can include multiple object detection results, and the object detection results are video frame images. The detection results that meet the conditions. For example, target detection result 1 includes video frame images. The first detection result that meets the criteria includes the predicted category and predicted location; or, the target detection result 1 includes the predicted category, predicted location, and confidence score of that detection result. Similarly, the target detection result m includes video frame images. The predicted category and predicted location in the m-th detection result that meets the conditions; or, the target detection result m includes the predicted category, predicted location and confidence score in the detection result.

[0070] Obviously, in video frame images When a video frame is considered a potential risk, if the video frame image... For each of the m detection results that meet the conditions, the target detection result list [object_list] includes m target detection results.

[0071] For example, by integrating all detected potential risk target information (i.e., m target detection results) into prompt words and designing structured dynamic prompts, it is possible to guide multimodal large models to conduct more comprehensive and integrated scenario analysis, rather than just focusing on the single most prominent risk target information. This allows for a more accurate understanding of complex risks in complex scenarios and reduces the false alarm rate.

[0072] For example, regarding scene description information, edge computing nodes can have a built-in scene description template. The target detection result list and scene type can be added to the scene description template to obtain the scene description information. Edge computing nodes can also generate scene description information based on the target detection result list and scene type. This embodiment does not impose any restrictions on this; the scene description information only needs to include the target detection result list and scene type. Regarding the target detection result list, it includes multiple target detection results. Each target detection result includes the predicted category and predicted location corresponding to a detected object, indicating that the detected object is a potential risk target. That is, a potential risk target includes the predicted category and predicted location of the detected object.

[0073] For example, a scenario description template could be: You are a senior security analyst evaluating a video stream from [Scene Type]. Preliminary analysis has identified the following [num] potential risk targets in the image: [object_list]. Please conduct a comprehensive analysis based on the following dimensions to determine if a real security threat exists. A real security threat refers to a target exhibiting clear aggressive, destructive, unauthorized intrusion, or seriously disruptive intent towards public order, and whose environment and behavioral patterns support the likelihood of such a threat. Of course, the above scenario description template is just an example; you can configure it according to your actual needs.

[0074] Based on the above scene description template, the scene type associated with the video capture device can be added to the placeholder of "Scene Type". The number of object detection results can be added to the placeholder of "num". For example, if the object detection result list [object_list] contains 3 object detection results, then add 3 to the placeholder of "num". The object detection result list can be added to the placeholder of "object_list", that is, add m object detection results. For example, in the placeholder of "object_list", add the predicted category, predicted location, and confidence score of object detection result 1, the predicted category, predicted location, and confidence score of object detection result 2, ..., the predicted category, predicted location, and confidence score of object detection result m in sequence.

[0075] For example, the prompt word includes deep reasoning information, which includes target nature reasoning information, behavioral intention reasoning information, scene context reasoning information, and counterfactual reasoning information. Alternatively, the deep reasoning information includes target nature reasoning information and behavioral intention reasoning information. Alternatively, the deep reasoning information includes target nature reasoning information, behavioral intention reasoning information, and scene context reasoning information. Alternatively, the deep reasoning information includes target nature reasoning information, behavioral intention reasoning information, and counterfactual reasoning information.

[0076] Target property inference information is used to instruct the multimodal large model to determine whether an object to be detected is a risky object. For example, for each target detection result in the target detection result list, it is determined whether the target object corresponding to that target detection result is a risky object. For instance, it is determined whether the target object corresponding to that target detection result is a risky object based on the predicted category and predicted location within that target detection result.

[0077] Behavioral intent reasoning information is used to instruct the multimodal large model to analyze whether the target object exhibits abnormal behavior based on video frame images when the detected object is a risky object. For example, if any target detection result in the target detection result list corresponds to a risky object, the multimodal large model is instructed to analyze whether the target object exhibits abnormal behavior based on video frame images. If none of the target detection results correspond to risky objects, then no analysis is performed to determine whether the target object exhibits abnormal behavior.

[0078] Scene context reasoning information is used to instruct the multimodal large model to analyze the scene risk level of the target object's environment based on the scene type when the target object exhibits abnormal behavior. For example, if the target object exhibits abnormal behavior, the multimodal large model is instructed to analyze the scene risk level based on the scene type in the scene description information; if the target object does not exhibit abnormal behavior, the scene risk level is not analyzed.

[0079] Counterfactual reasoning information is used to instruct the multimodal large model to interpret whether the target object is carrying a detected object based on video frame image analysis when the target object does not exhibit abnormal behavior. For example, when the target object does not exhibit abnormal behavior, it instructs the multimodal large model to provide a reasonable explanation for the target object carrying a detected object based on video frame image analysis, such as a reasonable explanation for the target object carrying a "knife".

[0080] For example, an example of deep reasoning could be: Please conduct a deep reasoning analysis from the following four aspects. Analyze these four aspects, all targets, and their context to determine if a real security threat exists. Target nature: Is the item a controlled item (e.g., a knife), an offensive tool, or a suspicious item? Does the item's appearance violate public place regulations? Behavioral intent: Does the target exhibit any unusual behavior (e.g., hiding, waving, approaching sensitive areas, following others)? Does the target show any pre-attack signs or attempt to evade detection? Context: Is the current scene a high-security area? Is the time nighttime or non-business hours? Counterfactual reasoning: If you believe this is not an unusual event, meaning the target does not pose a real security threat, what is the most reasonable explanation?

[0081] In the aforementioned deep reasoning information, target properties can represent target property reasoning information, used to instruct the multimodal large model to determine whether the object to be detected is a risky object. Behavioral intent can represent behavioral intent reasoning information, used to instruct the multimodal large model to analyze whether the target object exhibits abnormal behavior when the object to be detected is a risky object. Scene context can represent scene context reasoning information, used to instruct the multimodal large model to analyze the scene risk level (e.g., whether it is a high-security area) of the environment in which the target object is located when the target object exhibits abnormal behavior. Counterfactual reasoning represents counterfactual reasoning information, used to instruct the multimodal large model to analyze reasonable explanations for non-abnormal behavior when the target object does not exhibit abnormal behavior.

[0082] For example, the prompt may also include analysis result response information. This analysis result response information is used to instruct the multimodal large model to output the model inference result; the analysis result response information is used to instruct the multimodal large model to output the comprehensive judgment reasoning corresponding to the model inference result and the output format corresponding to the comprehensive judgment reasoning; and the analysis result response information is used to instruct the multimodal large model to output a decision score.

[0083] For example, the model inference results include risk object inference results. If the risk object inference results indicate that the object to be detected is a risk object, the model inference results can also include abnormal action inference results, which indicate whether the target object exhibits abnormal actions. Furthermore, the model inference results also include explanatory information about the target object carrying the object to be detected. The model inference results also include the scene risk level. Based on this, the analysis result response information is used to instruct the multimodal large model to output the model inference results.

[0084] For example, the analysis result response information is used to instruct the multimodal large model to output a decision score, which represents the confidence level of the multimodal large model in determining that the target object has abnormal behavior.

[0085] For example, the analysis result response information is used to instruct the multimodal large model to output a comprehensive judgment reason and output format. The comprehensive judgment reason indicates the reason why the multimodal large model gives the model reasoning result, and the output format indicates the format in which the multimodal large model outputs the comprehensive judgment reason.

[0086] For example, an example of an analysis result response message could be: Please provide your analysis results in a clear format, including two fields: "justification" (a paragraph explaining your comprehensive reasoning, with the important parts of the reasoning bolded) and "decision_score" (a score between 0 and 1). JSON format is preferred, but not limited to.

[0087] In the above analysis result response information, "justification" indicates that the multimodal large model needs to output a comprehensive reason for the judgment, and JSON format indicates the output format of the comprehensive reason for the judgment. Furthermore, "decision_score" indicates that the multimodal large model needs to output a decision score. Based on the above analysis result response information, the output of the multimodal large model is a string (not limited to JSON format), which, after parsing, yields a data object containing two key fields: justification: a natural language text describing the reason for the judgment; and decision_score: a decision score, a numerical value between 0 and 1, where a higher score indicates a higher probability of abnormal behavior.

[0088] In summary, edge computing nodes can construct prompt words, which can include scene description information, deep reasoning information, and analysis result response information. Of course, this is just an example of a prompt word, and there are no restrictions on the content of the prompt word. Edge computing nodes can construct prompt words according to actual needs.

[0089] Step 205: The edge computing node inputs the prompt words and video frame images into the multimodal large model, performs inference through the multimodal large model, obtains the model inference result, and outputs the model inference result.

[0090] Multimodal large models are artificial intelligence models capable of simultaneously processing multiple types of information, such as images and text, and are used to generate natural language descriptions and perform semantic analysis based on image content. By deploying lightweight multimodal large models on edge computing nodes, low-latency and highly privacy-preserving localized processing can be achieved.

[0091] For example, after obtaining the prompt words and video frame images (i.e., potentially risky video frames), the prompt words and video frame images can be input into a multimodal large model for inference. The input data of the multimodal large model can include prompt words and video frame images. The multimodal large model comprehensively analyzes the image content of the video frame images and the semantic cues of the prompt words to obtain the model's inference results, and then outputs the model's inference results.

[0092] For example, since the prompt words include target property inference information, and this inference information is used to instruct the multimodal large model to determine whether the object to be detected is a risky object, the multimodal large model needs to determine whether the object to be detected is a risky object. Since the prompt words include scene description information, and this scene description information includes a list of target detection results, the multimodal large model can obtain the predicted category and predicted location from each target detection result in the list.

[0093] After obtaining the predicted category and predicted location, a multimodal large model is used to determine the first image feature corresponding to the predicted category. Then, the multimodal large model extracts the region image of the predicted location from the video frame image and determines the second image feature of that region image. For example, because the multimodal large model has learning capabilities, it can learn the image features of each category supported by the object detection model. Based on this, after obtaining the predicted category, the multimodal large model can determine the first image feature corresponding to that category. For example, after obtaining the video frame image and the predicted location, the multimodal large model can extract the region image of the predicted location from the video frame image and determine the second image feature of that region image.

[0094] After obtaining the first and second image features, a multimodal large model calculates the similarity between them, such as distance similarity or cosine similarity. Based on this, if the similarity is greater than a similarity threshold (configured according to actual needs, such as 90% or 95%), the multimodal large model determines that the object to be detected corresponding to the target detection result is a risky object; if the similarity is not greater than the similarity threshold, the multimodal large model determines that the object to be detected corresponding to the target detection result is not a risky object.

[0095] Based on the above reasoning process, the model's reasoning result can include a risk object reasoning result, which indicates that the target object carries a risk object, or that the target object does not carry a risk object. For example, if the object to be detected corresponding to any target detection result is a risk object, then the risk object reasoning result can indicate that the target object carries a risk object; if the objects to be detected corresponding to all target detection results are not risk objects, then the risk object reasoning result can indicate that the target object does not carry a risk object.

[0096] For example, since the prompt words include behavioral intent inference information, and this information is used to instruct the multimodal large model to analyze whether the target object (such as the target object corresponding to any target detection result) exhibits abnormal behavior based on the video frame image when the object to be detected is a risky object, the multimodal large model can also analyze whether the target object exhibits abnormal behavior based on the video frame image, such as "hiding, waving, approaching sensitive areas, or following others." This embodiment does not limit how the analysis of whether the target object exhibits abnormal behavior is performed; for example, the multimodal large model can extract image features from the video frame image, and the similarity between the image features and the abnormal behavior image features can be used to analyze whether the target object exhibits abnormal behavior.

[0097] Based on the above reasoning process, the model reasoning result may include the abnormal action reasoning result, which indicates that the target object has abnormal actions, or that the target object does not have abnormal actions.

[0098] For example, since the prompt includes counterfactual reasoning information, and this counterfactual reasoning information is used to instruct the multimodal large model to analyze the explanatory information of the target object carrying the object to be detected based on video frame image analysis when the target object does not exhibit abnormal behavior, the multimodal large model can also analyze the explanatory information of the target object carrying the object to be detected based on the video frame image analysis. This includes reasonable explanations such as the target object carrying a risky object (the object to be detected is a risky object) or the target object not exhibiting abnormal behavior. This embodiment does not limit how the multimodal large model obtains the explanatory information; it simply provides the reason why the target object does not exhibit abnormal behavior.

[0099] Based on the above reasoning process, the model reasoning results can include explanatory information about the target object carrying the object to be detected, such as a reasonable explanation for carrying a risky object or a reasonable explanation for the target object not exhibiting abnormal behavior.

[0100] For example, since the prompt words include scene context inference information, and this information is used to instruct the multimodal large model to analyze the scene risk level of the target object's environment based on the scene type when the target object exhibits abnormal behavior, the multimodal large model obtains the scene type from the scene description information when the target object exhibits abnormal behavior. Then, it analyzes the scene risk level of the target object's environment based on this scene type. For instance, if the scene type indicates a higher scene security requirement, the scene risk level is higher; conversely, if the scene type indicates a lower scene security requirement, the scene risk level is lower. For example, the scene "school corridor" has a high security requirement and a relatively high scene risk level. This embodiment does not impose restrictions on how the multimodal large model obtains the scene risk level.

[0101] Based on the above reasoning process, the model reasoning result may include a scenario risk level, which is used to determine the initial alarm threshold for abnormal actions. The determination process is described in subsequent embodiments.

[0102] In summary, the model inference results can include inference results for risky objects. If the inference result for risky objects indicates that the object to be detected is a risky object, the model inference results can also include inference results for abnormal actions. If the inference result for abnormal actions indicates that the target object does not have any abnormal actions, the model inference results can also include explanatory information about the object to be detected carried by the target object. If the inference result for abnormal actions indicates that the target object has any abnormal actions, the model inference results can also include the scene risk level.

[0103] For example, since the prompt includes analysis result response information, this response information is used to instruct the multimodal large model to output model inference results. Based on this, after obtaining the model inference results, the multimodal large model can output model inference results, which may include risk object inference results, abnormal action inference results, and explanation information about the target object carrying the object to be detected. Alternatively, the model inference results may include risk object inference results, abnormal action inference results, and scene risk level.

[0104] The analysis result response information is used to instruct the multimodal large model to output a comprehensive judgment reason and output format. Therefore, the multimodal large model can also provide a comprehensive judgment reason for the model's inference results, indicating why the multimodal large model gave the above model inference results, which is an explanatory explanation of the model's inference results. Based on this, the multimodal large model can also output the comprehensive judgment reason in a specified format (JSON format).

[0105] The analysis results are used to instruct the multimodal large model to output a decision score. Therefore, the multimodal large model can also output a decision score, which represents the confidence level of the multimodal large model in determining that the target object has abnormal behavior. The higher the score, the higher the probability of abnormal behavior.

[0106] Step 206: The edge computing node obtains and stores the model inference results output by the multimodal large model. In addition, the edge computing node can also obtain and store the comprehensive judgment reasoning and decision score output by the multimodal large model.

[0107] Step 207: If the decision score is greater than the target alarm threshold, the edge computing node sends alarm data to the user device. This alarm data may include video frame images and model inference results, and may also include a comprehensive judgment reasoning from the multimodal large model output. Alternatively, if the decision score is not greater than the target alarm threshold, the edge computing node does not send alarm data to the user device.

[0108] For example, the model inference result may include a scene risk level, and the edge computing node can determine an initial alarm threshold (basic alarm threshold) based on this scene risk level. For instance, the edge computing node can pre-configure the mapping relationship between risk levels and alarm thresholds, such as manually setting the mapping relationship, without any restrictions. Based on this, after obtaining the scene risk level from the model inference result, the edge computing node queries the mapping relationship between the risk level and the alarm threshold using this scene risk level to obtain the initial alarm threshold corresponding to that scene risk level.

[0109] For example, after obtaining the initial alarm threshold, the edge computing node can use the initial alarm threshold as the target alarm threshold. Alternatively, the edge computing node can adjust the initial alarm threshold to obtain the target alarm threshold. The dynamic adjustment process of the initial alarm threshold is described below.

[0110] For example, the model inference result may include anomaly action inference results, which can be confirmed by the operator (e.g., confirming correctness or confirming false alarm). For instance, assuming the anomaly action inference result indicates that the target object does not exhibit any abnormal action, and the operator is aware that the target object in the video frame does not exhibit any abnormal action, then the anomaly action inference result is confirmed as "correct." If the operator is aware that the target object in the video frame exhibits abnormal action, then the anomaly action inference result is confirmed as "false alarm." Alternatively, assuming the anomaly action inference result indicates that the target object exhibits abnormal action, and the operator is aware that the target object in the video frame exhibits abnormal action, then the anomaly action inference result is confirmed as "correct." If the operator is aware that the target object in the video frame does not exhibit abnormal action, then the anomaly action inference result is confirmed as "false alarm."

[0111] Edge computing nodes can maintain a historical false alarm feedback library. The historical false alarm feedback library records the abnormal action inference results, false alarm data (indicating whether there is a false alarm), the prediction category, scene type and video frame image corresponding to the abnormal action inference results. See Table 1 for an example of the historical false alarm feedback library.

[0112] Table 1

[0113]

[0114] In Table 1, the abnormal action inference result indicates whether the target object within the video frame exhibits abnormal action or not. A false positive of "Yes" indicates the abnormal action inference result is correct and there are no false positives; a false positive of "No" indicates the abnormal action inference result is incorrect and there are false positives. The predicted category is the category of the object to be detected. When the object X to be detected within the video frame is a risky object, the predicted category represents the category of object X. The risky object inference result can include the predicted category of the object to be detected. The scene type indicates the scene type associated with the video acquisition device.

[0115] For example, after obtaining the model inference results, the edge computing node can calculate the reference false alarm rate corresponding to the predicted category and scene type within a historical time period. The reference false alarm rate represents the probability that the inference result of the abnormal action is a false alarm. For instance, the end time of the historical time period can be the current time, the start time of the historical time period can be the creation time of the historical false alarm feedback database, or the end time of the historical time period can be the current time, and the duration of the historical time period can be a fixed duration (such as 3 months).

[0116] Edge computing nodes can obtain the predicted category of the object to be detected from the model inference results and the scene type associated with the video acquisition device. Then, they can filter all candidate records (abnormal action inference results) corresponding to the predicted category and scene type from the historical false alarm feedback library. Based on these candidate records, the number of false alarm candidate records and the number of correct candidate records can be determined, and then a reference false alarm rate can be calculated, such as the number of false alarm candidate records / (number of false alarm candidate records + number of correct candidate records).

[0117] Based on this, if the reference false alarm rate is greater than the first tolerance threshold (configurable according to actual needs), the edge computing node increases the initial alarm threshold to obtain the target alarm threshold, i.e., automatically fine-tunes the increase of the initial alarm threshold to obtain the target alarm threshold, thereby reducing future false alarms. If the reference false alarm rate is less than the second tolerance threshold (configurable according to actual needs, and the second tolerance threshold is less than the first tolerance threshold), the edge computing node decreases the initial alarm threshold to obtain the target alarm threshold, i.e., fine-tunes the decrease of the initial alarm threshold to obtain the target alarm threshold for missed alarm scenarios. If the reference false alarm rate is not less than the second tolerance threshold and not greater than the first tolerance threshold, then the initial alarm threshold is determined as the target alarm threshold.

[0118] This application proposes an anomaly analysis device based on real-time target detection and a multimodal large model. See [link to relevant documentation]. Figure 3 The diagram shown is a structural schematic of the device, which may include:

[0119] The video capture module is used to acquire video streams from video capture devices.

[0120] The real-time initial screening module is used to input video frame images into the object detection model to obtain predicted categories and predicted locations. The video frame images include target objects (such as target users), the predicted category can represent the category of the object to be detected carried by the target object, and the predicted location can represent the location of the object to be detected.

[0121] For example, the real-time initial screening module uses a target detection model to quickly analyze the video stream and filters out potentially risky video frames based on a preset risk library. It then outputs the potentially risky video frames and the detected data, such as the predicted category, predicted location, and confidence score corresponding to the video frame image.

[0122] The video frame optimization module is used to perform specified types of processing on the selected video frame images. These processing types can include, but are not limited to, enhancement and / or denoising. By performing enhancement and / or denoising, image quality can be optimized, ensuring clear and reliable input data for subsequent depth analysis.

[0123] The dynamic prompting module is used to construct prompts corresponding to video frame images (i.e., potentially risky video frames). For example, based on the data output by the real-time initial screening module and a preset template, the dynamic prompting module can automatically construct natural language prompts to guide multimodal large model analysis.

[0124] The multimodal large model validation module is used to input prompts and video frame images into the multimodal large model, perform inference through the multimodal large model, obtain the model's inference results, and output the model's inference results. For example, the multimodal large model validation module performs in-depth analysis of the scene and provides reasons and decision scores.

[0125] The decision response module is used to acquire and store the model inference results output by the multimodal large model. It parses the model inference results output by the multimodal large model, integrates an adaptive adjustment mechanism based on the initial alarm threshold and historical false alarm feedback according to the scene risk level, dynamically acquires the currently valid target alarm threshold, and if the decision score is greater than the target alarm threshold, decides to send alarm data to the user device.

[0126] As can be seen from the above technical solutions, in this embodiment, by constructing a hierarchical collaborative architecture of real-time initial screening and semantic deep analysis, high-efficiency target detection is combined with a multimodal large model with strong generalization capabilities to achieve efficient and accurate abnormal action recognition. This not only significantly improves the cognitive understanding of complex scenes and effectively reduces the false alarm rate, but also greatly saves computing resources by performing deep processing only on key frame images. For example, by introducing a real-time initial screening function, a high-real-time target detection model is used to quickly scan the video stream, analyze the abnormal situation of the current scene, and obtain as much information as possible. Moreover, only key video frames with potentially risky targets are processed, effectively solving the problem of resource waste and excessive latency caused by full deep analysis of all video frames. This can significantly reduce computing overhead, and is especially suitable for high-risk scenarios requiring high concurrency and low latency response, such as subway stations and bank lobbies. For example, by introducing deep semantic analysis, a four-dimensional structured semantic reasoning mechanism for identifying real security threats is proposed. By constructing prompt words, four independent reasoning dimensions—target nature, behavioral intent, scene context, and counterfactual reasoning—are introduced into the multimodal large-scale model. This guides the multimodal large-scale model to comprehensively analyze target nature, behavioral intent, scene context, and counterfactual reasoning, constructing independent semantic scores for each dimension and generating corresponding feature vectors. After weighted fusion, a unified semantic reasoning tensor is formed to represent causal semantic relationships in complex scenarios. This layer-by-layer guidance enables the multimodal large-scale model to make deep judgments, achieving high-precision, low-false-positive, and interpretable intelligent early warning. This overcomes the limitation of prompt words merely serving as text guidance, formalizing the logical reasoning patterns of human experts into a computable semantic reasoning structure. This gives the multimodal large-scale model multi-dimensional analysis and causal explanation capabilities, enabling it to represent causal semantic relationships in complex scenarios. It can identify the "causal chain of threat generation" in abnormal scenarios, outputting structured explanations and decision scores, effectively reducing false alarm rates and improving interpretability, while also significantly enhancing the credibility of the multimodal large-scale model's results.

[0127] By combining high-efficiency target detection with a multimodal large model with strong generalization capabilities, a dynamic prompting mechanism for identifying "real security threats" is proposed. This mechanism enables low-latency and high-privacy intelligent analysis on edge computing nodes, which not only significantly improves the cognitive understanding of complex scenarios and effectively reduces the false alarm rate, but also greatly saves computing resources by performing deep processing only on key frames, thus having broad application prospects.

[0128] This paper proposes an edge-end adaptive feedback verification mechanism. By combining the decision scores and judgment reasons output by a multimodal large model with data from a historical false alarm feedback library, an alarm threshold model is constructed that dynamically adjusts based on scene risk level and false alarm rate. This achieves "self-learning and self-optimization" capabilities in different deployment environments, effectively solving the problems of poor scene generalization and high false alarm rate caused by fixed thresholds. Furthermore, it enables alarm sensitivity to self-optimize according to different scene characteristics and actual data, improving long-term robustness and accuracy in different deployment environments. The paper proposes deploying the multimodal large model on edge computing nodes, combining a keyframe selection strategy and image optimization. This improves detection speed while reducing redundant information interference by inputting only keyframes, ensuring analysis depth while achieving low-latency response and localized data processing. Compared to methods relying on cloud-based large models, this avoids network transmission latency and privacy leakage risks, making it suitable for scenarios with high real-time and security requirements. For abnormal scenarios, combining the target detection model and the multimodal large model can quickly identify abnormal scenarios and provide corresponding decisions, greatly improving real-time performance without requiring manual verification, reducing the workload of manual review, reducing labor costs, and improving overall efficiency. Leveraging the powerful generalization reasoning capabilities of multimodal large models reduces the false alarm rate and helps improve operational efficiency.

[0129] Based on the same concept as the methods described above, this application proposes an anomaly analysis device based on real-time target detection and a multimodal large model. (See also...) Figure 4 The diagram shown is a structural schematic of the anomaly analysis device based on real-time target detection and multimodal large model. The device may include:

[0130] The acquisition module 41 is used to input video frame images into the target detection model to obtain predicted category and predicted position; wherein, the video frame image includes a target object, the predicted category represents the category of the object to be detected carried by the target object, and the predicted position represents the position of the object to be detected;

[0131] Construction module 42 is used to construct prompt words corresponding to the video frame image, the prompt words include the predicted category and the predicted position, and input the prompt words and video frame image into the multimodal large model;

[0132] Processing module 43 is used to determine the first image feature corresponding to the predicted category through the multimodal large model, obtain the region image of the predicted position from the video frame image, and determine the second image feature of the region image; if the similarity between the first image feature and the second image feature is greater than the similarity threshold, then the object to be detected is determined to be a risky object, and the multimodal large model is used to analyze whether the target object has abnormal actions based on the video frame image, and output the model inference result;

[0133] The model inference results include risk object inference results. If the risk object inference results indicate that the object to be detected is a risk object, the model inference results also include abnormal action inference results, which indicate whether the target object has any abnormal actions.

[0134] For example, the prompt may include scene description information and deep inference information. The scene description information indicates that the object to be detected is a potential risk target, and the potential risk target includes the predicted category and the predicted location. The deep inference information includes target property inference information and behavioral intent inference information. The target property inference information instructs the multimodal large model to determine whether the object to be detected is a risky object. The behavioral intent inference information instructs the multimodal large model to analyze whether the target object exhibits abnormal behavior based on the video frame image, provided that the object to be detected is a risky object. Based on this, the multimodal large model determines a first image feature corresponding to the predicted category, obtains a region image of the predicted location from the video frame image, and determines a second image feature of the region image. If the similarity between the first image feature and the second image feature is greater than a similarity threshold, then the object to be detected is determined to be a risky object, and the multimodal large model analyzes whether the target object exhibits abnormal behavior based on the video frame image.

[0135] For example, the deep reasoning information also includes counterfactual reasoning information, which is used to instruct the multimodal big model to analyze the explanatory information of the target object carrying the object to be detected based on the video frame image when the target object does not have abnormal actions;

[0136] The processing module 43 is further configured to, after analyzing whether the target object has abnormal actions based on the video frame image using the multimodal large model, if the target object does not have abnormal actions, analyze the explanatory information of the object to be detected carried by the target object based on the video frame image using the multimodal large model; wherein the model inference result also includes the explanatory information.

[0137] For example, the scene description information also includes the scene type associated with the video acquisition device, which is used to acquire the video frame images; wherein, the processing module 43 is further used to, after analyzing whether the target object has abnormal actions based on the video frame images using the multimodal large model, if the target object has abnormal actions, analyze the scene risk level of the environment in which the target object is located based on the scene type using the multimodal large model; the model inference result also includes the scene risk level; the scene risk level is used to determine the initial alarm threshold for the abnormal actions.

[0138] For example, the processing module 43 is further configured to obtain the decision score output by the multimodal large model; wherein the decision score represents the confidence level of the multimodal large model in determining that the target object has abnormal behavior; if the decision score is greater than a target alarm threshold, alarm data is sent to the user device, the alarm data including the video frame image and the model inference result; wherein the target alarm threshold is the initial alarm threshold, or the target alarm threshold is obtained by adjusting the initial alarm threshold; wherein the processing module 43 is further configured to statistically analyze the relationship between the prediction and the historical time period. The reference false alarm rate corresponding to the category and the scenario type, where the reference false alarm rate represents the probability value that the inference result of the abnormal action is a false alarm; if the reference false alarm rate is greater than a first tolerance threshold, the initial alarm threshold is increased to obtain the target alarm threshold; if the reference false alarm rate is less than a second tolerance threshold, the initial alarm threshold is decreased to obtain the target alarm threshold, where the second tolerance threshold is less than the first tolerance threshold; if the reference false alarm rate is not less than the second tolerance threshold and the reference false alarm rate is not greater than the first tolerance threshold, the initial alarm threshold is determined as the target alarm threshold.

[0139] For example, the prompt word further includes analysis result response information; wherein, the analysis result response information is used to instruct the multimodal large model to output the model inference result;

[0140] The analysis result response information is also used to instruct the multimodal large model to output the comprehensive judgment reason corresponding to the model inference result and the output format corresponding to the comprehensive judgment reason; and the analysis result response information is also used to instruct the multimodal large model to output a decision score.

[0141] For example, the acquisition module 41 is further configured to acquire the confidence score corresponding to the predicted category; if the confidence score is greater than the preliminary screening threshold, the video frame image is determined to be a potentially risky video frame, and the construction module performs the operation of constructing the prompt word corresponding to the video frame image; if the confidence score is not greater than the preliminary screening threshold, the video frame image is discarded.

[0142] Based on the same application concept as the above method, this application proposes an electronic device, see [link to application]. Figure 5 As shown, it includes: a processor 51 and a machine-readable storage medium 52, the machine-readable storage medium 52 storing machine-executable instructions that can be executed by the processor 51; the processor 51 is used to execute the machine-executable instructions to implement an anomaly analysis method based on real-time target detection and multimodal large model.

[0143] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can realize an anomaly analysis method based on real-time target detection and multimodal large model.

[0144] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0145] Based on the same application concept as the above method, this application embodiment also provides a computer program product, which may include a computer program that, when executed by a processor, implements an anomaly analysis method based on real-time target detection and multimodal large model.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An anomaly analysis method based on real-time target detection and multimodal large model, characterized in that, The method includes: The video frame image is input into the object detection model to obtain the predicted category and predicted location; wherein, the video frame image includes the target object, the predicted category represents the category of the object to be detected carried by the target object, and the predicted location represents the location of the object to be detected; Construct prompt words corresponding to the video frame image, the prompt words including the predicted category and the predicted position, and input the prompt words and the video frame image into the multimodal large model; The multimodal large model determines the first image feature corresponding to the predicted category, obtains the region image of the predicted location from the video frame image using the multimodal large model, and determines the second image feature of the region image using the multimodal large model; if the multimodal large model determines that the similarity between the first image feature and the second image feature is greater than a similarity threshold, then the object to be detected is determined to be a risky object, and the multimodal large model analyzes whether the target object has abnormal actions based on the video frame image, and outputs the model inference result; The model inference results include risk object inference results. If the risk object inference results indicate that the object to be detected is a risk object, the model inference results also include abnormal action inference results, which indicate whether the target object has any abnormal actions.

2. The method according to claim 1, characterized in that, The prompt words include scene description information and deep reasoning information. The scene description information is used to indicate that the object to be detected is a potential risk target. The potential risk target includes the predicted category and the predicted location. The deep reasoning information includes target property reasoning information and behavioral intention reasoning information. The target property reasoning information is used to instruct the multimodal large model to determine whether the object to be detected is a risky object. The behavioral intent reasoning information is used to instruct the multimodal large model to analyze whether the target object has abnormal behavior based on the video frame image when the object to be detected is a risky object.

3. The method according to claim 2, characterized in that, The deep reasoning information also includes counterfactual reasoning information, which is used to instruct the multimodal big model to analyze the explanatory information of the target object carried by the object to be detected based on the video frame image when the target object does not have abnormal actions; The method further includes, after analyzing whether the target object has abnormal actions based on the video frame images using the multimodal large model, the method further includes: if the target object does not have abnormal actions, then analyzing the explanatory information of the object to be detected carried by the target object based on the video frame images using the multimodal large model; wherein the model inference result also includes the explanatory information.

4. The method according to claim 2, characterized in that, The scene description information also includes the scene type associated with the video acquisition device, which is used to acquire the video frame images; The method further includes, after analyzing whether the target object exhibits abnormal behavior based on the video frame images using the multimodal large model, the method further includes: if the target object exhibits abnormal behavior, then analyzing the scene risk level of the environment in which the target object is located based on the scene type using the multimodal large model; wherein the model inference result also includes the scene risk level; The scenario risk level is used to determine the initial alarm threshold for the abnormal action.

5. The method according to claim 4, characterized in that, After analyzing whether the target object exhibits abnormal behavior based on the video frame images using the multimodal large model and outputting the model inference result, the method further includes: Obtain the decision score output by the multimodal large model; wherein the decision score represents the confidence level of the multimodal large model in determining that the target object has abnormal behavior; If the decision score is greater than the target alarm threshold, alarm data is sent to the user equipment. The alarm data includes the video frame image and the model inference result. The target alarm threshold is the initial alarm threshold, or the target alarm threshold is obtained by adjusting the initial alarm threshold. Among them, the reference false alarm rate corresponding to the prediction category and the scene type within the statistical historical time period is calculated, and the reference false alarm rate represents the probability value that the abnormal action reasoning result is a false alarm; If the reference false alarm rate is greater than the first tolerance threshold, the initial alarm threshold is increased to obtain the target alarm threshold; if the reference false alarm rate is less than the second tolerance threshold, the initial alarm threshold is decreased to obtain the target alarm threshold, wherein the second tolerance threshold is less than the first tolerance threshold. If the reference false alarm rate is not less than the second tolerance threshold and the reference false alarm rate is not greater than the first tolerance threshold, then the initial alarm threshold is determined as the target alarm threshold.

6. The method according to claim 1 or 5, characterized in that, The prompt words also include analysis result response information; wherein, the analysis result response information is used to instruct the multimodal large model to output the model inference result; The analysis result response information is used to instruct the multimodal large model to output the comprehensive judgment reason corresponding to the model inference result and the output format corresponding to the comprehensive judgment reason; The analysis result response information is used to instruct the multimodal large model to output a decision score.

7. The method according to claim 1, characterized in that, After inputting the video frame image into the object detection model to obtain the predicted category and predicted location, the method further includes: Obtain the confidence score corresponding to the predicted category; If the confidence score is greater than the initial screening threshold, the video frame image is determined to be a potentially risky video frame, and the operation of constructing prompt words corresponding to the video frame image is performed. If the confidence score is not greater than the initial screening threshold, the video frame image is discarded.

8. An anomaly analysis device based on real-time target detection and multimodal large model, characterized in that, The device includes: The acquisition module is used to input video frame images into the target detection model to obtain predicted categories and predicted locations; wherein, the video frame images include target objects, the predicted category represents the category of the object to be detected carried by the target object, and the predicted location represents the location of the object to be detected; A construction module is used to construct prompt words corresponding to the video frame image, the prompt words including the predicted category and the predicted position, and input the prompt words and video frame image into the multimodal large model; The processing module is configured to determine the first image feature corresponding to the predicted category using the multimodal large model, obtain the region image of the predicted location from the video frame image using the multimodal large model, and determine the second image feature of the region image using the multimodal large model; if the multimodal large model determines that the similarity between the first image feature and the second image feature is greater than a similarity threshold, then the object to be detected is determined to be a risky object; the multimodal large model analyzes whether the target object has abnormal actions based on the video frame image, and outputs the model inference result; The model inference results include risk object inference results. If the risk object inference results indicate that the object to be detected is a risk object, the model inference results also include abnormal action inference results, which indicate whether the target object has any abnormal actions.

9. The apparatus according to claim 8, characterized in that, The prompts include scene description information and deep reasoning information. The scene description information is used to indicate that the object to be detected is a potential risk target. The potential risk target includes the predicted category and the predicted location. The deep reasoning information includes target property reasoning information and behavioral intention reasoning information. The target property reasoning information is used to instruct the multimodal large model to determine whether the object to be detected is a risk object. The behavioral intent reasoning information is used to instruct the multimodal big model to analyze whether the target object has abnormal actions based on the video frame image when the object to be detected is a risky object. Alternatively, the deep reasoning information may also include counterfactual reasoning information, which is used to instruct the multimodal large model to analyze the explanatory information of the target object carried by the object to be detected based on the video frame image when the target object does not have abnormal actions; The processing module is further configured to, after analyzing whether the target object has abnormal actions based on the video frame images using the multimodal large model, if the target object does not have abnormal actions, analyze the explanatory information of the object to be detected carried by the target object based on the video frame images using the multimodal large model; wherein the model inference result also includes the explanatory information; Alternatively, the scene description information may also include the scene type associated with the video acquisition device, which is used to acquire the video frame images; wherein, the processing module is further configured to, after analyzing whether the target object has abnormal actions based on the video frame images using the multimodal large model, if the target object has abnormal actions, analyze the scene risk level of the environment in which the target object is located based on the scene type using the multimodal large model; wherein, the model inference result may also include the scene risk level; the scene risk level is used to determine the initial alarm threshold for the abnormal actions; Alternatively, the processing module is further configured to obtain a decision score output by the multimodal large model; wherein the decision score represents the confidence level of the multimodal large model in determining that the target object has abnormal behavior; if the decision score is greater than a target alarm threshold, alarm data is sent to the user equipment, the alarm data including the video frame image and the model inference result; wherein the target alarm threshold is the initial alarm threshold, or the target alarm threshold is obtained by adjusting the initial alarm threshold; The processing module is further configured to: calculate the reference false alarm rate corresponding to the predicted category and the scene type within a historical time period, wherein the reference false alarm rate represents the probability value that the abnormal action reasoning result is a false alarm; if the reference false alarm rate is greater than a first tolerance threshold, then the initial alarm threshold is increased to obtain the target alarm threshold; if the reference false alarm rate is less than a second tolerance threshold, then the initial alarm threshold is decreased to obtain the target alarm threshold, wherein the second tolerance threshold is less than the first tolerance threshold; if the reference false alarm rate is not less than the second tolerance threshold and the reference false alarm rate is not greater than the first tolerance threshold, then the initial alarm threshold is determined as the target alarm threshold. Alternatively, the prompt word may also include analysis result response information; wherein the analysis result response information is used to instruct the multimodal large model to output the model inference result; The analysis result response information is also used to instruct the multimodal large model to output the comprehensive judgment reason corresponding to the model inference result and the output format corresponding to the comprehensive judgment reason; and the analysis result response information is also used to instruct the multimodal large model to output a decision score; Alternatively, the acquisition module is further configured to acquire the confidence score corresponding to the predicted category; if the confidence score is greater than the preliminary screening threshold, the video frame image is determined to be a potentially risky video frame, and the construction module performs the operation of constructing the prompt word corresponding to the video frame image; if the confidence score is not greater than the preliminary screening threshold, the video frame image is discarded.

10. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-7.

Images