Abnormal behavior detection method and device and electronic equipment

Abstract

The invention provides an abnormal behavior detection method and device and electronic equipment, and relates to the technical field of behavior detection.The method comprises the steps that a monitoring data set is obtained, and the monitoring data set comprises monitoring data collected by using a non-fixed camera; target features are detected from the monitoring data set through a pre-trained target detection small model, monitoring data containing the target features are marked as target images, and the target features comprise any one or more of personnel, vehicles and ships; identifying key visual features from the target image by using a pre-trained multi-modal large model, and converting the key visual features into text data; and comparing the text data with a preset scene knowledge base by using a pre-trained large language model, judging whether the text data is abnormal behavior data, and determining whether to trigger an alarm. The key visual features can be comprehensively extracted, the recognition rate of the key visual features is improved, and abnormal behaviors can be accurately detected.

Description

Technical Field

[0001] The present disclosure relates to the technical field of behavior detection, and in particular to an abnormal behavior detection method, device, and electronic device. Background Art

[0002] In scenarios where non-fixed cameras are used to monitor abnormal behavior in important areas, in order to meet the needs of manual duty, non-fixed cameras are mainly dome cameras, and the viewing angle and focal length often change. Abnormal behaviors monitored in important areas are often sporadic, and the amount of sample data containing abnormal behaviors is small. It is difficult to obtain a large number of labeled samples for model training. The relevant technology has insufficient generalization ability in the few-sample learning scenario, and the recognition rate of abnormal behavior is low, and it is unable to accurately detect abnormal behavior.

[0003] Currently, no effective technical solutions have been proposed to address the problems of insufficient generalization capabilities of related technologies in few-sample learning scenarios, low recognition rates of abnormal behaviors, and inability to accurately detect abnormal behaviors. Summary of the Invention

[0004] The main purpose of the present disclosure is to provide an abnormal behavior detection method, device and electronic device to solve the problem that the related technology has insufficient generalization ability in a few-sample learning scenario, has a low recognition rate for abnormal behavior, and cannot accurately detect abnormal behavior.

[0005] To achieve the above objectives, the first aspect of the present disclosure provides a method for detecting abnormal behavior, comprising:

[0006] Acquire a monitoring data set, wherein the monitoring data set includes monitoring data collected using a non-fixed camera;

[0007] Detecting target features from the monitoring data set using a pre-trained target detection model, and marking the monitoring data containing the target features as a target image, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of a person, a vehicle, and a vessel;

[0008] Using a pre-trained multimodal large model to identify key visual features from the target image and converting the key visual features into text data, wherein the pre-trained multimodal large model is a pre-trained multimodal large model;

[0009] The text data is compared with a preset scenario knowledge base using a pre-trained large language model to determine whether the text data is abnormal behavior data and whether to trigger an alarm, wherein the pre-trained large language model is a pre-trained large language model.

[0010] Optionally, obtaining the monitoring data set includes:

[0011] Collecting surveillance data using a non-fixed camera in a preset scenario, and labeling each image in the surveillance data;

[0012] Multi-source heterogeneous data is fused into the monitoring data, wherein the multi-source heterogeneous data includes data collected by radar, sound sensor or vibration sensor.

[0013] Optionally, the pre-trained target detection model is a target detection algorithm based on a convolutional neural network, including any one or more of a single-stage detector and a YOLO series model.

[0014] Optionally, the key visual features include the target features and environmental features;

[0015] The method of using a pre-trained multimodal large model to identify key visual features from the target image and converting the key visual features into text data includes:

[0016] Identifying key visual features and environmental features from the target image using a pre-trained multimodal large model, wherein the pre-trained multimodal large model includes any one or more of CLIP, Visual-BERT, OFA model, RWKV series model, and VisualRWKV model, and the environmental features include any one or more of barbed wire, rivers, fences, and sentry posts;

[0017] Determine the temporal and spatial correlation between the target features and the environmental features by asking preset questions;

[0018] The target features, environmental features and spatiotemporal association relationships are converted into text data.

[0019] Optionally, the using of a pre-trained large language model to compare the text data with a preset scenario knowledge base, determining whether the text data is abnormal behavior data, and determining whether to trigger an alarm includes:

[0020] Optimize prompt words in a pre-trained large language model, compare the text data with a pre-set scenario knowledge base, and determine whether the text data meets the abnormal behavior of scenario control, wherein the pre-trained large language model includes any one or more of ChatGPT, BERT, and DeepSeek, the scenario knowledge base includes rules and knowledge of scenario control, and the abnormal behavior includes any one or more of crossing the border, wading across the border, approaching barbed wire, and climbing barbed wire;

[0021] If the text data conforms to the abnormal behavior controlled by the scenario, the text data is abnormal behavior data, and an alarm is triggered.

[0022] Furthermore, if the text data does not conform to the abnormal behavior of the scenario control, the text data is not abnormal behavior data and no alarm is triggered.

[0023] Optionally, the method further includes:

[0024] A three-level fusion architecture is established based on the pre-trained target detection small model, the pre-trained multimodal large model and the pre-trained large language model, wherein the pre-trained target detection small model is the bottom layer of the three-level fusion architecture, the pre-trained multimodal large model is the middle layer of the three-level fusion architecture, and the pre-trained large language model is the top layer of the three-level fusion architecture.

[0025] A second aspect of the present disclosure provides an abnormal behavior detection device, comprising:

[0026] an acquisition unit, configured to acquire a monitoring data set, wherein the monitoring data set includes monitoring data collected using a non-fixed camera;

[0027] a detection unit, configured to detect target features from the monitoring data set using a pre-trained target detection model, and mark the monitoring data containing the target features as a target image, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of a person, a vehicle, and a vessel;

[0028] a recognition unit, configured to recognize key visual features from the target image using a pre-trained multimodal large model and convert the key visual features into text data, wherein the pre-trained multimodal large model is a pre-trained multimodal large model;

[0029] A comparison unit is used to compare the text data with a preset scenario knowledge base using a pre-trained large language model to determine whether the text data is abnormal behavior data and whether to trigger an alarm, wherein the pre-trained large language model is a pre-trained large language model.

[0030] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the abnormal behavior detection method provided by any one of the first aspects.

[0031] The fourth aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the abnormal behavior detection method provided in any one of the first aspects.

[0032] In the abnormal behavior detection method provided in the embodiments of the present disclosure, a monitoring data set is obtained, wherein the monitoring data set includes monitoring data collected using a non-fixed camera; a pre-trained target detection model is used to detect target features from the monitoring data set, and the monitoring data containing the target features are marked as target images, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of people, vehicles, and ships; due to the fast processing characteristics of the pre-trained target detection model, monitoring data containing target features can be quickly identified from massive data in a short period of time;

[0033] A pre-trained multimodal large model is used to identify key visual features from the target image and convert the key visual features into text data, wherein the pre-trained multimodal large model is a pre-trained multimodal large model. The powerful generalization ability of the pre-trained multimodal large model reduces the dependence on target and abnormal behavior samples, thereby solving the problem of insufficient control scene samples. In addition, the use of multimodal fusion can comprehensively extract key visual features, improve the recognition rate of key visual features, and solve the problem of difficulty in identifying special targets.

[0034] The pre-trained large language model is used to compare the text data with a pre-set scenario knowledge base to determine whether the text data represents abnormal behavior and whether to trigger an alarm. The pre-trained large language model is a pre-trained large language model. The pre-trained large language model's powerful semantic understanding and logical reasoning capabilities can accurately determine whether an alarm should be triggered, resolving the issue of related technologies' insufficient generalization capabilities in small-sample learning scenarios, resulting in low recognition rates for abnormal behavior and an inability to accurately detect abnormal behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the following briefly introduces the drawings required for use in the specific embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 A flowchart of the abnormal behavior detection method provided in an embodiment of the present disclosure;

[0037] Figure 2 A block diagram of an abnormal behavior detection device provided in an embodiment of the present disclosure;

[0038] Figure 3A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0040] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present disclosure described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatus.

[0041] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] In scenarios where non-fixed cameras are used to monitor abnormal behavior in important areas such as borders, in order to meet the needs of manual duty, non-fixed cameras are mainly dome cameras, and their viewing angles and focal lengths often change. Abnormal behaviors monitored in important areas are often sporadic, and the amount of sample data containing abnormal behaviors is small, making it difficult to obtain a large number of labeled samples for model training. Related technologies lack generalization capabilities in few-sample learning scenarios, and the recognition rate of abnormal behaviors is low, making it impossible to accurately detect abnormal behaviors. They face complex challenges such as dynamic scene adaptation, special target detection, and few-sample anomaly recognition.

[0043] In order to solve the above problems, the present disclosure provides an abnormal behavior detection method, such as Figure 1 As shown, the method includes the following steps S11 to S14:

[0044] Step S11: Acquire a monitoring data set, wherein the monitoring data set includes monitoring data collected by non-fixed cameras. There are a large number of non-fixed cameras in important areas, which can continuously generate a large amount of monitoring data.

[0045] In an optional embodiment of the present disclosure, step S11 includes:

[0046] Use non-fixed cameras to collect surveillance data in preset scenarios and annotate each image in the surveillance data. The preset scenario can be a pre-set control scenario. For each image obtained by the non-fixed camera in the control scenario, the target category, precise location, and whether there is any abnormal behavior can be annotated for each image. The target category can be people, vehicles, ships, etc.

[0047] Fusion of heterogeneous multi-source data within surveillance data. This data includes data collected by radar, acoustic sensors, or vibration sensors. This fusion process can be extended to include processing heterogeneous multi-source data from radar, acoustic sensors, vibration sensors, and more.

[0048] Step S12: Use the pre-trained target detection model to detect target features from the monitoring data set, and mark the monitoring data containing the target features as a target image, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of personnel, vehicles and ships; the pre-trained target detection model has been pre-trained and does not need to perform the training process again, and the target image can be an alarm picture or an alarm video.

[0049] The present disclosure utilizes the fast processing characteristics of a pre-trained target detection small model to quickly identify monitoring data containing target features from massive data in a short period of time.

[0050] In an optional embodiment of the present disclosure, the pre-trained target detection small model in step S12 is a target detection algorithm based on a convolutional neural network, including any one or more of a single-stage detector and a YOLO series model.

[0051] The pre-trained object detection model includes a single-stage detector (Single Shot MultiBox Detector, referred to as SSD) and any one or more of the YOLO series models, such as the YOLO v8 model.

[0052] By pre-training a small target detection model, a preliminary and rapid screening of massive amounts of surveillance data can be performed. Due to the large number of non-fixed cameras in important areas, a large amount of surveillance data is continuously generated. By using the fast processing characteristics of the pre-trained target detection model, surveillance data containing target features can be identified from massive amounts of data in a short period of time, so as to narrow the scope when processing image data later.

[0053] Step S13: Use the pre-trained multimodal big model to identify key visual features from the target image and convert the key visual features into text data, wherein the pre-trained multimodal big model is a pre-trained multimodal big model; the pre-trained multimodal big model has been pre-trained and does not need to be trained again; the pre-trained multimodal big model has powerful cross-modal information fusion capabilities, deep semantic understanding capabilities and logical reasoning capabilities, and can simultaneously process multiple types of data such as images, videos, and audio, and fully explore the complementary information between different modal data, so as to more comprehensively and accurately understand the border monitoring scene. Through the pre-trained multimodal big model, real-time perception, precise analysis and intelligent decision-making of complex scenes in important areas can be achieved, effectively improving the efficiency and safety of management and control.

[0054] The present invention reduces the dependence on target and abnormal behavior samples by pre-training the powerful generalization ability of the multimodal large model, does not require a large amount of sample annotation, and can realize abnormal behavior detection of special targets such as barbed wire with zero or few samples, thereby solving the problem of few and scarce samples in management and control scenarios; and the use of multimodal fusion can comprehensively extract key visual features, improve the recognition rate of key visual features, and solve the problem of difficult identification of special targets such as barbed wire.

[0055] In an optional embodiment of the present disclosure, the key visual features include target features and environmental features; the key visual features may be the appearance features, clothing features, and items carried by a person;

[0056] Wherein, step S13 includes:

[0057] Using a pre-trained multimodal large model to identify target features and environmental features from a target image, the pre-trained multimodal large model includes any one or more of CLIP (Contrastive Language-Image Pretraining), Visual-BERT, OFA (One-For-All) model, RWKV (Recurrent Weighted Key-Value) series models, and Visual RWKV model; target features include any one or more of people, vehicles, and ships; and environmental features include any one or more of barbed wire, rivers, fences, and sentry posts; for example, the pre-trained multimodal large model may be the 6B model of Visual RWKV 6.0;

[0058] By using pre-set questions, the temporal and spatial correlation between target features and environmental features can be determined. By using pre-set questions to the machine, human-machine question and answer, human-machine interaction, and text-based question and answer methods can be used to determine the temporal and spatial correlation between target features and environmental features. The temporal and spatial correlation can include a person approaching a wire fence, a person climbing a wire fence, a vehicle approaching a fence, a boat crossing a river, etc.

[0059] Convert target features, environmental features, and spatiotemporal relationships into text data.

[0060] Pre-training a multimodal large model can convert image data into text data and extract key visual features from the image data. For example, for a picture containing a suspicious person, the pre-trained multimodal large model can identify the person's appearance, clothing characteristics, and items carried, and convert these key visual features into text data, providing a data format that is easier to understand and process for subsequent analysis.

[0061] Step S14: Use pre-trained large language models (LLMs) to compare text data with a preset scenario knowledge base to determine whether the text data is abnormal behavior data and whether to trigger an alarm. The pre-trained large language model is a pre-trained large language model. Abnormal behavior data can include people wading, crossing the border, wading across the border, people staying abnormally near barbed wire, vehicles approaching barbed wire, smuggling, etc. The pre-trained large language model has been pre-trained and does not need to be trained again. The pre-trained large language model can be optimized through prompt words and continuously optimized to a scenario-fine-tuned pre-trained large language model. Through fine-tuning of the control scenario, the pre-trained large language model has strong semantic understanding and logical reasoning capabilities. It can determine whether an alarm needs to be triggered based on the input text data and the rules and knowledge of the control scenario.

[0062] The present disclosure uses the powerful semantic understanding and logical reasoning capabilities of the pre-trained large language model to accurately determine whether an alarm needs to be triggered, solving the problem that related technologies have insufficient generalization capabilities in small-sample learning scenarios, have low recognition rates for abnormal behaviors, and cannot accurately detect abnormal behaviors.

[0063] In an optional embodiment of the present disclosure, step S14 includes:

[0064] Prompt word optimization is performed in the pre-trained large language model, and the text data is compared with the pre-set scenario knowledge base to determine whether the text data meets the abnormal behavior of scenario control. The pre-trained large language model includes any one or more of ChatGPT, BERT and DeepSeek. The scenario knowledge base includes the rules and knowledge of scenario control. Abnormal behaviors include any one or more of crossing the border, wading across the border, approaching barbed wire and climbing barbed wire. Abnormal behaviors also include abnormal stays of people near barbed wire, vehicles approaching barbed wire, smuggling, etc.

[0065] If the text data matches the abnormal behavior of scenario control, the text data is abnormal behavior data and an alarm is triggered.

[0066] In an optional implementation of the present disclosure, if the text data does not conform to the abnormal behavior of the scenario control, the text data is not abnormal behavior data and no alarm is triggered.

[0067] For example, when the pre-trained large language model receives the text data described by the pre-trained multimodal large model, "A non-uniformed person is wandering near a border protection facility, carrying a suspected package item", the pre-trained large language model can comprehensively analyze this information while continuously optimizing the manually proposed prompt words, and determine whether this behavior is consistent with abnormal behavior in border control, and thus decide whether to issue an alarm.

[0068] In an optional embodiment of the present disclosure, the method further includes:

[0069] A three-level fusion architecture is established based on a small pre-trained target detection model, a large pre-trained multimodal model, and a large pre-trained language model. The small pre-trained target detection model forms the bottom layer of the three-level fusion architecture, the large pre-trained multimodal model forms the middle layer, and the large pre-trained language model forms the top layer. By using the small pre-trained target detection model as the bottom layer, the large pre-trained multimodal model as the middle layer, and the large pre-trained language model as the top layer of the three-level fusion architecture, a three-level fusion architecture or framework is established. This architecture is used to accurately detect abnormal behavior in control scenarios.

[0070] This disclosure breaks through related technical bottlenecks through cross-layer technology collaboration. The first-level detection layer uses a lightweight pre-trained target detection model to achieve rapid recognition of people, vehicles, and ships. The second-level multimodal understanding layer uses a pre-trained multimodal large model to perform semantic understanding of target images and supplement surrounding facilities and environmental information. The third-level reasoning layer builds a scene-specific logical knowledge base and uses a pre-trained large language model to analyze the spatiotemporal relationship between targets and scenes, enabling intelligent decision-making in complex scenarios. For example, it can realize the risk judgment of crossing the boundary, the analysis of abnormal behaviors such as people wading in water and climbing barbed wire. The three-level architecture is progressive, and through the collaboration of different models at multiple levels, the accuracy and reliability of alarms are improved.

[0071] From the above description, it can be seen that the present disclosure achieves the following technical effects:

[0072] This disclosure uses the fast processing characteristics of the pre-trained target detection small model to quickly identify monitoring data containing target features from massive data in a short period of time;

[0073] By leveraging the powerful generalization capabilities of pre-trained multimodal large models, the system reduces reliance on target and abnormal behavior samples, thereby resolving the issue of limited control scenario samples. Furthermore, the use of multimodal fusion enables comprehensive extraction of key visual features, improving their recognition rate and resolving the difficulty in identifying special targets.

[0074] By pre-training the large language model's powerful semantic understanding and logical reasoning capabilities, it is possible to accurately determine whether an alarm needs to be triggered, solving the problem of insufficient generalization ability of related technologies in small-sample learning scenarios, low recognition rate of abnormal behaviors, and inability to accurately detect abnormal behaviors.

[0075] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0076] The present disclosure also provides an abnormal behavior detection device for implementing the above method embodiment. Figure 2 As shown, the abnormal behavior detection device 20 includes:

[0077] an acquisition unit 21, configured to acquire a monitoring data set, wherein the monitoring data set includes monitoring data collected using a non-fixed camera;

[0078] A detection unit 22 is configured to detect target features from a monitoring data set using a pre-trained target detection model, and mark the monitoring data containing the target features as a target image, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of a person, a vehicle, and a vessel;

[0079] a recognition unit 23 for identifying key visual features from a target image using a pre-trained multimodal large model and converting the key visual features into text data, wherein the pre-trained multimodal large model is a pre-trained multimodal large model;

[0080] The comparison unit 24 is used to compare the text data with the preset scenario knowledge base using a pre-trained large language model to determine whether the text data is abnormal behavior data and whether to trigger an alarm, wherein the pre-trained large language model is a pre-trained large language model.

[0081] The specific manner in which each unit in the above device embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0082] The present disclosure also provides an electronic device, such as Figure 3 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 3 A processor 31 is taken as an example.

[0083] The controller may further include an input device 33 and an output device 34 .

[0084] The processor 31, memory 32, input device 33 and output device 34 may be connected via a bus or other means. Figure 3 The bus connection is taken as an example.

[0085] The processor 31 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. The general-purpose processor can be a microprocessor or any conventional processor.

[0086] Memory 32, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the control method in the embodiments of the present disclosure. Processor 31 executes the non-transitory software programs, instructions, and modules stored in memory 32 to execute various server functional applications and data processing, thereby implementing the abnormal behavior detection method of the above-mentioned method embodiment.

[0087] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the processing device operated by the server, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include a memory remotely located relative to the processor 31, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0088] The input device 33 can receive input digital or character information and generate key signal input related to user settings and function control of the processing device of the server. The output device 34 can include a display device such as a display screen.

[0089] One or more modules are stored in the memory 32 and when executed by one or more processors 31, perform the following operations: Figure 1 The method shown.

[0090] Those skilled in the art will appreciate that all or part of the processes in the above method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FM), a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above types of memory.

[0091] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting abnormal behavior, characterized in that: include: Acquire a monitoring data set, wherein the monitoring data set includes monitoring data collected using a non-fixed camera; Detecting target features from the monitoring data set using a pre-trained target detection model, and marking the monitoring data containing the target features as a target image, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of a person, a vehicle, and a vessel; Using a pre-trained multimodal large model to identify key visual features from the target image and converting the key visual features into text data, wherein the pre-trained multimodal large model is a pre-trained multimodal large model; The text data is compared with a preset scenario knowledge base using a pre-trained large language model to determine whether the text data is abnormal behavior data and whether to trigger an alarm, wherein the pre-trained large language model is a pre-trained large language model.

2. The method according to claim 1, characterized in that The obtaining of the monitoring data set includes: Collecting surveillance data using a non-fixed camera in a preset scenario, and labeling each image in the surveillance data; Multi-source heterogeneous data is fused into the monitoring data, wherein the multi-source heterogeneous data includes data collected by radar, sound sensor or vibration sensor.

3. The method according to claim 1, characterized in that The pre-trained target detection model is a target detection algorithm based on a convolutional neural network, including any one or more of a single-stage detector and a YOLO series model.

4. The method according to claim 1, wherein The key visual features include the target features and environmental features; The method of using a pre-trained multimodal large model to identify key visual features from the target image and converting the key visual features into text data includes: Identifying target features and environmental features from the target image using a pre-trained multimodal large model, wherein the pre-trained multimodal large model includes any one or more of CLIP, Visual-BERT, OFA model, RWKV series model, and Visual RWKV model, and the environmental features include any one or more of barbed wire, rivers, fences, and sentry posts; Determine the temporal and spatial correlation between the target features and the environmental features by asking preset questions; The target features, environmental features and spatiotemporal association relationships are converted into text data.

5. The method according to claim 1, wherein The method of comparing the text data with a preset scenario knowledge base using a pre-trained large language model to determine whether the text data is abnormal behavior data and whether to trigger an alarm includes: Optimize prompt words in a pre-trained large language model, compare the text data with a pre-set scenario knowledge base, and determine whether the text data meets the abnormal behavior of scenario control, wherein the pre-trained large language model includes any one or more of ChatGPT, BERT, and DeepSeek, the scenario knowledge base includes rules and knowledge of scenario control, and the abnormal behavior includes any one or more of crossing the border, wading across the border, approaching barbed wire, and climbing barbed wire; If the text data conforms to the abnormal behavior controlled by the scenario, the text data is abnormal behavior data, and an alarm is triggered.

6. The method according to claim 5, characterized in that If the text data does not conform to the abnormal behavior of the scenario control, the text data is not abnormal behavior data and no alarm is triggered.

7. The method according to claim 1, characterized in that The method further comprises: A three-level fusion architecture is established based on the pre-trained target detection small model, the pre-trained multimodal large model and the pre-trained large language model, wherein the pre-trained target detection small model is the bottom layer of the three-level fusion architecture, the pre-trained multimodal large model is the middle layer of the three-level fusion architecture, and the pre-trained large language model is the top layer of the three-level fusion architecture.

8. An abnormal behavior detection device, characterized in that: include: an acquisition unit, configured to acquire a monitoring data set, wherein the monitoring data set includes monitoring data collected using a non-fixed camera; a detection unit, configured to detect target features from the monitoring data set using a pre-trained target detection model, and mark the monitoring data containing the target features as a target image, wherein the pre-trained target detection model is a pre-trained target detection model, and the target features include any one or more of a person, a vehicle, and a vessel; a recognition unit, configured to recognize key visual features from the target image using a pre-trained multimodal large model and convert the key visual features into text data, wherein the pre-trained multimodal large model is a pre-trained multimodal large model; A comparison unit is used to compare the text data with a preset scenario knowledge base using a pre-trained large language model to determine whether the text data is abnormal behavior data and whether to trigger an alarm, wherein the pre-trained large language model is a pre-trained large language model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the abnormal behavior detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the abnormal behavior detection method described in any one of claims 1 to 7.

Images