Target behavior detection method and device in video monitoring, equipment and storage medium

Abstract

The invention discloses a target behavior detection method and device in video monitoring, equipment and a storage medium. The method comprises the following steps: acquiring video stream data of a monitoring area; detecting an image in the video stream data based on one or more preset behavior detection models to obtain a detection result image of the preset behavior detection model; and filtering the detection result image by using a trained multi-modal large language model to obtain a monitoring target image. According to the technical scheme provided by the invention, the multi-mode large language model is introduced as a false alarm filtering mechanism for secondary analysis, different application scenes can be easily adapted, and the detection of the specified behavior is realized in combination with the required behavior detection model, so that diversified requirements can be flexibly adapted, false alarms are effectively reduced, and the detection efficiency is improved. The detection accuracy and reliability of the whole scheme are improved, high accuracy can be kept even under the complex background or illumination condition, and the user experience is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of video analysis technology, and in particular to a method, device, equipment and storage medium for detecting target behavior in video surveillance. Background Art

[0002] With the rapid development of smart cities, Industry 4.0, and intelligent transportation, higher requirements are placed on the real-time processing and analysis capabilities of video content. Currently, the technical solutions in this field mainly rely on deep learning models such as convolutional neural networks to achieve high-precision object detection and image segmentation in video surveillance. These models need to be trained with a large amount of labeled data to improve detection accuracy. However, in actual applications, false alarms still occur frequently due to factors such as environmental changes, lighting conditions, and complex backgrounds.

[0003] Current false positive handling methods usually include post-processing steps after applying a single traditional model, such as setting a confidence threshold to filter out low-confidence detection results. In addition, there are studies that combine the results of multiple traditional models to reduce false positives through voting mechanisms or weighted averages. However, these methods all require manual parameter adjustment, lack flexibility, and have limited effects in complex scenarios and low accuracy. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, device and storage medium for detecting target behavior in video surveillance, so as to more intelligently and flexibly realize the detection of specified behaviors in video surveillance, and effectively filter out false alarms to improve the overall detection performance.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting target behavior in video surveillance, the method comprising:

[0006] Get the video stream data of the monitoring area;

[0007] Detecting images in the video stream data based on one or more preset behavior detection models to obtain detection result images of the preset behavior detection models;

[0008] The detection result image is filtered using a trained multimodal large language model to obtain a monitoring target image.

[0009] Optionally, the filtering of the detection result image using a trained multimodal large language model to obtain a monitoring target image includes:

[0010] Obtaining a corresponding preset prompt word according to the behavior type corresponding to the detection result image;

[0011] The detection result image and the corresponding preset prompt word are input into the multimodal large language model for question and answer, and it is determined whether the detection result image is the monitoring target image according to the question and answer result.

[0012] Optionally, there are multiple preset prompt words; inputting the detection result image and the corresponding preset prompt words into the multimodal large language model for question and answer, and determining whether the detection result image is the monitoring target image according to the question and answer result, includes:

[0013] The detection result image is questioned and answered multiple times using the multimodal large language model through each of the preset prompt words, and whether the detection result image is the monitoring target image is determined based on the multiple question and answer results.

[0014] Optionally, after acquiring the video stream data of the monitoring area, the method further includes:

[0015] Writing the video stream data into a memory mapped file;

[0016] The detecting the image in the video stream data based on one or more preset behavior detection models includes:

[0017] The video frame data are read in parallel from the memory mapping file and provided to the plurality of preset behavior detection models for detection respectively.

[0018] Optionally, before detecting the image in the video stream data based on one or more preset behavior detection models, the method further includes:

[0019] The preset behavior detection model used is determined based on user configuration.

[0020] Optionally, the method further includes:

[0021] The preset behavior detection model is added, removed or replaced based on user operations.

[0022] In a second aspect, an embodiment of the present invention further provides a target behavior detection device in video surveillance, the device comprising:

[0023] A video stream data acquisition module is used to acquire video stream data of the monitoring area;

[0024] An image detection module, configured to detect images in the video stream data based on one or more preset behavior detection models to obtain a detection result image of the preset behavior detection model;

[0025] The image filtering module is used to filter the detection result image using the trained multimodal large language model to obtain a monitoring target image.

[0026] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0027] one or more processors;

[0028] A memory for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the target behavior detection method in video surveillance provided by any embodiment of the present invention.

[0030] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target behavior detection method in video surveillance provided by any embodiment of the present invention.

[0031] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program, and when the program is executed by a processor, it implements the target behavior detection method in video surveillance provided by any embodiment of the present invention.

[0032] An embodiment of the present invention provides a method for detecting target behavior in video surveillance, which first obtains video stream data of a surveillance area, then detects images in the video stream data based on one or more preset behavior detection models, obtains a detection result image of the preset behavior detection model, and then uses a trained multimodal large language model to filter the obtained detection result image to obtain a final surveillance target image. The method for detecting target behavior in video surveillance provided by an embodiment of the present invention can be easily adapted to different application scenarios by introducing a multimodal large language model as a false alarm filtering mechanism for secondary analysis, and can realize the detection of specified behaviors in combination with the required behavior detection model, so that it can flexibly adapt to diverse needs, and at the same time effectively reduce false alarms, improve the accuracy and reliability of the overall solution detection, and maintain a high accuracy even under complex backgrounds or lighting conditions, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a method for detecting target behavior in video surveillance provided in Embodiment 1 of the present invention;

[0034] Figure 2 A schematic diagram of the structure of a target behavior detection device in video surveillance provided by Embodiment 2 of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0037] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0038] Embodiment 1

[0039] Figure 1 This is a flow chart of the target behavior detection method in video surveillance provided by the first embodiment of the present invention. This embodiment can be applied to various application scenarios, such as urban streets, industrial production sites, etc., to monitor and identify specified behaviors in real time to achieve functions such as risk prevention. The method can be executed by the target behavior detection device in video surveillance provided by the embodiment of the present invention. The device can be implemented by hardware and / or software, and can generally be integrated into a computer device. Figure 1 As shown, the specific steps include:

[0040] S11, obtaining video stream data of the monitoring area.

[0041] S12: Detect images in the video stream data based on one or more preset behavior detection models to obtain detection result images of the preset behavior detection models.

[0042] S13. Use the trained multimodal large language model to filter the detection result image to obtain a monitoring target image.

[0043] Specifically, the video stream data of the monitoring area can be captured by a camera or other video source, and each frame of the obtained video stream data can be encoded into a preset data format for subsequent processing. The preset behavior detection model can be a model based on a deep learning method that is pre-trained for the current application scenario, and various object detection and image segmentation algorithms can be specifically used. The preset behavior detection models used in different application scenarios can be different, and the number can be one or more. Each preset behavior detection model can realize real-time and accurate detection of different behaviors of one or more objects, so that the advantages of various object detection and image segmentation algorithms can be combined to improve the accuracy of the detection results of the preset behavior detection model itself. For example, taking the unloading port of the factory as an example, the requirements during a certain period of time may include key behaviors such as detecting whether the staff wears a safety helmet, whether there is a hook in front of the vehicle when unloading, and whether there is someone on the vehicle when unloading. These requirements require the system to have high-precision and real-time detection capabilities. For each behavior detection, the corresponding preset behavior detection model can be selected to perform safety helmet detection, hook detection, and human body detection, etc.

[0044] After obtaining the video stream data, the images in the video stream data can be detected in parallel based on the preset behavior detection model used in real time. Each preset behavior detection model can obtain a detection result image accordingly, which can be obtained by intercepting the relevant video frame when the preset behavior detection model detects the corresponding potential unsafe behavior. Subsequently, each detection result image can be analyzed by a multimodal large language model specially trained for image recognition for the specified behavior and judged whether it meets the preset conditions corresponding to the behavior, so as to achieve false alarm filtering of the detection result image according to the judgment result, and obtain the final monitoring target image. The use of a multimodal large language model will not affect the main detection function for video monitoring, but can also improve the accuracy of abnormal situation judgment and reduce the dependence on specific types of training data. Exemplarily, when the preset behavior detection model for helmet detection detects a detection result image without wearing a helmet, if the multimodal large language model still judges that the staff in the detection result image is not wearing a helmet, the detection result image can be used as the monitoring target image, and if the multimodal large language model judges that the staff in the detection result image is wearing a helmet, the detection result image can be filtered out as a false alarm result. After obtaining the monitored target image, the specified alarm operation or other response mechanism can be triggered.

[0045] On the basis of the above technical solution, optionally, the detection result image is filtered using a trained multimodal large language model to obtain a monitoring target image, including: obtaining a corresponding preset prompt word according to a behavior type corresponding to the detection result image; inputting the detection result image and the corresponding preset prompt word into the multimodal large language model for question and answer, and determining whether the detection result image is the monitoring target image according to the question and answer result.

[0046] Specifically, after obtaining the detection result image, the corresponding preset prompt words can be obtained according to the behavior type that the preset behavior detection model used is responsible for detecting. The preset prompt words can be pre-configured according to the behavior types targeted by various preset behavior detection models. Then the detection result image and its corresponding preset prompt words can be input into the multimodal large language model together. After receiving the detection result image and its corresponding preset prompt words, the multimodal large language model can use its own contextual understanding and pattern recognition capabilities to analyze the detection result image, and give answers according to the requirements of the corresponding preset prompt words, so that it can be determined whether the detection result image is a monitoring target image that needs to be alarmed based on the question and answer results. Exemplarily, the preset prompt words are "Is the staff wearing a safety helmet? Answer yes or no".

[0047] Further optionally, there are multiple preset prompt words; inputting the detection result image and the corresponding preset prompt words into the multimodal large language model for question and answer, and determining whether the detection result image is the monitoring target image according to the question and answer results, includes: using the multimodal large language model to conduct multiple questions and answers on the detection result image through each of the preset prompt words, and determining whether the detection result image is the monitoring target image according to the multiple question and answer results.

[0048] Specifically, for a detection result image, when asking questions to a multimodal large language model, multiple question-answering feedback can be performed on the detection result image through different preset prompt words, and a more comprehensive final recognition judgment can be made based on the multiple question-answering results. Different preset prompt words can be used to ask questions about the detection result image around the same behavior type to obtain more details for judgment. Specifically, preset rules or thresholds can be combined to further improve the flexibility and adaptability of the overall solution and optimize the quality of the final output results. Further, questions and answers can be performed in sequence according to multiple preset prompt words, and after completing the question and answer of the current preset prompt word, it is determined whether more preset prompt words are needed for further question and answer based on the current question and answer results. If further question and answer is needed, new preset prompt words can be obtained and input into the multimodal large language model with the corresponding detection result image for question and answer, and the question and answer are repeated until it is considered that sufficient information has been obtained to make a final judgment. For example, in a scenario where it is determined whether there is a hook in front of the vehicle when unloading, if the multimodal large language model gives a "yes" answer when answering the preset prompt "Is the dump box on the vehicle in a raised state? Answer yes or no", then you can further ask "Is there a hook in front of the vehicle? Answer yes or no". At this time, if the answer "yes" is given and it is consistent with the detection result of the corresponding preset behavior detection model, then the detection result can be considered correct, and the corresponding alarm operation can be triggered and monitoring can continue. At this time, if the answer "no" is given and it is inconsistent with the detection result of the corresponding preset behavior detection model, then the detection result can be considered to be a false alarm result, and the detection result can be ignored or further verification can be performed. As another example, if the multimodal large language model gives a "no" answer when answering the preset prompt word "Is the staff wearing a safety helmet? Answer yes or no", you can further ask "Please confirm whether there is an object similar to a safety helmet on the head of the staff, but it may not be accurately identified due to angle or light problems, answer yes or no", and if a "yes" answer is given at this time, it can be considered that there may be a false alarm, and the staff can be further verified or reminded to wear the safety helmet correctly. Experiments have proved that the scheme provided in this embodiment reduces the false alarm rate by 60% to 85% compared with the traditional scheme. The specific effect varies depending on the application scenario, but all are significantly reduced. For example, in the above-mentioned factory discharge port application scenario, the average false alarm rate of the traditional scheme is 80%, and the average false alarm rate of the scheme provided in this embodiment is 10%. When applied in an industrial production environment, the overall recognition accuracy of the scheme provided in this embodiment is improved by 15% to 35% compared with the traditional scheme.

[0049] On the basis of the above technical solution, optionally, after acquiring the video stream data of the monitored area, it also includes: writing the video stream data into a memory mapping file; detecting images in the video stream data based on one or more preset behavior detection models, including: reading video frame data from the memory mapping file in parallel, and providing them to multiple preset behavior detection models for detection respectively.

[0050] Specifically, the encoded video frame data can be directly written into a pre-allocated memory mapping file, and each preset behavior detection model can correspond to an algorithm module, and each algorithm module runs independently, so that multiple algorithm modules can quickly access the video frame data for detection. Each algorithm module can read the latest video frame from the memory mapping file in parallel and perform the corresponding object detection or image segmentation task. Traditional solutions usually need to pass data to each design module in turn, using a linear data processing flow, which may lead to uneven resource allocation or redundant calculations, especially when processing large-scale video streams, which will affect the overall system efficiency. In contrast, the solution provided in this embodiment can also enable video frame data to be shared in parallel among multiple algorithm modules by using memory mapping file technology, thereby effectively reducing the overhead of data transmission and copying, making full use of the multi-core processing capabilities of modern computing devices, and significantly improving the overall processing efficiency, especially when processing high-resolution video streams. The advantage is obvious. Experiments have proved that the solution provided in this embodiment has a processing speed improvement of more than 20% compared with the traditional linear processing solution.

[0051] Based on the above technical solution, optionally, before detecting the image in the video stream data based on one or more preset behavior detection models, it also includes: determining the preset behavior detection model to be used based on user configuration.

[0052] Specifically, an algorithm management system similar to a routing mechanism can be designed, and a modular design method can be adopted to allow users to flexibly configure and select one or more preset behavior detection models to be used according to the needs of the current actual scene, so as to ensure that the system can quickly adapt to diverse and changing monitoring needs. Taking the above factory discharge port scene as an example, in a specific time period or a specific area, it may only be necessary to enable the helmet detection model, while in other time periods or areas, it may be necessary to enable multiple models at the same time for joint detection. Specifically, a friendly user interface can be provided, and alternative preset behavior detection models can be provided to the user through the user interface, so that the user can configure the preset behavior detection model currently required to be loaded by checking. Exemplarily, in the above factory discharge port application scenario, the user can check the options such as the helmet detection model, the hook detection model and the human body detection model. Further, the user can also be allowed to configure the detailed properties of each preset behavior detection model, such as the detection accuracy, processing speed and resource occupancy, so that the user can adjust the parameters and settings of the model according to the needs of the actual application scenario to achieve the best detection effect.

[0053] On the basis of the above technical solution, optionally, the method further includes: adding, removing or replacing the preset behavior detection model based on user operation. Specifically, the above algorithm management system can be adopted. Through a flexible architecture, users can also be allowed to add, remove or replace the alternative preset behavior detection models in the system according to the needs of the actual application scenario. It can be easily integrated into the existing system without affecting the remaining modules. The system can automatically adapt to achieve the above target behavior detection method. For example, for different application scenarios, from urban streets to industrial production sites, etc., corresponding preset behavior detection models can be added for new scenarios, and the original preset behavior detection models for old scenarios can be removed to adapt to changes in different application scenarios. For example, when the performance of the algorithm itself is improved for some preset behavior detection models, the improved model can be used to replace the original model to improve the performance of the overall solution. Thereby achieving strong versatility and practicality, and meeting the detection needs in diverse scenarios well through adaptive configuration capabilities.

[0054] The system architecture designed by the traditional solution is rigid. When facing new detection requirements, algorithm performance improvements or application scenario changes, the entire system often needs to be adjusted or even reconstructed on a large scale, which not only increases the complexity of system maintenance, but also limits its ability to quickly adapt to new requirements. At the same time, the traditional solution also focuses on the accuracy of specific detection tasks when dealing with false alarms, while ignoring other potential important information. This design also needs to retrain or significantly adjust the model structure when the application scenario changes, which reduces the versatility and practicality. In addition, in order to achieve a good false alarm control effect, a large number of accurately labeled data sets are usually required for training. However, it is very difficult and time-consuming to obtain sufficient and qualified training samples, especially in complex backgrounds or rare events, which also limits the wide application of detection technology. The solution provided in this embodiment uses a modular design, and each algorithm module is a relatively independent unit, so that the update or replacement of a single algorithm module will not affect the operation of the entire system, and can support flexible algorithm combinations and configuration adjustments without large-scale adjustments or reconstruction of the entire system, thereby simplifying the later maintenance and upgrade work. As new detection algorithms continue to emerge, users can integrate new technologies into existing systems through simple operations, thereby ensuring the long-term technological advancement and competitiveness of the system. At the same time, this flexibility enables the system to quickly adapt to a variety of complex and changing application scenarios, whether it is high-precision specific target recognition or a wide range of general object detection, thereby enhancing versatility and practicality. Further combined with the above-mentioned flexible algorithm management, multimodal large language model false alarm filtering, and memory-mapped file data processing mechanism, etc., it can achieve wide application and accurate and efficient operation in complex monitoring environments, thereby significantly improving the performance and user experience of the video surveillance system, and can also bring higher economic benefits and social value to users.

[0055] The technical solution provided by the embodiment of the present invention first obtains the video stream data of the monitoring area, then detects the images in the video stream data based on one or more preset behavior detection models, obtains the detection result image of the preset behavior detection model, and then uses the trained multimodal large language model to filter the obtained detection result image to obtain the final monitoring target image. By introducing a multimodal large language model as a false alarm filtering mechanism for secondary analysis, it can be easily adapted to different application scenarios, and combined with the required behavior detection model to achieve the detection of specified behaviors, so that it can flexibly adapt to diverse needs, while also effectively reducing false alarms, improving the accuracy and reliability of the overall solution detection, and maintaining a high accuracy even under complex backgrounds or lighting conditions, thereby improving the user experience.

[0056] Embodiment 2

[0057] Figure 2This is a schematic diagram of the structure of the target behavior detection device in video surveillance provided by the second embodiment of the present invention. The device can be implemented by hardware and / or software, and can generally be integrated into a computer device to execute the target behavior detection method in video surveillance provided by any embodiment of the present invention. Figure 2 As shown, the device comprises:

[0058] The video stream data acquisition module 21 is used to acquire the video stream data of the monitoring area;

[0059] An image detection module 22, configured to detect images in the video stream data based on one or more preset behavior detection models to obtain a detection result image of the preset behavior detection model;

[0060] The image filtering module 23 is used to filter the detection result image using the trained multimodal large language model to obtain a monitoring target image.

[0061] The technical solution provided by the embodiment of the present invention first obtains the video stream data of the monitoring area, then detects the images in the video stream data based on one or more preset behavior detection models, obtains the detection result image of the preset behavior detection model, and then uses the trained multimodal large language model to filter the obtained detection result image to obtain the final monitoring target image. By introducing a multimodal large language model as a false alarm filtering mechanism for secondary analysis, it can be easily adapted to different application scenarios, and combined with the required behavior detection model to achieve the detection of specified behaviors, so that it can flexibly adapt to diverse needs, while also effectively reducing false alarms, improving the accuracy and reliability of the overall solution detection, and maintaining a high accuracy even under complex backgrounds or lighting conditions, thereby improving the user experience.

[0062] On the basis of the above technical solution, optionally, the image filtering module 23 includes:

[0063] A preset prompt word acquisition unit, used to acquire a corresponding preset prompt word according to the behavior type corresponding to the detection result image;

[0064] The model question-answering unit is used to input the detection result image and the corresponding preset prompt word into the multimodal large language model for question-answering, and determine whether the detection result image is the monitoring target image according to the question-answering result.

[0065] On the basis of the above technical solution, optionally, the preset prompt words are multiple; and the model question-answering unit is specifically used for:

[0066] The detection result image is questioned and answered multiple times using the multimodal large language model through each of the preset prompt words, and whether the detection result image is the monitoring target image is determined based on the multiple question and answer results.

[0067] On the basis of the above technical solution, optionally, the target behavior detection device in video surveillance further includes:

[0068] A file writing module, used for writing the video stream data into a memory mapping file after acquiring the video stream data of the monitoring area;

[0069] The image detection module 22 is specifically used for:

[0070] The video frame data are read in parallel from the memory mapping file and provided to the plurality of preset behavior detection models for detection respectively.

[0071] On the basis of the above technical solution, optionally, the target behavior detection device in video surveillance further includes:

[0072] The detection model configuration module is used to determine the preset behavior detection model to be used based on user configuration before detecting the image in the video stream data based on one or more preset behavior detection models.

[0073] On the basis of the above technical solution, optionally, the target behavior detection device in video surveillance further includes:

[0074] The detection model updating module is used to add, remove or replace the preset behavior detection model based on user operations.

[0075] The target behavior detection device in video surveillance provided by the embodiment of the present invention can execute the target behavior detection method in video surveillance provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0076] It is worth noting that in the above-mentioned embodiment of the target behavior detection device in video surveillance, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0077] Embodiment 3

[0078] Figure 3 The schematic diagram of the structure of the computer device provided for the third embodiment of the present invention shows a block diagram of an exemplary computer device suitable for implementing the implementation mode of the present invention. Figure 3The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 3 As shown, the computer device includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 3 Taking a processor 31 as an example, the processor 31, memory 32, input device 33 and output device 34 in the computer device can be connected through a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0079] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the target behavior detection method in video surveillance in the embodiment of the present invention (for example, the video stream data acquisition module 21, the image detection module 22 and the image filtering module 23 in the target behavior detection device in video surveillance). The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 32, that is, realizes the above-mentioned target behavior detection method in video surveillance.

[0080] The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The input device 33 can be used to obtain video stream data of the monitoring area, and generate key signal input related to user settings and function control of the computer device, etc. The output device 34 can be used to output or display the obtained monitoring target image, etc.

[0082] Embodiment 4

[0083] Embodiment 4 of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a method for detecting target behavior in video surveillance when executed by a computer processor, the method comprising:

[0084] Get the video stream data of the monitoring area;

[0085] Detecting images in the video stream data based on one or more preset behavior detection models to obtain detection result images of the preset behavior detection models;

[0086] The detection result image is filtered using a trained multimodal large language model to obtain a monitoring target image.

[0087] The storage medium may be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system, which is connected to the computer system via a network (such as the Internet). The second computer system may provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that may be executed by one or more processors.

[0088] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided in an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the target behavior detection method in video surveillance provided in any embodiment of the present invention.

[0089] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0091] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0092] Embodiment 5

[0093] Embodiment 5 of the present invention also provides a computer program product, which includes a computer program (also referred to as code, instruction), which can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the target behavior detection method in video surveillance provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.

[0094] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for detecting target behavior in video surveillance, characterized in that: include: Get the video stream data of the monitoring area; Detecting images in the video stream data based on one or more preset behavior detection models to obtain detection result images of the preset behavior detection models; The detection result image is filtered using a trained multimodal large language model to obtain a monitoring target image.

2. The method for detecting target behavior in video surveillance according to claim 1, characterized in that: The method of filtering the detection result image using the trained multimodal large language model to obtain a monitoring target image includes: Obtaining a corresponding preset prompt word according to the behavior type corresponding to the detection result image; The detection result image and the corresponding preset prompt word are input into the multimodal large language model for question and answer, and it is determined whether the detection result image is the monitoring target image according to the question and answer result.

3. The method for detecting target behavior in video surveillance according to claim 2, characterized in that: There are multiple preset prompt words; inputting the detection result image and the corresponding preset prompt words into the multimodal large language model for question and answer, and determining whether the detection result image is the monitoring target image according to the question and answer result, including: The detection result image is questioned and answered multiple times using the multimodal large language model through each of the preset prompt words, and whether the detection result image is the monitoring target image is determined based on the multiple question and answer results.

4. The method for detecting target behavior in video surveillance according to claim 1, characterized in that: After obtaining the video stream data of the monitoring area, the method further includes: Writing the video stream data into a memory mapped file; The detecting the image in the video stream data based on one or more preset behavior detection models includes: The video frame data are read in parallel from the memory mapping file and provided to the plurality of preset behavior detection models for detection respectively.

5. The method for detecting target behavior in video surveillance according to claim 1, characterized in that: Before detecting the image in the video stream data based on one or more preset behavior detection models, the method further includes: The preset behavior detection model used is determined based on user configuration.

6. The method for detecting target behavior in video surveillance according to claim 1, characterized in that: The method further comprises: The preset behavior detection model is added, removed or replaced based on user operations.

7. A target behavior detection device in video surveillance, characterized in that: include: A video stream data acquisition module is used to acquire video stream data of the monitoring area; An image detection module, configured to detect images in the video stream data based on one or more preset behavior detection models to obtain a detection result image of the preset behavior detection model; The image filtering module is used to filter the detection result image using the trained multimodal large language model to obtain a monitoring target image.

8. A computer device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the target behavior detection method in video surveillance as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the target behavior detection method in video surveillance as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the target behavior detection method in video surveillance as described in any one of claims 1 to 6.

Images