Setting behavior recognition method and device based on image data

Abstract

The application discloses a setting behavior recognition method and device based on image data. The method comprises the following steps: periodically sampling video frames collected by an image collection unit to obtain sampling video data; performing object recognition on the sampling video data, and taking video data containing a setting object and having a quantity of setting objects greater than or equal to a setting threshold as valid video data; selecting a corresponding visual processing model and a natural language processing type; generating a prompt sentence for multi-modal content analysis for the visual processing model; performing text description on the valid video data by using the visual processing model to generate an image caption for the valid video data; inputting the image caption into the corresponding processing model to determine whether a setting behavior exists in the valid video data and determine a probability of the setting behavior, and outputting the determined probability. The application reduces the misjudgment of specific behaviors in the video frames and saves a large amount of processing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to intelligent video monitoring technology based on large model technology, and in particular to a set behavior recognition method and device based on image data. BACKGROUND

[0002] Currently, public places install cameras to collect corresponding video streams, and use human key point detection, behavior analysis and the like in the field of computer vision to detect public security events. Although such methods can achieve certain detection effects, their detection effects in complex scenes are not good. Specifically, due to the diversity of human actions, a high false positive rate is caused. In addition, if the camera is blocked, the possibility of misjudgment is also increased. In addition, although the camera can capture the behavior of the public security event well, for those uncommon or irregular action combinations, the recognition accuracy is greatly reduced. Therefore, the video detection technology of the existing camera does not perform well in complex scenes, resulting in a low accuracy of video recognition. SUMMARY

[0003] The present application provides a set behavior recognition method and device based on image data to at least solve the above technical problems in the prior art.

[0004] According to a first aspect of the present application, a set behavior recognition method based on image data is provided, comprising:

[0005] Periodically sampling the video frames collected by the image collection unit to obtain sampled video data;

[0006] Performing object recognition on the sampled video data, and regarding the video data containing a set object and having a number of set objects greater than or equal to a set threshold in the sampled video data as valid video data;

[0007] Selecting a corresponding visual processing model and a natural language processing type; and generating a prompt sentence for multi-modal content analysis for the visual processing model;

[0008] Generating an image caption for the valid video data by using the visual processing model to perform text description on the valid video data;

[0009] Inputting the image caption into the corresponding processing model to determine whether a set behavior exists in the valid video data and determine a probability of existence of the set behavior, and outputting the determined probability.

[0010] In some executable embodiments, the determination of the probability of existence of the set behavior comprises:

[0011] generating image captions for the effective video data frame by frame, triggering the natural language processing to perform multi-modal content analysis on the image captions of each frame, determining whether the frame image exists a set behavior, and a first probability of the set behavior;

[0012] and inputting the image captions and the video data frame into a multi-modal large model, determining whether the frame image exists a set behavior and a second probability of the set behavior through the multi-modal large model;

[0013] taking an average of the first probability and the second probability as a probability of the set behavior existing in the effective video data.

[0014] In some executable embodiments, the object recognition on the sampled video data includes:

[0015] For the sampled video data, an object in a video frame is detected by using a yolo in a target detection algorithm, the number of objects in the video frame is counted, and a video frame with a number of objects less than a set threshold is removed from the sampled video data as effective video data.

[0016] In some executable embodiments, the detection of the object in the video frame by using the yolo in the target detection algorithm includes:

[0017] extracting image information features of each video frame of the sampled video data through a backbone network, and performing convolution processing on the image information features to obtain local feature information of each video frame;

[0018] performing normalization processing on the local feature information of each video frame to reduce the covariant offset of the local feature information;

[0019] inputting the normalized local feature information into an activation function to determine the object in the video frame.

[0020] In some executable embodiments, the object is a human, and the set behavior includes an excessive contact behavior of a limb.

[0021] According to a second aspect of the present application, a set behavior recognition device based on image data is provided, including:

[0022] a sampling unit configured to periodically sample video frames collected by an image collection unit to obtain sampled video data;

[0023] an identification unit configured to perform object recognition on the sampled video data, and take video data containing a set object and having a number of set objects greater than or equal to a set threshold in the sampled video data as effective video data.

[0024] a selection unit configured to select a corresponding visual processing model and a natural language processing type;

[0025] a first generation unit configured to generate a prompt sentence for multi-modal content analysis for the visual processing model;

[0026] a second generation unit configured to generate an image caption for valid video data by performing text description on the valid video data using the visual processing model;

[0027] a determination unit configured to input the image caption into a corresponding processing model to determine whether a set behavior exists in the valid video data and determine a probability of existence of the set behavior, and output the determined probability.

[0028] In some executable embodiments, the determination unit is further configured to:

[0029] generate an image caption for each frame of the valid video data, trigger the natural language processing type to perform multi-modal content analysis on the image caption of each frame, determine whether a set behavior exists in the frame image, and determine a first probability of the set behavior;

[0030] input the image caption and the frame of the video data into a multi-modal large model, and determine whether a set behavior exists in the frame image and a second probability of the set behavior by using the multi-modal large model;

[0031] average the first probability and the second probability as the probability of existence of the set behavior for the valid video data.

[0032] In some executable embodiments, the recognition unit is further configured to:

[0033] For the sampled video data, use a yolo in a target detection algorithm to detect objects in the video frames, count the number of objects in the video frames, and remove video frames with a number of objects less than a set threshold from the sampled video data as valid video data.

[0034] In some executable embodiments, the recognition unit is further configured to:

[0035] extract image information features of each video frame of the sampled video data by using a backbone network, and perform convolution processing on the image information features to obtain local feature information of each video frame;

[0036] perform normalization processing on the local feature information of each video frame to reduce the covariant offset of the local feature information;

[0037] input the normalized local feature information into an activation function to determine the objects in the video frames.

[0038] In some executable embodiments, the subject is a human; and the set behavior includes an overcontacting behavior of a limb.

[0039] According to a third aspect of the present application, an electronic device is provided, comprising:

[0040] at least one processor; and

[0041] a memory connected with the at least one processor in communication; wherein,

[0042] the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the set behavior recognition method based on image data.

[0043] According to a fourth aspect of the present application, a non-transitory computer readable storage medium is provided, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the set behavior recognition method based on image data.

[0044] The set behavior recognition method based on image data and the device, the electronic device, and the storage medium of the present application reduce the dependence on on-site supervisors by interval sampling of video frames and filtering of invalid frames to reduce unnecessary calculations, thereby saving a large amount of processing resources.

[0045] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and other objects, features and advantages of the example embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0047] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0048] Figure 1 A flowchart of a set behavior recognition method based on image data according to an embodiment of the present application is shown;

[0049] Figure 2A composition structure schematic diagram of a behavior recognition device based on image data according to an embodiment of the present application is shown.

[0050] Figure 3 A composition structure schematic diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] To make the objectives, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] Figure 1 A flow schematic diagram of a behavior recognition method based on image data according to an embodiment of the present application is shown. Figure 1 As shown, the behavior recognition method based on image data according to an embodiment of the present application includes the following processing steps:

[0053] Step 101: periodically sampling the video frames collected by the image collection unit to obtain sampled video data.

[0054] In the embodiments of the present application, the image collection unit can be a camera, such as a video monitoring camera, or a depth camera such as a binocular camera, which can acquire images or videos of the environment to monitor the set area.

[0055] In the embodiments of the present application, the main purpose is to monitor some public security events, mainly to monitor some fighting events, so that public security events can be discovered and stopped in time to ensure the stability of social security and the safety of people's life.

[0056] In the embodiments of the present application, if image recognition is performed on all video frames, it will result in a very large amount of data processing, leading to slow data processing speed. Since social security events are small probability events and generally difficult to occur, only some remote or hidden areas may have such events occasionally. Moreover, even if a public security event occurs, it will be continuous, and the embodiments of the present application are aimed at the video frames collected by the camera, which uses a periodic sampling method to acquire video frames in a period of time, and through image analysis of the periodically sampled video frames, it can be determined whether a public security event has occurred or not.

[0057] In the embodiments of the present application, the sampling period can be 30 seconds, or 1 minute, 2 minutes, etc. For example, every two minutes, 30 seconds of video frames are sampled to analyze the sampled video frames to determine whether a corresponding public security event has occurred or not.

[0058] Taking a fight as an example of a public security event, generally speaking, the fight is a continuous action, and the duration of the fight is short for tens of seconds, and long for several minutes or even longer. Even if the duration is only ten seconds, the corresponding video frames also span dozens or even hundreds of frames in the video stream. If the fight detection is performed on each frame in the video stream, a huge amount of computing resources will be consumed. Moreover, it is unnecessary to a certain extent because the fight spans many frames in the video stream. In a fixed frame interval sampling manner, a part of the video frames are selected from the video stream for fight detection, and the detection result can represent the result of fight detection on all video frames in the video stream to a certain extent. In view of the above analysis, the corresponding video frames are extracted from the video stream in a fixed frame interval sampling manner, and then the fight is detected in combination with subsequent related algorithms.

[0059] In step 102, object recognition is performed on the sampled video data, and video data containing a set object and having a number of set objects greater than or equal to a set threshold is taken as valid video data.

[0060] In the embodiment of the application, the object to be recognized in the video image frame is a person.

[0061] For the sampled video data, the objects in the video frame are detected by using yolo in the target detection algorithm, the number of objects in the video frame is counted, and the video frame with the number of objects less than the set threshold is removed from the sampled video data as valid video data. The yolo network structure can use yolov8 to recognize the objects and their number in the video image frame. Specifically, the image information features of each video frame of the sampled video data are extracted by a backbone network, the image information features are convoluted to obtain the local feature information of each video frame, the local feature information of each video frame is normalized to reduce the covariance shift of the local feature information, and the normalized local feature information is input into an activation function to determine the objects in the video frame.

[0062] Taking fighting as an example, the participants of the behavior are people, and at least two people are involved. Therefore, the prerequisite for the existence of fighting behavior in a video image frame is that there are at least two people in the video image frame. If the number of people in a video frame is less than 2, then there is generally no fighting behavior in the video frame. In view of this, for any video frame, first, the people in the video frame are detected by using yolov8 in the target detection algorithm, and then the number of people people_num is counted. If the value of people_num is less than 2, it means that there is generally no fighting behavior in the video frame, and the video frame is defined as an invalid video frame, and no subsequent judgment is needed. The video frame is discarded and is not input into the visual large model and the multi-modal large model for multi-modal analysis, which can save algorithm resources and reduce unnecessary algorithm time consumption. If the value of people_num is greater than or equal to 2, it means that there may be fighting behavior in the video frame, and the video frame is defined as a valid video frame. In this case, the video frame is input into the visual large model and the multi-modal large model for subsequent analysis and judgment.

[0063] In step 103, a corresponding visual processing model and a natural language processing model are selected, and a prompt sentence for multi-modal content analysis is generated for the visual processing model.

[0064] In the embodiments of the present application, the visual processing large model can select at least one of the following three types:

[0065] Textually Prompted Models: including contrastive, generative, hybrid and conversational, etc. As an example, CLIP (Contrastive Language-Image Pre-training) is a typical model thereof, which is a multi-modal model based on contrastive learning. It is pre-trained through a large-scale image-text pair dataset to learn the matching relationship between images and texts. It encodes images and texts into the same vector space, so that similar images and texts are closer in space, thereby realizing cross-modal semantic understanding and retrieval.

[0066] Visually Prompted Models: These include models like SAM (Segment Anything Model) and SegGPT, which perform image segmentation and other tasks through visual prompts. SAM segments specific objects from images through user prompts such as clicks, boxes, masks, and text. SAM has the ability to generalize to zero-shot, meaning it can segment visual objects in images even if those objects were not present in the training set. The SAM model consists of an image encoder, prompt encoder, and mask decoder, allowing it to quickly (about 50 milliseconds) predict masks based on prompts in a browser.

[0067] Heterogeneous Modalities-Based Models: These include models like ImageBind and Valley, which aim to handle multiple types of input data and enable cross-modal learning. As an example, Stable Diffusion is a typical model that generates high-quality images from text descriptions through text embedding by a text encoder, latent space sampling, step-by-step denoising generation by a U-Net network, and image decoding by a VAE decoder.

[0068] Before using the visual large model to describe the obtained video frames, a prompt needs to be written for the visual large model to focus on the behavior of the characters in the image. The content of the prompt for the visual large model can be written as: "analyze and describe the specific actions of the characters in the image, and try to explain the possible purposes or emotional states behind their behavior."

[0069] Step 104, using the visual processing model to generate a text description of the valid video data, generating an image caption for the valid video data.

[0070] Then, use the visual large model to describe the obtained video frames and generate an "image caption", which is the description of the image by the visual large model. After obtaining the image caption generated by the visual large model, input the image caption into the natural language processing large model, and use the powerful natural language processing capability of the natural language processing large model to analyze the above image caption. In order to enable the natural language processing large model to better analyze whether there is a fight behavior from the image caption, a prompt needs to be written in advance for the natural language processing large model, and the content of the prompt is as follows:

[0071] "You are a professional image analysis expert, your task is to judge whether there is a fight behavior in the image according to the provided image caption, and give the probability of the existence of such behavior. Please read the following image caption carefully and answer the following questions:

[0072] 1. Image caption: [Insert image caption here]

[0073] 2. Question:

[0074] Please provide the probability P1 (0-100%) that there is a fight in the image.

[0075] Please answer the question in percentage.

[0076] Step 105, input the image caption into the corresponding processing model to determine whether the set behavior exists in the effective video data and determine the probability of the existence of the set behavior, and output the determined probability.

[0077] In the embodiments of the present application, image captions are generated frame by frame for effective video data, triggering the natural language processing type to perform multi-modal content analysis on the image captions of each frame to determine the probability of the existence of the set behavior in the frame image; the average value of the probabilities of all frame images is taken as the probability of the existence of the set behavior in the effective video data.

[0078] For an effective video frame, it is determined that the number of people in the frame is not less than 2, and the video frame is selected to be input into a visual large model, and a corresponding image caption is generated. The image caption is input into a multi-modal large model together with the image caption, and the multi-modal large model is used to determine whether the set behavior exists in the video frame. The set behavior includes excessive contact behavior of limbs, such as fighting behavior. The embodiments of the present application output the probability P2 of the existence of fighting behavior through the multi-modal large model. In order to make the focus of the multi-modal large model more targeted and output the desired results, a prompt also needs to be written for the multi-modal large model. The content of the prompt of the multi-modal large model is as follows:

[0079] "You are a professional image and text analysis expert, and your task is to extract relevant information from the provided image and its caption. Please carefully review the following image and its caption, and answer the following questions:

[0080] 1. Image: [Insert video frame here]

[0081] 2. Image caption: [Insert image caption corresponding to the video frame here]

[0082] 3. Question:

[0083] - Please provide the probability P2 (0-100%) that there is a fight in the image.

[0084] Please answer the question in percentage.

[0085] For any video frame, the above corresponding probability P1 and probability P2 are weighted and summed, denoted as probability P. For example, the weighted arithmetic mean of probability P1 and probability P2 is taken as the basis for determining whether the set behavior occurs in the video frame, that is, the probability of the existence of fighting behavior in the video frame. At this time, the weights of the two probabilities are both 1. If the value of probability P is greater than a set threshold such as 60%, it is determined that there is fighting behavior in the video frame. Otherwise, it is determined that there is no fighting behavior in the video frame. The probability P in the video frame can also be the geometric mean of probability P1 and probability P2, as the probability of the existence of fighting behavior in the video frame.

[0086] Figure 2 The composition structure of the set behavior recognition device based on image data of the embodiment of the application is shown in the schematic diagram, as shown in the figure, Figure 2 The set behavior recognition device based on image data of the embodiment of the application includes:

[0087] The sampling unit 20 is configured to periodically sample the video frames collected by the image collection unit to obtain sampled video data.

[0088] The recognition unit 21 is configured to perform object recognition on the sampled video data, and take the video data containing the set object and having a number of set objects greater than or equal to a set threshold as valid video data.

[0089] The selection unit 22 is configured to select a corresponding visual processing model and a natural language processing type.

[0090] The first generation unit 23 is configured to generate a prompt sentence for multi-modal content analysis for the visual processing model.

[0091] The second generation unit 24 is configured to perform text description on the valid video data by using the visual processing model, and generate an image caption for the valid video data.

[0092] The determination unit 25 is configured to input the image caption into the natural language processing type, trigger the natural language processing type to perform multi-modal content analysis on the image caption, determine whether the set behavior exists in the valid video data, and determine the probability of the existence of the set behavior, and output the determined probability.

[0093] In some executable embodiments, the determination unit 25 is further configured to:

[0094] generate an image caption for the valid video data frame by frame, trigger the natural language processing type to perform multi-modal content analysis on the image caption of each frame, determine whether the set behavior exists in the frame image, and determine the first probability of the set behavior;

[0095] and inputting the image caption and the video data frame into a multi-modal large model, determining, by the multi-modal large model, whether the frame image exists the set behavior and a second probability of the set behavior;

[0096] averaging the first probability and the second probability as a probability of the set behavior of the effective video data.

[0097] In some executable embodiments, the identification unit 21 is further configured to:

[0098] For the sampled video data, objects in the video frame are detected by using yolo in the target detection algorithm, the number of objects in the video frame is counted, and video frames with the number of objects less than a set threshold are removed from the sampled video data as effective video data.

[0099] In some executable embodiments, the identification unit 21 is further configured to:

[0100] Image information features of each video frame of the sampled video data are extracted by a backbone network, and the image information features are convoluted to obtain local feature information of each video frame.

[0101] The local feature information of each video frame is normalized to reduce the covariant offset of the local feature information.

[0102] The normalized local feature information is input into an activation function to determine the object in the video frame.

[0103] In some executable embodiments, the object is a human, and the set behavior includes excessive contact behavior of the limbs.

[0104] In exemplary embodiments, the foregoing units and the like can be implemented by one or more central processing units (CPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, micro controllers (MCUs), microprocessors (Microprocessors), or other electronic elements.

[0105] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0106] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0107] Figure 3 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. Figure 3 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0108] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, data transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0109] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the setting behavior recognition method based on image data. For example, in some embodiments, the setting behavior recognition method based on image data can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the setting behavior recognition method based on image data described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the setting behavior recognition method based on image data by any other suitable means, such as by means of firmware.

[0110] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0111] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0112] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include but are not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0115] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0116] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0117] In addition, the terms "first", "second", are only used for descriptive purposes and should not be construed as indicating or implying relative importance or an implied indication of the number of technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0118] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A setting behavior recognition method based on image data, characterized by, The method comprises: periodically sampling video frames collected by an image collection unit to obtain sampled video data; performing object recognition on the sampled video data, and taking video data containing a set object and having a number of set objects greater than or equal to a set threshold as valid video data; selecting a corresponding visual processing model and a natural language processing model, and generating a prompt sentence for multi-modal content analysis of the visual processing model; performing text description on the valid video data by using the visual processing model to generate an image caption for the valid video data; inputting the image caption into the corresponding processing model to determine whether a set behavior exists in the valid video data and determine a probability of existence of the set behavior, and outputting the determined probability; the determination of the probability of existence of the set behavior comprises: generating an image caption for each frame of the valid video data, triggering the natural language processing model to perform multi-modal content analysis on the image caption of each frame to determine whether the set behavior exists in the frame image and a first probability of the set behavior; and inputting the image caption and the video data frame into a multi-modal large model to determine whether the set behavior exists in the frame image and a second probability of the set behavior; taking an average of the first probability and the second probability as the probability of existence of the set behavior of the valid video data.

2. The setting behavior recognition method based on image data according to claim 1, characterized in that, the object recognition on the sampled video data comprises: for the sampled video data, detecting objects in the video frames by using yolo in a target detection algorithm, counting the number of objects in the video frames, and removing video frames having a number of objects less than a set threshold from the sampled video data as valid video data.

3. The setting behavior recognition method based on image data according to claim 2, characterized in that, the detection of objects in the video frames by using yolo in the target detection algorithm comprises: extracting image information features of each video frame of the sampled video data by using a backbone network, and performing convolution processing on the image information features to obtain local feature information of each video frame; performing normalization processing on the local feature information of each video frame to reduce the covariant offset of the local feature information; inputting the normalized local feature information into an activation function to determine the objects in the video frames.

4. The setting behavior recognition method based on image data according to any one of claims 1 to 3, characterized in that, The object is a person, and the set behavior includes excessive contact behavior of a limb.

5. An apparatus for setting a behavior recognition based on image data, characterized by, The device comprises: a sampling unit configured to periodically sample video frames collected by an image collection unit to obtain sampled video data; an identification unit configured to perform object recognition on the sampled video data, and take video data containing a set object and having a number of set objects greater than or equal to a set threshold as valid video data; a selection unit configured to select a corresponding visual processing model and a natural language processing model; a first generation unit configured to generate a prompt sentence for multi-modal content analysis of the visual processing model; a second generation unit configured to perform text description on the valid video data by using the visual processing model to generate an image caption for the valid video data; The determination unit is configured to input the image caption into a corresponding processing model to determine whether the set behavior exists in the effective video data and determine a probability of existence of the set behavior, and output the determined probability. The determination unit is further configured to: generate an image caption for each frame of the effective video data, trigger the natural language processing model to perform multi-modal content analysis on the image caption of each frame, determine whether the set behavior exists in the frame image, and determine a first probability of the set behavior; input the image caption and the frame of the video data into a multi-modal large model, and determine whether the set behavior exists in the frame image and a second probability of the set behavior through the multi-modal large model; and average the first probability and the second probability as the probability of existence of the set behavior in the effective video data.

6. The behavior recognition apparatus based on image data according to claim 5, wherein, The recognition unit is further configured to: for the sampled video data, detect objects in the video frames by using a yolo in a target detection algorithm, count the number of objects in the video frames, and remove video frames with a number of objects less than a set threshold from the sampled video data as effective video data.

7. The setting behavior recognition apparatus based on image data according to claim 6, characterized by, The recognition unit is further configured to: extract image information features of each video frame of the sampled video data through a backbone network, and perform convolution processing on the image information features to obtain local feature information of each video frame; perform normalization processing on the local feature information of each video frame to reduce the covariant offset of the local feature information; input the normalized local feature information into an activation function to determine the objects in the video frames.

8. The setting behavior recognition apparatus based on image data according to any one of claims 5 to 7, characterized by, The objects are people, and the set behavior includes excessive contact behavior of limbs.

Images