Video behavior recognition method, system, device and program product

Abstract

The invention provides a video behavior recognition method, system and device and a program product, and the method comprises the steps: obtaining a to-be-recognized video frame group and a to-be-recognized text cue word, the to-be-recognized video frame group comprising one or more to-be-recognized video frames; inputting the to-be-recognized video frame group into an image processing model, partitioning each to-be-recognized video frame through the image processing model, calculating importance scores of a plurality of to-be-selected image blocks obtained through partitioning, selecting the to-be-recognized image blocks based on the importance scores of the plurality of to-be-selected image blocks, and extracting to-be-recognized visual features of the to-be-recognized image blocks; inputting the to-be-recognized text cue words into a text network, and extracting to-be-recognized text features of the to-be-recognized text cue words through the text network; and inputting the to-be-recognized visual features and the to-be-recognized text features into the large language model to obtain a behavior recognition result output by the large language model. According to the invention, the reasoning efficiency and the accuracy of video behavior recognition are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video behavior recognition method, system, device, and program product. Background Technology

[0002] Current road surveillance systems typically use detection models, such as the YOLO series (YOLO (You Only Look Once) is an end-to-end real-time object detection algorithm), to identify traffic violations. These models require specifying the categories to be detected before training, such as whether a driver is not wearing a seatbelt or whether someone is using a mobile phone while driving. Once trained, the model cannot recognize newly added categories; for example, a model trained for the aforementioned two types of behavior cannot identify pedestrians climbing over guardrails. Furthermore, some violations are difficult to determine from a single image, such as vehicles running red lights at intersections. These violations require analysis of multiple frames of continuous video, which traditional detection models struggle to handle.

[0003] If a detection model is not used, and instead a large multimodal model is used for recognition, problems such as slow recognition speed, high resource consumption, and low recognition rate will occur. Road violations typically occur with a low probability and occupy a small proportion of the image. To identify violations as accurately as possible, the model needs to work around the clock, increasing the number of inferences. Furthermore, each inference requires input from multiple frames, including all image patches corresponding to each frame, resulting in a large input and slow inference speed. At the same time, because large multimodal models perform inferences frequently, they consume more resources than smaller models.

[0004] There are several reasons for the low recognition rate of large multimodal models. First, existing multimodal models are usually not specifically trained for road monitoring scenarios. Second, because large multimodal models are slow to infer, in order to ensure real-time, 24 / 7 monitoring, the monitoring video is usually processed by skipping frames. For example, out of 100 video frames, only 10 frames are actually extracted at equal intervals. Excessive frame skipping intervals may result in the loss of key frames of illegal behavior, thus reducing the recognition rate. Summary of the Invention

[0005] To address the problems in the existing technology, the purpose of this application is to provide a video behavior recognition method, system, device, and program product, which reduces the amount of inference data for each inference of a multimodal large model by filtering video frame image blocks, thereby improving inference efficiency and the accuracy of video behavior recognition.

[0006] The first aspect of this application provides a video behavior recognition method, comprising the following steps: Obtain the video frame group to be identified and the text prompt word to be identified. The video frame group to be identified includes one or more video frames to be identified, and the text prompt word is used to describe the purpose of behavior recognition. The video frame group to be identified is input into the image processing model. The image processing model divides each video frame to be identified into blocks. The importance score of the multiple candidate image blocks obtained by the block division is calculated. The candidate image block is selected based on the importance scores of the multiple candidate image blocks, and the visual features to be identified of the candidate image block are extracted. Input the text prompt words to be identified into the text network, and extract the text features of the text prompt words to be identified through the text network; The visual features and text features to be identified are input into the large language model to obtain the behavior recognition results output by the large language model.

[0007] In some embodiments, before inputting the visual features to be identified and the text features to be identified into the large language model, the method further includes: obtaining a reference video frame group, which includes one or more reference video frames; inputting the reference video frame group into an image processing model, dividing each reference video frame into blocks through the image processing model, and extracting reference visual features from the multiple reference image blocks obtained by the block division. The visual features to be identified, the reference visual features, and the text features to be identified are all input into the large language model.

[0008] In some embodiments, the image processing model is configured to identify behavior categories in the video frames to be identified and to calculate importance scores relative to the identified behavior categories for multiple image blocks to be selected.

[0009] In some embodiments, the following training steps are also included: Acquire sample data, which includes multiple sample video frame groups and multiple sample text prompts. Each sample video frame group includes one or more sample video frames. The sample video frame group is input into the image processing model. The image processing model divides each sample video frame into blocks. The importance score of the multiple sample image blocks is calculated. The training image blocks are selected based on the importance scores of the multiple sample image blocks, and the training visual features of the training image blocks are extracted. Input the sample text prompts into the text network to obtain the training text features; A sample video frame group and a sample text prompt word are used as a training sample group. The training visual features and training text features corresponding to the training sample group are input into the large language model to obtain the behavior recognition result output by the large language model. The first loss function value is calculated based on the preset labels and behavior recognition results corresponding to the training sample group, and the image processing model and large language model are trained based on the first loss function value.

[0010] In some embodiments, the image processing model includes an image segmentation network, an image patch selector, an image patch filtering module, and a visual network. The image segmentation network is used to segment the input video frame into blocks, the image patch selector is used to calculate the importance score of the image patch obtained by the image segmentation network, the image patch filtering module is used to filter the image patch based on the importance score, and the visual network is used to extract visual features from the image patch obtained by the image patch filtering module. The training steps also include: The sample video frame group is input into the pre-trained visual reference model. The pre-trained visual reference model is configured to divide each sample video frame into blocks, identify the behavior category based on the sample video frame group, and use the gradient weighted class activation mapping method to calculate the importance score of each sample image block obtained by the block to the identified behavior category. The second loss function value is calculated based on the importance scores of multiple sample image patches output by the image patch selector and the importance scores of multiple sample image patches output by the pre-trained visual reference model. The image segmentation network and image block selector are trained based on the second loss function value.

[0011] In some embodiments, the training step involves obtaining sample data using the following steps: Obtain a group of sample video frames for one or more behavior categories; Input the category text of one or more behavior categories into a pre-trained language model, and obtain the sample text prompt words corresponding to each behavior category through the pre-trained language model.

[0012] In some embodiments, the sample data further includes one or more reference video frame groups, each reference video frame group including one or more reference video frames; In the training step, when inputting the training visual features and training text features corresponding to the training sample group into the large language model, it also includes inputting the reference visual features of a reference video frame group into the large language model.

[0013] A second aspect of this application also provides a video behavior recognition system for implementing the video behavior recognition method of the first aspect, the system comprising: The data acquisition module is used to acquire the video frame group to be identified and the text prompt words to be identified. The video frame group to be identified includes one or more video frames to be identified, and the text prompt words are used to describe the purpose of behavior recognition. The image processing module is used to input the group of video frames to be identified into the image processing model, divide each video frame to be identified into blocks through the image processing model, calculate the importance score of multiple candidate image blocks obtained by the block division, select the candidate image block based on the importance score of multiple candidate image blocks, and extract the candidate visual features of the candidate image block. The text processing module is used to input the text prompt words to be recognized into the text network and extract the text features of the text prompt words to be recognized through the text network; The behavior recognition module is used to input the visual features and text features to be recognized into the large language model and obtain the behavior recognition results output by the large language model.

[0014] A third aspect of this application also provides a video behavior recognition device, comprising: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the video behavior recognition method described above by executing the executable instructions.

[0015] A fourth aspect of this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video behavior recognition method described above.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0017] The video behavior recognition method, system, device, and program products of this application have the following beneficial effects: By employing this application, a group of video frames to be identified and text prompts are obtained. After obtaining the visual features and text features to be identified, a large language model is used to obtain the behavior recognition result. When extracting the visual features to be identified, each video frame to be identified is divided into blocks, and the image blocks with higher importance scores are selected from the candidate image blocks based on importance. The visual features corresponding to the image blocks to be identified are used as the visual features input into the large language model. On the one hand, this can reduce the amount of features to be analyzed and processed for each video frame, only processing the effective image blocks with high importance scores, without processing all image blocks, thus improving video processing efficiency and enabling the large language model to process more video frames in a certain period of time, reducing the time interval for extracting frames from video data. On the other hand, by selecting the image blocks to be identified with higher importance scores, it is beneficial to remove invalid interference information, allowing the large language model to focus more on image information that has a greater impact on the recognition result. At the same time, by processing more video frames, it is beneficial to improve the accuracy of video behavior recognition. This video behavior recognition method is based on text prompts to identify behaviors in videos. It is not limited to a specific behavior category and can recognize video behaviors of multiple different behavior categories without having to provide a separate detection and recognition model for each behavior category. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart of a video behavior recognition method according to an embodiment of this application; Figure 2 This is a schematic diagram of the architecture of a video behavior recognition model according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of behavior recognition of video frame groups based on text prompts according to an embodiment of this application; Figure 4 This is a schematic diagram of the model training process for behavior recognition of video frame groups based on text prompts according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the process of behavior recognition of a video frame group to be identified based on text prompts and reference video frame groups according to an embodiment of this application; Figure 6 This is a schematic diagram of the model training process for behavior recognition of a video frame group to be recognized based on text prompts and reference video frame groups according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a video behavior recognition system according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a video behavior recognition device according to an embodiment of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0021] Furthermore, the accompanying drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. Although the terms "first" or "second," etc., are used in this specification to denote certain features, these are only for indicating function and not as a limitation on the number or importance of specific features.

[0022] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0023] like Figure 1 As shown in the figure, this application provides a video behavior recognition method, including the following steps: S100: Obtain the video frame group to be identified and the text prompt word to be identified. The video frame group to be identified includes one or more video frames to be identified, and the text prompt word is used to describe the purpose of behavior recognition. In this embodiment, the video frame group to be identified may include a set number of video frames, such as one or more frames. This set number can be a preset fixed value or a dynamically adjusted value. For example, it can be set according to a specified behavior category. For some specified behavior categories (such as a driver not wearing a seatbelt), only a small number of video frames are needed for identification, so the set number can be set to a smaller value, or even 1. For some specified behavior categories (such as speeding, running a red light, etc.), more video frames are needed for identification, so the set number can be set to a larger value, such as 20, 25, 50, etc. Based on this, obtaining the video frame group to be identified in step S100 may include: determining the number of video frames in each video frame group based on the specified behavior category, and obtaining the video frame group to be identified corresponding to the number of video frames. Here, the specified behavior category can be the target behavior category included in the text prompt, or the specified behavior category identified by a pre-trained behavior category recognition model based on the video frame group to be identified, or the specified behavior category input by the user. In this embodiment, the text prompts are used to describe the purpose of behavior recognition. The purpose of behavior recognition includes two categories: the first category is to inquire whether a target behavior category has been identified in the video frame, with corresponding text prompts such as "Is there a pedestrian running a red light in the following video?" or "Please observe and analyze whether there are any illegal behaviors in the following video, such as a pedestrian running a red light?"; the second category is to inquire about the categories of abnormal behaviors present in the video frame, with corresponding text prompts such as "Is there any abnormal behavior in the following video? If so, please specify the category of abnormal behavior," or "Please identify what abnormal behaviors are present in the following video," etc. Corresponding to the first category of text prompts, the final behavior recognition result obtained by the video behavior recognition method is the result of whether the video frame to be recognized matches the target behavior category in the text prompts, such as "Yes" indicating a match and "No" indicating a mismatch; corresponding to the second category of text prompts, the final behavior recognition result obtained by the video behavior recognition method is the identified abnormal behavior category, such as outputting "Pedestrian running a red light detected" or "Driver not wearing a seatbelt detected," etc. S200: Input the group of video frames to be identified into the image processing model, divide each video frame to be identified into blocks through the image processing model, calculate the importance score of multiple candidate image blocks obtained by the block division, select the candidate image block based on the importance score of multiple candidate image blocks, and extract the candidate visual features of the candidate image block. In this embodiment, the image processing model can divide each video frame to be identified into blocks according to a preset segmentation principle. For example, each video frame to be identified can be divided into n×m image blocks, that is, each row can be divided into n image blocks and each column can be divided into m image blocks. n and m can be set as needed, such as setting the division into 16×16 image blocks, and using the image block as the smallest unit of importance analysis. In this embodiment, the image processing model is further configured to identify behavior categories in the video frame to be identified, and to calculate an importance score for multiple candidate image blocks relative to the identified behavior category. The importance score corresponds to the degree of association between the image block and the identified behavior category. The higher the importance score, the higher the degree of association between the image block and the identified behavior category. In other embodiments, the importance score can also be obtained in other ways, such as pre-specifying one or more behavior categories and calculating an importance score for multiple candidate image blocks relative to the specified behavior category, or using other image importance evaluation systems to score each image block. In this embodiment, the image blocks to be identified are selected based on the importance scores of multiple candidate image blocks. For example, the selection is based on a comparison of the importance scores of multiple candidate image blocks with a preset importance threshold. If the importance score of a candidate image block is greater than the preset importance threshold, then the candidate image block is selected as the image block to be identified. If the importance score of a candidate image block is less than or equal to the preset importance threshold, then the candidate image block is filtered out and not used as the image block for subsequent visual feature extraction. After the image blocks to be identified are selected, the visual features to be identified are extracted from the image blocks to be identified, which are used as the visual tokens to be identified input into the large language model. S300: Input the text prompt words to be recognized into the text network, and extract the text features of the text prompt words to be recognized through the text network; In this embodiment, the text network is implemented, for example, using a word segmenter; the extracted text features to be identified are used as the text tokens to be identified input into the large language model. S400: Input the visual features and text features to be identified into the large language model to obtain the behavior recognition results output by the large language model; This large language model is a multimodal large model and can adopt any deep learning-based large language model, including but not limited to Llama, Qwen, ChatGLM, etc. As mentioned above, the purpose of behavior recognition includes two categories, the corresponding text prompts also include two categories, and the corresponding behavior recognition results also have two possibilities. For text prompts that ask whether the target behavior category has been identified in the video frame, the behavior recognition result output by the large language model is the result of whether the target behavior category in the video frame to be identified matches the target behavior category in the text prompt, such as "yes" indicating that the two match and "no" indicating that the two do not match. For text prompts that ask about the abnormal behavior category present in the video frame, the behavior recognition result output by the large language model is the abnormal behavior category that has been identified, such as outputting "pedestrian running a red light detected" or "driver not wearing a seat belt detected".

[0024] By employing the video behavior recognition method of this application, after obtaining the video frame group to be recognized and the text prompt words in step S100, the visual features to be recognized and the text features to be recognized are obtained in steps S200 and S300 respectively, and the behavior recognition result is obtained by using a large language model in step S400. In step S200, each video frame to be recognized is divided into blocks, and the image blocks to be recognized with higher importance scores are selected from the image blocks to be selected based on importance. The visual features to be recognized corresponding to the image blocks to be recognized are used as the visual features input into the large language model. On the one hand, this can reduce the amount of features to be analyzed and processed for each video frame, only processing the effective image blocks with high importance scores, without processing all image blocks, thus improving video processing efficiency and enabling the large language model to process more video frames in a certain period of time, reducing the time interval for extracting frames from video data. On the other hand, by selecting the image blocks to be recognized with higher importance scores, it is beneficial to remove invalid interference information, allowing the large language model to focus more on the image information that has a greater impact on the recognition result. At the same time, by processing more video frames, it is beneficial to improve the accuracy of video behavior recognition. This video behavior recognition method is based on text prompts to identify behaviors in videos. It is not limited to a specific behavior category and can recognize video behaviors of multiple different behavior categories without having to provide a separate detection and recognition model for each behavior category.

[0025] The video behavior recognition method of this application can be applied to road monitoring scenarios to collect monitoring videos in real time and analyze whether illegal behaviors exist and the types of illegal behaviors in the monitoring videos. However, this application is not limited to this. The video behavior recognition method of this application can identify real-time video streams or pre-recorded video data. The application scenarios of this method are not limited to road monitoring scenarios, but can also be used in community monitoring scenarios, park monitoring scenarios, etc. The target behavior categories analyzed are not limited to illegal behavior categories, but can also be other abnormal behavior categories such as falling or fighting.

[0026] Figure 2 A schematic diagram of the architecture of a video behavior recognition model according to an embodiment of this application is shown. Figure 2 As shown, the video behavior recognition model includes an image processing model, a text network, and a large language model. Between the image processing model and the large language model, there is also a mapping layer (MLP) to align the visual features output by the image processing model with the input of the large language model in terms of dimensions.

[0027] Figure 3 This is a schematic diagram illustrating the process of behavior recognition for a group of video frames to be identified based on text prompts, according to an embodiment of this application. Figure 2 and Figure 3 As shown, the image processing model includes an image segmentation network, an image patch selector, an image patch filtering module, and a visual network. The image segmentation network segments the input video frames into blocks. The image patch selector calculates importance scores for the image patches obtained from the segmentation network. The image patch filtering module filters the image patches based on their importance scores. The visual network extracts visual features from the image patches filtered by the image patch filtering module. The text network is, for example, a word segmenter. The image segmentation network is implemented, for example, using a convolutional network; the image patch selector is implemented, for example, using a Transformer module; and the visual network is implemented, for example, using a Transformer module, but this application is not limited to these methods.

[0028] In this process, after the image block filtering module selects the image blocks to be identified for each video frame, the multiple image blocks to be identified are stitched together and then input into the video network. Here, stitching together multiple image blocks to be identified means that the feature vectors of multiple image blocks to be identified are stitched together in sequence to obtain a stitched image block vector to be identified, which is then input into the video network for image feature extraction.

[0029] Based on this, in step S200, the video frame group to be identified is input into the image processing model, including: dividing each video frame to be identified into blocks through an image block network; identifying the behavior category of the video frame group to be identified through an image block selector, and calculating the importance score of the multiple image blocks to be selected for the identified behavior category; selecting the image block to be identified based on the importance score of the multiple image blocks to be selected through an image block filtering module; and extracting the visual features to be identified from the image blocks to be identified through a visual network.

[0030] In some embodiments, in step S200, the image block selector performs behavior category identification on the video frame group to be identified, and may identify one or more behavior categories. When multiple behavior categories are identified, the image block selector calculates the importance score of each of the multiple candidate image blocks obtained by segmentation relative to each identified behavior category. For example, if the image block selector identifies two behavior categories in the video frame group to be identified, denoted as behavior category 1 and behavior category 2, then the image block selector calculates the importance score of each candidate image block relative to behavior category 1 and the importance score of each candidate image block relative to behavior category 2. For different behavior categories, the degree of association between different image blocks in the video frame and the behavior category may be different, so the importance score of an image block relative to behavior category 1 may be different from the importance score of the same image block relative to behavior category 2. For example, in the same video frame, the upper left corner may capture a pedestrian running a red light, and the lower right corner may capture a driver not wearing a seat belt. The image block selector may then identify two behavior categories for this video frame: pedestrian running a red light and driver not wearing a seat belt. Among the multiple image blocks divided in the video frame, the image block corresponding to the upper left corner of the original image has a higher importance score relative to pedestrians running red lights, while the image block corresponding to the lower right corner of the original image has a higher importance score relative to drivers not wearing seat belts.

[0031] In some embodiments, when the importance score output by the image patch selector includes the importance scores of the video group to be identified relative to multiple behavior categories, the image patch filtering module can filter the image patches to be identified corresponding to different behavior categories according to the output of the image patch selector, and input them into the visual network respectively. The visual network extracts the visual features corresponding to different behavior categories, and inputs the visual features and text features corresponding to different behavior categories together into the large language model. The large language model can simultaneously recognize multiple behavior categories. For example, the image patch filtering module selects a first part of the image patches to be identified corresponding to behavior category 1 according to the importance score of each image patch to be selected relative to behavior category 1, and selects a second part of the image patches to be identified corresponding to behavior category 2 according to the importance score of each image patch to be selected relative to behavior category 2. The first part of the image patches to be identified and the second part of the image patches to be identified may be completely different, or there may be some overlap between the image patches. The first and second image blocks to be identified are input into the visual network respectively, and the first visual features corresponding to the first image block and the second visual features corresponding to the second image block are extracted. The first visual features, the second visual features and the text features are simultaneously input into the large language model. The large language model can simultaneously recognize behavior category 1 and behavior category 2. The large language model can output the recognition results of the two behavior categories separately, or it can only output the actual recognized behavior category.

[0032] In this embodiment, when a group of video frames to be identified includes multiple video frames, the image blocks with higher importance scores are selected from the image blocks of each video frame. The image blocks from multiple video frames are then concatenated, and a visual token is extracted using a visual network. For example, if one video frame yields 5 image blocks and the next video frame yields 8, the features of the 8 image blocks are concatenated to the features of the 5 image blocks. Therefore, in each action recognition, important image information from multiple video frames can be combined for joint recognition, which helps improve the accuracy of recognizing the behavior category corresponding to continuous actions.

[0033] Figure 4 This is a schematic diagram illustrating the model training process for behavior recognition of video frame groups based on text prompts, according to an embodiment of this application. Figure 4 As shown, in this embodiment, the video behavior recognition method further includes training a video behavior recognition model using the following steps: Acquire sample data, which includes multiple sample video frame groups and multiple sample text prompts. Each sample video frame group includes one or more sample video frames. The sample video frame group is input into the image processing model. Each sample video frame is divided into blocks by the image segmentation network. The importance score of the multiple sample image blocks obtained by the image block selector is calculated. The training image block is selected by the image block filtering module based on the importance scores of the multiple sample image blocks. The training visual features of the training image block are extracted by the visual network. After the training visual features are dimensionally aligned by the mapping layer, they are used as the visual training token input to the large language model. The sample text prompts are input into the text network to obtain the training text features; these training text features are used as the training text tokens input into the large language model. A sample video frame group and a sample text prompt word are used as a training sample group. The training visual features and training text features corresponding to the training sample group are input into the large language model to obtain the behavior recognition result output by the large language model. The first loss function value is calculated based on the preset labels and behavior recognition results corresponding to the training sample group, and the image processing model and large language model are trained based on the first loss function value.

[0034] The image patch selector calculates importance scores for multiple sample image patches obtained from segmentation. This includes: the image patch selector identifies the behavior category of a sample video frame group, and calculates an importance score for each identified behavior category for each of the multiple sample image patches obtained from segmentation. The image patch selector may identify one or more behavior categories for a sample video frame group. When the image patch selector identifies multiple behavior categories in a sample video frame group, the image patch selector outputs the importance scores of each sample image patch for different behavior categories to the image patch filtering module. The image patch filtering module filters out training image patches for different behavior categories based on the importance scores corresponding to different behavior categories, inputs them into the visual network to obtain training image features corresponding to different behavior categories, and inputs the training image features of different behavior categories and text features together into the large language model, enabling the large language model to simultaneously identify multiple behavior categories.

[0035] During training, the behavior recognition result output by the large language model may include one or more output tokens and the probability of each output token. The cross-entropy loss is calculated based on the probability of each output token and a preset label to obtain the first loss function value. For example, when the sample text prompt is a first-class text prompt, the output token of the large language model may include a token corresponding to "yes," a token corresponding to "no," or a token corresponding to other text content. The preset label is a text token determined based on the actual matching between the sample video frame group and the sample text prompt. For example, if the behavior category in the sample text prompt actually exists in the sample video frame group, the preset label is the text token corresponding to "yes"; if the behavior category in the sample text prompt does not actually exist in the sample video frame group, the preset label is the text token corresponding to "no." When the sample text prompt is a second-class text prompt, the output token of the large language model may include tokens corresponding to different behavior categories. The preset label is the text token corresponding to the actual abnormal behavior category in the sample video frame group. For example, a word segmenter can be used to segment the labeled abnormal categories (such as illegal category segmentation) to obtain the actual value token, which serves as the preset label.

[0036] like Figure 4As shown, in this embodiment, the training steps of the video behavior recognition model further include training the image segmentation network and the image patch selector based on a pre-trained visual reference model. The pre-trained visual reference model is used as a baseline model, and the image segmentation network and the image patch selector simulate the function of the pre-trained visual reference model by calculating the loss between the output of the pre-trained visual reference model and the output of the image patch selector. The pre-trained visual reference model can be any pre-trained visual base model based on deep learning, including but not limited to CLIP, DINOv2, and VideoMAE.

[0037] Specifically, the training steps for the video behavior recognition model also include the following steps: The sample video frame group is input into the pre-trained visual reference model. The pre-trained visual reference model is configured to divide each sample video frame into blocks, identify the behavior category based on the sample video frame group, and use the gradient weighted class activation mapping (Grad-CAM) method to calculate the importance score of each sample image block obtained by the block to the identified behavior category. Here, the importance score obtained by the gradient weighted class activation mapping method is in the form of the baseline GradCAM result. The second loss function value is calculated based on the importance scores of multiple sample image patches output by the image patch selector and the importance scores of multiple sample image patches output by the pre-trained visual reference model. The image segmentation network and image block selector are trained based on the second loss function value.

[0038] Specifically, when calculating the second loss function value each time, the second loss function value is calculated based on the benchmark GradCAM result of the same sample video frame and the image patch features output by the image patch selector.

[0039] By training the image segmentation network and image patch selector through backpropagation based on the second loss function value, the image segmentation network and image patch selector are made to function as a pre-trained visual reference model. As mentioned above, the pre-trained visual reference model can segment the input video frames into blocks, identify behavior categories based on the video frames, and calculate the importance score of each segmented image patch to the identified behavior category. After iterative optimization training based on the second loss function value, the image segmentation network can segment the input video frames into blocks, while the image patch selector can identify behavior categories based on the input video frames and calculate the importance score of each segmented image patch to the identified behavior category.

[0040] When training the pre-trained visual reference model, it can be trained using sample video frames corresponding to various behavior categories and the importance scores of image patches within those frames corresponding to different behavior categories. This enables the pre-trained visual reference model to also recognize and calculate the corresponding importance scores for different behavior categories. In some embodiments, when the pre-trained visual reference model identifies behavior categories from a group of input video frames, it can identify not only one behavior category but also different behavior categories, each with a predicted score (corresponding to the presence of that behavior category in the video frame). The pre-trained visual reference model then calculates the importance score of each image patch corresponding to that behavior category for each behavior category. Correspondingly, the image selector identifies multiple different behavior categories and calculates the importance score of each image patch corresponding to each behavior category. When calculating the second loss function, for each identified behavior category, a second loss function value is calculated based on the output of the pre-trained visual reference model and the output of the image selector. Finally, the image patching network and the image patch selector can be jointly trained based on the second loss function values ​​for multiple behavior categories. For example, if a pre-trained visual reference model identifies behavior category 1 and behavior category 2 for a sample video frame group, the output of the pre-trained visual reference model includes the importance score of each sample image patch corresponding to behavior category 1 and the importance score of each sample image patch relative to behavior category 2. The output of the image patch selector includes the importance score of each sample image patch corresponding to behavior category 1 and the importance score of each sample image patch relative to behavior category 2. A second loss function value is calculated based on the importance score of the pre-trained visual reference model output relative to behavior category 1 and the importance score of the image patch selector output relative to behavior category 1.

[0041] The training of the first and second loss function values ​​can be performed simultaneously. That is, for each training sample group, the sample video frames of that group are input into the image processing model and simultaneously into the pre-trained visual reference model. The sample text prompts from that group are input into the word segmenter. After calculating the first and second loss function values, backpropagation is performed to optimize these values ​​and train the overall video behavior recognition model. Alternatively, the training of the first and second loss function values ​​can be performed separately. For example, the image processing module can be trained first using the second loss function value, and then the entire video behavior recognition model can be trained using the first loss function value; or, the entire video behavior recognition model can be trained first using the first loss function value, and then the image processing module can be trained using the second loss function value.

[0042] Pre-trained visual reference models based on the Gradient Weighted Class Activation Mapping (Grad-CAM) method are complex to deploy due to their large model data volume, complex computation process, and low data processing efficiency. Directly using this pre-trained visual reference model for image segmentation and image patch selection during inference can negatively impact the execution efficiency of the video behavior recognition process. This application employs the aforementioned training steps, using this pre-trained visual reference model as a benchmark to train the image segmentation network and image patch selector, resulting in an image processing model that is easier to deploy and more computationally efficient. Furthermore, it achieves image patch importance scores consistent with the output of the pre-trained visual reference model.

[0043] Optionally, the pre-trained visual reference model is constructed based on Transformer architectures such as VideoMAE / ViT combined with the gradient-weighted class activation mapping (Grad-CAM) method. The steps of the pre-trained visual reference model in predicting the importance of a given image patch using the Grad-weighted class activation mapping (Grad-CAM) method are as follows: a. First, the gradient of the score of the image patch based on the recognized behavior category c is calculated with respect to the feature map activation values ​​of the MLP (Multi-Layer Perceptron) in the last few blocks of the pre-trained visual reference model (e.g., the last 1 to 3 Transformer Blocks near the output).

[0044] in It is the score of behavior category c before softmax (the prediction score of behavior category c by the pre-trained visual reference model). In position The activation values ​​of the feature map with feature dimension d are obtained; after obtaining the gradient scores, average pooling is performed on all image patches to generate an importance weight for each feature of behavior category c. .

[0045] b. After obtaining the importance weights for each feature, calculate the importance score of each image patch to behavior category c using a linear combination method based on the following formula;

[0046] in, The feature dimension is Feature map activation, It is the importance weight of feature d under behavior category c. This is the final importance score of the image patch for behavior category c; c. Use min-max normalization to scale the importance scores of all image patches to the range of 0 to 1.

[0047] Optionally, when the pre-trained visual reference model identifies multiple behavior categories for a sample video frame group, the importance score of each sample image patch is calculated using steps a to c above for each behavior category.

[0048] In this embodiment, the pre-trained visual reference model obtains the gradient weighted class activation mapping (Grad-CAM) result through one backpropagation. After calculating the importance score of the sample image patch based on the image patch selector, the image patch feature output by the image patch selector is obtained. The dimension and form of the image patch feature are the same as those of Grad-CAM. The size of the image patch feature is adjusted to align with the size of Grad-CAM. Then, the mean squared error loss (MSE Loss) of the image patch feature and Grad-CAM is calculated to obtain the second loss function value.

[0049] The pre-trained visual reference model can identify behavior categories either based on a set of segmented image patches or on a group of unsegmented video frames from the original input. Similarly, the trained image patch selector can identify behavior categories based on either a set of segmented image patches or on a group of unsegmented video frames from the original input.

[0050] In this embodiment, the training steps of the video behavior recognition model involve obtaining sample data using the following steps: Obtain a group of sample video frames for one or more behavior categories; Input the category text of one or more behavior categories into a pre-trained language model, and obtain the sample text prompt words corresponding to each behavior category through the pre-trained language model.

[0051] For example, when applied to road monitoring scenarios, acquiring sample data includes: Collect training data from road surveillance video scenes and label the types of illegal behaviors; when labeling illegal behaviors in each video segment, the location of each type of illegal behavior can also be labeled. For the labeled illegal behavior categories, a pre-trained language model is used to expand and generate sample text prompts for model training. This pre-trained language model can also be pre-trained using a large language model, which is different from the large language model for behavior recognition mentioned above. This pre-trained language model can be any deep learning-based large language model, including but not limited to L1lama, Qwen, ChatGPT, etc.

[0052] For the collected video training data, video frames are obtained in sequence to form multiple sample video frame groups, each of which includes one or more sample video frames.

[0053] The following example uses the InternVL2.5 multimodal large-scale language model based on the PyTorch deep learning framework, the InternVL2.5 image processing model as the visual network, the VideoMAE pre-trained visual reference model, and the DeepSeek-R1 pre-trained language model to illustrate the training process of this video recognition model, taking pedestrians running red lights and drivers not wearing seat belts as specific violation categories. The parameters and model types described in the following steps are merely examples and are not intended to limit the scope of protection of this application.

[0054] Step 1: Use an online search engine to find 100 videos each of pedestrians running red lights and drivers not wearing seat belts, and label the violation category for each video; Step 2: Use the DeepSeek-R1 model to generate 100 text prompts for each of the two violation categories: pedestrians running red lights and drivers not wearing seat belts. For example, "Is there a pedestrian running a red light in the following video?" or "Please observe and analyze whether there is any illegal behavior in the following video, such as pedestrians running red lights." Step 3: Extract frames from the illegal video collected in Step 1 at a rate of 25 frames per second, and use the VideoMAE model for inference and backpropagation to obtain Grad-CAM and normalize it. Input the same sample video frame group into the image segmentation network and image block selector in sequence, and output the image block feature map. Use upsampling to align the image block feature map with the Grad-CAM size, and calculate the mean squared error loss between the image block feature map and Grad-CAM. Step 4: Set the importance threshold to 0.5, filter the image block regions corresponding to the image feature map pixels with importance scores less than or equal to the importance threshold, and send the filtered image blocks with importance scores greater than the importance threshold into the visual network and obtain the visual token through the mapping layer. Step 5: Use a word segmenter to segment the text prompt words obtained in Step 2 to obtain the text token; Step 6: Concatenate the visual token obtained in Step 4 and the text token obtained in Step 5, and feed them into the large language model based on InternVL2.5 to obtain the output token and the probability of each output token; when the output token corresponds to the identified illegal behavior category, use a word segmenter to segment the labeled illegal category to obtain the real value token, and calculate the cross-entropy loss between the output token and the real value token. Step 7: Use the backpropagation algorithm to train the video behavior recognition model with the goal of optimizing the cross-entropy loss and mean squared error loss. This includes training the image block network, image block selector, visual network, and large language model.

[0055] After obtaining the video behavior recognition model through the above steps, the model trained in step 7 can be deployed in a road monitoring scenario to collect monitoring videos in real time and analyze whether there are illegal behaviors in the videos. The behavior recognition method proposed in this application comprehensively utilizes multimodal large model technology, large language model technology, feature selection technology, etc., which can process monitoring videos in real time in video monitoring scenarios, identify illegal behaviors in the videos, significantly improve the detection rate of illegal behaviors and the running speed, and reduce resource consumption.

[0056] The above embodiments illustrate the reasoning and training processes of the recognition method using video behavior recognition based on text prompts as an example. Optionally, when performing video behavior recognition, not only text prompts can be used as prompt information, but reference videos can also be used as prompt information. For example, when inputting the visual token and text token to be recognized into the large language model, the visual token of the reference video can also be input simultaneously. The text prompt to be recognized can be a third-category text prompt, corresponding to the behavior recognition objective of identifying whether there is a target behavior category in the reference video. For example, the text prompt to be recognized could be "Does the video to be recognized contain an illegal behavior consistent with the content of the reference video?". Existing technologies often only allow for behavior recognition of videos based on text prompts, and cannot simultaneously perform behavior recognition of videos based on both text prompts and reference video prompts. This application, however, provides users with more recognition options by simultaneously providing text prompts and reference videos.

[0057] Figure 5This is a schematic diagram illustrating the process of behavior recognition based on text prompts and reference video frame groups according to an embodiment of this application. Optionally, before step S400: inputting the visual features and text features to be recognized into the large language model, the method further includes: obtaining a reference video frame group, which includes one or more reference video frames; inputting the reference video frame group into an image processing model, dividing each reference video frame into blocks by the image processing model, and extracting reference visual features from the multiple reference image blocks obtained by the block division. Specifically, after the image block division network of the image processing model divides the reference video frame group into multiple reference image blocks, the multiple reference image blocks are directly input into the visual network to extract reference visual features. Since the reference video frames themselves contain images corresponding to the target behavior category, there is no need to further filter image blocks based on the behavior category.

[0058] In some embodiments that simultaneously provide text prompts and reference video frame groups, in step S400, the visual features to be identified, the reference visual features, and the text features to be identified are input into the large language model. The behavior recognition result output by the large language model includes "yes" or "no," where "yes" corresponds to the video frame group to be identified containing the same target behavior category as the reference video frame group, and "no" corresponds to the video frame group to be identified not containing the target behavior category of the reference video frame.

[0059] Figure 6 This is a schematic diagram illustrating the model training process for behavior recognition based on text prompts and reference video frame groups according to an embodiment of this application. In this embodiment, the sample data further includes one or more reference video frame groups, each of which includes one or more reference video frames. The reference video frame groups used in the actual inference process can be the same as those used during training, but this application is not limited to this. In the actual inference process, one or more video frames containing the target behavior category that were not used in the training can also be used as reference video frame groups.

[0060] In the training steps of the video action recognition model, when inputting the training visual features and training text features corresponding to the training sample group into the large language model, it also includes inputting the reference visual features of a reference video frame group into the large language model. Other training steps of the video action recognition model in this embodiment are the same as those described above. Figure 4 The corresponding training steps are the same, and will not be repeated here.

[0061] In some embodiments, only one video behavior recognition model can be trained. This model can recognize behaviors and output recognition results not only based on the video frame group to be recognized and the text prompt word to be recognized, but also based on visual prompt information (visual prompt words) provided by the video frame group to be recognized, the text prompt word to be recognized, and the reference video frame group. In training the video behavior recognition model, the sample data includes two types of training data sets: one type includes a sample video frame group and a text prompt word; the other type includes a sample video frame group, a text prompt word, and a reference video frame group. Using these two types of training data sets to train the video behavior recognition model allows the large language model to recognize behaviors either based solely on the visual token and text token to be recognized, or based on the visual token, text token, and reference visual token during the inference phase.

[0062] like Figure 7 As shown in the figure, this application embodiment also provides a video behavior recognition system for implementing the above-described video behavior recognition method. The system includes: The data acquisition module M100 is used to acquire the video frame group to be identified and the text prompt words to be identified. The video frame group to be identified includes one or more video frames to be identified, and the text prompt words are used to describe the purpose of behavior recognition. The image processing module M200 is used to input the group of video frames to be identified into the image processing model, divide each video frame to be identified into blocks through the image processing model, calculate the importance score of multiple candidate image blocks obtained by the block division, select the candidate image block based on the importance score of multiple candidate image blocks, and extract the candidate visual features of the candidate image block. The text processing module M300 is used to input the text prompt words to be recognized into the text network and extract the text features of the text prompt words to be recognized through the text network; The behavior recognition module M400 is used to input the visual features and text features to be recognized into the large language model and obtain the behavior recognition results output by the large language model.

[0063] In the video behavior recognition system of this application, the functions of each module can be implemented using the specific implementation methods of the video behavior recognition method described above, which will not be elaborated here. By adopting this video behavior recognition system, the beneficial effects of the video recognition method described above can be obtained when each module implements its process, which will not be elaborated here.

[0064] This application also provides a video behavior recognition device, including a processor; a memory storing executable instructions of the processor; wherein the processor is configured to perform the steps of the video behavior recognition method by executing the executable instructions.

[0065] Those skilled in the art will understand that various aspects of this application can be implemented as systems, methods, or computer program products. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "platform."

[0066] The following reference Figure 8 To describe an electronic device 600 according to this embodiment of the present application. Figure 8 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0067] like Figure 8 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0068] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the video behavior recognition method section of this specification according to various exemplary embodiments of this application. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0069] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0070] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0071] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0072] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0073] In the video behavior recognition device, when the program in the memory is executed by the processor, it implements the steps of the video behavior recognition method. Therefore, the device can also achieve the technical effects of the video behavior recognition method.

[0074] An exemplary embodiment of this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the video behavior recognition method described above.

[0075] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. Exemplarily, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0076] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0077] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, C++, and Python. Program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0078] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this application, such as the steps of the video behavior recognition method described above.

[0079] When the computer program is executed by the processor, it implements the steps of the video behavior recognition method described above. Therefore, the computer program product can also achieve the technical effects of the video behavior recognition method described above.

[0080] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the scope of protection of this application.

Claims

1. A video behavior recognition method, characterized in that, Includes the following steps: Obtain a group of video frames to be identified and a text prompt word to be identified. The group of video frames to be identified includes one or more video frames to be identified, and the text prompt word is used to describe the purpose of behavior recognition. The group of video frames to be identified is input into an image processing model. The image processing model divides each video frame to be identified into blocks. An importance score is calculated for multiple candidate image blocks obtained from the block division. Based on the importance scores of the multiple candidate image blocks, a candidate image block is selected, and the visual features to be identified of the candidate image block are extracted. The text prompt words to be identified are input into a text network, and the text network is used to extract the text features to be identified from the text prompt words. The visual features to be identified and the text features to be identified are input into a large language model to obtain the behavior recognition result output by the large language model.

2. The video behavior recognition method according to claim 1, characterized in that, Before inputting the visual features to be identified and the text features to be identified into the large language model, the method further includes: obtaining a reference video frame group, the reference video frame group including one or more reference video frames; inputting the reference video frame group into the image processing model, dividing each reference video frame into blocks through the image processing model, and extracting reference visual features from the multiple reference image blocks obtained by the block division; The visual features to be identified, the reference visual features, and the text features to be identified are input together into the large language model.

3. The video behavior recognition method according to claim 2, characterized in that, The image processing model is configured to identify behavior categories for the video frames to be identified and to calculate importance scores for the plurality of image blocks to be selected relative to the identified behavior categories.

4. The video behavior recognition method according to claim 1, characterized in that, It also includes the following training steps: Acquire sample data, which includes multiple sample video frame groups and multiple sample text prompts, with each sample video frame group including one or more sample video frames; The sample video frame group is input into the image processing model. The image processing model divides each sample video frame into blocks. An importance score is calculated for the multiple sample image blocks obtained by the block division. Training image blocks are selected based on the importance scores of the multiple sample image blocks, and training visual features of the training image blocks are extracted. The sample text prompts are input into the text network to obtain the training text features; A sample video frame group and a sample text prompt word are used as a training sample group. The training visual features and training text features corresponding to the training sample group are input into the large language model to obtain the behavior recognition result output by the large language model. The first loss function value is calculated based on the preset labels and behavior recognition results corresponding to the training sample group, and the image processing model and the large language model are trained based on the first loss function value.

5. The video behavior recognition method according to claim 4, characterized in that, The image processing model includes an image segmentation network, an image block selector, an image block filtering module, and a visual network. The image segmentation network is used to segment the input video frame into blocks. The image block selector is used to calculate the importance score of the image blocks obtained by the image segmentation network. The image block filtering module is used to filter the image blocks based on the importance score. The visual network is used to extract visual features from the image blocks filtered by the image block filtering module. The training steps also include: The sample video frame group is input into a pre-trained visual reference model. The pre-trained visual reference model is configured to divide each sample video frame into blocks, identify behavior categories based on the sample video frame group, and use the gradient weighted class activation mapping method to calculate the importance score of each sample image block obtained by the block to the identified behavior category. The second loss function value is calculated based on the importance scores of the multiple sample image patches output by the image patch selector and the importance scores of the multiple sample image patches output by the pre-trained visual reference model; The image segmentation network and the image block selector are trained based on the second loss function value.

6. The video behavior recognition method according to claim 4, characterized in that, In the training step, sample data is obtained using the following steps: Obtain a group of sample video frames for one or more behavior categories; One or more behavior categories are input into a pre-trained language model, and sample text prompts corresponding to each behavior category are obtained through the pre-trained language model.

7. The video behavior recognition method according to claim 4, characterized in that, The sample data also includes one or more reference video frame groups, each of which includes one or more reference video frames. In the training step, when inputting the training visual features and training text features corresponding to the training sample group into the large language model, it also includes inputting the reference visual features of a reference video frame group into the large language model.

8. A video behavior recognition system, characterized in that, The system for implementing the video behavior recognition method according to any one of claims 1 to 7, the system comprising: The data acquisition module is used to acquire a group of video frames to be identified and a text prompt word to be identified. The group of video frames to be identified includes one or more video frames to be identified, and the text prompt word is used to describe the purpose of behavior recognition. The image processing module is used to input the group of video frames to be identified into the image processing model, divide each video frame to be identified into blocks through the image processing model, calculate the importance score of multiple candidate image blocks obtained by the block division, select the candidate image block based on the importance score of the multiple candidate image blocks, and extract the candidate visual features of the candidate image block. The text processing module is used to input the text prompt words to be identified into a text network and extract the text features of the text prompt words to be identified through the text network. The behavior recognition module is used to input the visual features to be recognized and the text features to be recognized into the large language model to obtain the behavior recognition results output by the large language model.

9. A video behavior recognition device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the video behavior recognition method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video behavior recognition method according to any one of claims 1 to 7.

Images