Behavior detection method, system, device and program product

Abstract

The invention provides a behavior detection method, system and device and a program product. The method comprises the following steps: acquiring to-be-detected video data and a current behavior description text; inputting the current frame of the video data and the current behavior description text into a feature extraction model, and obtaining a current fusion feature output by the feature extraction model; the current fusion feature is input into a gating network, the gating network is configured to generate a current confidence coefficient based on the current fusion feature, a historical fusion feature corresponding to a historical frame of the video data and a historical confidence coefficient corresponding to the historical fusion feature, and whether detection continues or not is judged according to the current confidence coefficient; when the gating network judgment continues to detect, the current fusion features are input into the large language model, a matching result output by the large language model is obtained, and the large language model is configured to judge whether the current frame of the video data is matched with the behavior description text or not based on the current fusion features. According to the invention, the efficiency and accuracy of behavior detection are effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a behavior detection method, system, device, and program product. Background Technology

[0002] Personnel behavior detection is a method of personnel monitoring that captures and analyzes behavioral patterns of people in specific scenarios to identify abnormal behaviors that deviate from the normal behavioral baseline. It is commonly used in security monitoring, industrial production safety, and network behavior supervision. Existing solutions for detecting abnormal personnel behavior events typically employ techniques for identifying typical abnormal behaviors such as fighting or falling to the ground. These techniques usually involve: first, using deep learning CNN (Convolutional Neural Network) algorithms to detect the target location of people in an image; then, using a keypoint detection classification model to identify whether an event needs to be detected. However, these processes are overly complex, and deviations in any step can lead to abnormal results. Furthermore, judging single-frame images or single targets is unlikely to yield satisfactory results for large-scale interactive events like fighting.

[0003] Based on the Transformer model and the development of multimodal models based on it, methods for detecting human behavior have emerged that combine real-time video and multimodal models. Compared to behavior detection schemes based on CNN algorithms, these methods can effectively improve the detection of abnormal behavior events. Current conventional schemes combining real-time video and multimodal models typically use image encoding models to obtain image features, which are then input into a large language model for analysis. However, large language models require deep parsing of each frame, resulting in high resource consumption and demanding hardware requirements. In practical video applications, due to hardware performance limitations, existing methods struggle to perform continuous analysis of continuous video, significantly impacting the accuracy of behavior detection. Summary of the Invention

[0004] In view of the problems in the prior art, the purpose of this application is to provide a behavior detection method, system, device and program product, which effectively improves the efficiency and accuracy of behavior detection.

[0005] The first aspect of this application provides a behavior detection method, comprising the following steps: Acquire the video data to be detected and the current behavior description text; Input the current frame of the video data and the current behavior description text into the feature extraction model to obtain the current fused features output by the feature extraction model; The current fusion feature is input into the gating network. The gating network is configured to generate a current confidence score based on the current fusion feature, the historical fusion feature corresponding to the historical frame of the video data, and the historical confidence score corresponding to the historical fusion feature. The current confidence score is used to determine whether to continue detection. The current confidence score represents the confidence score of the current frame corresponding to the current fusion feature matching the current action text. When the gating network decides to continue detection, it inputs the current fused features into the large language model and obtains the matching result output by the large language model. The large language model is configured to determine whether the current frame of the video data matches the behavior description text based on the current fused features.

[0006] In some embodiments, the feature extraction model includes an image encoder, a text encoder, a cross-modal attention layer, and a feature fusion layer; The image encoder is configured to convert the input image into a visual feature vector; the text encoder is configured to convert the current behavior description text into a text feature vector. The cross-modal attention configuration calculates attention weights based on visual feature vectors and text feature vectors, generates current image features based on visual feature vectors and attention weights, and generates current text features based on text feature vectors and attention weights. The feature fusion layer is configured to obtain the current fused features based on the current image features and the current text features.

[0007] In some embodiments, inputting the current fused features into a large language model includes the following steps: The current fused features are input into the feature mapping layer, which is configured to map the current fused features to the feature space of the large language model. The current fused features, after being processed by the feature mapping layer, are input into the large language model.

[0008] In some embodiments, the feature mapping layer includes at least one downsampling layer, a linear layer, a layer normalization layer, a random deactivation layer, and at least one upsampling layer, which are connected in series.

[0009] In some embodiments, the following training steps are also included: A large language model is trained based on a first training dataset, which includes multiple first sample fusion features. Each first sample fusion feature is obtained based on a first sample image and a first sample behavior description text. The parameters of the large language model are frozen, and the feature extraction model and gating network are trained based on the second training dataset. The second training dataset includes multiple second sample data pairs, each of which includes a second sample image and a second sample behavior description text.

[0010] In some embodiments, training a feature extraction model and a gating network based on a second training dataset includes the following steps: The second sample data is input to the feature extraction model to obtain the second sample fusion features output by the feature extraction model. The second sample behavior description text includes causal relationship labels used to describe the cause of the behavior and the behavior action. The fused features of the second sample are input into the gating network to obtain the judgment result output by the gating network; When the gating network determines that detection should continue, it inputs the fused features of the second sample into the large language model to obtain the matching result output by the large language model. The loss function value is calculated based on the matching results output by the large language model and the matching labels of the second sample data pair. The feature extraction model and gating network are then trained based on the loss function value.

[0011] In some embodiments, the feature extraction model includes an image encoder, a text encoder, and a cross-modal attention layer; training the feature extraction model and the gating network based on a second training dataset includes the following steps: The second sample data is used as input to the feature extraction model. Each second sample behavior description text is divided into subject, reason, and predicate according to the subject, reason, and action of the behavior. The feature extraction model is configured to extract sample visual feature vectors based on the second sample images and sample text feature vectors based on the second sample behavior description texts. The sample text feature vectors include subject feature vectors, reason feature vectors, and predicate feature vectors. Sample image features and sample text features are calculated through cross-attention based on the sample visual feature vectors and sample text feature vectors. Sample text features include subject features, reason feature vectors, and predicate features. Loss function values ​​are calculated for the sample image features and the subject features, reason feature vectors, and predicate features, respectively. The feature extraction model is trained based on the calculated loss function values.

[0012] A second aspect of this application also provides a behavior detection system for implementing the behavior detection method of the first aspect, the system comprising: The data acquisition module is used to acquire the video data to be detected and the current behavior description text; The feature extraction module is used to input the current frame of the video data and the current behavior description text into the feature extraction model to obtain the current fused features output by the feature extraction model; The gating network module is used to input the current fusion feature into the gating network. The gating network is configured to generate the current confidence score based on the current fusion feature, the historical fusion feature corresponding to the historical frame of the video data, and the historical confidence score corresponding to the historical fusion feature. It then determines whether to continue detection based on the current confidence score. The current confidence score represents the confidence score of the current frame corresponding to the current fusion feature matching the current action text. The matching and detection module is used to input the current fused features into the large language model when the gating network decides to continue detection, and obtains the matching result output by the large language model. The large language model is configured to determine whether the current frame of the video data matches the behavior description text based on the current fused features.

[0013] A third aspect of this application also provides a behavior detection device, comprising: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the behavior detection method described above by executing the executable instructions.

[0014] A fourth aspect of this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the behavior detection method described above.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0016] The behavior detection method, system, device, and program products of this application have the following beneficial effects: By adopting the scheme of this application, a two-stage behavior detection and matching is provided. First, based on the feature extraction model, the current frame of the video data and the current behavior description text corresponding to the current fusion features are extracted. The first-stage detection is implemented based on a gating network. The gating network determines whether to perform the second-stage detection based on the current confidence level. This avoids full-frame detection by a large language model, thus avoiding the large language model repeatedly processing some repetitive detection work with low confidence levels. This is beneficial to improving the efficiency of behavior detection based on continuous video frames, thereby achieving continuous detection of continuous video, which is conducive to improving video detection accuracy. It also reduces the resource consumption and running burden of this behavior detection method and reduces the hardware requirements for operation. The gating network combines temporal information to improve detection robustness. Compared with single-frame detection, it effectively utilizes historical fusion features and historical confidence levels for comprehensive decision-making, reduces the impact of detection noise, and makes the generation of confidence levels more comprehensive and the decision more accurate. When deciding to proceed with the second phase of detection, the current fusion model with high confidence is further matched and analyzed using a large language model. Compared to conventional language models that need to cover a wide range of ideas, the large language model in this application focuses more on behavior recognition and description, which improves the accuracy of behavior detection based on continuous video data while ensuring the efficiency of the behavior detection method. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of a behavior detection method according to an embodiment of this application; Figure 2 This is a schematic diagram of a behavior detection process according to an embodiment of this application; Figure 3 This is a schematic diagram of the feature extraction process of a feature extraction model according to an embodiment of this application; Figure 4 This is a schematic diagram of a gating network according to an embodiment of this application; Figure 5 This is a schematic diagram of a feature mapping layer according to an embodiment of this application; Figure 6 This is a schematic diagram of the model training process according to an embodiment of this application; Figure 7 This is a schematic diagram of the training process of a feature extraction model according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a behavior detection system according to an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a behavior detection device according to an embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0020] Furthermore, the accompanying drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. Although the terms "first" or "second," etc., are used in this specification to denote certain features, these are only for indicating function and not as a limitation on the number or importance of specific features.

[0021] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0022] Figure 1The flowchart of a behavior detection method according to an embodiment of this application is shown. Figure 2 The implementation process of a behavior detection method according to an embodiment of this application is illustrated. For example... Figure 1 and Figure 2 As shown, in this embodiment, the behavior detection method includes the following steps: S100: Obtain the video data to be detected and the current behavior description text; This application is used to detect whether there is a behavior corresponding to behavior description text in video data. Behavior description text is natural language text that describes the behavior of the target detection, such as "a person is running" or "a person falls down". It can also be further supplemented with causal relationships, such as "a person is running to catch a bus" or "a person falls down because of an attack". Behavior description text is used to describe the behavior type of the target detection behavior. Each behavior type is determined by a specific predicate, or by a specific subject and a specific predicate. The subject is the executing entity, such as "a person" or "a vehicle", and the predicate is the specific behavior action performed, such as "running" or "falling down". Thus, "a person falls down" corresponds to one behavior type, and "a person runs" corresponds to another behavior type. Behavior description text includes text describing the behavior action, or it includes text describing the subject and the behavior action. The video data to be detected here can be real-time video data obtained from monitoring equipment or recorded video data; the current behavior description text can be a natural language description text input by the user or a natural language description text extracted from a storage location of behavior description text. S200: Input the current frame of the video data and the current behavior description text into the feature extraction model to obtain the current fused features output by the feature extraction model; The feature extraction model is configured to extract current image features based on the current frame of the video data, extract current text features based on the current behavior description text, and obtain current fused features based on the current image features and current text features. For example, the current image features and current text features are concatted together to obtain the concatenated current fused features. S300: Input the current fused features into the gating network. The gating network is configured to generate the current confidence based on the current fused features, the historical fused features corresponding to the historical frames of the video data, and the historical confidence corresponding to the historical fused features. It then determines whether to continue detection based on the current confidence. The current confidence represents the confidence that the current frame corresponding to the current fused features matches the current action text. The number of historical fusion features and historical confidence scores stored in the gating network can be preset. The gating network maintains a list of historical fusion features and a list of historical confidence scores. The gating network can realize the first stage detection of the current fusion feature. Based on the detected current confidence score, it is determined whether to continue to the second stage detection in step S400. For the current fusion feature with obviously low confidence score, there is no need to continue to input it into the large language model for detection. This avoids the large language model performing full frame-by-frame detection of video data, which is conducive to improving the efficiency of behavior detection based on video data. Traditional single-frame detection relies solely on the features of the current frame, making it susceptible to noise (such as changes in lighting, motion blur, occlusion, etc.) and prone to misjudgment. In contrast, the gating network in this application introduces historical frame fusion features and historical confidence when generating the current confidence score. By utilizing temporal information, it effectively improves detection robustness and avoids false positives and false negatives. By combining historical frame fusion features and historical confidence scores to make a comprehensive judgment on the current fusion features, it can avoid the influence of noise and is more comprehensive than single-frame image feature analysis. The process of determining whether to continue testing based on the current confidence level includes: determining whether the current confidence level is greater than a preset confidence threshold; if so, determining to continue testing; if the current confidence level is less than or equal to the preset confidence threshold, determining not to continue testing and directly outputting a mismatch result. S400: When the gated network decides to continue detection, it inputs the current fused features into the large language model and obtains the matching result output by the large language model. The large language model is configured to determine whether the current frame of the video data matches the behavior description text based on the current fused features. The matching results output by the large language model include matching and non-matching. Matching means that the behavior type described by the current behavior description text was detected in the current frame of the video data to be detected, and non-matching means that the behavior type described by the current behavior description text was not detected in the current frame of the video data to be detected. In this embodiment, the large language model is used for the second stage of detection. The gating network determines whether the current fusion feature with high confidence is matched. This not only ensures the efficiency of behavior detection, but also further improves the accuracy of behavior detection.

[0023] By employing the behavior detection method of this application, a two-stage behavior detection matching is provided. First, steps S100 and S200 extract the current fusion features corresponding to the current frame and current behavior description text of the video data based on the feature extraction model. Step S300 implements the first-stage detection based on a gating network. The gating network determines whether to perform the second-stage detection based on the current confidence level, avoiding full frame-by-frame detection by a large language model. This avoids the large language model repeatedly processing repetitive detection work with low confidence levels, which is beneficial to improving the efficiency of behavior detection based on continuous video frames, thereby achieving continuous detection of continuous video, improving video detection accuracy, and reducing the resource consumption and operational burden of the behavior detection method, reducing the hardware requirements for operation. The gating network combines temporal information to improve detection robustness. Compared with single-frame detection, it effectively utilizes historical fusion features and historical confidence levels for comprehensive decision-making, reducing the impact of detection noise, making the generation of confidence levels more comprehensive, and the decision more accurate. When determining to continue the second stage of detection, step S400 uses a large language model to further match and analyze the current fusion model with high confidence. Compared with conventional language models that need to cover a wide range of opinions, the large language model of this application focuses more on behavior recognition and description, which improves the accuracy of behavior detection based on continuous video data while ensuring the efficiency of the behavior detection method.

[0024] Because this application improves the efficiency and accuracy of behavior detection based on continuous video data, it can be well applied to the detection and analysis of abnormal behavior in real-time video streams in monitoring scenarios. However, this application is not limited to this. In other embodiments, this application can be applied not only to the detection of abnormal behaviors (such as fighting, falling, running, etc.), but also to the detection of normal behaviors (such as walking, sitting still, etc.). This application can also be applied to the behavior detection and analysis of real-time video streams in other scenarios, or it can also be applied to the behavior detection and analysis of pre-recorded video data.

[0025] Figure 3 The process of feature extraction using a feature extraction model according to an embodiment of this application is illustrated. For example... Figure 3As shown, the feature extraction model includes an image encoder, a text encoder, a cross-modal attention layer, and a feature fusion layer. The image encoder is configured to convert the input image into a visual feature vector. The text encoder is configured to convert the current behavior description text into a text feature vector. The cross-modal attention layer is configured to calculate attention weights based on the visual and text feature vectors, generate current image features based on the visual feature vectors and attention weights, and generate current text features based on the text feature vectors and attention weights. The feature fusion layer is configured to obtain the current fused features based on the current image features and current text features. For example, the feature fusion layer could be a Concat layer. The cross-modal attention layer in the feature extraction model enables precise semantic alignment and deep fusion of image and text features. The cross-modal attention layer can calculate attention weights for different regions of the image based on the text semantics, allowing the image encoder to accurately focus on the key parts corresponding to the text when calculating image features. For example, when the behavior description text mentions "running," the text-image attention weights obtained by the cross-modal attention layer will have a higher attention weight for the leg region in the image, so the image encoder will focus more on the image region with the higher weight when calculating image features. Cross-modal attention layers can assign higher attention weights to parts of the text that semantically match the image based on the visual details of the image. For example, if the behavioral description text is "a person is running next to a tree," and the core action in the image is "a person running," then in the image-text attention weights obtained by the cross-modal attention layer, the weight of "running" will be increased, while the weight of "tree" will be decreased. Therefore, the text encoder will focus more on the text content with higher weights when calculating text features. Simultaneously, during the training of the feature extraction model, the cross-modal attention layer can deeply infer the implicit relationships between the image and the text. For example, if the behavioral description text mentions "picking up an object," and the image shows two pieces of information: "bending over" and "hand touching an object," the cross-modal attention layer will associate these two pieces of information and determine them as the action corresponding to "picking up."

[0026] like Figure 3As shown, after inputting the current frame and the current behavior description text into the feature extraction model, the processing steps of the feature extraction model include: the image encoder extracts features from the current frame image to obtain a visual feature vector, and the text encoder extracts features from the current behavior description text to obtain a text feature vector. The cross-modal attention layer calculates text-image attention weights and image-text attention weights based on the visual and text feature vectors. It then generates current image features based on the text-image attention weights and the visual feature vectors, such as updating the visual feature vectors based on the text-image attention weights and weighting the visual feature vectors according to the text-image attention weights. Finally, the feature fusion layer obtains the current fused features based on the current image features and the current text features.

[0027] In this embodiment, both the image encoder and the text encoder are smaller than large language models. For example, the image encoder uses a ViT (Vision Transformer)-based image model, which segments the input image into fixed-size image blocks, converts them into vector sequences, and extracts global features from the vector sequences through a multi-head self-attention layer and a feedforward neural network of the Transformer encoder. By pre-training the image encoder on a massive dataset of action behaviors (such as walking, running, fighting, falling, etc.), the image encoder can significantly outperform traditional open-source general-purpose small models (such as the CLIP model) in action recognition. The text encoder uses a Transformer-based text model. By introducing cross-modal attention layers in both the image encoder and the text encoder, image features and text features undergo cross-attention calculation. Through the introduction of image-text attention weights and text-image attention weights, and the weighted enhancement of visual feature vectors and text feature vectors, early deep fusion of vision and semantics is achieved, prompting features to begin semantic association from the bottom layer. Since the current image features and the current text features are respectively the features after deep image-text semantic fusion through cross-modal attention layers, the concatenated current fused features are the features after deep image-text semantic fusion.

[0028] Figure 4 The structure of a gating network according to an embodiment of this application is shown. Figure 4As shown, the gating network in this embodiment is a metacognitive gating network, which can be implemented based on a miniature Transformer network or an LSTM (Long Short-Term Memory) network. The hidden layer of the gating network stores and maintains a historical fusion feature queue and a historical confidence queue. The number of features stored in the historical fusion feature queue and the number of confidence values ​​stored in the historical confidence queue can be set as needed. The historical fusion feature queue, the historical confidence queue, and the current fusion feature are fused together by a Concat layer and then input into the confidence calculation layer. The confidence calculation layer calculates the current confidence value corresponding to the current fusion feature based on the input features. The current confidence value represents the probability of detecting the behavior corresponding to the current behavior description text in the current video frame corresponding to the current fusion feature. After calculating the current confidence value, the confidence calculation layer stores the current fusion feature and the current confidence value into the historical fusion feature queue and the historical confidence queue in the hidden layer for use in the confidence calculation of the next video frame. The decision layer determines whether to continue the second stage of detection based on the current confidence level and the preset confidence threshold. If the current confidence level is greater than the preset confidence threshold, the decision layer outputs the current fused features to the large language model. If the current confidence level is less than or equal to the preset confidence threshold, the decision layer directly outputs the result that the current video frame and the current behavior description text do not match.

[0029] like Figure 2 As shown, the feature spaces of the feature extraction models are different from those of the large language model. A feature mapping layer is added between the output of the gating network and the input of the large language model. This feature mapping layer is configured to map the current fused features output by the gating network to the feature space of the large language model. Here, the feature spaces are different because the feature vectors of the feature extraction model and the feature vectors of the large language model have different dimensions, different meanings of each dimension, and different numerical ranges. Step S400: Input the current fused features into the large language model, including the following steps: The current fused features output by the gated network are input into the feature mapping layer, and the current fused features are mapped to the feature space of the large language model through the feature mapping layer; The current fused features, after being processed by the feature mapping layer, are input into the large language model.

[0030] like Figure 5As shown, in this embodiment, the feature mapping layer includes at least one downsampling layer, a linear layer, a layer normalization layer, a dropout layer, and at least one upsampling layer, sequentially connected in series. First, multiple downsampling layers are used to compress the current fused features and extract core features. The linear layer adjusts the dimensions. The linear layer is used for dimension alignment and linear space mapping, aligning the input features with the target feature space (the feature space of the large language model) through a linear transformation. Then, the layer normalization layer aligns the numerical distribution, unifying the feature distribution, and the dropout layer prevents overfitting. Finally, multiple upsampling layers progressively reconstruct the features, with each upsampling layer fusing the output of the corresponding downsampling layer to maintain feature integrity. The final output is then fed into the second-stage large language model. This feature mapping layer not only maps the fused features from the first stage to the feature space of the large language model but also, through the combination of downsampling and upsampling, makes the transformed features richer than those obtained through direct linear mapping.

[0031] Combination Figure 2 As can be seen, this embodiment provides a two-stage multimodal model joint processing method. The first-stage network architecture includes a feature extraction model and a gating network, while the second-stage network architecture includes a feature mapping layer and a large language model. The first stage performs preliminary identification based on the feature extraction model and the gating network, using the gating network as a scheduler to decide whether to proceed to the second stage. Simultaneously, to better adapt the small model features of the first stage to the large model features of the second stage, a feature mapping layer is introduced after the gating network to map the low-dimensional small model features to the high-dimensional large model feature space. The second stage uses the large language model to further detect and analyze the fused features with high confidence, ensuring both the efficiency and recognition effectiveness of the behavior detection method.

[0032] Figure 2 The model architecture shown can be collectively referred to as the behavior detection model, which includes a feature extraction model, a gating network, a feature mapping layer, and a large language model. During inference, the video data to be detected and the current behavior description text (text data) are input into the behavior detection model, which can then match and detect whether the behavior corresponding to the current behavior description text exists in each video frame of the video stream.

[0033] like Figure 6 As shown, in this embodiment, the behavior detection method further includes the following steps: training a behavior detection model: S510: Train a large language model based on the first training dataset. The first training dataset includes multiple first sample fusion features. Each first sample fusion feature is obtained based on a first sample image and a first sample behavior description text. S520: Freeze the parameters of the large language model, and train the feature extraction model and gating network based on the second training dataset. The second training dataset includes multiple second sample data pairs, each of which includes a second sample image and a second sample behavior description text. In this embodiment, in step S520, while training based on the second training dataset, the feature mapping layer is trained simultaneously, that is, the parameters in the feature extraction model, gating network, and feature mapping layer are trained and optimized simultaneously.

[0034] Therefore, the training process of the behavior detection model is divided into two steps. Step S510 corresponds to the first training step, which trains only the large language model. Step S520 corresponds to the second training step, which trains the feature extraction model and the feature mapping layer of the gating network based on the large language model with frozen parameters. In the first training step, only the large language model is trained. A large amount of abnormal behavior description data in monitoring scenarios is collected and labeled in advance, including: extracting multiple first sample images from abnormal behavior images in monitoring scenarios, labeling the first sample images to obtain the first sample behavior description text corresponding to the first sample images, using an image encoder and a text encoder (which can be the image encoder and text encoder in the feature extraction model, or other mature image encoders and text encoders) to extract the first sample fusion features corresponding to each first sample image and first sample behavior description text, inputting the first sample fusion features into the large language model to obtain the matching results output by the large language model, calculating the first loss function value based on the matching results and matching labels, and optimizing the training of the large language model.

[0035] In this embodiment, during the training and inference process of the large language model, the processing of the input fusion features (the first sample fusion features during training and the current fusion features during inference) can be as follows: Based on the fusion features input to the large language model through the gating network, the large language model first performs fast shallow inference. If the internal confidence is high (e.g., the internal confidence is greater than or equal to a preset threshold), the matching result is directly output. If there is still uncertainty (e.g., the internal confidence is less than a preset threshold), the large language model further activates deeper network layers to perform multi-step thought chain inference.

[0036] In step S520, training the feature extraction model, gating network, and feature mapping layer based on the second training dataset includes the following steps: The second sample data is input into the feature extraction model to obtain the second sample fusion features output by the feature extraction model. The second sample behavior description text in the second sample data pair includes causal relationship labels for describing the cause and action of the behavior. In the causal relationship labels, the cause of the behavior corresponds to the cause in the causal relationship, and the action of the behavior corresponds to the result in the causal relationship. For example, if a causal relationship label includes "running because of being attacked", then the cause of the behavior is "being attacked" and the action of the behavior is "running". Or if a causal relationship label includes "running because of chasing a bus", then the cause of the behavior is "chasing a bus" and the action of the behavior is "running". Optionally, the second sample behavior description text may further include the subject of the behavior, such as a causal relationship label including "a person runs because of being attacked" or "a person runs because of chasing a bus". The fused features of the second sample are input into the gating network to obtain the judgment result output by the gating network; When the gating network determines that detection should continue, it inputs the fused features of the second sample into the large language model to obtain the matching result output by the large language model. The loss function value is calculated based on the matching results output by the large language model and the matching labels of the second sample data pair. The feature extraction model, gating network and feature mapping layer are trained based on the loss function value.

[0037] In this embodiment, the second sample behavior description text in the second sample data pair is used for behavior cause and behavior action. That is, in the second training step, the behavior type is further subdivided according to the behavior cause. For example, "a person runs because they are attacked" and "a person runs because they are chasing a bus" have the same subject (a person) and behavior action (running), but they correspond to different behavior types because of different behavior causes (being attacked and chasing a bus). In this embodiment, by determining the behavior type according to a specific subject (behavior subject), a specific cause state (behavior cause), and a specific predicate (behavior action) in the second training step, the behavior type is further subdivided. The behavior cause in the causal relationship label corresponds to the behavior cause in the second sample image. Through parameter optimization training of the feature extraction model, gating network, and feature mapping layer, the overall behavior detection model can fully explore the causal relationship between behavior cause and behavior action, realize causal relationship perception, and have a more detailed detection function. It can not only distinguish different behavior subjects and different behavior actions, but also further distinguish different behavior causes. For example, for two images containing the same action but different causes, the final matching result output by the large language model corresponds to two different action types, enhancing the ability to judge the causal differences between similar actions.

[0038] In the second training step, the behavior description text of the second sample can further include causal difference labels describing the causes of the differential behavior and the behavioral actions. Unlike the causal relationship labels where the behavior cause corresponds to the behavior cause in the second sample image, the differential behavior cause in the causal difference labels differs from the behavior cause in the second sample image. When training with the second sample image and causal relationship labels, the output of the large language model corresponds to the matching label "yes"; when training with the second sample image and causal difference labels, the output of the large language model corresponds to the matching label "no". Through training, the behavior detection model can more accurately identify and detect the correct behavior cause in the image, while effectively avoiding interference from other differential behavior causes.

[0039] In this embodiment, during the second step of training, a causal perception-based training method is adopted. This not only improves the model's ability to identify basic abnormal behaviors through training but also further enhances its ability to judge causal differences between similar behaviors. Specifically, correct causal relationship labels are added to the second sample images. For example, if a second sample image is labeled as "a person is running to catch a bus," the model training process needs to narrow the distance between the image features of the second sample image and the causal relationship label. Simultaneously, causal difference labels can be added to the second sample images. For example, if a second sample image is labeled as "a person is running because they were attacked" (different causal adverb) and "a person is falling because they were attacked" (different causal adverb and predicate), the model training process needs to widen the distance between the image features of the second sample data pair and the causal difference labels. In specific implementation, traditional single-category labels (such as "a person is running," "a person is falling") are converted into subject, causal adverb, and predicate structure labels (such as "a person is running," "a person is falling")... Due to a certain incident The algorithm uses the cross-entropy loss function to independently calculate the contrastive learning differences for the subject, adverbial of cause, and predicate, thereby enhancing the ability to judge causal differences between similar behaviors.

[0040] like Figure 7 As shown, in step S520, the feature extraction model, gating network, and feature mapping layer are trained based on the second training dataset, including the following steps: The second sample data is input into the feature extraction model. The text segmentation layer (split layer) divides each second sample behavior description text into subject (behavior subject), causal adverbial (behavior cause), and predicate (behavior action) according to the behavior subject, behavior cause, and behavior action, respectively. The second sample data pair may include a second sample image and the corresponding causal relationship label or causal difference label. The causal relationship label or causal difference label is used as the second sample behavior description text corresponding to the second sample image. The feature extraction model is configured to use an image encoder to extract sample visual feature vectors based on a second sample image, and a text segmentation layer to segment the sample behavior description text into subject, causal adverbial, and predicate. A text encoder is then used to extract sample text feature vectors based on the second sample behavior description text. The sample text feature vectors include subject feature vectors, causal adverbial feature vectors, and predicate feature vectors. Based on the sample visual feature vectors and sample text feature vectors, sample image features and sample text features are calculated through cross-attention. The sample text features include subject features, causal adverbial features, and predicate features. Loss function values ​​are calculated for the sample image features and the subject features, causal adverbial features, and predicate features, respectively. The feature extraction model is trained based on the calculated loss function values.

[0041] In step S520, while training the feature extraction model based on causal relationships, the feature extraction model, gating network, and feature mapping layer are also trained simultaneously based on the matching results and matching labels output by the large language model. For example, in step S520, the second sample image pairs (each second sample image pair includes a second sample image and a causal relationship label or causal difference label corresponding to the second sample image) are input into the feature extraction model to obtain the sample fusion features output by the feature extraction model. After the sample fusion features are passed through the gating network and feature mapping layer and input into the large language model, the first loss function value is calculated based on the matching results and matching labels output by the large language model. At the same time, the second loss function value (such as the cross-entropy loss function) is calculated based on the image features and subject features in the feature extraction model, the third loss function value (such as the cross-entropy loss function) is calculated based on the image features and predicate adverbial features in the feature extraction model, and the fourth loss function value (such as the cross-entropy loss function) is calculated based on the image features and predicate adverbial features in the feature extraction model. The feature extraction model, gating network, and feature mapping layer are trained based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value. In step S520, the first loss function value is used to train the parameters of the feature extraction model, gating network, and feature mapping layer, improving the overall behavior detection model's accuracy in detecting behavior types after subdividing the causes of behavior. This not only enhances the ability to recognize basic behavior types (determined based on the subject and predicate) but also enhances the ability to judge the causal type between similar behavior types. The second, third, and fourth loss function values ​​are used to train the feature extraction model. By optimizing the parameters of the image encoder, text encoder, and cross-modal attention layer in the feature extraction model, the distance between the text features obtained by the text encoder and the positive labels of the same cause is reduced, while the distance between the text features obtained by the text encoder and the negative labels of different causes is increased. This makes the text features obtained by the text encoder contain more accurate causal features, which is beneficial to improving the accuracy of the final large language model in recognizing behavior causes based on the fused features output by the feature extraction model.

[0042] like Figure 8 As shown in the embodiments of this application, a behavior detection system is also provided for implementing the behavior detection method of the first aspect. The system includes: The data acquisition module M100 is used to acquire the video data to be detected and the current behavior description text; The feature extraction module M200 is used to input the current frame of the video data and the current behavior description text into the feature extraction model to obtain the current fused features output by the feature extraction model; The gated network module M300 is used to input the current fusion feature into the gated network. The gated network is configured to generate the current confidence based on the current fusion feature, the historical fusion feature corresponding to the historical frame of the video data, and the historical confidence corresponding to the historical fusion feature. It then determines whether to continue detection based on the current confidence. The current confidence represents the confidence that the current frame corresponding to the current fusion feature matches the current behavioral text. The matching detection module M400 is used to input the current fused features into the large language model when the gating network decides to continue detection, and obtains the matching result output by the large language model. The large language model is configured to determine whether the current frame of the video data matches the behavior description text based on the current fused features.

[0043] In the behavior detection system of this application, the functions of each module can be implemented using the specific implementation methods of the behavior detection method described above, which will not be elaborated here. By using the behavior detection system of this application, when the functions of each module are implemented, the technical effects of the aforementioned behavior detection method can be obtained, which will not be elaborated here.

[0044] This application also provides a behavior detection device, including a processor; a memory storing executable instructions of the processor; wherein the processor is configured to perform the steps of the behavior detection method by executing the executable instructions.

[0045] Those skilled in the art will understand that various aspects of this application can be implemented as systems, methods, or computer program products. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "platform."

[0046] The following reference Figure 9 To describe an electronic device 600 according to this embodiment of the present application. Figure 9 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0047] like Figure 9 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0048] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the behavior detection method section of this specification according to various exemplary embodiments of this application. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0049] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0050] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0051] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0052] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0053] In the behavior detection device, when the program in the memory is executed by the processor, it implements the steps of the behavior detection method. Therefore, the device can also achieve the technical effects of the behavior detection method described above.

[0054] An exemplary embodiment of this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the behavior detection method described above.

[0055] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. Exemplarily, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0056] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file storing the computer program, an installation package, or other digital file.

[0057] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, C++, and Python. Program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0058] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this application, such as the steps of the behavior detection method described above.

[0059] When the computer program is executed by the processor, it implements the steps of the behavior detection method described above. Therefore, the computer program product can also achieve the technical effects of the behavior detection method described above.

[0060] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the scope of protection of this application.

Claims

1. A behavior detection method, characterized in that, Includes the following steps: Acquire the video data to be detected and the current behavior description text; Input the current frame of the video data and the current behavior description text into the feature extraction model to obtain the current fused features output by the feature extraction model; The current fusion feature is input into a gating network, which is configured to generate a current confidence score based on the current fusion feature, the historical fusion features corresponding to the historical frames of the video data, and the historical confidence scores corresponding to the historical fusion features. The gating network then determines whether to continue detection based on the current confidence score. The current confidence score represents the confidence score that the current frame corresponding to the current fusion feature matches the current behavioral text. When the gating network determines to continue detection, it inputs the current fusion features into the large language model to obtain the matching result output by the large language model. The large language model is configured to determine whether the current frame of the video data matches the behavior description text based on the current fusion features.

2. The behavior detection method according to claim 1, characterized in that, The feature extraction model includes an image encoder, a text encoder, a cross-modal attention layer, and a feature fusion layer; The image encoder is configured to convert the input image into a visual feature vector; the text encoder is configured to convert the current behavior description text into a text feature vector. The cross-modal attention configuration is to calculate attention weights based on the visual feature vector and the text feature vector, generate current image features based on the visual feature vector and the attention weights, and generate current text features based on the text feature vector and the attention weights; The feature fusion layer is configured to obtain the current fused feature based on the current image features and the current text features.

3. The behavior detection method according to claim 1, characterized in that, The current fused features are input into the large language model, including the following steps: The current fused features are input into a feature mapping layer, which is configured to map the current fused features to the feature space of the large language model. The current fused features, after being processed by the feature mapping layer, are input into the large language model.

4. The behavior detection method according to claim 3, characterized in that, The feature mapping layer includes at least one downsampling layer, a linear layer, a layer normalization layer, a random deactivation layer, and at least one upsampling layer, which are connected in series.

5. The behavior detection method according to claim 1, characterized in that, It also includes the following training steps: The large language model is trained based on a first training dataset, which includes multiple first sample fusion features, each of which is obtained based on a first sample image and a first sample behavior description text. The parameters of the large language model are frozen, and the feature extraction model and the gating network are trained based on the second training dataset. The second training dataset includes multiple second sample data pairs, and each second sample data pair includes a second sample image and a second sample behavior description text.

6. The behavior detection method according to claim 5, characterized in that, The process of training the feature extraction model and the gating network based on the second training dataset includes the following steps: The second sample data is input into the feature extraction model to obtain the second sample fusion feature output by the feature extraction model. The second sample behavior description text includes causal relationship tags for describing the cause of the behavior and the behavior action. The second sample fusion feature is input into the gating network to obtain the judgment result output by the gating network; When the gating network determines that detection should continue, it inputs the second sample fusion feature into the large language model to obtain the matching result output by the large language model. The loss function value is calculated based on the matching result output by the large language model and the matching label of the second sample data pair, and the feature extraction model and the gating network are trained based on the loss function value.

7. The behavior detection method according to claim 5, characterized in that, The feature extraction model includes an image encoder, a text encoder, and a cross-modal attention layer; training the feature extraction model and the gating network based on the second training dataset includes the following steps: The second sample data is input into the feature extraction model. Each second sample behavior description text is divided into subject, reason, and predicate according to the subject, reason, and action of the behavior. The feature extraction model is configured to extract sample visual feature vectors based on the second sample image and sample text feature vectors based on the second sample behavior description text. The sample text feature vectors include subject feature vectors, reason feature vectors, and predicate feature vectors. Sample image features and sample text features are calculated through cross-attention based on the sample visual feature vectors and the sample text feature vectors. The sample text features include subject features, reason feature vectors, and predicate features. Loss function values ​​are calculated for the sample image features and the subject features, reason feature vectors, and predicate features, respectively. The feature extraction model is trained based on the calculated loss function values.

8. A behavior detection system, characterized in that, The system for implementing the behavior detection method according to any one of claims 1 to 7, the system comprising: The data acquisition module is used to acquire the video data to be detected and the current behavior description text; The feature extraction module is used to input the current frame of the video data and the current behavior description text into the feature extraction model to obtain the current fused features output by the feature extraction model; A gated network module is used to input the current fusion feature into the gated network. The gated network is configured to generate a current confidence score based on the current fusion feature, the historical fusion feature corresponding to the historical frame of the video data, and the historical confidence score corresponding to the historical fusion feature, and to determine whether to continue detection based on the current confidence score. The current confidence score represents the confidence score that the current frame corresponding to the current fusion feature matches the current behavioral text. The matching detection module is used to input the current fusion features into the large language model when the gated network determines to continue detection, and obtain the matching result output by the large language model. The large language model is configured to determine whether the current frame of the video data and the behavior description text match based on the current fusion features.

9. A behavior detection device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the behavior detection method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the behavior detection method according to any one of claims 1 to 7.

Images