Open vocabulary group behavior detection method for indoor security and protection monitoring video scene

Abstract

The invention provides an open vocabulary group behavior detection method for an indoor security and protection monitoring video scene, and belongs to the field of video action recognition. The method comprises the following steps: S1, collecting and processing an indoor scene monitoring video, obtaining effective video clips containing people, and obtaining a ternary annotation result lt of each effective video clip; a video, a text and a mark gt; ; s2, for each frame of a video and a corresponding text, respectively using Swin-B and BERT structures of a CLIP pre-training model as an image encoder and a text encoder; the Swin-B and BERT parameters are updated and constrained by regular terms, and finally an image-text encoder is determined; s3, constructing, training and determining an open vocabulary group behavior detection model; and S4, inputting an actual monitoring video to the open vocabulary group behavior detection model to obtain a behavior category of each group. According to the method, the group to which each person in the indoor security and protection monitoring video belongs can be identified, the behaviors of each group are classified, and the open vocabulary detection requirement can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of video action recognition, and in particular to an open vocabulary group behavior detection method for indoor security monitoring video scenes. Background Art

[0002] The purpose of group behavior detection for security surveillance video scenes is to use computer technology to automatically locate groups for each frame of surveillance images, that is, to identify how many groups there are in the current frame and which group each person belongs to, and to classify the activities performed by each group, which is of great significance to public safety. Since the number of groups and their members in each frame are unknown, this task requires simultaneous identification of group behavior participants, perception of the spatiotemporal correlation between different participants, and identification of individual behaviors, which is very challenging.

[0003] Due to these difficulties, existing methods for understanding group activities are mostly limited to a single task of classifying an entire video clip into one of the predefined activity categories, known as group behavior recognition. The conventional setting of group behavior recognition usually assumes that each video clip contains only a single group activity, and the participants of the activity have been pre-determined by manual annotation. However, these assumptions do not hold true in many real surveillance scene videos containing group behaviors, that is, there are often multiple groups performing independent activities in the video, as well as unrelated individuals that do not belong to any group. For example, a video surveillance screen of a coffee shop usually contains multiple group behaviors such as checking out, eating, chatting, queuing, and studying at the same time.

[0004] Therefore, although group behavior recognition has existed as a representative task of group behavior understanding for more than ten years, its actual application value in real security monitoring video scenes has great limitations. For example, the patent document with publication number CN117789094A provides a group behavior detection and recognition method based on deep learning. Although it comprehensively considers multi-level information of individual motion characteristics, group member interaction characteristics and group overall characteristics, it still assumes that there is only one group in a picture, and those that do not belong to this group are abnormal interference items, which does not conform to the actual situation of security monitoring.

[0005] For example, the patent document with publication number CN107506734A provides a method for detecting and locating sudden abnormal events in groups, and the patent document with publication number CN105574489A provides a method for detecting violent group behavior based on hierarchical cascade. The core of both methods is to automatically identify the images into two categories: normal and abnormal. It is impossible to output which group each member in the monitoring image belongs to and the specific behavior category of each group.

[0006] In addition, all existing group behavior detection methods for security surveillance video scenarios are closed-set detections, that is, the group behavior categories in the test set or actual usage scenarios are consistent with those in the training set, resulting in poor generalization performance of the model. The group behavior detection model trained in one surveillance scenario cannot be applied to new surveillance scenarios. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to propose a group behavior detection method that can simultaneously identify which group each person in an indoor security monitoring video belongs to, classify the behavior of each group, and meet the needs of open vocabulary detection.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: an open vocabulary group behavior detection method for indoor security monitoring video scenes, comprising the following steps: S1: Collect and process surveillance video data of indoor scenes, obtain valid video clips containing people, and combine manual fine-grained annotation and automatic global semantic annotation to obtain the ternary annotation results <video, text, logo> of each valid video clip; S2: Build and train an image-text encoder based on a video-text multimodal encoding and decoding model, specifically: According to the ternary annotation results, for each frame of the video and the corresponding text, the Swin-B structure of the CLIP pre-trained model is used as the image encoder and the BERT structure is used as the text encoder; the parameters of Swin-B and BERT are optimized in the training phase, and the regularization term is used to constrain the update of Swin-B and BERT parameters and determine the final image-text encoder; S3: Construct and train an open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model, and perform parameter optimization to determine the open vocabulary group behavior detection model; S4: Input the actual surveillance video into the determined open vocabulary group behavior detection model to obtain the behavior category of each group.

[0009] The present invention utilizes video and text information, extracts features through the Swin-B structure in the CLIP pre-training model as an image encoder and the BERT structure as a text encoder, measures the importance of parameters by using the degree of gradient change during parameter fine-tuning during training, and designs a regularization term to constrain the parameter update process of the visual encoder and text encoder. It not only effectively utilizes the parameters of the pre-training model to adapt it to the group behavior detection task, but also avoids the unconstrained update of the Swin-B parameters and the BERT parameters, which causes the forgetting of the knowledge learned in the CLIP image-text overall semantic similarity task, reduces knowledge forgetting, and makes it suitable for open vocabulary scenarios.

[0010] Preferably, the specific process of step S1 is: S11: Collect surveillance video data of indoor scenes; S12: Define various types of group behavior, including fighting, dancing, queuing, checking out, eating, studying or working, taking pictures, chatting, playing ball, walking, running, and loner; S13: setting the length of the video segment and dividing the monitoring data video to obtain a plurality of video segments; S14: using a pedestrian detection and tracking tool to perform human body detection and tracking on all video clips, obtaining a person detection and tracking frame, removing video clips that do not contain people, and obtaining the remaining valid video clips; S15: annotate the effective video clips using manual fine-grained annotation and automatic global semantic annotation to obtain a triplet annotation result <video, text, logo>; S16: Divide the valid video clips into a training set, a validation set, and a test set.

[0011] Preferably, the specific process of step S15 is: S151: Manual fine-grained labeling: Based on the obtained person detection and tracking frame, label each tracked person with the group number and the corresponding group behavior category, and the minimum rectangular bounding box formed by the persons with the same group number is used as the location information of the group; S152: Based on manual fine-grained annotation, automatic global semantic annotation is performed: for a video to be annotated , if it is determined that j group behavior category, it is marked as j Strings, according to j Total number of strings and group behavior categories A Automatically generated ( A - j ) additional strings, indicating the group behavior categories that are not present in the video clip; S153: Utilization A String to video Labeling, get triple labeling results <video ,text , logo >, where the text Video All defined group behavior categories exist in 1 or 0, respectively, indicating the text Affirmation and negation, i.e. signs 1 means video There is text in The group behavior category represented by 0 means video Text does not exist in Represents the category of group behavior.

[0012] The present invention provides an effective data preparation and annotation strategy, including a combination of manual fine-grained annotation and automatic global semantic annotation. In particular, the automatic global semantic annotation process can quickly generate a string describing the content of a video clip, and the generated string not only contains the existing group behavior categories, but also clearly indicates the types of behaviors that are not included, thereby automatically generating a large number of <video, text, logo> ternary annotation results, greatly enriching the diversity of training data and supporting model training under an open vocabulary setting.

[0013] Preferably, in step S16, when the valid video clips are divided into a training set, a validation set and a test set, the group behavior categories in the test set are restricted to not appear in the training set, but the group behavior categories in the training set and the validation set are consistent.

[0014] The present invention limits the group behavior categories in the test set to not appear in the training set when dividing the effective video clips, but the group behavior categories of the training set and the verification set are consistent, which can meet the open vocabulary setting and identify the behavior categories that do not appear in the training set. That is, the present invention can work effectively even when the group behavior categories in the test set or the actual use scenario and the behavior categories in the training set are basically not overlapped, thereby improving the generalization ability of group behavior detection in new monitoring scenarios.

[0015] Preferably, the specific process of step S2 is: S21: For Video For each frame of the video, the Swin-B structure in the CLIP pre-trained model is used as the image encoder to extract the image features of each frame. The parameters of Swin-B are optimized and fine-tuned in the training phase. Represented as a feature sequence , according to the feature sequence Get the video Video feature representation ; S22: For text , using the BERT structure in the CLIP pre-trained model as a text encoder to extract text Text feature representation , BERT’s parameters are optimized and fine-tuned during the training phase; S23: Construct a video-text multimodal encoding and decoding model to represent video features and text feature representation Splicing along the feature channel dimension to obtain video-text multimodal fusion features; S24: Obtaining probability values ​​based on video-text multimodal fusion features , set as a flag The probability value is 1, then the sign The probability of being 0 is 1- ; S25: According to the signs Determine the loss value of the video-text multimodal encoding and decoding model for the probability of 1 and 0 ; S26: Setting the regularized constraint loss function , as a regular term to constrain the update process of the Swin-B parameters of the visual encoder and the BERT parameters of the text encoder; S27: Define the total training loss function , and use the gradient descent method to optimize the video-text multimodal encoding and decoding model on the training set so that the validation set Minimum, finalized image encoder and text encoder are obtained.

[0016] The present invention utilizes regularized constraint loss function Constraining the update of Swin-B parameters and BERT parameters can avoid the problem of knowledge forgetting caused by unconstrained update of Swin-B parameters and BERT parameters in completely different scenarios where the targeted group behavior detection task and the pre-trained image-text overall semantic similarity comparison learning task of the CLIP multimodal model belong to completely different scenarios, thereby losing the ability to represent group behavior concept features that do not exist in the training set. That is, the present invention can better realize open vocabulary group behavior detection.

[0017] Preferably, in step S21, according to the feature sequence Get the video Video feature representation The specific process is: feature sequence Mapped into features through a fully connected layer ;feature After a tangent nonlinear transformation layer, the features are obtained ;feature After a linear mapping layer and a Softmax layer, the weight vector is obtained , the value of which indicates the weight of the feature of the corresponding frame in the final video feature representation of the entire video; feature and the weight vector Multiply to get the video feature representation after frame fusion .

[0018] Preferably, the specific process of step S3 is: S31: Get every video All frame image data, the temporal position information of all individuals in each frame, the group number of the individual, and the group behavior category corresponding to the group are recorded, and the video is obtained. Corresponding <Video ,text , logo >Triple labeling results; S32: Design two encoders and decoders consisting of a standard four-layer Transformer structure. The output of the encoder is used as the input of the decoder to build a video-individual position information multimodal cross-attention model; S33: Processing video with the optimized image encoder Swin-B All frames of , obtain the feature representation of all frames, use a fully connected layer to process the temporal position information of all individuals in each frame to obtain the individual position embedding representation sequence, S34: The feature representations of all frames and the embedding representations of all individual positions in each frame are input into the encoder of the video-individual position information multimodal cross attention model; S35: The decoder outputs the group belonging embedding representation vector and group behavior category embedding representation vector corresponding to all individuals; S36: Using clustering loss function and classification loss function as supervision signals to guide the optimization of open vocabulary group behavior detection model based on video-individual position information multimodal cross-attention model; S37: After the optimization is completed, a determined open vocabulary group behavior detection model is obtained.

[0019] The present invention processes video clips of continuous frames, combines the position information of each individual and the features of the video frames, and realizes multimodal cross-attention modeling of video-individual position information through the Transformer structure. It can identify the group to which each individual in the video belongs and classify the behavior of each group, thereby achieving more accurate group behavior detection.

[0020] Preferably, in step S31, the video is obtained The specific process of obtaining the temporal position information of all individuals in each frame is as follows: using the pedestrian detection and tracking tool to obtain the minimum rectangular bounding box of the area where all individuals are located in each frame, Indicates The center point coordinates and width and height of the minimum rectangular bounding box of each individual m The temporal position information of each individual is expressed as ,in Indicates that the video The total number of individuals in n Video No. n frame.

[0021] Preferably, the specific process of step S36 is: S361: Through clustering loss function Constraint Video The group belonging embedding representation vectors of all individuals belonging to the same group are consistent; S362: Processing Video with Optimized Text Encoder BERT Corresponding text , obtain the vector representation based on the text description of the group behavior category; Through the classification loss function Constrain the similarity between the embedding representation vector of the group behavior category corresponding to each individual and the vector representation based on the text description of the group behavior category to be close to 1; S363: Under the constraint of , the parameter optimization training of the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model is completed by the stochastic gradient descent method. Preferably, the specific process of step S4 is: S41: Segment the actual surveillance video to obtain video clips ; S42: Obtain video clips using pedestrian detection and tracking tools The temporal position information of all individuals in each frame; S43: Use the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model to obtain the group belonging embedding representation vectors and group behavior category embedding representation vectors of all individuals; S44: Cluster the group belonging embedding representation vectors of all individuals through a clustering algorithm, and individuals with the same group behavior category belong to the same group; S45: There are a total of a For each individual, perform an inner product operation on the group behavior category embedding representation vector corresponding to all individuals belonging to the same group and the vector representation based on the text description of the group behavior category, and take the category corresponding to the maximum value of the inner product operation result as the behavior category of each individual; Select a The most common behavior category among the individuals is taken as the final group behavior category of the group. a The behavior categories of each individual are different, so choose a The behavior category with the largest inner product operation result among the individuals is taken as the final group behavior category of the group.

[0022] Compared with the prior art, the advantages of the present invention are: (1) when multiple persons appear in the surveillance video at the same time, the prior art can only output one group behavior category for the entire picture, while the present invention can simultaneously detect which group each person belongs to and identify the behavior category corresponding to the group in one model, and can also identify outliers who do not belong to any group; (2) when performing group behavior detection, the regularization term is designed to constrain the parameter update process of the visual encoder and the text encoder, which can not only effectively utilize the parameters of the pre-trained model to adapt it to the group behavior detection task, but also reduce knowledge forgetting and meet the needs of open vocabulary detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of an embodiment of the present invention; Figure 2 It is an image-text feature extractor pre-training model framework of the base video-text multimodal model in an embodiment of the present invention; Figure 3 It is an open vocabulary group behavior detection model framework based on the video-individual position information multimodal cross-attention model in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Example like Figure 1 As shown, this embodiment provides an open vocabulary group behavior detection method for indoor security monitoring video scenes, including the following steps: S1: Collect and process surveillance video data of indoor scenes, obtain valid video clips containing people, and combine manual fine-grained annotation and automatic global semantic annotation to obtain the ternary annotation results <video, text, logo> of each valid video clip. The specific process is as follows: S11: Collect surveillance video data of real indoor scenes, including libraries, restaurants, indoor gymnasiums, etc. This embodiment takes the above three types of scenes as examples, and each type contains video data of a period of time shot from multiple different surveillance camera angles. This embodiment selects 24 hours of video data shot from 3 different surveillance camera angles, totaling 216 hours of video data; S12: define possible group behavior categories. This embodiment defines 12 categories, including fighting, dancing, queuing, checking out, eating, studying or working, taking pictures, chatting, playing ball, walking, running, and being out of the group; S13: Set the length of the video segment. In this embodiment, 3 seconds is used as a video segment length. The original 216 hours of video data is segmented to obtain 259,200 video segments. The original surveillance video frame rate is 25 frames per second, that is, each video segment has 75 frames; S14: using an existing open source pedestrian detection and tracking tool to perform human body detection and tracking on all video clips, obtaining a person detection and tracking frame, and removing video clips that do not contain people. In this embodiment, a total of 170,000 valid video clips are obtained; S15: Annotate 170,000 video clips using manual fine-grained annotation and automatic global semantic annotation, specifically: S151: First, perform manual fine-grained labeling: on the basis of obtaining the person detection tracking frame, label each tracked person with the group number and the corresponding group behavior category to which it belongs, and use the minimum rectangular bounding box formed by the persons with the same group number as the location information of the group; S152: Based on manual fine-grained annotation, automatic global semantic annotation is performed to fully utilize the semantic information of the existing image-text multimodal large model: For a video clip to be labeled , if it is determined that j group behavior category, it is marked as j Strings, according to j Total number of strings and group behavior categories A Automatically generated ( A - j ) additional strings representing group behavior categories that are not present in the video clip.

[0026] In this embodiment, the video clip to be marked There are three behaviors in , namely, queuing, chatting and outliers. j =3, the annotation result is 3 strings, namely "This video has a group of people queuing up", "This video shows a group of people chatting" and “There are outliers in this video.” Based on these three annotation results, an additional nine character strings can be automatically generated based on the behavior category data, that is, the defined group behavior categories, which are 12 categories in this embodiment: "This video does not contain a fighting group," "This video does not contain a dancing group", "This video does not contain a checkout group", "This video does not contain a group of people eating," "This video does not contain a learning (working) group", "This video does not contain a group of people taking photos," "This video does not contain a group of people playing ball." "This video does not contain a group of people walking," "This video does not contain a group of runners", which shows that for each video clip Quickly obtain 12 string annotations; S153: Use 12 strings to identify video clips Labeling, get triple labeling results <video ,text , logo >. The text Is Automatic conversion is obtained, in this embodiment , Video All defined group behavior categories exist in 1 or 0, respectively, indicating the text Affirmation and negation, i.e. signs 1 means video Text exists in The group behavior category represented by 0 means video Text does not exist in The triplet annotation results of this embodiment are shown in the following table:

[0027] The fast global semantic annotation provided in this embodiment can quickly generate a character string describing the content of a video clip. The generated character string not only includes the existing group behavior categories, but also clearly points out the behavior types that are not included, and can automatically generate a large number of <video, text, logo> ternary annotation results, which greatly enriches the diversity of training data and supports model training under open vocabulary settings.

[0028] S16: 170,000 video clips are divided into training set, validation set and test set. In order to meet the open vocabulary setting, the group behavior categories in the test set are restricted to not appear in the training set during the division, but the group behavior categories in the training set and the validation set are consistent. It can still work effectively when there is basically no overlap between the group behavior categories in the test set or actual use scenarios and the behavior categories in the training set, which improves the generalization ability of group behavior detection in new monitoring scenarios.

[0029] S2: Figure 2 As shown in the figure, an image-text encoder based on the video-text multimodal encoding and decoding model is constructed and trained. The specific process is as follows: S21: For Video Each frame, in this embodiment, has 75 frames in total. The Swin-B structure in the CLIP pre-training model commonly used in the field of image feature extraction is used as an image encoder to extract the image features of each frame. Here, the parameters of the CLIP pre-training model Swin-B are optimized and fine-tuned in the training stage to obtain a visual feature representation that is more conducive to group behavior detection. The image feature dimension of each frame is 1*512, that is, the video Expressed as , The dimension is 75*512; In order to reduce the consumption of feature calculation resources, the feature sequence After a fully connected layer, it is mapped into a feature with a dimension of 75*256 , in this embodiment, the fully connected layer here consists of 256 neurons; feature After a tangent linear transformation layer, the feature size is 75*256. ;feature After a linear mapping layer and a Softmax layer, a weight vector with a dimension of 75*1 and a value range of 0~1 is obtained. , the value of which indicates the weight of the corresponding frame feature in the final video feature representation of the entire video; Will and Multiply to get the video feature representation after frame fusion ; S22: For text , using the BERT structure in the CLIP pre-training model commonly used in the field of text feature extraction as a text encoder to extract text The feature representation , the feature dimension is 1*512. Here, the parameters of BERT are optimized and fine-tuned in the training phase, aiming to obtain text feature representation that is more conducive to group behavior detection; S23: Construct a video-text multimodal encoding and decoding model to represent video features and text feature representation Splicing along the feature channel dimension to obtain a video-text multimodal fusion feature with a dimension of 1*768; S24: The fused feature is decoded by a fully connected layer and a Sigmoid layer to obtain a probability value , set as a flag The probability value is 1, then the sign The probability of being 0 is 1 - ; S25: According to the signs The calculation formula for the loss value of the video-text multimodal encoding and decoding model is obtained by using the probability of 1 and 0: ; S26: Since the group behavior detection task targeted by the present invention and the pre-trained image-text overall semantic similarity comparison learning task of the CLIP multimodal model belong to completely different task scenarios, updating the parameters of Swin-B and BERT without constraints will lead to the problem of knowledge forgetting, that is, forgetting the knowledge learned in the CLIP image-text overall semantic similarity task, losing the ability to represent group behavior concept features that do not exist in the training set, which is not conducive to open vocabulary group behavior detection.

[0030] Therefore, the present invention sets a regularization term to constrain the parameter update process of the visual encoder and the text encoder, specifically: define the regularization constraint loss function for: ,in K represents the total number of parameters in the image encoder Swin-B and the text encoder BERT, k Indicates k parameters; and Respectively represent The current fine-tuned updated parameters and the original pre-trained parameters. Indicates the first The importance of maintaining the original pre-training task knowledge memory is calculated as follows: ,in, represents the number of all <video, text, logo> triplets in the training set, Indicates training samples; S27: Define the total training loss function , and use the gradient descent method to optimize the video-text multimodal encoding and decoding model on the training set so that the validation set Minimum, optimized and fine-tuned final image encoder and text encoder for the open vocabulary group behavior detection task in indoor security surveillance video scenes.

[0031] S3: Figure 3 As shown, an open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model is constructed and trained, and parameter optimization is performed to determine the open vocabulary group behavior detection model. The specific process is as follows: S31: Process 75 consecutive video clips, and perform each video clip according to step S1. The following information is obtained: Information 1: 75 frames of image data, recorded as ; Information 2: The minimum rectangular bounding box of the area where all individuals are located in each frame, the group number of the individual and the group behavior category corresponding to the group, recorded as ,in Indicates the video clip The total number of individuals in Indicates Individuals, Indicates Individuals in the video No. n The center coordinates and width and height of the minimum rectangular bounding box of the frame, that is, The individual in n The timing position information of the frame, in this embodiment , if If the frame does not contain the individual, The value of Indicates Individuals in video clips The group number to which it belongs, Indicates Individuals in video clips The behavioral category of the group in which they are located; In this embodiment, m Individuals throughout the video clip The expression in is: ; Information 3: Video clips Corresponding 12 groups <Video ,text , logo >Triplet labeling results.

[0032] S32: Design two encoders and decoders consisting of a standard 4-layer Transformer structure. The output of the encoder is used as the input of the decoder to build a multimodal cross-attention model of video-head position information; S33: 75 frames of image data The optimized image encoder Swin-B calculates and outputs 75 frames of feature representation , each frame dimension is 1*256; M The individual position embedding representation sequence obtained by the fully connected layer (consisting of 256 neurons and a RELU nonlinear mapping function) is , M For Video The total number of individuals in In this embodiment , ; S34: The feature representation and individual position embedding representation sequence is input into the encoder of the video-individual position information multimodal cross attention model, that is, the total input feature of the encoder is ; S35: Decoder output M The corresponding individual M The group belonging embedding representation vector and M The group behavior type embedding representation vector, that is, the output of the decoder is composed of 2 M vector representation, denoted by ,in ( ) represent the Each individual corresponds to a group belonging embedding representation vector and a group behavior category representation vector; S36: Use clustering loss function and classification loss function as supervisory signals to guide the optimization of the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model, specifically: S361: Since different video clips contain different numbers of groups of video clip meanings, in order to enable the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model to be applicable to all video clips to achieve end-to-end training optimization effect, this embodiment defines that a video clip contains a maximum of 6 groups, namely , define the clustering loss function The calculation formula is:

[0033] Clustering loss function Constraint Video The group belonging embedding representation vectors of all individuals belonging to the same group are consistent; S362: Through classification loss function Constrain the embedding representation vector of the group behavior category corresponding to each individual and the vector representation based on the text description of the group behavior category The closest ones are: First, the 12 sentences are trained through the text encoder BERT based on the image-text feature extractor pre-training step of the video-text multimodal codec model, that is, the optimized text encoder BERT, to obtain 12 group behavior category texts. Vector representation of description , and then embed the group behavior category of each individual output by the encoder , and calculate its inner product with each group behavior category text Vector representation of description (In this embodiment ), so that when When the similarity is close to 1, When the similarity is close to 0, Indicates The behavior category of the group to which each individual belongs, and the classification loss is defined The calculation formula is: ; S363: Under the constraint of , the parameter optimization training of the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model is completed by the stochastic gradient descent method; S37: After the optimization is completed, a determined open vocabulary group behavior detection model is obtained.

[0034] The embodiment of the present invention processes video clips of continuous frames, combines the position information of each individual and the features of the video frames, and implements multimodal cross-attention modeling of video-individual position information through the Transformer structure. It can identify the group to which each individual in the video belongs and classify the behavior of each group, thereby achieving more accurate group behavior detection.

[0035] S4: Input the actual surveillance video into the determined open vocabulary group behavior detection model to obtain the behavior category of each group, specifically: S41: In practical applications, the indoor security surveillance video is first segmented with a step size of 75 frames. ; S42: Obtain video clips using the same pedestrian detection and tracking algorithm as in step S1 The temporal position information of all individuals in each frame: , , m Indicates a video clip Middle m Individuals, M represents the total number of individuals; S43: Obtaining the open vocabulary group behavior detection model determined in step S3 M The group belonging embedding representation vector of each individual and M The group behavior category embedding representation vector of individuals is obtained, that is, 2 M The vectors represent: ; S44: KMeans clustering algorithm (the number of cluster centers is set to 6) Clustering is performed, and individuals with the same group behavior category belong to the same group; S45: Finally, an inner product operation is performed on the group behavior category embedding representation vector corresponding to all individuals belonging to the same group and the vector representation based on the text description of the group behavior category, and the category corresponding to the maximum value of the inner product operation result is taken as the behavior category of each individual; In the same group, there are a Individuals, select a The most common behavior category among the individuals is taken as the final group behavior category of the group. a The behavior categories of each individual are different, so choose a The behavior category with the largest inner product calculation result among the individuals is taken as the final group behavior category of the group; For example: If the group k There are 5 individuals, namely , and their corresponding group behavior category embedding representation vectors are Each group behavior category embedding representation vector is combined with 12 vector representations based on the text description of the group behavior category. Perform an inner product operation and take the category corresponding to the maximum value of the inner product operation as the behavior category of the individual, recorded as [ ],in Indicates the category number, is the cosine similarity obtained by the inner product operation; statistics The mode in is used as the final group behavior category. If the behavior categories of the five individuals are not equal, choose The category corresponding to the maximum value is taken as the final group behavior category.

[0036] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open vocabulary group behavior detection method for indoor security surveillance video scenes, characterized in that: The following steps are involved: S1: Collect and process surveillance video data of indoor scenes, obtain valid video clips containing people, and combine manual fine-grained annotation and automatic global semantic annotation to obtain the ternary annotation results <video, text, logo> of each valid video clip; S2: Build and train an image-text encoder based on a video-text multimodal encoding and decoding model, specifically: According to the ternary annotation results, for each frame of the video and the corresponding text, the Swin-B structure of the CLIP pre-trained model is used as the image encoder and the BERT structure is used as the text encoder; the parameters of Swin-B and BERT are optimized in the training phase, and the regularization term is used to constrain the update of Swin-B and BERT parameters and determine the final image-text encoder; S3: Construct and train an open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model, and perform parameter optimization to determine the open vocabulary group behavior detection model; S4: Input the actual surveillance video into the determined open vocabulary group behavior detection model to obtain the behavior category of each group.

2. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 1 is characterized in that: The specific process of step S1 is: S11: Collect surveillance video data of indoor scenes; S12: Define various types of group behavior, including fighting, dancing, queuing, checking out, eating, studying or working, taking pictures, chatting, playing ball, walking, running, and loner; S13: setting the length of the video segment and dividing the monitoring data video to obtain a plurality of video segments; S14: using a pedestrian detection and tracking tool to perform human body detection and tracking on all video clips, obtaining a person detection and tracking frame, removing video clips that do not contain people, and obtaining the remaining valid video clips; S15: annotate the effective video clips using manual fine-grained annotation and automatic global semantic annotation to obtain a triplet annotation result <video, text, logo>; S16: Divide the valid video clips into a training set, a validation set, and a test set.

3. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 2 is characterized in that: The specific process of step S15 is: S151: Manual fine-grained labeling: Based on the obtained person detection and tracking frame, label each tracked person with the group number and the corresponding group behavior category, and the minimum rectangular bounding box formed by the persons with the same group number is used as the location information of the group; S152: Based on manual fine-grained annotation, automatic global semantic annotation is performed: for a video to be annotated , if it is determined that j group behavior category, it is marked as j Strings, according to j Total number of strings and group behavior categories A Automatically generated ( A - j ) additional strings, indicating the group behavior categories that are not present in the video clip; S153: Utilization A String to video Labeling, get triple labeling results <video ,text , logo >, where the text Video All defined group behavior categories exist in 1 or 0, respectively, indicating the text Affirmation and negation, i.e. signs 1 means video There is text in The group behavior category represented by 0 means video Text does not exist in Represents the category of group behavior.

4. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 2 is characterized in that: In step S16, when the valid video clips are divided into a training set, a validation set and a test set, the group behavior categories in the test set are restricted to not appear in the training set, but the group behavior categories in the training set and the validation set are consistent.

5. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 4 is characterized in that: The specific process of step S2 is: S21: For Video For each frame of the video, the Swin-B structure in the CLIP pre-trained model is used as the image encoder to extract the image features of each frame. The parameters of Swin-B are optimized and fine-tuned in the training phase. Represented as a feature sequence , according to the feature sequence Get the video Video feature representation ; S22: For text , using the BERT structure in the CLIP pre-trained model as a text encoder to extract text Text feature representation , BERT’s parameters are optimized and fine-tuned during the training phase; S23: Construct a video-text multimodal encoding and decoding model to represent video features and text feature representation Splicing along the feature channel dimension to obtain video-text multimodal fusion features; S24: Obtaining probability values ​​based on video-text multimodal fusion features , set as a flag The probability value is 1, then the sign The probability of being 0 is 1- ; S25: According to the signs Determine the loss value of the video-text multimodal encoding and decoding model for the probability of 1 and 0 ; S26: Setting the regularized constraint loss function , as a regular term to constrain the update process of the Swin-B parameters of the visual encoder and the BERT parameters of the text encoder; S27: Define the total training loss function , and use the gradient descent method to optimize the video-text multimodal encoding and decoding model on the training set so that the validation set Minimum, finalized image encoder and text encoder are obtained.

6. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 5, characterized in that: In step S21, according to the feature sequence Get the video Video feature representation The specific process is: feature sequence Mapped into features through a fully connected layer ;feature After a tangent nonlinear transformation layer, the features are obtained ;feature After a linear mapping layer and a Softmax layer, the weight vector is obtained , the value of which indicates the weight of the feature of the corresponding frame in the final video feature representation of the entire video; feature and the weight vector Multiply to get the video feature representation after frame fusion .

7. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 6, characterized in that: The specific process of step S3 is: S31: Get every video All frame image data, the temporal position information of all individuals in each frame, the group number of the individual, and the group behavior category corresponding to the group are recorded, and the video is obtained. Corresponding <Video ,text , logo >Triple labeling results; S32: Design two encoders and decoders consisting of a standard four-layer Transformer structure. The output of the encoder is used as the input of the decoder to build a video-individual position information multimodal cross-attention model; S33: Processing video with the optimized image encoder Swin-B All frames of the image are processed to obtain the feature representation of all frames, and a fully connected layer is used to process the temporal position information of all individuals in each frame to obtain the individual position embedding representation sequence; S34: The feature representations of all frames and the embedding representations of all individual positions in each frame are input into the encoder of the video-individual position information multimodal cross attention model; S35: The decoder outputs the group belonging embedding representation vector and group behavior category embedding representation vector corresponding to all individuals; S36: Using clustering loss function and classification loss function as supervision signals to guide the optimization of open vocabulary group behavior detection model based on video-individual position information multimodal cross-attention model; S37: After the optimization is completed, a determined open vocabulary group behavior detection model is obtained.

8. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 7, characterized in that: In step S31, obtain the video The specific process of obtaining the temporal position information of all individuals in each frame is as follows: using the pedestrian detection and tracking tool to obtain the minimum rectangular bounding box of the area where all individuals are located in each frame, Indicates The center point coordinates and width and height of the minimum rectangular bounding box of each individual m The temporal position information of each individual is expressed as ,in Indicates that the video The total number of individuals in n Video No. n frame.

9. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 8, characterized in that: The specific process of step S36 is: S361: Through clustering loss function Constraint Video The group belonging embedding representation vectors of all individuals belonging to the same group are consistent; S362: Processing Video with Optimized Text Encoder BERT Corresponding text , obtain the vector representation based on the text description of the group behavior category; Through the classification loss function Constrain the similarity between the embedding representation vector of the group behavior category corresponding to each individual and the vector representation based on the text description of the group behavior category to be close to 1; S363: Under the constraint of , the parameter optimization training of the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model is completed by the stochastic gradient descent method.

10. The open vocabulary group behavior detection method for indoor security monitoring video scenes according to claim 9, characterized in that: The specific process of step S4 is: S41: Segment the actual surveillance video to obtain video clips ; S42: Obtain video clips using pedestrian detection and tracking tools The temporal position information of all individuals in each frame; S43: Use the open vocabulary group behavior detection model based on the video-individual position information multimodal cross-attention model to obtain the group belonging embedding representation vectors and group behavior category embedding representation vectors of all individuals; S44: Cluster the group belonging embedding representation vectors of all individuals through a clustering algorithm, and individuals with the same group behavior category belong to the same group; S45: performing an inner product operation on the group behavior category embedding representation vector corresponding to all individuals belonging to the same group and the vector representation based on the text description of the group behavior category, and taking the category corresponding to the maximum value of the inner product operation result as the behavior category of each individual; In the same group, there are a Individuals, select a The most common behavior category among the individuals is taken as the final group behavior category of the group. a The behavior categories of each individual are different, so choose a The behavior category with the largest inner product operation result among the individuals is taken as the final group behavior category of the group.

Images