Target individual behavior description method based on parameter persistent evolution

Abstract

The invention discloses a target individual behavior description method based on parameter persistent evolution, which relates to the field of computer vision, and comprises the following steps: extracting frames from a video, and performing feature extraction on each frame of extracted image by using a pre-trained vision model to obtain frame features; detecting and tracking a target individual in the video by using a YOLOv7 algorithm in combination with a StrongSORT algorithm and an OsNet algorithm, and extracting individual features by using the visual model; a Deformable Transform deformable decoder is utilized to combine the individual features with the frame features, and a query feature is generated; and connecting the query features to a positioning head and a subtitle generation head, describing target individual behaviors, and performing model optimization by adopting a loss function. According to the method, the problem that behavior description cannot be carried out in the video by taking an individual as a unit in an existing description method is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for describing target individual behavior based on continuous parameter evolution. Background Art

[0002] In the field of computer vision and behavior analysis, the description of target individual behavior has wide applications in video surveillance, especially when analyzing multiple individuals in complex scenes. This technology can not only provide detailed situational analysis, but also accurately assess potential risks to ensure the safety and harmony of public places. Currently, video-level captioning methods cannot provide fine-grained descriptions of the specific behavior of each individual, making it difficult to accurately identify the individual. To address this challenge, a target individual behavior description method based on continuous parameter evolution is proposed.

[0003] Existing video-level description methods do not perform well in fine-grained behavior description. Many existing methods can only perform well on frame-level video description tasks, such as describing what happened in an entire video, or describing several events in a video in units of events. However, in actual scenarios, it is often necessary to provide fine-grained behavior descriptions of key targets with specific features, such as "the behavior of a person in a red jacket interacting with others at the entrance of a supermarket." These requirements place higher demands on the model's behavior description capabilities.

[0004] At present, most existing technologies mainly rely on deep learning methods for video behavior analysis, which usually include frame-level feature extraction, event detection, and description generation. Although these methods perform well in specific scenarios and tasks, they still have certain limitations when dealing with variable real-world scenarios. For example, whether describing the video or dividing the video into several events and describing each event, they are all based on videos provided by the Internet, not videos in real-world surveillance; secondly, the existing description methods cannot describe the behavior of individuals in the video, but can only describe the entire video or the events that occurred in the video. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for describing target individual behaviors based on continuous parameter evolution, which solves the problem that the existing description methods are unable to describe the behaviors of individuals in a video.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for describing target individual behavior based on continuous parameter evolution, comprising the following steps: S1: Extract frames from the video and use the pre-trained visual model to extract features from each frame to obtain frame features; S2: Use the YOLOv7 algorithm combined with the StrongSORT algorithm and the OsNet algorithm to detect and track the target individuals in the video, and use the visual model to extract individual features; S3: Using a Deformable Transformer deformable decoder to combine the individual features and the frame features to generate a query feature; S4: Connect the query features to the positioning head and the subtitle generation head to describe the target individual behavior, and use the loss function to optimize the model to complete the target individual behavior description based on the continuous evolution of parameters.

[0007] Furthermore, the S1 includes the following sub-steps: S11: uniformly sample video frames from the input video at a fixed frame rate; S12: extracting features from the video frame using a pre-trained visual model to obtain a feature map; S13: folding the spatial dimension of the feature map into one dimension to form a feature vector of a set size; S14: Input the feature vector as data into the Transformer encoder to obtain frame features.

[0008] Furthermore, S2 includes the following sub-steps: S21: Use the YOLOv7 algorithm combined with the StrongSORT algorithm and the OsNet algorithm to detect each individual in the video, and mark the bounding box of the detected individual in the video; S22: cropping the video frame according to the bounding box to extract a set of image regions containing only the target individual; S23: uniformly sample each target individual in the image region set, uniformly adjust the size, and input the adjusted image into the visual model for individual feature extraction; S24: The extracted individual features are pooled, and the format of the pooled features is converted through a fully connected layer to complete the extraction of individual features.

[0009] Furthermore, the Deformable Transformer in S3 adopts a multi-scale deformation attention mechanism, and the calculation formula of the multi-scale deformation attention mechanism is:

[0010] in, is a multi-scale deformation attention mechanism, represents the query feature vector currently being processed, Represents the query vector The corresponding standardized reference point position, represents the feature map, Indicates the scale level of the feature map, Indicates the value of the scale layer of the feature map, represents the number of sampling points around each query reference point, Indicates the value of the number of sampling points around each query reference point, Indicated by and Calculated sampling positions The attention weight, is the feature transformation matrix, It is in the feature map According to the sampling position after deformation Extracted features.

[0011] Furthermore, the sampling position The calculation formula is:

[0012] in, represents the conversion function, represents the query point, Indicates the offset.

[0013] Furthermore, the loss function in S4 is for:

[0014] in, represents the generalized IOU loss between the predicted time period and the actual time period, represents the classification loss between the predicted classification score and the actual label, represents the cross entropy loss between the predicted word probability and the actual value, Represents the weights of various losses.

[0015] The beneficial effect of the present invention is that the present invention proposes a target individual behavior description method based on continuous parameter evolution. The method can dynamically obtain the key behavior characteristics of the target individual in the video by utilizing a multi-head self-attention mechanism and a Deformable Transformer deformable encoder, and realize accurate individual behavior description through a continuously optimized parameter evolution algorithm.

[0016] In view of the problem that the existing technology is insufficient in the fine-grained description of individual behavior in complex video surveillance scenarios, the present invention effectively solves the stability and accuracy problems of detecting and tracking individuals, especially in the long-term video data processing, and performs well. The model uses YOLOv7 combined with StrongSORT and OsNet algorithms for individual detection and tracking, extracts features through pre-trained visual models, and uses a multi-scale deformation attention mechanism to achieve feature fusion. Experimental results show that this method is superior to existing methods in multiple key performance indicators and significantly improves the accuracy and detail of behavior description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of the method for describing target individual behaviors based on continuous evolution of parameters.

[0018] Figure 2 Schematic diagram of the target individual behavior description method based on continuous parameter evolution. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0020] like Figure 1 As shown, a method for describing target individual behavior based on continuous parameter evolution includes the following steps: S1: Extract frames from the video and use the pre-trained visual model to extract features from each frame to obtain frame features; S2: Use the YOLOv7 algorithm combined with the StrongSORT algorithm and the OsNet algorithm to detect and track the target individuals in the video, and use the visual model to extract individual features; S3: Using a Deformable Transformer deformable decoder to combine the individual features and the frame features to generate a query feature; S4: Connect the query features to the positioning head and the subtitle generation head to describe the target individual behavior, and use the loss function to optimize the model to complete the target individual behavior description based on the continuous evolution of parameters.

[0021] like Figure 2As shown, the technical solution used in the present invention includes the following main modules: The model mainly includes two parts. The first part is to process feature encoding; the process first extracts frames from the video, and then uses the pre-trained visual model to extract frame-level features. The extracted features and their corresponding positions are embedded and input into the deformation encoder. Then, YOLOv7 is used with StrongSORT combined with OsNet to detect and track individuals in the video, and the video is cropped and frames are extracted according to the bounding box in the time period when each individual appears and disappears. Frames containing only individual information will use the same pre-trained visual model for feature extraction, and the features of each individual obtained are input into the Deformable Transformer deformable decoder as a query. The second part is decoding, in which the features of the individual in the video are extracted and combined with the frame features in the encoder, and then put into the decoder. The decoder outputs query features, which are connected to the positioning head and the subtitle generation head to generate subtitles for each individual. The loss function includes positioning loss and subtitle loss, which are used to calculate the time when the individual appears and disappears, and to compare the generated subtitles with the real subtitles.

[0022] The S1 includes the following sub-steps: S11: uniformly sample video frames from the input video at a fixed frame rate; S12: extracting features from the video frame using a pre-trained visual model to obtain a feature map; S13: folding the spatial dimension of the feature map into one dimension to form a feature vector of a set size; S14: Input the feature vector as data into the Transformer encoder to obtain frame features.

[0023] In one embodiment of the present invention, a consistent frame rate (such as ) uniformly samples the video, and each frame image is represented as ,in Represents a frame in a video, which is an image. represents the size of the image, and 3 represents the three color channels of the image ( aisle), Indicates the height of the image. Represents the width of the image. Features are extracted through the I3D model and expressed as , Indicates one of the characteristics, represents the extracted feature dimension, represents the number of feature channels, represents the height of the feature map, Represents the width of the feature map. , , . Process the video frame sequence to obtain the frame feature set , the feature vector of each frame is expressed as Then, use Convolution changes the channel dimension from Reduce to , generate a new feature map In order to meet the input requirements of the encoder, The spatial dimension of is folded into one dimension, forming feature map.

[0024] The S2 includes the following sub-steps: S21: Use the YOLOv7 algorithm combined with the StrongSORT algorithm and the OsNet algorithm to detect each individual in the video, and mark the bounding box of the detected individual in the video; S22: cropping the video frame according to the bounding box to extract a set of image regions containing only the target individual; S23: uniformly sample each target individual in the image region set, uniformly adjust the size, and input the adjusted image into the visual model for individual feature extraction; S24: The extracted individual features are pooled, and the format of the pooled features is converted through a fully connected layer to complete the extraction of individual features.

[0025] Use YOLOv7 combined with StrongSORT and OsNet to detect and track the target individuals in the video. YOLOv7 is used to detect the position of the individual in the video frame, StrongSORT is used to track the detected individuals in the video, and OsNet is a deep learning model specially designed for re-identification (reID). It can identify unique features from the individual images extracted during the tracking process. These features help StrongSORT deal with occlusion and interaction between individuals encountered in complex scenes to ensure the accuracy of tracking. Frames with individual bounding boxes are obtained through this algorithm , and cut out individual regions according to the bounding box to obtain the region set In order to accurately capture the action characteristics of each participant during the interaction, frame feature extraction is performed on each individual and uniform sampling is performed. Frame, input size is , extract individual features through I3D model and generate feature set .

[0026] The extracted individual feature vector is used as the query and input into the multi-head self-attention layer of the Deformable Transformer deformable decoder to calculate the attention weights between the features to capture the time-sensitive characteristics of each individual. Figure 2 The Deformable Transformer decoder structure in the paper uses the Multi-Scale Deformable Attention Mechanism (MSDAtt), which is an attention mechanism designed specifically for image and video processing. It extends the self-attention mechanism in the standard Transformer model to more effectively handle the multi-scale features and deformation alignment problems of visual data. In visual tasks, especially those involving large changes in the size, shape, and posture of objects, the processing efficiency and effect are improved by focusing on the sparse sampling points in the image feature map. The Multi-Scale Deformable Attention Mechanism (MSDAtt) solves the problem of slow convergence of the Transformer when processing image feature maps by focusing on a set of sparse sampling points around the reference point.

[0027] The Deformable Transformer in S3 adopts a multi-scale deformation attention mechanism, and the calculation formula of the multi-scale deformation attention mechanism is:

[0028] in, It is a multi-scale deformation attention mechanism used to calculate the sum of weighted features when processing video or image data. represents the query feature vector currently being processed, Represents the query vector The corresponding standardized reference point position, represents the feature map, Indicates the scale level of the feature map, The multi-scale deformation attention mechanism (MSDAtt) improves its ability to adapt to different object sizes and perspectives by processing feature maps of multiple scales. Represents the number of sampling points around each query reference point. These sampling points help the model capture more local details. Indicates the value of the number of sampling points around each query reference point, Indicated by and Calculated sampling positions The attention weights determine the importance of the features of different sampling points in the final output. is the feature transformation matrix, which is used to linearly transform the features at the sampling position to better match the query vector Combined, It is in the feature map According to the sampling position after deformation Extracted features.

[0029] The sampling position The calculation formula is:

[0030] in, represents the conversion function, which converts the query point Map to This part is responsible for adjusting the query point according to the position of the reference point in the feature hierarchy to adapt to feature maps of different scales. represents the query point, Represents the offset, which adjusts the position of the query point based on the learned offset parameter to more accurately focus on the relevant features in the image. This offset is learned through the network and can adapt to different objects and scenes.

[0031] The above sampling position formula is a key part of the multi-scale deformation attention mechanism (MSDAtt), which is used to calculate the location of the sample points after deformation. This formula is an important part of the feature extraction process of videos or images because it helps the model focus on key areas in the image, especially those areas whose shape and position may change over time.

[0032] The decoding network in the present invention consists of a Deformable Transformer decoder, a localization head, and a captioning head. The task of the decoder is to generate behavior descriptions and location information based on given individual features. The Deformable Transformer decoder is based on the multi-scale deformable attention mechanism (MSDAtt). By focusing on the sparse sampling points around a specific reference point, it solves the slow convergence problem of the traditional self-attention mechanism when processing image feature maps. The input received by the decoder includes multiple query features and its corresponding reference point , these query features interact in each layer of the decoder. MSDAtt performs attention calculations on the feature map at multiple sampling points at different scales and outputs a context vector , these vectors are further processed to generate descriptions of individual behaviors. The localization head is used to predict the bounding box of each individual in the video and the start and end positions of the time segment. The caption generation head uses the generated context vectors Generate a natural language description of individual actions. Each generated caption sequence It is composed of multiple words, representing the behavior sequence of the target individual in the video.

[0033] During the training process, the model generates The behavior description of each individual. In order to ensure the consistency between the predicted events and the actual data, the Hungarian algorithm is used to determine the optimal binary matching result. The matching loss is composed of the weighted sum of the generalized IOU loss, the classification loss, and the caption loss.

[0034] The loss function in S4 for:

[0035] in, Represents the generalized IOU loss between the predicted time period and the actual time period, which is used to measure the accuracy of positioning. Represents the classification loss between the predicted classification score and the actual label, which is used to measure the accuracy of the classification. Represents the cross entropy loss between the predicted word probability and the actual value, which is used to measure the accuracy of subtitle generation. Represents the weights of various losses.

[0036] The trained model can be used in various video surveillance scenarios to monitor and analyze the behavior of individuals in the video in real time. The model can accurately capture and describe individual behaviors in complex scenarios, providing effective technical support for security monitoring, intelligent traffic management and other fields.

[0037] During the experiment, the monitoring dataset was randomly divided into training set, validation set and test set. In order to evaluate the subtitle generation performance of the proposed method, four indicators, BLEU4, METEOR, CIDER and ROUGE-L, were used. These indicators were evaluated by calculating the average matching accuracy of the generated subtitles and the benchmark real subtitles at different IOU thresholds (0.3, 0.5, 0.7, 0.9). However, these indicators do not fully cover the narrative quality of subtitles or the performance of subtitles in the overall coverage of video stories. Therefore, SODA_c was further used for comprehensive evaluation.

[0038] In the tracking and detection stages, we experimented with YOLOv5 combined with DeepSORT and YOLOv7 combined with StrongSORT and OsNet. For the subtitle generation task of the UCF-crime dataset, we tested three different feature extraction methods: C3D, I3D, and CLIP. We used the C3D model pre-trained on the Sports1M dataset to extract frame-level features; the I3D model used a version pre-trained on the Kinetics dataset; and CLIP used the pre-trained model officially released by OpenAI.

[0039] In terms of model structure, a two-layer DeformableTransformer with multi-scale (four-level) deformable attention is used. The hidden dimension of the Deformable Transformer is 512 in the MSDAtt layer and 2,048 in the feedforward layer. After each target person is detected and tracked, its features are extracted using the corresponding feature extraction method and input into the Deformable Transformer as a query. Finally, LSTM is used as the caption generator, and the hidden dimension of LSTM is 512.

[0040] During model training, the Adam optimizer was used and the initial learning rate was set to 5e -5 , each mini-batch processes one video. The entire training process is completed by two NVIDIA A100 GPUs. Compared with the baseline method, the present invention achieves more accurate behavior description.

[0041] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A method for describing target individual behavior based on continuous parameter evolution, characterized in that: The following steps are involved: S1: Extract frames from the video and use the pre-trained visual model to extract features from each frame to obtain frame features; S2: Use the YOLOv7 algorithm combined with the StrongSORT algorithm and the OsNet algorithm to detect and track the target individuals in the video, and use the visual model to extract individual features; S3: Using a Deformable Transformer deformable decoder to combine the individual features and the frame features to generate a query feature; S4: Connect the query features to the positioning head and the subtitle generation head to describe the target individual behavior, and use the loss function to optimize the model to complete the target individual behavior description based on the continuous evolution of parameters.

2. The target individual behavior description method based on continuous parameter evolution according to claim 1 is characterized in that: The S1 includes the following sub-steps: S11: uniformly sample video frames from the input video at a fixed frame rate; S12: extracting features from the video frame using a pre-trained visual model to obtain a feature map; S13: folding the spatial dimension of the feature map into one dimension to form a feature vector of a set size; S14: Input the feature vector as data into the Transformer encoder to obtain frame features.

3. The target individual behavior description method based on continuous parameter evolution according to claim 1 is characterized in that: The S2 includes the following sub-steps: S21: Use the YOLOv7 algorithm combined with the StrongSORT algorithm and the OsNet algorithm to detect each individual in the video, and mark the bounding box of the detected individual in the video; S22: cropping the video frame according to the bounding box to extract a set of image regions containing only the target individual; S23: uniformly sample each target individual in the image region set, uniformly adjust the size, and input the adjusted image into the visual model for individual feature extraction; S24: The extracted individual features are pooled, and the format of the pooled features is converted through a fully connected layer to complete the extraction of individual features.

4. The target individual behavior description method based on continuous parameter evolution according to claim 1 is characterized in that: The Deformable Transformer in S3 adopts a multi-scale deformation attention mechanism, and the calculation formula of the multi-scale deformation attention mechanism is: in, is a multi-scale deformation attention mechanism, represents the query feature vector currently being processed, Represents the query vector The corresponding standardized reference point position, represents the feature map, Indicates the scale level of the feature map, Indicates the value of the scale layer of the feature map, represents the number of sampling points around each query reference point, Indicates the value of the number of sampling points around each query reference point, Indicated by and Calculated sampling positions The attention weight, is the feature transformation matrix, It is in the feature map According to the sampling position after deformation Extracted features.

5. The target individual behavior description method based on continuous parameter evolution according to claim 4 is characterized in that: The sampling position The calculation formula is: in, represents the conversion function, represents the query point, Indicates the offset.

6. The target individual behavior description method based on continuous parameter evolution according to claim 1 is characterized in that: The loss function in S4 for: in, represents the generalized IOU loss between the predicted time period and the actual time period, represents the classification loss between the predicted classification score and the actual label, represents the cross entropy loss between the predicted word probability and the actual value, Represents the weights of various losses.

Images