A time sequence event detection method for multi-event instance video

Abstract

This invention discloses a method for temporal event detection in multi-event instance videos. The method includes the following steps: 1. Acquiring video data; 2. Constructing a video feature extraction model; 3. Using the video feature extraction model to extract features, obtaining video frame feature vector groups; 4. Correcting the annotation files corresponding to the video frame feature vectors, obtaining frame-level annotations and segment-level annotations; 5. Constructing an event boundary prediction model; 6. Constructing a multi-instance discrimination model; 7. Constructing a video segment interaction model; 8. Training the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model to obtain a joint model; 9. Calling the joint model to detect target events in the video. This invention improves the accuracy of temporal event detection through video feature modeling and a contrastive learning strategy, solving the problem of easy misjudgment and missed judgment in current complex video scenarios (multi-instance problems), and can significantly improve the performance of temporal event detection.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a computer video analysis method, specifically a method for detecting temporal events in multi-event instance videos. Background Technology

[0002] With the rapid development of mobile devices and the internet, thousands of hours of video are uploaded to the internet every second. This massive amount of video information far exceeds the processing capacity of traditional manual systems. Therefore, video content analysis methods have attracted considerable interest from both academia and industry. Temporal event detection is a hot topic in video content analysis, focusing on locating event instances (i.e., determining the start and end times of events) within untrimmed video based on video content. This task holds significant potential in video retrieval, video highlight generation, and video surveillance. However, due to the complexity of video content, using artificial intelligence algorithms for event detection remains a challenging problem in the field.

[0003] While existing temporal event detection algorithms have achieved good performance in simple video scenarios (single-event instance videos), their performance on multi-event instance videos is far from satisfactory. The main challenge in multi-event video event detection is the indistinct background and blurred boundaries between multiple event instances. This often causes algorithms to detect multiple events as a single event, resulting in a large number of missed or false detections of target events in the video. This limits the deployment and application of algorithms in complex everyday videos. Therefore, the multi-event instance problem is a major challenge in the field of video temporal event detection. Summary of the Invention

[0004] To overcome the aforementioned shortcomings of existing technologies, this invention proposes a temporal event detection method for multi-event instance videos. This invention improves the accuracy of temporal event detection through video feature modeling and a contrastive learning strategy, solving the problems of misjudgment and missed detection in complex video scenarios (multi-instance problems), and significantly improving the performance of temporal event detection.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for temporal event detection in multi-event instance videos includes the following steps:

[0007] Step 1: Obtain video data, which includes a file containing annotations for the start and end times of each video and the target events contained in each video;

[0008] Step 2: Construct a video feature extraction model based on action recognition;

[0009] Step 3: Perform video preprocessing on the video data obtained in Step 1, and then use a video feature extraction model to extract features to obtain the video frame feature vector group F;

[0010] Step 4: Correct the annotation file corresponding to the video frame feature vector group F obtained in Step 3, aligning the annotation file with the position on the feature vector group to obtain the frame-level annotation G suitable for the feature vectors. b With fragment-level annotation G s ;

[0011] Step 5: Construct an event boundary prediction model with an encoder-decoder structure as its backbone. The specific steps are as follows:

[0012] Step 51: Obtain input data. The input to the event boundary prediction model comes from the video frame feature vector group F obtained in Step 3.

[0013] Step 5.2: Construct a deep learning network structure of encoder-decoder. Provide the encoder with a set of video feature vectors F. After obtaining the set of video feature vectors F, the encoder performs information interaction between the feature vectors to generate global semantic information Fi of the set of video feature vectors F. enc The decoder has two inputs: the video feature vector set F and the global semantic information F. enc Through global semantic information F enc Update the video feature vector set F to generate F that is more representative of the beginning and end of the event. enc ;

[0014] Step 5.3: Construct a boundary classifier. The boundary classifier uses the output information F of the decoder. dec Predicting the boundaries of video events yields the boundary confidence score C. b ;

[0015] Step Six: Construct a multi-instance discrimination model based on contrastive learning selection. The specific steps are as follows:

[0016] Step 61: Perform boundary sampling. The specific operation is as follows:

[0017] a. Define the boundary region of the event instance, given an event instance. Let the start time and end time of the event be the start time and end time, respectively. Then, the start and end regions of the event are defined as follows:

[0018]

[0019] in, n represents the nth event instance, and δ is a hyperparameter that controls the size of the region;

[0020] b. Sample the video frame feature vector group F obtained in step three to form positive and negative sample pairs, and sample the feature vectors in the start and end regions of the same instance respectively. Forming positive sample pairs P Feature vectors were sampled from the start and end regions of the boundary regions of different instances. Forming negative sample pairs N ;

[0021] Step 62: Construct a contrastive learning scheme and optimize the multi-instance discrimination model. Where f() is the similarity calculation function, To measure distance;

[0022] Step 63: Update the embedding representation of the feature vectors of the boundary region using the optimization objective, making the embedding representations of the same instance close together and the embedding representations of different instances far apart. Name the updated feature vector group the instance-level feature vector group, which includes the event start vector group F. s Event End Vector Group F e ;

[0023] Step 7: Construct a video clip interaction model. The specific steps are as follows:

[0024] Step 71: The input to the video segment interaction model is the video frame feature vector group F. A sliding window operation is performed on the video time sequence to generate video segments of different scales.

[0025] Step 72: For the multi-scale segments generated in Step 71, the video frame feature vector group F obtained in Step 3 is unified to the same length through interpolation sampling to obtain multi-scale segment-level features P. f ; Construct a deep learning model to perform information interaction on fragment-level features;

[0026] Step 73: Construct a paragraph classifier and calculate the confidence C of the target event or background contained in each paragraph. s ;

[0027] Step 8: Using the annotation files obtained in Step 4, jointly train the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model to obtain a joint model of the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model. The specific training steps are as follows:

[0028] Step 81: Train the event boundary prediction model using frame-level labeled files, and define the loss function 1: L1 = L(C b G b ), and calculate the loss 1;

[0029] Step 82: Train the event video clip interaction model using the segment-level labeled files, and define loss function 2: L2 = L(C s G s ), and calculate the loss 2;

[0030] Step 83: Train a multi-instance discrimination model using a contrastive learning strategy, and define loss function 3: And calculate the loss as 3;

[0031] Step 84: Calculate the cumulative losses 1, 2, and 3 to obtain the total loss as L. all =λ1L1+λ2L2+λ3L3, where {λ1,λ2,λ3} are used to control the training weights of the three models, and the three models are trained by the gradient descent algorithm;

[0032] Step Nine: Use the joint model trained in Step Eight to perform target event detection on the video. The specific steps are as follows:

[0033] For any video segment, the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model generate boundary confidence, instance confidence, and segment confidence, respectively. The joint model trained in step eight uses the product of these three and sorts them to output the final video detection result.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] 1. By interacting with multi-level feature information, the problem of missed detection and false detection in multi-event instance videos is solved from the information level of video segments;

[0036] 2. By using a multi-instance discrimination model, the problem of missed detection and false detection in multi-event instance videos is solved at the video boundary level;

[0037] 3. The test results have higher recall and precision;

[0038] 4. It reduces the problem of inaccurate boundary localization in video event detection algorithms in complex video scenes, and avoids the omission and false judgment of the algorithm in continuous complex events, providing a new approach for subsequent video event detection methods. Attached Figure Description

[0039] Figure 1 This is a basic flowchart of the timing event detection method of the present invention;

[0040] Figure 2 This is a flowchart for constructing a boundary prediction model;

[0041] Figure 3 This is a flowchart of constructing a multi-instance discrimination model;

[0042] Figure 4 This is a flowchart for constructing an interactive model for video clips;

[0043] Figure 5 This is a concept diagram of video output. Detailed Implementation

[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0045] This invention provides a method for temporal event detection in multi-event instance videos, such as... Figure 1 As shown, the method includes the following steps:

[0046] Step 1: Obtain video data.

[0047] In this step, the video data includes a label file containing the start and end times of each video and the target events contained in each video.

[0048] In this step, there are multiple ways to obtain video data, such as: (1) taking photos or videos using electronic devices; (2) downloading videos via the Internet or other means; and (3) obtaining videos from other video providers.

[0049] Step 2: Construct a video feature extraction model based on action recognition to obtain the RGB and optical flow feature information of the video.

[0050] In this step, the video feature extraction network model typically uses an action recognition algorithm with pre-trained parameters, such as I3D, TSN, SlowFast, etc.

[0051] Step 3: Perform video preprocessing on the video data obtained in Step 1, and then use a video feature extraction model to extract features to obtain the video frame feature vector group F.

[0052] In this step, video preprocessing includes, but is not limited to, video cutting, video frame extraction, grayscale processing, and foreground extraction.

[0053] Step 4: Correct the annotation file corresponding to the video frame feature vector group F obtained in Step 3 to obtain the frame-level annotation G suitable for the feature vectors. b With fragment-level annotation G s .

[0054] In this step, the correction method is as follows: Align the positions of the annotation file with those of the feature vector group. This is because the annotation file is a video-level annotation, such as from minute to second. This corresponds to vector number n in the feature vector group.

[0055] Step 5: Construct an event boundary prediction model with an encoder-decoder structure as the backbone, and predict the event boundaries by considering the correlation between video frames.

[0056] In this step, the flowchart for constructing the model is as follows: Figure 2 As shown, the specific steps are as follows:

[0057] (1) Obtain input data. The input of the event boundary prediction model comes from the video frame feature vector group F obtained in step three.

[0058] (2) Construct a deep learning network structure of encoder-decoder, where the encoder aims to model the global information of the video using a set of video frame feature vectors F. Specifically, we provide the encoder with a set of video frame feature vectors F. After obtaining the set of video frame feature vectors F, the encoder interacts with the feature vectors within it to generate the global semantic information F of the set of video frame feature vectors F. enc (Also a set of feature vectors). The decoder has two inputs: the video frame feature vector set F and the global semantic information F. enc The aim is to utilize global semantic information F enc Update the video frame feature vector set F to generate F that is more representative of the beginning and end of the event. enc .

[0059] (3) Construct a boundary classifier, which uses the output information F of the decoder. dec Predict the boundaries of video events (predict each F) dec The feature vectors in the video are used as the confidence of the video event boundary to obtain the boundary confidence C. b .

[0060] Step 6: Construct a multi-instance discrimination model based on contrastive learning selection to prevent misjudgment and incorrect judgment of multi-instance videos by learning the differences in instance boundaries.

[0061] In this step, the process of building a multi-instance discrimination model is as follows: Figure 3 As shown, the specific steps are as follows:

[0062] (1) Perform boundary sampling, the specific operation is as follows:

[0063] a. Define the boundary region of the event instance, given an event instance. Let the start time and end time of the event be the start time and end time, respectively. Then, the start and end regions of the event can be defined as follows:

[0064]

[0065] in, n represents the nth event instance, and δ is a hyperparameter that controls the size of the region.

[0066] b. Sample the video frame feature vector group F obtained in step three to form positive and negative sample pairs, and sample the feature vectors in the start and end regions of the same instance respectively. Forming positive sample pairs P Feature vectors were sampled from the start and end regions of the boundary regions of different instances. Forming negative sample pairs N .

[0067] (2) Construct a contrastive learning scheme, and build the model with the following optimization objective: Where f() is the similarity calculation function, To measure distance, f() is preferably cosine similarity. This optimization objective updates the embedding representations of the feature vectors in the boundary region, making the embedding representations of the same instance similar and the embedding representations of different instances far apart. We name the updated feature vector set the instance-level feature vector set (including the event start vector set F). s Event End Vector Group F e This enables the model to distinguish the boundaries of continuous events, avoiding false positives and false negatives.

[0068] Step 7: Construct a video clip interaction model. Enhance the semantic representation of video clips by modeling the information between multi-scale video clips, so that the model can learn easily confused local details and global information in multi-instance videos and prevent misjudgment and incorrect judgment of multi-instance videos.

[0069] In this step, the video clip interaction model achieves accurate video event detection by analyzing the information exchange between multi-scale video clips. For example... Figure 4 As shown, the specific construction steps of the video clip interaction model are as follows:

[0070] (1) The input to the video segment interaction model is the video frame feature vector group F. A sliding window operation is performed on the video time sequence to generate video segments of different scales. The detailed steps of the sliding window operation are as follows:

[0071] a. Set up a series of window groups of different sizes. Where D is the number of windows and w is the window size.

[0072] b. Set different step sizes according to different window sizes. For each window, the step size is:

[0073] c. Based on the window size and sliding window, perform sliding window operations on the video time sequence to generate video segments of different scales.

[0074] (2) For the generated multi-scale segments, the video frame feature vector group F obtained in step three is unified to the same length through interpolation sampling to obtain multi-scale segment-level features P. f A deep learning model is constructed, which then performs information interaction on fragment-level features. Preferably, the model uses a self-attention-based structure to complete information interaction between multiple fragments.

[0075] (3) Construct a paragraph classifier to calculate the confidence C of the target event or background contained in each paragraph. s This model can distinguish the background present in continuous events, avoiding false positives and false negatives.

[0076] Step 8: Use the annotation file obtained in Step 4 to jointly train the event boundary prediction model, multi-instance discrimination model and video segment interaction model constructed above, and obtain the joint model of the event boundary prediction model, multi-instance discrimination model and video segment interaction model.

[0077] In this step, the event boundary prediction model is trained using frame-level labeled files, and loss function 1 is defined as: L1 = L(C b G b ), and calculate loss 1; train the event video segment interaction model using segment-level labeled files, and formulate loss function 2: L2 = L(C s G s ), and calculate loss 2; train a multi-instance discrimination model using a contrastive learning strategy, and formulate loss function 3: Calculate loss 3; calculate cumulative loss 1, loss 2, and loss 3 to obtain the total loss as L. all =λ1L1 + λ2L2 + λ3L3, where {λ1, λ2, λ3} are used to control the training weights of the three models. The three models are trained using the gradient descent algorithm, specifically: L is trained by gradient descent... all The loss is backpropagated to update the parameters of the three models mentioned above. The training stops when L... all It converges to a minimum value and remains at that minimum value during subsequent training, without decreasing further. Loss function 1 and loss function 2 include, but are not limited to, common regression or classification losses such as cross-entropy loss and L2-loss; loss function 3 includes, but is not limited to, margin loss, triplet loss, and softmax loss.

[0078] Step 9: Use the joint model trained in Step 8 to detect target events in the video.

[0079] In this step, for any video segment, the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model generate boundary confidence, instance confidence, and segment-level confidence, respectively. The joint model trained in step eight uses the product of these three and sorts them to output the final video detection result, such as... Figure 5 As shown.

Claims

1. A method for detecting temporal events in multi-event instance videos, characterized in that... The method includes the following steps: Step 1: Obtain video data, which includes a file containing annotations for the start and end times of each video and the target events contained in each video; Step 2: Construct a video feature extraction model based on action recognition; Step 3: Perform video preprocessing on the video data obtained in Step 1, and then use a video feature extraction model to extract features, obtaining video frame feature vector groups. ; Step 4: Process the video frame feature vector group obtained in Step 3 The corresponding annotation file is corrected, and the position of the annotation file is aligned with the feature vector group to obtain frame-level annotations suitable for the feature vectors. With fragment-level annotation ; Step 5: Construct an event boundary prediction model with an encoder-decoder structure as its backbone. The specific steps are as follows: Step 51: Obtain input data. The input to the event boundary prediction model comes from the video frame feature vector group obtained in Step 3. ; Step 52: Construct an encoder-decoder deep learning network structure to provide the encoder with video feature vector sets. The encoder obtains the video feature vector group. Then, information is exchanged between the feature vectors to generate a video feature vector group. global semantic information ; The decoder has two inputs: a set of video feature vectors. With global semantic information Through global semantic information Update video feature vector group This produces more symbolic representations of the beginning and end of an event. ; Step 53: Construct a boundary classifier. The boundary classifier uses the output information of the decoder. Predict the boundaries of video events to obtain boundary confidence scores. ; Step Six: Construct a multi-instance discrimination model based on contrastive learning selection. The specific steps are as follows: Step 61: Perform boundary sampling. a. Define the boundary region of the event instance, given an event instance. , , Let the start time and end time of the event be the start time and end time, respectively. Then, the start and end regions of the event are defined as follows: ； in, , Indicates the first One event instance, It is a hyperparameter that controls the size of the region; b. The video frame feature vector group obtained in step three... Samples are taken to form positive and negative sample pairs, and feature vectors are sampled from the start and end regions of the same instance. , To form a positive sample pair Feature vectors were sampled from the start and end regions of the boundary regions of different instances. , , forming negative sample pairs ; Step 62: Construct a contrastive learning scheme and optimize the multi-instance discrimination model. ,in, For similarity calculation function, To measure distance; Step 63: Update the embedding representation of the feature vectors of the boundary region using the optimization objective, making the embedding representations of the same instance close together and the embedding representations of different instances far apart. Name the updated feature vector group the instance-level feature vector group, which includes the event start vector group. Event End Vector Group ; Step 7: Construct a video clip interaction model. The specific steps are as follows: Step 71: The input to the video segment interaction model is a set of video frame feature vectors. Sliding window operations are performed on the video timeline to generate video clips of different scales; Step 72: For the multi-scale segments generated in Step 71, the video frame feature vector group obtained in Step 3... By unifying the sampling length through interpolation, multi-scale fragment-level features are obtained. ; Construct a deep learning model to perform information interaction on fragment-level features; Step 73: Build a paragraph classifier to calculate the confidence level of the target event or background contained in each paragraph. ; Step 8: Using the annotation files obtained in Step 4, jointly train the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model to obtain a joint model of the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model. The specific training steps are as follows: Step 81: Train the event boundary prediction model using frame-level labeled files, and define loss function 1: And calculate the loss 1; Step 82: Train the event video clip interaction model using clip-level labeled files, and define loss function 2: And calculate the loss 2; Step 83: Train a multi-instance discrimination model using a contrastive learning strategy, and define loss function 3: And calculate the loss as 3; Step 84: Calculate the cumulative losses 1, 2, and 3 to obtain the total loss. ,in The training weights are used to control the training of the three models, which are then trained using the gradient descent algorithm. Step 9: Use the joint model trained in Step 8 to detect target events in the video.

2. The method for temporal event detection in multi-event instance videos according to claim 1, characterized in that... In step 71, the specific steps of the sliding window operation are as follows: a. Set up a series of window groups of different sizes. ,in It is the number of windows. For window size; b. Set different step sizes according to different window sizes. For each window, the step size is: ; c. Based on the window size and sliding window, perform sliding window operations on the video time sequence to generate video segments of different scales.

3. The method for temporal event detection in multi-event instance videos according to claim 1, characterized in that... The specific steps of step nine are as follows: For any video segment, the event boundary prediction model, the multi-instance discrimination model, and the video segment interaction model generate boundary confidence, instance confidence, and segment confidence, respectively. The joint model trained in step eight uses the product of these three and sorts them to output the final video detection result.

Images