A method for continuous multi-stage modeling

Abstract

This invention provides a continuous multi-stage modeling method for a novel network used in continuous video streams for activity detection. The network takes an uncut video segment as input and outputs action suggestion categories for each short time segment. These categories are divided into six types: background, preparation, start, confirmation, end, and follow-up. The steps of the continuous multi-stage modeling method are as follows: behavior stage category pre-classification; feature extraction using various convolutional networks for action recognition; behavior stage classification; proposal generation; candidate region evaluation; since the output of the behavior stage classification closely matches the real-world situation, a pre-set score is assigned to each suggestion; non-maximum suppression (NMS) is used to remove redundant suggestions, finally obtaining the final behavior suggestion region.

Description

Technical Field

[0001] This invention relates to a novel temporal detection network, a new type of network for activity detection in continuous video streams. Background Technology

[0002] Temporal action detection focuses on predicting action categories and the temporal boundaries of action instances in undressed videos. Most action detection methods are inspired by the success of image object detection {Faster-CNN, YOLO, SSD}. Mainstream methods can be divided into two types: two-stage pipelines and single-stage pipelines. For two-stage pipelines, the first step is to generate proposals, and the second step is to classify the proposals. Based on previous work, under the same conditions, two-stage methods outperform single-stage methods. In proposal generation, earlier works primarily used sliding windows as candidate objects. Recently, many methods have referenced pre-defined fixed temporal anchors to generate proposals. For example, TAG divides active instances into three stages and then uses an action classifier to evaluate binary action probabilities. SMS assumes that each temporal window starts from a start frame, then one or more intermediate frames, and finally an end frame. BSN locates temporal boundaries with high probabilities and then evaluates the confidence of candidate proposals generated from these boundaries. BMN proposes a boundary matching mechanism to evaluate the confidence scores of densely distributed proposals. MGG proposes a multi-granularity generator for generating temporal action proposals from different granularities. Unlike the previous works, we use six stages to represent a complete action instance. Each stage corresponds to a period of action rather than a frame. We not only predict the probability that each frame belongs to an action, but also predict actions as a continuous multi-stage sequence.

[0003] Accurately pinpointing the start and end times of an action in a video remains an open challenge, especially when applying 2D object detection methods to action detection, which often yields unsatisfactory results.

[0004] One important reason is that in behavior detection, determining the start and end points of a behavior is often ambiguous, making it difficult to distinguish the start point from the features before and after it. Summary of the Invention

[0005] To address the problems of the prior art, this invention proposes a continuous multi-stage modeling method for novel networks used in continuous video streams for activity detection.

[0006] This invention divides a complete action into five stages: preparation, start, action confirmation, end, and follow-up. Adding a background stage that is not part of the action, each point in a video can correspond to one of these six stages. A video must contain a start, action confirmation, and end stage to be considered an action. This invention focuses on action proposal generation. Experiments on open data show that, using only RGB image sequences as input, and with the same feature extraction network, the proposed method achieves performance that rivals or even surpasses all two-stream networks.

[0007] The purpose of this invention is to propose a continuous multi-stage modeling method for novel networks used in continuous video streams for activity detection. For example... Figure 1 As shown, the network takes an uncut video segment as input and outputs action suggestion categories for each short time segment. There are six categories: Background, Ready, Start, Confirm, End, and Follow-up. Figure 1 The network structure is as follows: (a) Feature Extractor Subnet: 3D convolutional networks or two-stream networks, such as r-c3d, are used to encode the input video. (2) Action Stage Subnet: The input feature sequence is fed into a bidirectional LSTM network after passing through two conv-3d networks, followed by a classification network, and outputs the action completion category. (3) Proposal Generation Subnet: The action network is combined to generate proposals. (4) IoU Evaluation Subnet: The score is calculated for each generated proposal. (5) Soft-NMS Non-maximum Suppression.

[0008] The steps of the continuous multi-stage modeling method of the present invention are as follows:

[0009] Step 1: Behavioral Stage Category Classification Preset: Assume our video input v = {S1,…,St,…,Sn}, where St represents the t-th frame, n is the total number of frames in the video, {Ga,Gb} represents a set of proposals, Ga represents the start time, Gb represents the end time, and the duration D = Gb - Ga. To better utilize contextual information, especially to avoid rigidly distinguishing features before and after the start time, we first expand the proposal as follows: add a preparation point Gp = Ga - D / 2, and a follow-up point Gf = Gb + D / 2; secondly, we divide {Ga,Gb} into 3 segments, with two dividing points: the start point Gs = Ga + D / 3, and the end point Ge = Gb - D / 3. In this way, the expanded proposal is divided into 5 segments in sequence: preparation {Gp,Ga}, start {Ga,Gs}, confirmation {Gs,Ge}, end {Ge,Gb}, and follow-up {Gb,Gf}. The other frames belong to the background stage. Based on this division, we have six behavioral stage categories: background, preparation, start, confirmation, end, and follow-up. Each frame of the input video corresponds to one of six behavioral stage categories.

[0010] Step 2: Feature Extractor Subnet: To extract features from a given video, we can use various convolutional networks for action recognition. In our framework, we use a 3D convolutional network as the feature extractor (such as the classic C3D convolutional network) and also experiment with a two-stream network. Taking the C3D network as an example, the input for feature extraction is a series of RGB images with input frames of size [3×Li×Hi×Wi], and the output is basic features, where Li is the length of the input image, Hi is the height of the input image, and Wi is the width of the input image. The size of the output feature is [Co×Lo×Ho×Wo], where Co is the number of channels in the output feature, Lo is the length of the output feature, Ho is the height of the output feature, and Wo is the width of the output feature. Lo, Ho, and Wo are scaled by the network relative to the original input feature size. For example, assuming the scaling factor of feature extraction on the input length is sc, then the output feature length Lo = Li / sc.

[0011] Step 3: Action Stage Subnet: (See attached) Figure 2As shown, the action phase classification consists of a CNN network and an LSTM network. The CNN network scales or expands the extracted feature matrix to fit the input of the LSTM network, containing N convolutional layers (kernel size k, hidden size h) and a max-pooling layer. The output feature size of the CNN network is [Lc×Lo], where Lc is the number of feature channels and Lo is the feature length. The aspect ratio of the feature is scaled to 1, and Lc is also the number of input channels of the LSTM network. The LSTM network we use is an N-layer bidirectional LSTM, which can maximize the use of contextual information. The output of the CNN network is fed into the LSTM network, and the output action phase category sequence corresponds to the length Lo, i.e., the feature length remains unchanged. Let Q = [q1, q2, ..., qi, ..., qn] represent the action phase category sequence output after passing through the LSTM network, where qi refers to the i-th output in the sequence, and n = Lo is the total length of the output sequence. Since the output length Lo is scaled proportionally, the preset frame-level action phase categories are also scaled proportionally.

[0012] Step 4: Proposal Generation Subnet. The output of Step 3 is the operation category sequence Q, which may contain one or more stages such as preparation, start, confirmation, end, follow-up, and background. In the subsequences of sequence Q, let Qr = [r1, r2, ..., ri, ..., rn] represent the preparation sequence, where ri represents the category of the i-th output in Qr; Qs = [s1, s2, ..., si, ..., sn] represent the start sequence, where si represents the category of the i-th output in Qs; Qc = [c1, c2, ..., ci, ..., cn] represent the confirmation sequence, where ci represents the category of the i-th output in Qc; Qe = [e1, e2, ..., ei, ..., en] represent the end sequence, where ei represents the category of the i-th output in Qe; and Qf = [f1, f2, ..., fi, ..., fn] represent the follow-up sequence, where fi represents the category of the i-th output in Qf. (See attached...) Figure 3 As shown, we select the start and end positions according to the following rules and combine them into candidate proposals:

[0013] (1) If the position t in the behavior category sequence Q belongs to the first half of Qr, Qs, or Qc, then t is selected as the starting position. Then we can obtain the candidate starting point set Cs = {r1, ..., rn, s1, ..., sn, c1, ..., cn / 2}.

[0014] (2) If position e belongs to the latter half of Qe, Qf, or Qc, then select e as the ending position. Then we can obtain the candidate endpoint set Ce = {cn / 2, ..., cn, e1, ..., en, f1, ..., fn}.

[0015] (3) If the time interval from the start position t belonging to Cs to the end position e belonging to Ce contains at least one of the start phase, confirmation phase, and follow phase (but excludes the background phase), then we combine the time intervals contained in t and e to form {t, e}, which constitutes a candidate proposal interval {t, e}. By combining the t and e that meet the conditions in the two sets, we can finally obtain the candidate proposal set Ps.

[0016] Step 5: Candidate Region Evaluation (IoU Evaluation Subnet): The input features of the candidate region network are the outputs of the CNN in the behavior stage classification, with a size of [Lc×Lo]. We extract feature sequences from the sequence positions corresponding to the candidate proposal set Ps, obtaining the feature set P(f). Since the features in P(f) have different lengths, we use 3D RoI pooling to extract fixed-size features. The output of 3D RoI pooling is fed into two fully connected network layers with a hidden size of u, and the output is the score corresponding to the feature set P(f).

[0017] Step Six: Since the output of the behavioral stage classification closely matches real-world scenarios, we assign a preset score to each proposal. Assuming the sequence from "Prepare" to "Follow-up" is qc, we use the first "Start" as the starting point qs and the last "End" as the ending point qe. For a proposal generated from sequence qc, we calculate the distance between the proposal's starting point and qs, denoted as ds; and the distance between the proposal's ending point and qe, denoted as de. We can then calculate the pre-score PES as follows: d = (1 - ds × i) × (1 - de × i), where i is the decay rate. Then, for each proposal, we use the product of the PES score and the preset score as the final score.

[0018] Step 7: Non-maximum suppression (Soft-NMS): We use non-maximum suppression (NMS) to remove redundant suggestions and finally obtain the final behavior suggestion region. Attached Figure Description

[0019] Figure 1 The network structure diagram of this invention.

[0020] Figure 2 A schematic diagram showing the behavioral stage classification system composed of CNN and LSTM networks.

[0021] Figure 3 The output of step three is the operation category sequence Q, which is a diagram showing the order from preparation to completion. Detailed Implementation

[0022] To make the invention's content and technical solutions clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of the invention.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] The purpose of this invention is to propose a continuous multi-stage modeling method for novel networks used in continuous video streams for activity detection. For example... Figure 1 As shown, the network takes an uncut video segment as input and outputs action suggestion categories for each short time segment. There are six categories: Background, Ready, Start, Confirm, End, and Follow-up. Figure 1 The network structure is as follows: (a) Feature Extractor Subnet: RGB or dual-stream network, such as r-c3d, is used to encode the input video. (2) Action Stage Subnet: The input feature sequence is fed into a bidirectional LSTM network after passing through two conv-3d networks, and then into a classification network to output the action completion category. (3) Proposal Generation Subnet: The action network is combined to generate proposals. (4) IoU Evaluation Subnet (5) Soft-NMS.

[0025] The steps of the continuous multi-stage modeling method of the present invention are as follows:

[0026] Step 1: Behavioral Stage Category Classification Preset: Assume our video input v = {S1,…,St,…,Sn}, where St represents the t-th frame, n is the total number of frames in the video, and {Ga,Gb} represents a set of proposals, where Ga represents the start time and Gb represents the end time. The duration D = Gb - Ga. To better utilize contextual information, especially to avoid rigidly distinguishing features before and after the start time, we first expand the proposals as follows: Gp = Ga - D / 2, Gf = Gb + D / 2. Secondly, we divide {Ga,Gb} into 3 segments, with two dividing points: Gs = Ga + D / 3, Ge = Gb - D / 3. In this way, the expanded proposals are divided into 5 segments. Adding the category belonging to the background, we have six behavioral stage categories: Background, Preparation, Start, Confirmation, End, and Follow-up. Each frame of the input video corresponds to one of these six behavioral stage categories.

[0027] Step 2: Feature Extraction: To extract features from a given video, we can use various convolutional networks for action recognition. In our framework, we use the C3D network as the feature extractor and also experiment with a two-stream network. Taking C3D as an example, the input for feature extraction is an RGB input frame of size [3×32×256×256], and the output is basic features. The size of the output feature is [Co×Lo×Ho×Wo], and its length, width, and height are scaled relative to the original input frame by the network. Assuming the scaling factor of feature extraction on the input length is 8, then Lo = 32 / 8 = 4.

[0028] Step 3: Behavioral Stage Classification (see attached) Figure 2 As shown, the action phase classification consists of a CNN network and an LSTM network. The CNN network scales or expands the extracted feature matrix to fit the input of the LSTM network, containing two convolutional layers (kernel size 3, hidden size 512) and a max-pooling layer. The output size of the CNN network is [512×4], where 512 is the input channel of the LSTM network. The LSTM network we use is a two-layer bidirectional LSTM, which can maximize the use of contextual information. The output of the CNN network is fed into the LSTM network, and the output action phase category sequence corresponds to length Lo. Let Q = [q1, q2, ..., qn] be the output action phase category sequence, where n = Lo. Since the output length Lo is scaled proportionally, the preset frame-level action phase categories also need to be scaled down by the same proportion sc.

[0029] Step 4: Proposal Generation: The output of Step 3 is the operation category sequence Q, ordered from preparation to end. In the subsequences of sequence Q, let Qr = [r1, r2, ..., rn] represent the preparation sequence, Qs = [s1, s2, ..., sn] represent the start order, Qc = [c1, c2, ..., cn] represent the confirmation sequence, Qe = [e1, e2, ..., en] represent the end sequence, and Qf = [f1, f2, ..., fn] represent the subsequent order. (See attached...) Figure 3 As shown, we select the start and end positions according to the following rules and combine them into candidate proposals:

[0030] (1) If the position t in the behavior category sequence Q belongs to the first half of Qr, Qs, or Qc, then t is selected as the starting position. Then, we can obtain the candidate starting point set Cs = {r1, ..., s1, ..., c1, ..., cn / 2}.

[0031] (2) If position e belongs to the latter half of Qe, Qf, or Qc, then select e as the ending position. Then we can obtain the candidate endpoint set Ce = {cn / 2, ..., cn, ..., e1, ..., f1, ...}.

[0032] (3) If the time interval from the start position t belonging to Cs to the end position e belonging to Ce contains at least one of the start phase, confirmation phase, and follow phase (but excludes the background phase), then we combine the time intervals contained in t and e to form {t, e}, which constitutes a candidate proposal interval {t, e}. By combining the t and e that meet the conditions in the two sets, we can finally obtain the candidate proposal set Ps.

[0033] Step 5: Candidate Region Evaluation: The input features to the candidate region network are the outputs of the CNN in the behavior stage classification, with a size of [Lc×Lo×1×1]. We extract the feature sequences from the sequence positions corresponding to the candidate proposal set Ps, obtaining the feature set P(f). Since the features in P(f) have different lengths, we use 3D RoI pooling to extract features of a fixed size. The output of 3D RoI pooling is fed into two fully connected network layers with a hidden size of u, and the output is the score corresponding to the feature set P(f).

[0034] Step Six: Since the output of the behavioral stage classification closely matches real-world scenarios, we assign a preset score to each proposal. Assuming the sequence from "Prepare" to "Follow-up" is qc, we use the first "Start" as the starting point qs and the last "End" as the ending point qe. For a proposal generated from sequence qc, we calculate the distance between the proposal's starting point and qs, denoted as ds; and the distance between the proposal's ending point and qe, denoted as de. We can then calculate the pre-score as follows: d = (1 - ds × i) × (1 - de × i) (1) where i is the decay rate. Then, for each proposal, we use the product of the PES score and the preset score as the final score.

[0035] Step 7: Non-maximum suppression: We use non-maximum suppression (NMS) to remove redundant suggestions, and finally obtain the final behavior suggestion region.

[0036] The convolutional neural network in this invention can take various forms; as long as it is a multi-layer convolution, it can achieve the effect of this algorithm.

[0037] The feature extraction network of this invention can be a C3D network, a two-stream network, or any other 3D convolutional network.

[0038] The non-maximum suppression (NMS) of this invention can be various NMS algorithms, such as soft-NMS, hard-NMS, etc. As long as it can achieve the non-maximum suppression effect, it can produce normal output.

Claims

1. A method for continuous multi-stage modeling for networks performing activity detection in continuous video streams, characterized in that, The network takes an uncut video as input and outputs the action suggestion category corresponding to each short time segment. The categories are divided into 6 categories: background, preparation, start, confirmation, end, and follow-up. The network structure is as follows: (1) Feature extraction: 3D convolutional network or dual-stream network is used to encode the input video; (2) Behavior stage classification: The input feature sequence is fed into a bidirectional LSTM network after passing through a CNN network, and then into a classification network to output the action completion category; (3) Proposal generation network, which combines the action network to generate suggestions; (4) Proposal evaluation network; (5) Soft-NMS; The steps of the continuous multi-stage modeling method are as follows: Step 1: Behavioral Stage Category Classification Preset: Let the video input v = {S1,…,St,…,Sn}, where St represents the t-th frame, n is the total number of frames in the video, and {Ga,Gb} represent a set of suggestions, where Ga represents the start time and Gb represents the end time. Then the duration D = Gb - Ga. Utilizing contextual information to avoid rigidly distinguishing features before and after the start time, the suggestions are first expanded as follows: add a preparation point Gp = Ga - D / 2, and subsequent points Gf = Gb + D / 2. Secondly, {Ga,Gb} is divided into 3... The segment is divided into two parts: start point Gs = Ga + D / 3 and end point Ge = Gb - D / 3. In this way, the suggestions are expanded and sequentially divided into 5 segments: Preparation {Gp, Ga}, Start {Ga, Gs}, Confirmation {Gs, Ge}, End {Ge, Gb}, and Follow-up {Gb, Gf}. Other frames belong to the background stage. Based on this division, there are six behavioral stage categories: background, preparation, start, confirm, end, and follow-up. Each frame of the input video corresponds to one of these six behavioral stage categories. Step 2 Feature Extraction: To extract features from a given video, a 3D convolutional network or a two-stream network is used as the feature extractor; the 3D convolutional network includes a C3D network; wherein, the input of the C3D network feature extraction is a series of RGB images, the size of the input frame is [3×Li×Hi×Wi], and the output is a basic feature, where Li is the length of the input image, Hi is the height of the input image, and Wi is the width of the input image; the size of the output feature is: [Co×Lo×Ho×Wo], where Co is the number of channels of the output feature, Lo is the length of the output feature, Ho is the height of the output feature, and Wo is the width of the output feature; the network scales the size of the original input feature; let the scaling ratio of feature extraction on the input length be sc, then the output feature length Lo = Li / sc; Step 3: Action Stage Classification: Action stage classification consists of a CNN network and an LSTM network. The CNN network scales or expands the feature matrix extracted from the features to fit the input of the LSTM network. It contains N convolutional network layers and one max pooling layer. The kernel size of the convolutional network layer is k, and the hidden size is h. The output feature size of the CNN network is [Lc×Lo], where Lc refers to the number of channels of the output feature, Lo refers to the length of the output feature, and the aspect ratio of the output feature is scaled to 1. Lc is the number of input channels of the LSTM network. The LSTM network used is a bidirectional LSTM with N layers, which utilizes context information. The output of the CNN network is fed into the LSTM network, and the length Lo corresponding to the output action stage category sequence remains unchanged. Let Q = [q1, q2, ..., qi, ..., qn] represent the action stage category sequence output after passing through the LSTM network, where qi refers to the i-th output in the sequence, and n = Lo is the total length of the output sequence. Since the output length Lo is scaled proportionally, the preset frame-level action stage categories are also scaled proportionally. Step 4 suggests the following: The output of Step 3 is an operation category sequence Q, which contains one or more stages: preparation, start, confirmation, end, follow-up, and background. In the subsequences of sequence Q, let Qr = [r1, r2, ..., ri, ..., rn] represent the preparation sequence, where ri represents the category of the i-th output in Qr; Qs = [s1, s2, ..., si, ..., sn] represent the start sequence, where si represents the category of the i-th output in Qs; Qc = [c1, c2, ..., ci, ..., cn] represent the confirmation sequence, where ci represents the category of the i-th output in Qc; Qe = [e1, e2, ..., ei, ..., en] represent the end sequence, where ei represents the category of the i-th output in Qe; and Qf = [f1, f2, ..., fi, ..., fn] represent the follow-up sequence, where fi represents the category of the i-th output in Qf. The start and end positions are selected according to the following rules, and they are combined into candidate suggestions: (1) If the position t in the behavior category sequence Q belongs to the first half of Qr, Qs or Qc, then t is selected as the starting position; then the candidate starting point set Cs = {r1, ..., rn, s1, ..., sn, c1, ..., cn / 2} is obtained; (2) If position e belongs to the latter half of Qe, Qf, or Qc, then select e as the ending position; then obtain the candidate endpoint set Ce = {cn / 2, ..., cn, e1, ..., en, f1, ..., fn}; (3) If the time intervals from the start position t belonging to Cs to the end position e belonging to Ce contain at least one of the start phase, confirmation phase and follow phase, but do not include the background phase, then the time intervals contained in t and e are combined to form {t,e}, which constitutes a candidate suggestion interval {t,e}; the t and e that meet the conditions in the two sets are combined to finally obtain the candidate suggestion set Ps. Step 5: Candidate Region Evaluation: The input features of the candidate region network are the outputs of the CNN in the behavior stage classification, with a size of [Lc×Lo]. The feature sequence is truncated using the sequence position corresponding to the candidate proposal set Ps to obtain the feature set P(f). Since the features in P(f) have different lengths, 3D RoI pooling is used to extract features of a fixed size. The output of 3D RoI pooling is fed into two fully connected network layers with a hidden size of u, and the score corresponding to the feature set P(f) is output. Step Six: Since the output of the behavioral stage classification matches the real situation, a preset score is set for each suggestion. Let the sequence from "Prepare" to "Follow-up" be qc, the first "Start" be the starting point qs, and the last "End" be the ending point qe. For suggestions generated from the sequence qc, calculate the distance between the starting point of the suggestion and qs, denoted as ds; calculate the distance between the end of the suggestion and qe, denoted as de; then calculate the pre-score PES as follows: d = (1 - ds × i) × (1 - de × i), where i is the decay rate; then, for each suggestion, the product of the PES score and the preset score is used as the final score. Step 7: Soft-NMS: Use Soft-NMS to remove redundant suggestions, and finally obtain the final behavior suggestion area.

2. The method for continuous multi-stage modeling according to claim 1, characterized in that, The C3D network is used as a feature extractor. The input for feature extraction is RGB, the size of the input frame is [3×Li×Hi×Wi], and the output is basic features. The output feature dimensions are: [Co×Lo×Ho×Wo], where the length, width, and height of the output feature are scaled relative to the original input frame by the network. Let the scaling factor of feature extraction over the input length be sc, then Lo = Li / sc.

Images