A method for human motion timing detection based on multimodal information

Abstract

This invention discloses a method for temporal detection of human actions based on multimodal information, relating to the fields of video understanding and computer vision. The invention includes: collecting publicly available human action video data and performing preprocessing; generating text prompt sequences and extracting text features; constructing a multimodal temporal feature extractor to extract multimodal spatiotemporal feature sequences; constructing a multimodal temporal response module to generate action response sequences; constructing an action nomination generator to generate action nominations; and constructing a multimodal instance-level action detection module to implement human action detection. Through the above technical solutions, this invention fully utilizes the effective information from multimodal data to improve the accuracy of human action detection.

Description

Technical Field

[0001] This invention relates to the fields of video understanding and computer vision, and in particular to a method for detecting human motion timing under multimodal information. Background Technology

[0002] With the continuous development of multimedia and computer vision technologies, the number and duration of videos are constantly increasing. The application demand for human-centered intelligent monitoring, situational awareness, and intelligent video understanding is growing. How to achieve accurate detection of human actions in any uncropped video, that is, semantic category recognition and temporal coordinate localization of action instances in the video, is the key to the further development of the above-mentioned related technologies.

[0003] Existing temporal action detection methods achieve action detection through single visible optical modal and derived modal video data. Their technical characteristics focus on the temporal correlation between video frame sequences and the design of effective aggregation methods for multi-scale temporal information. However, this approach contradicts the increasing semantic complexity of videos, leading to performance bottlenecks. Their performance characteristics are such that they tend to focus on simple and easily distinguishable video temporal points, resulting in incomplete action instance detection and overlapping detection between instances. Ultimately, this manifests as unreliable detection results, hindering practical applications.

[0004] For action recognition, the upstream task of action detection, a feasible solution to the above contradictions is to introduce additional modal information. By utilizing the information characteristics of multiple modalities, easily confused background responses can be suppressed, and action fragment-level responses can be enhanced to achieve better recognition results. However, for action detection methods, how to combine the differences in responses to the same semantic target with multimodal information to achieve effective detection from fragments to instances, enhance the semantic discrimination capability of temporal detection, and ultimately construct a temporal action detection framework based on multimodal information to improve detection performance, the relevant technical means are still immature. Summary of the Invention

[0005] This invention aims to provide a method for detecting human motion timing based on multimodal information, thereby addressing the problem that current detection technologies cannot effectively utilize multimodal information to achieve effective detection of motion instances.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A method for human motion temporal detection based on multimodal information includes the following steps:

[0008] Step 1: Use publicly available human motion video data as the raw dataset, and label, clean, and preprocess the data for network training.

[0009] And divide it into training and test sets;

[0010] Step 2: Based on the labeled semantic categories, generate fragment-level and instance-level text prompt sequences respectively, construct a text feature encoder, and extract text semantic features;

[0011] Step 3: Construct a pre-trained temporal feature extractor and a feature-level multimodal attention module, and extract multimodal spatiotemporal feature sequences by combining text semantic features;

[0012] Step 4: Construct a multimodal temporal action sequence response module to generate fragment-level action response sequences;

[0013] Step 5: Construct an action nomination generator to generate action nominations based on fragment-level action response sequences;

[0014] Step 6: Construct a multimodal instance-level action detection module to detect action instances based on action nomination.

[0015] After processing publicly available human motion video data, this invention extracts text features using a text feature encoder based on semantic category information, and extracts multimodal spatiotemporal sequences by combining a temporal feature extractor and a multimodal attention module, achieving feature-level multimodal information fusion and feature representation. Next, a multimodal temporal motion sequence response module is constructed and optimized using temporal-level annotation to achieve segment-level separation of motion and background. Then, motion nominations are generated based on the temporal response, and instance-level motion prediction is performed with the assistance of text information, accurately adjusting motion boundaries, ultimately achieving accurate human motion detection under multimodal information collaboration.

[0016] Furthermore, step 1 includes the following steps:

[0017] Step 1.1: Obtain publicly available human body video data, and annotate the action instances in the video data to obtain an instance annotation set. Where s represents the start time of the action, e represents the end time of the action, c represents the semantic category of the action, and N I This indicates the number of action instances in the video;

[0018] Step 1.2: For each video containing human motion, extract frames at a specified frame rate (preferably 25 FPS) to obtain a visible optical image sequence, and normalize the resolution of each image in the sequence to a uniform size W×H, such as adjusting it to 224×224.

[0019] Step 1.3: Extract optical flow data from the sequence processed in Step 1.2 (preferably using the NVOFA algorithm) to obtain an optical flow image sequence, the length of which is the length L of the visible optical image sequence;

[0020] Step 1.4: Group the visible optical image sequence and the optical flow image sequence according to the same number of frames to obtain several visible optical video segment sequences. and optical flow video clip sequences Where, N um1 and N um2 These are the number of color channels and the number of optical flow channels, respectively.

[0021] Furthermore, step 2 includes the following steps:

[0022] Step 2.1: Combine the action semantic categories from all video annotations into a semantic category set. Based on the content of the set, each video is mapped to a coarse-grained semantic category, resulting in a coarse-grained semantic category set Y. C For example, Y C = {sport,daily,...}; where N c Represents the semantic category set Y S The size of the set;

[0023] Step 2.2, based on the semantic category set Y S Coarse-grained semantic category set Y C Text cues are constructed for visible light video clip sequences and optical flow video clip sequences, including text cues P for each action semantic category in the visible light modality. i,r And each action semantic category P of the optical flow mode i,f Based on the semantic category of the action, a multimodal text prompt sequence P in the visible light modality is obtained. r Multimodal text prompt sequence P of optical flow f ;

[0024] For example, the text prompts generated for each segment are as follows:

[0025]

[0026] Where i represents the i-th action semantic category, s C This indicates the coarse-grained semantic category of the video, where bg represents the background class, and r and f represent the visible light mode and optical flow mode, respectively. i Indicates the corresponding action semantic category;

[0027] Step 2.3, based on the semantic category set Y S Construct instance-level text hints P for each action category i,I Instance-level text hints based on all action categories P i,I Get instance-level text hint P I ;

[0028] For example, the instance-level text hint for a certain action category is specifically as follows:

[0029] P i,I='an action instance of {c i / bg}',

[0030] Where i represents the semantic category of the i-th action;

[0031] Step 2.4: Construct a pre-trained CLIP (Contrastive Language-Image Pre-training) network (preferably based on VIT-B / 16 pre-training), and use the text encoding branch of the CLIP network to process the multimodal text prompt sequence P. r With P f Instance-level text hint P I Feature encoding is performed to obtain the text features. and instance text features Where N represents the text feature dimension;

[0032] Step 2.5, based on the instance annotation set Y, from X T,r ,X T,f Text features corresponding to the semantic category of each segment in the video are extracted to construct a multimodal text feature sequence.

[0033] Furthermore, step 3 specifically includes:

[0034] Step 3.1: Construct a pre-trained TSN (Time-Sensitive Networking) network (preferably pre-trained with Kinetics-600) to extract features from multimodal spatiotemporal feature sequences;

[0035] The single-stream network of this TSN network adopts a BN-Inception structure (i.e., a normalized Inception structure), which includes several feature-level multimodal attention modules M. i (i = 1, ..., N) O ), N O The number of feature-level multimodal attention modules is preferably five, meaning the TSN network consists of alternating cascaded convolutional modules, feature-level multimodal attention modules, and Inception modules, with the number of feature-level multimodal attention modules and Inception modules being the same. The structure of each feature-level multimodal attention module includes a global average pooling layer, a linear layer, and a cosine similarity calculation layer. The input to the cosine similarity calculation layer also includes the module input. The cosine similarity is multiplied by the module input and then added back to the module input to obtain the final output of the module. Its mathematical expression can be represented as:

[0036]

[0037] Among them O i-1,t O′ i,t , X′ T,t and O i+1,t The subscript t in the middle represents the t-th segment, where O′ i,t This represents the input of the i-th feature-level multimodal attention module, which is the output of the previous feature-level multimodal attention module. The input of the first feature-level multimodal attention module is the output of the convolution module (convolution operation, batch normalization, and activation function operation). P avg (·) indicates a global average pooling layer, and Linear(·) indicates a linear layer that uses P avg (O i ′ ,t Mapped to X T,t In the same dimension, Sim(·) represents the cosine similarity function.

[0038] Step 3.2: Train the constructed TSN network. Preferably, retrain the TSN network on the Kinetics-600 dataset to optimize the network parameters of the linear layers in the multimodal attention modules at each feature level.

[0039] Step 3.3: Input the visible optical video clip sequence into the trained TSN network. and optical flow video clip sequences To extract visible optical features With optical flow characteristics Where N represents the feature dimension.

[0040] Furthermore, step 4 specifically includes:

[0041] Step 4.1: Construct a linear layer to display visible optical features With optical flow characteristics Equal-dimensional mapping to corresponding modal temporal action detection features

[0042] Step 4.2: Construct a visual responder consisting of a two-stream cascaded 1D convolutional layer, a ReLU layer, a 1D convolutional layer, and a Softmax layer, inputting temporal action detection features. Obtain the visual temporal action response sequence of the corresponding modality.

[0043] Step 4.3: Construct a two-stream text-visual joint responder to integrate text features. With visual feature F r ,F f Perform similarity calculations to obtain the text-time action response sequence of the corresponding modality.

[0044] Taking the visible optical mode as an example, the response of segment i to the j-th category is:

[0045] S Tr,i,j =|Sim(F r,i ,X T,r,j )|;

[0046] Step 4.4: Calculate the fragment-level loss functions for the visual modality and the text modality respectively, which are used to optimize the network parameters of the constructed linear layer, visual responder, and dual-stream text-visual joint responder;

[0047]

[0048] in, Let be the fragment-level loss functions for the visual modality and the text modality, respectively, and let CE(·) represent the cross-entropy loss function. i S represents the semantic category, i.e., the category label, of the corresponding fragment i in the instance annotation set Y. Vr,i S Vf,i S represents the visual temporal action response characteristics of the i-segment in the visible light mode and the optical flow mode, respectively. Tr,i S Tf,i These are text temporal action response features in the visible light mode and the optical flow mode, respectively.

[0049] Furthermore, step 5 specifically includes:

[0050] Step 5.1, fuse the multimodal response sequences from step 4:

[0051] S=α1S Vr +α2S Vf +α3S Tr +α4S Tf ,

[0052] Where S is the fused multimodal response sequence, and α1, α2, α3, α4 are the balance factor hyperparameters, where α1+α2+α3+α4=1;

[0053] Step 5.2, set the threshold set {ξ1,...,ξ} n}, consecutive segments exceeding a threshold are used as action nominations to obtain an action nomination set. Where s and e represent the start and end times of the action, respectively, and N... P Indicates the number of nominations.

[0054] Furthermore, step 6 specifically includes:

[0055] Step 6.1: For the nominations generated in Step 5.2, divide them into positive and negative samples based on the difference between their centers and the centers of instances in the instance annotation set Y. If x is positive, it is a positive sample; otherwise, it is a negative sample.c ,l c These represent the center coordinates and length of the real instance, respectively, and ψ is a preset threshold.

[0056] Step 6.2, Cascaded temporal action detection feature F r ,F f And extract the sub-feature sequence F from it based on the time coordinate of the nomination. I And generate instance-level features

[0057] F P =P avg (F I );

[0058] Step 6.3: Construct cascaded linear layers and ReLU activation function layers to process instance-level features F. P Mapping to features Then construct a linear layer classifier, inputting feature F. P 'Get the output instance classification probability'

[0059] Furthermore, step 6 also includes the instance-level text features X obtained based on the CLIP network. T,I Calculate the multimodal instance detection loss:

[0060]

[0061] Where FL(·) represents Focal loss, InfoNCE(·) represents InfoNCE loss, and N pos With N neg These represent the number of positive samples and the number of negative samples, respectively.

[0062] Loss detection using multimodal instances. Network optimization was performed on the instance-level action detection module (cascaded linear layers and ReLU activation function layers, linear layer classifier).

[0063] Furthermore, in step 6, during inference, the confidence score for the i-th action class is calculated for each nomination.

[0064]

[0065] Among them, X T,I,i This represents the instance-level text feature of the i-th action class.

[0066] Then, based on the confidence level, the non-maximum suppression algorithm (Soft-NMS algorithm) is used to suppress low-quality nominations, and the final detection result is obtained.

[0067] The technical solution provided by this invention brings at least the following beneficial effects:

[0068] This invention further introduces textual modality on the basis of existing visual multimodal information. By designing corresponding multimodal modules at the feature level, segment level and instance level, it makes full use of the complementary information between multiple modalities, suppresses the performance degradation of the final human action detection caused by noise in single modal data, and ultimately improves the action detection performance from multiple levels such as feature extraction, nomination generation and confidence calculation. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a schematic diagram of a human motion detection method based on multimodal information in an embodiment of the present invention;

[0071] Figure 2 This is a structural diagram of the temporal feature extractor in an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of the action nomination generation process in an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of the multimodal instance-level action detection module and action detection in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0075] The purpose of this invention is to provide a human motion detection method based on multimodal information, which features effective extraction of multimodal information and high detection accuracy.

[0076] To make the objectives, features, and advantages of this invention more apparent and understandable, the invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0077] like Figure 1 As shown, the human motion detection method based on multimodal information proposed in this invention includes the following steps:

[0078] Step 1: Collect and obtain publicly available human motion video data, and complete the data annotation, cleaning and preprocessing, and divide the data into training set and test set;

[0079] Step 2: Based on the labeled semantic categories, generate fragment-level (i.e., the smallest unit of feature extraction) and instance-level text prompt sequences (i.e., video segments corresponding to a single human action), construct a text feature encoder, and extract text semantic features;

[0080] Step 3: Construct a pre-trained temporal feature extractor and a feature-level multimodal attention module, and extract multimodal spatiotemporal feature sequences by combining text semantic features;

[0081] Step 4: Construct a multimodal temporal action sequence response module to generate fragment-level action response sequences;

[0082] Step 5: Construct an action nomination generator to generate action nominations based on fragment-level action response sequences;

[0083] Step 6: Construct a multimodal instance-level action detection module to achieve accurate detection of action instances based on action nomination;

[0084] The features of the steps described above will be further described in detail below with reference to specific embodiments:

[0085] In step 1, publicly available human body video data is obtained. This example uses the publicly available THUMOS14 dataset to clean and obtain instance annotations from the videos. Number of instances N I The value is 3003, where s represents the action start time, e represents the action end time, and c represents the action semantic category. For example, a single instance sample is {69.4, 78.7, HighJump}. Next, for each video, the frame rate is first unified to 30 FPS, then frames are extracted at 25 FPS to obtain a visible optical image sequence. The resolution of each image in the sequence is adjusted to 224×224, and the NVOFA algorithm (an optical flow extraction algorithm) is used to extract optical flow data, resulting in an optical image sequence. Then, taking test video number 254 as an example, the video has 6292 frames, grouped into sets of 16 frames, ultimately obtaining a video segment sequence and an optical flow video segment sequence of length 394. Finally, the publicly disclosed division ratio v is used. train :v test The dataset is divided into a ratio of 213:200.

[0086] In step 2, first from step 1 Extract all action semantic categories to form a set In this example, the set size is N. c =20; Map each video to a coarse-grained semantic category based on the category themes in the set. In this example, the data type is all sports, resulting in Y. C ={sport}; based on the semantic category set Y S Coarse-grained semantic category set Y C Text prompts are generated for visible optical video clip sequences. Taking a "high jump" action clip as an example, the generated clip text prompts are as follows:

[0087]

[0088] Where i represents the i-th action semantic category, and r and f represent the visible light mode and optical flow mode, respectively. The resulting multimodal text prompt P is then obtained. r With P f According to the semantic category set Y S To construct instance-level text suggestions, using the above example as an example, we obtain...

[0089] P i,I ='an action instance of highjump',

[0090] Finally, we get instance-level text hint P. i,I The CLIP network, pre-trained on VIT-B / 16, uses a text encoding branch to process multimodal text cue sequences P. r With P f Instance-level text hint P I Feature encoding is performed to obtain the text features. and instance text features Where N represents the text feature dimension, here N = 2048; based on the instance annotation set Y, from X T,r ,X T,f Text features corresponding to the semantic category of each segment in the video are extracted to construct a multimodal text feature sequence. Here, L = 394.

[0091] In step 3, a pre-trained Kinetics-600 TSN network is constructed, whose single-stream network uses a BN-Inception architecture, wherein the network is constructed as follows: Figure 2 The feature-level multimodal attention module shown is inserted into the TSN network. For the 5 modules M of the TSN network, i (i = 1, ..., 5), output O for each module i (i = 1, ..., 5), its feature-level multimodal attention module functions as follows:

[0092]

[0093] Among them O i-1,t O i ′ ,t , X′ T,t and O i+1,t The subscript t in P represents the t-th segment. avg (·) indicates a global average pooling layer, and Linear(·) indicates a linear layer that uses P avg (O i ′ ,t Mapped to X T,t In the same dimension, Sim(·) represents the cosine similarity function.

[0094] The above feature extraction network was retrained on the Kinetics-600 dataset, and the linear layer parameters in the feature-level multimodal attention module were optimized. Finally, a sequence of visible optical video clips was input. and optical flow video clip sequences in, The "3" in the text refers to the three channels in the RGB space of a visible optical video clip sequence. The 16 in the figure refers to the number of optical flow channels, used for extracting visible optical features. With optical flow characteristics Where N represents the feature dimension, and in this embodiment N = 2048.

[0095] In step 4, a linear layer is first constructed to visualize the optical features. With optical flow characteristics Equal-dimensional mapping to corresponding modal temporal action detection features Simultaneously, a visual responder is constructed consisting of a two-stream cascaded 1D convolutional layer (one-dimensional convolutional layer) - ReLU - 1D convolutional layer - Softmax layer, inputting temporal action detection features. Obtain the visual temporal action response sequence of the corresponding modality. Construct a two-stream text-visual joint responder to integrate text features With visual feature F r ,F f To calculate similarity, taking the visible optical modality as an example, the response of segment i to category j is:

[0096] S Tr,i,j =|Sim(F r,i ,X T,r,j )|;

[0097] Obtain the text-based temporal action response sequence of the corresponding modality. Finally, the fragment-level loss functions for the visual modality and the text modality are calculated and optimized separately:

[0098]

[0099] Where CE(·) represents the cross-entropy loss function, y i This indicates the semantic category of the segment corresponding to the instance annotation Y.

[0100] The action nomination generator described in step 5, which generates action nominations based on fragment-level action response sequences, is illustrated in the diagram below. Figure 3 As shown; firstly, the multimodal response sequences from step 4 are fused.

[0101] S=α1S Vr +α2S Vf +α3S Tr +α4S Tf ,

[0102] Where α1, α2, α3, and α4 are balance factor hyperparameters, where α1 + α2 + α3 + α4 = 1. Their values ​​are determined by the distribution of the data, and the magnitude of the values ​​determines the importance of different sequences. In this embodiment, α1 = α2 = 0.35, α3 = α4 = 0.15; a threshold set {ξ1, ..., ξ} is set. n In this example, the threshold is set to {0.5, 0.7, 0.9}. Consecutive segments exceeding this threshold are used as action nominations, resulting in the action nomination set. Where s and e represent the start and end times of the action, and N... P Indicates the number of nominations.

[0103] like Figure 4 As shown, in step 6, for the nominations finally generated in step 5, positive and negative samples are first divided according to the difference between their centers and the centers of instances in Y. If x is positive, it is a positive sample; otherwise, it is a negative sample. c ,l c Here, represent the center coordinates and length of a real instance, respectively. ψ is a threshold hyperparameter, set to 0.4 in this example. For instance, given a nomination {5,8}, its nearest neighbor instance is {6,7}. Based on calculation, this instance is classified as a positive sample. The cascaded temporal action detection feature F... r ,F f And extract the sub-feature sequence F from it based on the time coordinate of the nomination. I And generate instance-level features for

[0104] F P =P avg (F I );

[0105] Constructing cascaded linear-ReLU layers will F P Mapped to Next, a linear layer classifier is constructed, with the mapped features F as input. P 'Get the output instance classification probability' Combining instance-level text features X T,I The multimodal instance detection loss is calculated as follows:

[0106]

[0107] Among them, y i For the corresponding labels, FL(·) represents Focal loss, InfoNCE(·) represents InfoNCE loss, and N... pos With N neg These represent the number of positive samples and the number of negative samples after the positive and negative samples are separated, respectively. The instance-level action detection module was optimized. During inference, the confidence score for the i-th action class is calculated for each nomination.

[0108]

[0109] In the final post-processing stage, the Soft-NMS algorithm is used to suppress low-quality nominations. In this example, the Soft-NMS threshold is set to 0.4, which means that action classes with confidence scores lower than the Soft-NMS threshold are suppressed to obtain the final detection results.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0111] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for detecting human motion temporal sequence based on multimodal information, characterized in that, Includes the following steps: Step 1: Use publicly available human motion video data as the raw dataset, and label, clean, and preprocess the data for network training. And divide it into training and test sets; Step 2: Based on the labeled semantic categories, generate fragment-level and instance-level text prompt sequences respectively, construct a text feature encoder, and extract text semantic features; Step 3: Construct a pre-trained temporal feature extractor and a feature-level multimodal attention module, and extract multimodal spatiotemporal feature sequences by combining text semantic features; Step 4: Construct a multimodal temporal action sequence response module to generate fragment-level action response sequences; Step 5: Construct an action nomination generator to generate action nominations based on fragment-level action response sequences; Step 6: Construct a multimodal instance-level action detection module to detect action instances based on action nomination; Step 2 includes the following steps: Step 2.1: Combine the action semantic categories from all video annotations into a semantic category set. Based on the content of the set, each video is mapped to a coarse-grained semantic category, resulting in a coarse-grained semantic category set. ;in, Represents a set of semantic categories The size of the set; Step 2.2, based on the semantic category set coarse-grained semantic category set Text cues are constructed for visible light video clip sequences and optical flow video clip sequences, including text cues for each action semantic category in the visible light modality. and each action semantic category of optical flow mode Based on the semantic category of the action, a multimodal text prompt sequence for the visible light modality is obtained. Multimodal text prompt sequences of light flow ; Step 2.3, based on the semantic category set Build instance-level text hints for each action category. Instance-level text hints based on all action categories Get instance-level text hints ; Step 2.4: Construct a pre-trained CLIP network and use the text encoding branch of the CLIP network to process the multimodal text prompt sequence. and Instance-level text hints Feature encoding is performed to obtain the text features. and instance text features , where N represents the text feature dimension; Step 2.5, based on the instance annotation set ,from Text features corresponding to the semantic category of each segment in the video are extracted to construct a multimodal text feature sequence. .

2. The method as described in claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Obtain publicly available human body video data, and annotate the action instances in the video data to obtain an instance annotation set. Where s represents the start time of the action, e represents the end time of the action, and c represents the semantic category of the action. This indicates the number of action instances in the video; Step 1.2: For each video containing human motion, extract frames at a specified frame rate to obtain a sequence of visible optical images, and normalize the resolution of each image in the sequence to a uniform size. ; Step 1.3: Extract optical flow data from the sequence processed in Step 1.2 to obtain an optical flow image sequence, the length of which is the length L of the visible optical image sequence; Step 1.4: Group the visible optical image sequence and the optical flow image sequence according to the same number of frames to obtain several visible optical video segment sequences. and optical flow video clip sequences ,in, and These are the number of color channels and the number of optical flow channels, respectively.

3. The method as described in claim 2, characterized in that, Step 3 specifically includes: Step 3.1: Construct a pre-trained TSN network to extract features from multimodal spatiotemporal feature sequences; The TSN network consists of a structure in which convolutional modules, feature-level multimodal attention modules, and Inception modules are cascaded alternately, with the number of feature-level multimodal attention modules being the same as the number of Inception modules. The structure of the feature-level multimodal attention module includes, in sequence, a global average pooling layer, a linear layer, and a cosine similarity calculation layer. The input of the cosine similarity calculation layer also includes the module input. The cosine similarity is then multiplied by the module input and added to the module input to obtain the final output of the module. Step 3.2: Train the constructed TSN network; Step 3.3: Input the visible optical video clip sequence into the trained TSN network. and optical flow video clip sequences To extract visible optical features With optical flow characteristics , where N represents the feature dimension.

4. The method as described in claim 3, characterized in that, In step 3.1, the constructed TSN network uses the TSN network pre-trained on the Kinetics-600 action recognition dataset. In step 3.2, the TSN network is retrained on the Kinetics-600 dataset to optimize the network parameters of the linear layers in the multimodal attention modules at each feature level.

5. The method as described in claim 3, characterized in that, Step 4 specifically includes: Step 4.1: Construct a linear layer to display visible optical features With optical flow characteristics Equal-dimensional mapping to corresponding modal temporal action detection features ; Step 4.2: Construct a visual responder consisting of a two-stream cascaded 1D convolutional layer, a ReLU layer, a 1D convolutional layer, and a Softmax layer, inputting temporal action detection features. Obtain the visual temporal action response sequence of the corresponding modality. Step 4.3: Construct a two-stream text-visual joint responder to integrate text features. With visual features Perform similarity calculations to obtain the text-time action response sequence of the corresponding modality. ; Step 4.4: Calculate the fragment-level loss functions for the visual modality and the text modality respectively, which are used to optimize the network parameters of the constructed linear layer, visual responder, and dual-stream text-visual joint responder; ， in, , These are the fragment-level loss functions for the visual modality and the text modality, respectively. Represents the cross-entropy loss function. This indicates the semantic category of the corresponding fragment i in the instance annotation set Y. , These represent the visual temporal action response characteristics of the i-segment in the visible light mode and the optical flow mode, respectively. , These are text temporal action response features in the visible light mode and the optical flow mode, respectively.

6. The method as described in claim 5, characterized in that, Step 5 specifically includes: Step 5.1, fuse the multimodal response sequences from step 4: ， in, To fuse multimodal response sequences, The preset balance factor, and ; Step 5.2, Set the threshold set Continuous segments exceeding a threshold are used as action nominations to obtain an action nomination set. Where s and e represent the start time and end time of the action, respectively. Indicates the number of nominations.

7. The method as described in claim 6, characterized in that, Step 6 specifically includes: Step 6.1: For the nominations generated in Step 5.2, divide them into positive and negative samples based on the difference between their centers and the centers of instances in the instance annotation set Y. If the sample is positive, it is considered a positive sample; otherwise, it is considered a negative sample. These represent the center coordinates and length of the real instance, respectively. For preset thresholds; Step 6.2, Cascaded Timing Action Detection Features And extract sub-feature sequences from them based on the time coordinates of the nominations. And generate instance-level features : ； Step 6.3: Construct cascaded linear layers and ReLU activation function layers to process instance-level features. Mapping to features Then construct a linear layer classifier, inputting features. Obtain the classification probability of the output instance. .

8. The method as described in claim 7, characterized in that, In step 6, based on instance-level text features Calculate the multimodal instance detection loss: in, Represents Focal loss. Indicates Info NCE loss, and These represent the number of positive samples and the number of negative samples, respectively. Loss detection using multimodal instances. Network optimization is performed on the instance-level action detection module, which consists of cascaded linear layers, ReLU activation function layers, and a linear layer classifier.

9. The method as described in claim 7, characterized in that, In step 6, during inference, the confidence score for the i-th action class is calculated for each nomination. in, Represents the instance-level text features of the i-th action class; Then, based on the confidence level, a non-maximum suppression algorithm is used to suppress low-quality nominations, and the final detection result is obtained.

Images