Method and system for video temporal localization towards action-interaction target perception based on interactive attention

Abstract

A video temporal localization method and system based on interactive attention for action-interactive target perception includes the following steps: 1) collecting video sequence data of industrial processes, corresponding temporal action annotation data, and human-object bounding box coordinates, and performing preprocessing; 2) constructing a video temporal localization network based on interactive attention for action-interactive target perception, building a human-object joint representation including entity spatial coordinates and local visual features, inputting the aligned global temporal features and local object features into an interactive perception gated temporal encoder, using a human-object interaction cross-attention mechanism to suppress background noise and inject fine-grained interactive semantics, and outputting dynamically adapted action bounding boxes through a multi-hole gated temporal encoding module driven by interactive features. 3) Construct a multi-objective joint loss function to perform a one-to-one optimal match between the predicted action set output by the network and the real action set; calculate the joint loss based on the matching results, and use the backpropagation algorithm to update the network parameters until convergence, thereby obtaining the trained video action detection model; 4) Adopt a forward feature propagation and decoding method to block the loss calculation and gradient backpropagation data path in the training stage, input the industrial process video to be detected and the local object information extracted in real time into the trained model, output a prediction set containing action category confidence and action start and end timestamps, and remove low confidence results.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a deep learning-based video understanding and analysis technology for complex industrial environments. Specifically, this invention integrates a video action detection method and system that combines human-object interaction masking attention and void gating, making it particularly suitable for the recognition and localization of personnel's temporal actions in industrial equipment operation scenarios. Background Technology

[0002] Temporal Action Detection (TAD) is a key task in the field of computer vision, aiming to identify the types of actions occurring in videos and accurately locate the start and end times of these actions. With the advancement of Industry 4.0 and intelligent manufacturing, the digital and intelligent management of production processes has become a necessity. In complex industrial scenarios, the automated monitoring of worker operations has extremely high application value.

[0003] The mainstream methods for existing temporal action detection typically adopt an architecture of "pre-extracted features + temporal regression". Specifically, they use video base models (such as InternVideo, VideoMAE, etc.) pre-trained on massive datasets (such as Kinetics-400) to extract global spatiotemporal features of the video, and then input them into a temporal detection network (such as ActionFormer) for action classification and boundary regression.

[0004] However, in industrial motion detection applications targeting industrial equipment scenarios, the existing technologies that directly utilize global features have significant drawbacks and shortcomings. First, there is severe interference from complex background noise: industrial workshop scenes are typically cluttered with numerous static devices, and compared to everyday videos, background pixels constitute a much larger proportion of industrial videos. This causes the model to overemphasize the features of static devices in the background, neglecting the areas where actual actions occur, resulting in a large amount of noise mixed into the feature representation. Second, fine-grained interactive actions are difficult to distinguish: industrial process actions often have high similarity and fine-grained features; different actions appear very similar in the overall picture, with the only difference being the worker's hand movements and the specific parts being interacted with. Existing basic models typically only focus on overall optical flow or motion trends, lacking explicit modeling of the microscopic interaction relationship between "human-hand-object." Finally, semantic information about objects is missing: in industrial operations, the definition of actions is often bound to specific objects. Existing methods based on global features fail to effectively integrate the precise location and category information provided by object detection into the temporal motion detection framework, resulting in a lack of semantic perception of the operated objects.

[0005] In summary, how to effectively integrate fine-grained object-level spatial information into temporal detection models in complex industrial scenarios with subtle movements, in order to enhance the model's understanding of "human-object interaction," is a technical challenge that urgently needs to be solved in the field of industrial visual inspection. Summary of the Invention

[0006] This invention addresses the shortcomings of existing industrial scene temporal action detection technologies, such as high feature extraction noise due to complex background environments, easy category confusion due to subtle differences in process actions, and low temporal boundary positioning accuracy due to the lack of local interaction semantics. It proposes a video temporal positioning method and system based on interactive attention and oriented towards action interaction target perception.

[0007] This invention constructs a human-object joint representation that includes entity spatial coordinates and local visual features. It effectively suppresses complex background noise and accurately injects fine-grained interactive semantics into global video features by utilizing a human-object interaction mask attention mechanism. Furthermore, by combining an interaction-aware gated temporal coding and integrated decoding architecture, it achieves video temporal action detection that is anti-interference, fine-grained, and has strong spatiotemporal interpretability.

[0008] The first aspect of this invention provides a video temporal localization method based on interactive attention for action-oriented interactive target perception, the method comprising the following steps: S1. Collect video sequence data of industrial processes, as well as corresponding time-series action annotation data, including action category, action start and end time, human-object bounding box coordinates, etc.; preprocess all data sequences to establish an input stream, and divide them into training set, validation set, and test set; S2. Construct a video temporal localization network based on interactive attention for action interaction target perception; the network structure is as follows: starting from the input layer, it is subsequently connected to an interactive perception gated encoder, a multi-scale adapter, a query selector, an ensemble decoder, and a classification and regression output layer; construct a human-object joint representation containing entity spatial coordinates and local visual features, input the aligned global temporal features and local object features into the interactive perception gated temporal encoder, use the human-object interaction cross-attention mechanism to suppress background noise and inject fine-grained interactive semantics, output the spatiotemporal coding features that dynamically adapt to the action boundary through the multi-hole gated temporal coding module driven by the interaction state features, and further output the action category probability and the precise temporal start and end boundary through the ensemble decoder; The preprocessed input stream is fed into a video temporal localization network based on interactive attention and action-oriented target perception for forward feature propagation and decoding. Global temporal features of the video sequence are acquired, and local object attribute sets (including local visual features, spatial bounding box coordinates, and validity masks) of the corresponding frames are extracted from the input stream. The aligned global temporal features and local object features are then fed into an interactive perception-gated temporal encoder, which includes a person-object interaction cross-attention module and a multi-dilatational gated temporal coding module with multiple parallel convolutional branches of different dilatational rates. The spatiotemporal features processed by the interactive perception-gated temporal encoder are then processed by a multi-scale adapter to form multi-scale spatiotemporal features under different receptive fields. These multi-scale spatiotemporal features are then fed into a middle-neighbor region integrated decoder after passing through a query selector. Through a dual-branch network of a classification head and a regression head, the video sequence's action category probability and precise temporal start and end boundaries are output. S3. Construct a multi-objective joint loss function, and use the bipartite graph matching algorithm to perform a one-to-one optimal match between the predicted action set output by the network and the real action set; calculate the joint loss composed of classification focus loss, center point-width distance loss and intersection-union ratio loss based on the matching results, and use the backpropagation algorithm to update the network parameters until convergence, and obtain the trained video action detection model; S4. In the inference stage, a forward feature propagation and decoding method is adopted to block the data path of loss calculation and gradient backpropagation in the training stage. The industrial process video to be detected and the local object information extracted in real time are input into the trained model, and the prediction set containing the action category confidence and the start and end timestamps of the action is directly output, and low confidence results are removed.

[0009] Further, step S1 includes: S11: Data source configuration: Collect process operation videos captured by industrial site monitoring cameras as raw input; collect annotation files containing the start and end times and categories of various process actions as ground truth.

[0010] S12: Data Preprocessing: Global spatiotemporal features are extracted using a pre-trained video base model (such as InternVideo2); bounding box information of people, hands, and interactive objects in each frame is extracted using an Open-Vocabulary Object Detector. Uniform resolution adjustment is performed on the video frame sequence; based on the bounding boxes from the object detector, regions of interest (RoIs) are cropped on the original image, and visual features of the RoIs are extracted using a lightweight base network; simultaneously, the bounding box coordinates are normalized and mapped to high-dimensional position embeddings.

[0011] S13: Mask generation: Based on the bounding box coordinates obtained in step S12, a dynamic geometric mask is generated in the spatial dimension. This mask matrix assigns high weight only to the coordinate regions where the hand and the interactive object are located, and assigns very low weight or zero weight to the background region.

[0012] S14: Data cleaning and dataset partitioning: Remove invalid samples that are blurry or severely occluded, and divide the processed feature sequence data into training set, validation set and test set.

[0013] Furthermore, in step S2, the interactive perception gated encoder is sequentially connected to the person-object cross-attention module and the interactive-driven multi-hole gated encoding module. The interactive perception gated encoder sequentially inputs the video feature tensor and the stitched and aligned multi-object attribute tensor, and outputs a unified temporal feature after fusion processing. The multi-scale adapter contains multiple sequentially connected mask adjusters, and the features are downsampled by iteratively looping through the mask adjusters. The query selector is sequentially connected to the candidate segment generation module, the validity screening module, and the Top-K dynamic selection module. The integrated decoder contains multiple decoder layers, and the decoder layers are sequentially connected to a multi-head self-attention module, a deformable cross-attention module, a boundary-aware cross-attention module, and a feedforward neural network. The classification and regression output layer contains two parallel prediction network branches, where the classification head adopts a single-layer linear network structure and the regression head adopts a multi-layer perceptron structure.

[0014] Furthermore, step S2 specifically includes: S21: Temporal alignment of input data: Obtain global temporal features of the video sequence and extract the set of local object attributes of the corresponding frame from the input stream (including local visual features, spatial bounding box coordinates, object category labels and validity masks); for samples of different lengths, perform padding alignment in the temporal dimension, flatten and stitch the local object attributes to construct a global video feature tensor and a multi-object attribute tensor of consistent length.

[0015] S22: Feature Fusion and Background Suppression Based on Person-Object Interaction Mask Attention Mechanism: The aligned global temporal features and local object features are input into the person-object interaction cross-attention module in the interaction-aware gated temporal encoder for feature fusion. The specific feature fusion process is as follows: the global video feature tensor is dimensionality-reduced and mapped to a preset hidden layer dimension through a linear projection layer to generate a temporal query vector Q; the multi-object attribute tensor is parsed to extract local visual features; the local visual features are mapped through a linear projection layer to construct a key vector K and a value vector V that integrate spatial geometric semantic priors; an attention key-value mask matrix is ​​generated based on the parsed validity mask. To prevent gradient anomalies, a safe indexing mechanism is introduced for completely empty frames without valid objects, forcing their attention to be allocated to valid zero vectors or padding items; the mask matrix is ​​injected into the multi-head cross-attention calculation so that the query vector Q is aligned only with the real interactive entity regions K and V in the image, thereby removing background noise; the calculation results are sequentially processed through residual connections, layer normalization, and feedforward neural networks to output high-purity person-object interaction features.

[0016] S23: Interaction-state driven multi-hole gated temporal coding: The human-object interaction state features fused with position encoding are input into the multi-hole gated coding module for dynamic modeling of the temporal receptive field. The coding module contains multiple parallel convolutional branches with different dilation rates. Since the input features are highly enriched with spatial geometric and semantic mutation information of human-object interaction, the gating mechanism in the coding module is directly driven by the interaction state features: when the feature representation undergoes intense physical interaction, the gating network adaptively activates fine-grained dilated convolutional branches to capture instantaneous high-frequency actions; during periods of no interaction and smoothness, wide receptive field branches are retained to extract long temporal context, thereby outputting high-resolution spatiotemporal coding features that dynamically adapt to action boundaries.

[0017] S24: Joint prediction based on center-neighbor region integrated decoder: The spatiotemporal encoded features are input into the center-neighbor region integrated decoder network after passing through a multi-scale adapter and a query selector; the multi-scale adapter transforms the features into multi-scale representations with different receptive fields of duration through downsampling, and the query selector selects the query vectors and their initial positions that are most likely to correspond to the action instances, achieving initial focusing from dense features to key targets; the decoder performs multi-layer feature iterative updates in combination with the query vectors through a cross-attention mechanism, and finally outputs the detection results through a dual-branch prediction head: the classification head outputs the log probability of the action category corresponding to each temporal query; the regression head outputs the normalized center point and width of the action interval, and finally completes the high-precision temporal localization of fine-grained process actions in the video.

[0018] Furthermore, the global feature injection and background suppression process in step S22 includes: generating a temporal query vector Q from global video features through linear projection; fusing local visual features, coordinate features, and category labels to construct a key vector K and a value vector V; generating an attention key-value mask matrix using the parsed validity mask; and introducing a secure indexing mechanism to force the complete empty frames; and injecting the mask matrix into the multi-head cross-attention calculation so that the query vector Q is aligned only with the real interactive entity regions in the image, thereby removing background noise.

[0019] Furthermore, the multi-hole gated temporal coding in step S23 includes: setting multiple parallel convolutional branches with different hole rates, and using human-object interaction state features to explicitly drive the gating mechanism to calculate the activation weights of each branch; when the feature representation undergoes intense physical interaction, the gating network adaptively activates the low-hole-rate branch to capture instantaneous high-frequency actions; during periods of no interaction and smoothness, the high-hole-rate branch is retained to extract long temporal context.

[0020] Furthermore, the integrated decoder in step S24 employs a dual-branch prediction head. The classification head outputs the log-probability of the action category corresponding to each temporal query; the regression head outputs the normalized action interval center point and width through center point mapping and width exponential transformation, in order to decode and generate accurate temporal boundary intervals.

[0021] Furthermore, the construction of the multi-objective joint loss function in step S3 includes: a weighted sum of action classification loss and boundary regression loss; the classification loss uses Focal Loss to address the imbalance between positive and negative samples; the regression loss uses DIoU Loss to constrain the overlap between the predicted time period and the actual time period; during the collaborative update of parameters, the gradient signal generated by the joint loss is used to update the network parameters end-to-end.

[0022] Further, step S3 includes: S31: End-to-end target assignment based on bipartite graph matching: Obtain a fixed set of action predictions from the current forward propagation output of the network (including the predicted action class logit and the normalized temporal boundary interval). Calculate the matching loss between the prediction set and the real action set using a bipartite graph matching algorithm. This matching loss comprehensively considers both class prediction cost and temporal boundary error, thereby achieving a one-to-one optimal match between the predicted target and the ground truth annotation.

[0023] S32: Construct a joint objective function: composed of a weighted sum of action classification loss and boundary regression loss; the classification loss uses Focal Loss to address the imbalance between positive and negative samples; the regression loss uses DIoU Loss to constrain the overlap between the predicted time period and the actual time period; during the collaborative update of parameters, the gradient signal generated by the joint loss is used to update the network parameters end-to-end until the model reaches convergence on the validation set, thus completing the training of the model.

[0024] Further, step S4 includes: The video of the industrial process to be detected and the local object attributes extracted in real time by the target detector are input into the trained video action detection model. After forward feature propagation and decoding, a prediction set containing the action category confidence and the start and end timestamps of the action is directly output. Prediction results with confidence below a preset threshold are removed, and the final worker operation behavior recognition and temporal localization results are output, realizing the automated monitoring of complex industrial production processes.

[0025] A second aspect of the present invention relates to a video temporal localization system for action-oriented interactive target perception based on interactive attention, comprising: The multi-source data acquisition and preprocessing module is used to acquire videos from industrial sites and their corresponding labeled ground truth values. It uses a pre-trained video basic model to extract global spatiotemporal features and establishes a parallel data stream based on an open vocabulary object detector to extract the bounding box coordinates and local visual features of people, hands and interactive objects in video frames. The Person-Object Interaction Mask Attention and Background Suppression Execution Module is used to input the aligned global video features and multi-object attribute tensors into the cross-attention module. The global features are used as query vectors and the local features are used as key vectors. The bounding box coordinates are used to generate a dynamic geometric mask. By injecting the mask into the cross-attention calculation, non-interactive areas are forcibly filtered out, and high-purity person-object interaction state features with background noise eliminated are output. The interactive state-driven temporal coding and multi-scale adaptation module is used to input the interactive state features into a multi-diffuse gated temporal coding module containing multiple convolutional branches with different dilation rates. The interactive state features are used to explicitly drive the gating mechanism to adaptively switch the temporal receptive field and output high-resolution spatiotemporal coding features. Subsequently, the multi-scale adapter is processed to generate multi-scale feature pyramids under receptive fields of different durations. The middle-neighbor region integrated decoding and action set prediction module is used to input the multi-scale spatiotemporal features into the decoder network after being filtered by the query selector. It combines the initial query vector to perform multi-layer cross-attention feature iteration and directly predicts the log probability of the output action category and the normalized precise temporal start and end boundaries through a dual-branch network of classification head and regression head. An end-to-end joint optimization module based on bipartite graph matching is used to construct a multi-objective joint loss function that includes classification focus loss and temporal intersection-union regression loss, and to use the joint loss signal for backpropagation and collaborative updating of network parameters. The online inference execution module receives the video sequence of the industrial process to be detected and the local object information extracted in real time during the inference stage. After the forward feature propagation and decoding of the target model, it directly outputs the prediction set and the final behavior recognition and temporal localization results.

[0026] The innovation of this invention lies in: Existing methods typically introduce object awareness during the training phase, using entity bounding boxes only as training labels to calculate the loss function and supervise model training, allowing the model to predict the position and category of hands and objects—a form of implicit feature enhancement. Other methods often employ an early, simple feature concatenation strategy, merely stitching local entity features to global video features along the channel dimension. This approach leads to a severe disconnect between spatial and temporal features. Neither simple feature concatenation nor implicit supervision enhancement can handle extremely complex industrial backgrounds. During inference, the network remains highly susceptible to severe interference from the large, static backgrounds of industrial workshops.

[0027] This invention proposes a person-object interaction mask attention mechanism, which uses information from multiple interactive objects in video frames (including object bounding boxes, object local features, and object spatial geometric masks) as explicit prior inputs to the network. It aligns and injects the local semantic information of the interaction between people, hands, and objects with global video features. This overcomes the problems of existing methods, such as the severe separation between spatial and temporal features, and the network's susceptibility to severe interference from extremely complex industrial backgrounds, such as large static areas in industrial workshops, during inference.

[0028] This invention proposes a method for explicit-driven multi-hole coding gating based on human-object interaction features. It utilizes the "human-object interaction state features" output by the human-object cross-attention module as the underlying signal. When strong human-object physical contact is perceived, the "fine-grained hole branch" is actively activated to capture high-frequency micro-movements. When the hand is idle or there is no physical target, the high-frequency branch is closed to preserve long-term temporal context. This overcomes the shortcomings of existing models, such as fixed receptive fields or gating mechanisms that rely solely on linear mapping of features for blind self-learning. It effectively solves the pain point of small differences in movements and easy confusion in industrial scenarios.

[0029] The advantages of this invention are: for massive industrial production videos with extremely complex backgrounds and subtle differences in process actions, a video action detection method with strong anti-interference ability and high recognition and localization accuracy by fusing human-object interaction mask attention and multi-hole gating is proposed. By effectively fusing entity space boundary masks and local interaction semantic information, static background noise is isolated, and the adaptive switching of the temporal receptive field is driven by the interaction state, which greatly improves the model's attention to the region of subtle "human-object interaction" actions and the moment of physical contact, and enhances the ability to recognize fine-grained human actions and accurately locate temporal boundaries in complex scenes. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall process architecture of the method of the present invention.

[0031] Figure 2 This is a schematic diagram of the structure of the person-object interaction mask attention module in the method of this invention.

[0032] Figure 3 This is a functional block diagram of the method of the present invention, showing the logical links and data flow of core modules such as data preprocessing, interactive sensing gate encoder decoder, etc. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Those skilled in the art should understand that the embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0034] Example 1

[0035] This embodiment provides a video temporal localization method based on interactive attention for action-oriented interactive target perception.

[0036] The method disclosed in this embodiment is preferably implemented in the PyTorch 2.2 deep learning framework and executed on a high-performance computing workstation equipped with an NVIDIA GeForce RTX 4090 GPU. To verify the effectiveness of this invention, this embodiment constructs an experimental environment based on real industrial production line monitoring data. The specific implementation process follows the steps described in S1 to S4 above: Step S1: Collect video image sequence data of industrial processes, as well as corresponding temporal action annotation data, including action category, action start and end time, and bounding box coordinates of the person; preprocess all data sequences to establish a parallel aligned input stream at the frame level and object level, and divide it into training set, validation set, and test set.

[0037] In this embodiment, the specific details are as follows: S11 data source configuration: Video data acquisition: High-definition industrial cameras were deployed in a food packaging workshop, capturing data at a resolution of 1920. The video is a 1080p, 30fps overhead monitoring video, with the camera mounted at a height that ensures it captures the entire workflow of the workers. The video content covers standard procedures such as "opening the machine cover," "operating the machine," and "closing the machine cover."

[0038] Motion data annotation: The video is manually annotated. The annotation content includes: motion category, motion start timestamp, and motion end timestamp.

[0039] S12 Data Preprocessing and Alignment: Video preprocessing: Scaling the original video to 224 at 30fps. A resolution of 224 is used as input to the video's basic model to obtain the video's global temporal features.

[0040] Object extraction preprocessing: The pre-trained LLMDet open-vocabulary object detector is used to infer the video. Based on the action annotation data, different prompt words are applied for object detection inference within the corresponding start and end time intervals for different action categories. The bounding boxes of objects with a confidence greater than 0.3 in each frame are extracted. Then, the bounding box coordinates (x1, y1, x2, y2) are normalized to the interval [0, 1] (for example, if the action "operate screen" appears between 09:31 and 09:40, the corresponding prompt word is "Hand.Person.Screen.", and the object detector will extract the bounding boxes of "hand, person, screen" objects with a confidence greater than 0.3 in each frame within the time period of 09:31-09:40 based on the prompt word).

[0041] Local object feature extraction: For the valid bounding boxes output by the object detector, a region of interest (ROI) cropping operation is performed on the original image of the corresponding frame to isolate local regions such as people, hands, and target objects from the complex background. Subsequently, the cropped local image patches are input into a lightweight basic convolutional network to extract the local object visual features.

[0042] Coordinate normalization and high-dimensional position embedding: Considering that local object features lose their absolute position information in the original image after cropping, this step performs geometric compensation on the original coordinate information of the bounding box. Specifically, the coordinates of the top left and bottom right corners of the bounding box are divided by the absolute width and height of the image, respectively, and mapped to the normalized interval [0, 1]. Subsequently, using a multilayer perceptron or linear projection layer, the normalized discrete coordinate vector is mapped into continuous high-dimensional position embedding features.

[0043] S13 mask generation: First, obtain the set of bounding box coordinates of the local objects extracted in step S12 on the original video frame; initialize a dimension H The all-zero matrix W serves as the basic mask tensor. Subsequently, based on the mapped bounding box coordinates, regional weight assignment operations are performed in the spatial dimension. Specifically, for pixel grid coordinate regions falling inside the bounding boxes of the hand and interactive objects, their corresponding mask element values ​​are modified to high-weight activation values; for regions outside the bounding boxes, their mask element values ​​are kept to low weight. Since the positions of the hand and objects in the video sequence continuously shift over time, this spatial geometric mask is dynamically generated in the temporal dimension and updated frame by frame, forming a three-dimensional spatiotemporal mask tensor.

[0044] S14 Data Cleaning and Dataset Partitioning: After removing invalid samples that do not involve workers or have blurry or distorted images from the camera, the processed video clips are divided into training, validation, and test sets according to a preset ratio for subsequent model training and evaluation.

[0045] Step S2: Temporal motion detection and localization based on person-object interaction. Step S2 combines explicit spatial geometric priors with dynamic gating mechanisms to achieve suppression of complex industrial backgrounds and accurate capture of fine-grained movements. The specific implementation steps are as follows: S21 constructs a video temporal localization network based on interactive attention for action-interactive target perception. Its structure is as follows: starting from the input layer, it is subsequently connected to the interactive perception gate encoder, multi-scale adapter, query selector, ensemble decoder, and classification and regression output layer.

[0046] The interactive perception gated encoder is sequentially connected to a person-object cross-attention module and an interactive-driven multi-hole gated encoding module. The interactive perception gated encoder sequentially inputs a video feature tensor and a stitched-aligned multi-object attribute tensor, and outputs a unified temporal feature after fusion processing. The multi-scale adapter contains multiple sequentially connected mask adjusters, which iteratively downsample the features. The query selector is sequentially connected to a candidate segment generation module, a validity screening module, and a Top-K dynamic selection module. The integrated decoder contains multiple decoder layers, which are sequentially connected to a multi-head self-attention module, a deformable cross-attention module, a boundary-aware cross-attention module, and a feedforward neural network. The classification and regression output layer contains two parallel prediction network branches, where the classification head uses a single-layer linear network structure and the regression head uses a multi-layer perceptron structure.

[0047] Timing alignment of S22 input data: Obtain the global video feature sequence in the current batch, denoted as ,in For batch size, To ensure a uniform maximum timing length, in this embodiment, It has 3200 dimensions.

[0048] Extract the corresponding local object attribute set from the input stream. Since there may be multiple interactive objects in each frame, set the maximum allowed number of objects to be... In this embodiment =10; The extracted object attributes include local visual features, normalized spatial bounding box coordinates, object category labels, and spatial masks; The above attributes are concatenated along the feature dimension so that the multi-object features of a single frame are flattened into a compact one-dimensional vector.

[0049] Since it's impossible for every frame of a video to contain characters and interacting objects, for each video sample, based on the original video's time sequence, local object features corresponding to the frame are inserted at the time points where character and object interaction occurs. During the remaining time periods without characters or character / object interaction, zero vectors are inserted along the time dimension to construct a local object attribute feature tensor sequence with the same length as the global video feature sequence. , where D is the total attribute dimension of a single object.

[0050] S23 uses a feature fusion and background suppression based on a person-object interaction mask attention mechanism to align the features. and The input is fed into the interaction cross-attention module, which injects the semantic prior of the local interaction object into the global temporal stream.

[0051] The global video features are reduced in dimensionality using a linear transformation layer to generate a temporal query vector. ,in As the hidden layer feature dimension, this embodiment sets =512.

[0052] (1)

[0053] Analyze multi-object attribute tensors and process visual features through independent projection networks. Coordinate features and category labels (Processed through an Embedding layer), the three are finally added element-wise to construct a key vector that integrates spatial geometry and semantic priors. Sum value vector .

[0054] (2)

[0055] from Parse the validity mask to generate the mask matrix. Here, True represents the background or invalid region, which is assigned a very large negative value when participating in the calculation. To prevent completely empty frames from occurring in extremely complex industrial scenes, i.e., a frame in which no valid entities are detected at all, causing the attention activation function Softmax to calculate NaN, the following safe index forced alignment mechanism is introduced: when it is detected that all objects in a frame are masked (i.e., ... (If all are invalid), the mask of its 0th object will be forcibly removed.

[0056] Inject the aforementioned mask matrix into multi-head attention computation This makes the query vector Aligning only with actual interactive objects in the image, background noise is mathematically and physically removed: (3) The output after the above cross-attention calculation The system sequentially passes through residual connections, layer normalization, and a feedforward neural network to output high-purity human-object interaction state features. .

[0057] S24 interactive state-driven multi-hole gating timing coding: Interactive features that incorporate position encoding Input is sent to the multi-cavity gated encoding module. The encoding module contains... Each with different void ratios Parallel convolutional branches. Due to The system already contains highly enriched information about the strength of physical interactions, and the gating mechanism is directly driven by this feature to explicitly calculate the activation weights of each branch. : (4) Based on this gating weight, temporal features from different receptive fields are dynamically integrated to output spatiotemporal encoded features. : (5) Its physical significance lies in the fact that when intense physical interactions occur, the gating network actively amplifies the weights of fine-grained void branches (low void ratio). To capture the boundaries of high-frequency instantaneous movements; in scenes where no characters appear during the hand's idle period, the weight of the wide receptive field branch (high void ratio) is increased to capture long-term temporal context information, effectively addressing the problem of small differences in industrial movements.

[0058] S25 is based on joint prediction using a center-neighbor region integrated decoder: Will The sequence is mapped to an action query sequence, input into the ensemble decoder for multi-level feature crossing and iterative updates, and finally outputs the detection result through a dual-branch prediction head. Action category prediction: Decoding features using a multilayer perceptron. Mapped to the log odds of each possible action category, and the probability distribution is output through an activation function: (6) Action temporal boundary prediction: The center point coordinates and relative width of the action are regressed using another set of MLPs. An exponential transform is used to ensure the positive value of the width, and the normalized center point is output based on the fundamental time bias. With width : (7) Finally, based on the aforementioned center point and width, the system decodes and generates precise temporal boundary intervals for each action in the industrial video, thus achieving high-precision positioning.

[0059] Step S3: End-to-end network training based on bipartite graph matching and joint loss function: This step aims to construct an end-to-end training mechanism. By designing a reasonable matching strategy and joint objective function, it guides the interactive-aware gated encoder and ensemble decoder in the network model to perform collaborative parameter updates. The specific implementation steps are as follows: S31 end-to-end target assignment based on bipartite graph matching: Obtaining the prediction and true set: Obtain a fixed number of values ​​output by the network decoder during the current forward propagation. Action prediction set For the first One prediction ,in This represents the predicted probability distribution for action categories. The center point and relative width are normalized.

[0060] Let the set of actual process actions in the current video segment be . ,in The number of actual actions (usually) Fill the real set with an empty set so that its size also reaches the specified value. .

[0061] Matching cost function calculation and optimal permutation search: To find the optimal combination between the prediction set and the true set. Calculate the bipartite graph matching cost loss for all possible permutations. This cost is a combination of classification cost and temporal boundary cost: (8) For non-empty actions, the pairwise matching cost is defined as: (9) The matching process solves for the optimal allocation index in polynomial time by calling the Hungarian algorithm, ensuring that each real action is precisely assigned to a unique prediction slot, thus eliminating duplicate predictions at the source.

[0062] Construction of the S32 joint objective function and end-to-end parameter optimization: After determining the optimal matching index between the predicted target and the ground truth annotations, this embodiment constructs a joint objective loss function for backpropagation to update network weights. This objective function consists of an action classification loss. Action boundary regression loss The weighted composition is not only computed at the last layer of the decoder, but also incorporates a deep supervision (Auxiliary Loss) mechanism.

[0063] Action classification loss: Considering that the vast majority of time frames in the video stream belong to the background or action transition period, Focal Loss is used in the classification head to solve this serious class imbalance problem. Let the one-hot encoding of the true label be... The calculation formula is as follows: (10) in, The predicted probability of the class log odds output by the model after Sigmoid activation; This represents the total number of truly valid actions within the current batch, used for loss normalization; in this embodiment, the focus parameter... Set to 2.0, balance parameter Set to 0.25.

[0064] Action boundary regression loss: Temporal boundary regression loss It consists of two parts: L1 regression loss for center point and width, and DIoU loss for temporal overlap.

[0065] Compared to the standard temporal IoU loss, DIoU additionally considers the normalized distance penalty between the center point of the predicted interval and the center point of the true interval, which can continuously provide gradients when the two intervals do not intersect, thereby significantly improving the speed and accuracy of temporal boundary regression. (11) In the formula, The squared Euclidean distance between the predicted center and the true center. It is the square of the span of the smallest closure interval containing both.

[0066] Combining the above losses yields the final joint loss:

[0067] Step S4: During the inference phase, the data paths for loss calculation and gradient backpropagation related to training are blocked. The video image sequence of the test set and the detected object information are input into the trained model to infer the enhanced temporal features containing high-frequency interaction information and transmit them to the temporal action detection network to directly output the final action detection results.

[0068] S41: Inference Deployment Configuration: During the model deployment phase, the label data and loss function calculation modules will no longer be loaded; Specifically, the ground truth values ​​used to provide supervision signals and the joint objective loss function used for parameter updates during the training phase are only needed during model learning. During the inference phase, the system retains only the forward inference path. By removing the backpropagation computation graph, memory usage is reduced by approximately 60%, significantly alleviating memory pressure.

[0069] S42: End-to-end feature mapping and prediction: The test set video sequence is preprocessed via dual streams and then input into the feature extraction network and the video temporal localization network based on interactive attention for action-oriented interactive target perception; the object interaction fusion output is performed to enhance the temporal features, and these features are input into the interaction state-driven multi-hole gating coding module; the joint neighborhood decoder generates an action query vector sequence based on the received encoded features, and uses the classification head and regression head to solve each query vector to obtain the final action instance.

[0070] Example 2

[0071] This embodiment relates to a video temporal localization system based on interactive attention for action-oriented interactive target perception. The system is constructed based on the method described in Embodiment 1. Through modular design, the system decouples complex data processing, feature fusion, and spatiotemporal prediction processes, and optimizes performance for industrial scenarios. Specifically, the system includes: a multi-source data acquisition and preprocessing module; a person-object interaction mask attention and background suppression execution module; an interaction state-driven temporal coding and multi-scale adaptation module; a mid-neighbor region integrated decoding and action set prediction module; and an end-to-end collaborative training and inference module.

[0072] Multi-source data acquisition and preprocessing module: This module is configured as the data access layer and the central hub for the underlying feature extraction of the system. It is responsible for converting unstructured industrial monitoring videos into network-computable aligned feature tensors. The specific execution logic includes methods such as global temporal feature extraction, local object feature extraction, and feature temporal alignment.

[0073] The Person-Object Interaction Mask Attention and Background Suppression Module is the core computational unit of the system. It injects local interactive object semantics into global temporal features to eliminate interference from large areas of static background and irrelevant noise in industrial workshops. Specifically, it includes functions such as fine-grained feature fusion, dynamic geometric mask injection, security mechanisms, and feature output.

[0074] Interactive State-Driven Temporal Coding and Multi-Scale Adaptation Module: This module is responsible for dynamically modeling high-purity interactive state features over time to address the pain point of subtle differences and easy confusion in industrial actions. Specifically, it includes functions such as multi-hole gating coding, adaptive receptive field switching, and multi-scale adaptation.

[0075] The intermediate neighbor region integrated decoding and action set prediction module: This module is responsible for the final temporal boundary localization and action classification tasks, specifically including functions such as multi-layer feature iteration and dual-branch joint prediction.

[0076] End-to-end collaborative training and inference module: This module is the main control unit of the system, switching between training and inference modes according to the actual application scenario. Training mode: Obtain the predicted set output by the network and the set of manually labeled real actions, and use a bipartite graph matching algorithm to achieve one-to-one optimal matching and eliminate duplicate predictions. Construct a multi-objective joint loss function, and use the gradient signal of this joint loss to backpropagate and collaboratively update the parameters of each network module in the system end-to-end.

[0077] Inference Mode: When deployed on an actual industrial production line, it receives industrial video to be detected and real-time extracted local object information. The module blocks all backpropagation paths, and after forward feature propagation and decoding, directly outputs a set of prediction results. Built-in logic removes predictions with confidence levels below a preset threshold, and finally outputs behavior recognition and temporal localization results containing "action category, start timestamp, end timestamp, and confidence level".

[0078] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A video temporal localization method based on interactive attention for action-oriented interactive target perception, characterized in that, Includes the following steps: S1. Collect video sequence data of industrial processes, corresponding temporal action annotation data, and human-object bounding box coordinates; perform global and local feature extraction, region of interest cropping, coordinate normalization, high-dimensional position embedding, and dynamic geometric mask generation on the video sequence data to establish a frame-level and object-level parallel aligned input stream, and divide the processed data into training set, validation set, and test set; S2. Construct a video temporal localization network based on interactive attention for action interaction target perception. Its structure is as follows: starting from the input layer, it is followed by an interactive perception gated encoder, a multi-scale adapter, a query selector, an ensemble decoder, and a classification and regression output layer. Construct a human-object joint representation containing entity spatial coordinates and local visual features. Input the aligned global temporal features and local object features into the interactive perception gated temporal encoder. Use the human-object interaction cross-attention mechanism to suppress background noise and inject fine-grained interactive semantics. Output the spatiotemporal coding features that dynamically adapt to the action boundary through the multi-hole gated temporal coding module driven by the interaction state features. Further output the action category probability and the precise temporal start and end boundaries through the ensemble decoder. S3. Construct a multi-objective joint loss function, and use the bipartite graph matching algorithm to perform a one-to-one optimal match between the predicted action set output by the network and the real action set; calculate the joint loss composed of classification focus loss, center point-width distance loss and intersection-union ratio loss based on the matching results, and use the backpropagation algorithm to update the network parameters until convergence, and obtain the trained video action detection model; S4. In the inference stage, a forward feature propagation and decoding method is adopted to block the data path of loss calculation and gradient backpropagation in the training stage. The industrial process video to be detected and the local object information extracted in real time are input into the trained model, and the prediction set containing the action category confidence and the start and end timestamps of the action is directly output, and low confidence results are removed.

2. The method according to claim 1, characterized in that, Step S1 includes: S11: Data source configuration: Collect process operation videos captured by industrial site monitoring cameras as raw input; collect labeled files containing the start and end times and categories of various process actions as ground truth; extract global spatiotemporal features using a pre-trained video base model; extract bounding box information of people, hands and interactive objects in each frame using an open vocabulary object detector; S12: Preprocessing and Alignment: Perform uniform resolution adjustment on the video frame sequence; based on the bounding boxes of the object detector, perform region of interest cropping on the original image and extract local object visual features; at the same time, normalize the bounding box coordinates and map them into high-dimensional position embeddings; S13: Mask generation: Generate a dynamic geometric mask in the spatial dimension based on the bounding box coordinates, assign high weights to the areas of the human hand and interactive objects, and low weights to the background areas, forming a three-dimensional spatiotemporal mask tensor that is updated frame by frame. S14: Data cleaning and dataset partitioning: Remove invalid samples that are blurry or severely occluded, and divide the processed feature sequence data into training set, validation set and test set.

3. The method according to claim 1, characterized in that, In step S2, the interactive perception gate encoder is sequentially connected to the person-object cross-attention module and the interactive-driven multi-hole gate encoding module. The interactive perception gate encoder sequentially inputs the video feature tensor and the spliced ​​and aligned multi-object attribute tensor, and outputs a unified temporal feature after fusion processing. The multi-scale adapter includes multiple mask adjusters connected in sequence. By iteratively looping through the mask adjusters, the features are downsampled and transformed. The query selector is connected in sequence to the candidate fragment generation module, the validity screening module, and the Top-K dynamic selection module. The integrated decoder includes multiple decoder layers, in which a multi-head self-attention module, a deformable cross-attention module, a boundary-aware cross-attention module, and a feedforward neural network are connected in sequence. The classification and regression output layer includes two parallel prediction network branches, where the classification head adopts a single-layer linear network structure and the regression head adopts a multi-layer perceptron structure.

4. The method according to claim 3, characterized in that, Step S2 includes: S21. Temporal alignment of input data: Obtain global temporal features of the video sequence and extract the set of local object attributes of the corresponding frame from the input stream (including local visual features, spatial bounding box coordinates, object category labels and validity masks); for samples of different lengths, perform padding alignment in the temporal dimension, flatten and stitch the local object attributes to construct a global video feature tensor and a multi-object attribute tensor of consistent length. S22. Feature fusion and background suppression based on person-object interaction mask attention mechanism: The aligned global temporal features and local object features are input into the person-object interaction cross attention module in the interaction-aware gated temporal encoder for feature fusion. S23. Interaction-state driven multi-hole gating temporal coding: The human-object interaction state features fused with position coding are input into the multi-hole gating coding module for dynamic modeling of the temporal receptive field; S24. Joint prediction based on center-neighbor region integrated decoder: The spatiotemporal encoded features are input into the center-neighbor region integrated decoder network after passing through a multi-scale adapter and a query selector; the multi-scale adapter transforms the features into multi-scale representations with different receptive fields of duration through downsampling, and the query selector selects the query vectors and their initial positions that are most likely to correspond to the action instances, thereby achieving initial focusing from dense features to key targets; the decoder performs multi-level feature iterative updates by combining the query vectors through a cross-attention mechanism.

5. The method according to claim 4, characterized in that, The global feature injection and background suppression process in step S22 includes: generating a temporal query vector Q from global video features through linear projection; fusing local visual features, coordinate features, and category labels to construct a key vector K and a value vector V; generating an attention key-value mask matrix using the parsed validity mask and introducing a secure indexing mechanism to force the complete empty frames; and injecting the mask matrix into the multi-head cross-attention calculation so that the query vector Q is aligned only with the real interactive entity regions in the image, thereby removing background noise.

6. The method according to claim 1, characterized in that, The multi-hole gated temporal coding in step S23 includes: setting multiple parallel convolutional branches with different hole rates, and using human-object interaction state features to explicitly drive the gating mechanism to calculate the activation weights of each branch; when the feature representation undergoes intense physical interaction, the gating network adaptively activates the low-hole rate branch to capture instantaneous high-frequency actions; during periods of no interaction and smoothness, the high-hole rate branch is retained to extract long temporal context.

7. The method according to claim 1, characterized in that, The integrated decoder in step S24 uses a dual-branch prediction head. The classification head outputs the log-probability of the action category corresponding to each time series query. The regression head outputs the normalized action interval center point and width through center point mapping and width exponential transformation to decode and generate accurate time series boundary intervals.

8. The method according to claim 1, characterized in that, The construction of the multi-objective joint loss function in step S3 includes: a weighted sum of action classification loss and boundary regression loss; the classification loss uses Focal Loss to address the imbalance between positive and negative samples; the regression loss uses DIoU Loss to constrain the overlap between the predicted time period and the actual time period; during the collaborative update of parameters, the gradient signal generated by the joint loss is used to update the network parameters end-to-end.

9. The method according to claim 1, characterized in that, The temporal action detection network output mechanism in step S4 includes: using the features output by the decoder to calculate the action; mapping the decoded features to a category probability vector through a classification generation layer; mapping the decoded features to a center point offset and width offset relative to the reference anchor point through a regression generation layer; finally, decoding the offset obtained from regression into an absolute timestamp, and performing non-maximum suppression to remove redundant predictions, and outputting the final action instance.

10. A video temporal localization system based on interactive attention for action-oriented interactive target perception, characterized in that, include: The multi-source data acquisition and preprocessing module is used to acquire video and labeled ground truth, extract global spatiotemporal features, local object bounding boxes and RoI visual features, and generate dynamic geometric masks. The module for handling person-object interaction masking attention and background suppression uses global features as query vectors and local features as key vectors, injecting dynamic geometric masks to filter background noise in non-interactive areas and outputting high-purity person-object interaction features. The module for interaction state-driven temporal coding and multi-scale adaptation uses interaction state features to drive a multi-hole gating mechanism to adaptively switch the temporal receptive field, outputting high-resolution spatiotemporal encoded features and generating a multi-scale feature pyramid. The module for integrated decoding of neighboring regions and action set prediction directly predicts the log-probability of the output action category and precise temporal start and end boundaries through a dual-branch network of classification and regression heads. The end-to-end joint optimization module based on bipartite graph matching constructs a joint objective function including classification focus loss and regression loss, and uses bipartite graph matching to achieve one-to-one optimal allocation and parameter updates. The online inference execution module is used to block the backpropagation path during the inference phase and directly output the behavior recognition and temporal localization results based on the local object information extracted in real time.

Images