Action detection method and device based on feature decoupling and timing cooperative calibration

Abstract

The application discloses a motion detection method and device based on time sequence cooperative calibration of feature decoupling, which comprises the following steps: monitoring video, extracting segment-level time sequence features through a backbone network, and then dividing the features into a worker branch and a device branch; using a feature decoupling module to output independent pure features with semantic anchoring through the back propagation of the detection head and the use of the adversarial training; outputting proposals through an adaptive segment generation module, and enhancing the proposals through a time sequence memory network; inputting the enhanced features and the proposals into a cross-subject time sequence cooperative module, performing classification and regression after time sequence RoI alignment, using a structured rule library generated by a large language model, combining scene segmentation and MLP dynamic prediction rule weights, and constructing a cooperative loss to force the time sequence alignment of worker motion and device state; and finally generating a structured text through a time axis generator. The application effectively eliminates the problems of feature mixing and detection inconsistency under the condition of only a small amount of labeling, and improves the motion detection accuracy and cross-subject time sequence consistency.

Description

Technical Field

[0001] This invention relates to the technical field of video action recognition, and in particular to an action detection method and apparatus based on feature decoupling and temporal collaborative calibration. Background Technology

[0002] With the widespread adoption of video surveillance technology, fixed cameras are deployed extensively in various scenarios, continuously collecting single-channel monitoring video and generating massive amounts of visual data. These videos cover worker actions and equipment operating status, which are of significant value for production process analysis, operational standard verification, and anomaly early warning. However, most current monitoring videos still rely on manual viewing or simple storage, lacking automated analysis methods, making it difficult to accurately extract the timing information of worker actions and equipment status from long-term continuous video.

[0003] While existing temporal action detection methods have achieved action localization and recognition in general scenarios, they face two core problems in real-world action recognition scenarios. First, workers and equipment in surveillance videos are visually highly mixed, and in practical applications, only a few worker action labels and equipment status labels are available, lacking strong annotations such as subject detection boxes or tracking trajectories. Traditional solutions typically rely on object detection or tracking algorithms to first locate the subject and then extract features separately. Such methods require significant subject annotation costs and are prone to failure when workers and equipment occlude or overlap. Second, there is an inherent temporal causal relationship between worker actions and equipment status. For example, after a worker completes a placement operation, the equipment should perform a clamping action. However, existing methods mostly employ independent detection strategies, lacking modeling of cross-subject temporal collaborative relationships, leading to logical conflicts in detection results along the timeline.

[0004] Furthermore, existing methods primarily rely on data-driven training, failing to effectively utilize the prior knowledge inherent in operational procedures and technological processes. A few methods that attempt to introduce rule constraints employ fixed rule formats, which cannot adapt to dynamic changes in different production scenarios, easily leading to over- or under-constraints, thus affecting detection accuracy and robustness.

[0005] Therefore, there is an urgent need for an action detection method that can automatically separate the visual features of workers and equipment under weak supervision conditions that require only a small number of worker actions and equipment status labels, and use prior temporal knowledge to perform temporal calibration on the detection results. Summary of the Invention

[0006] The present invention aims to at least partially address the aforementioned shortcomings of the prior art and provide an action detection method and apparatus based on feature decoupling and temporal collaborative calibration.

[0007] To address this, the present invention proposes a motion detection method based on feature decoupling and temporal collaborative calibration, which can automatically separate the visual features of workers and equipment, and use prior temporal correlation to mutually calibrate the detection results, eliminating feature mixing and detection inconsistency, and improving the accuracy and efficiency of worker motion and equipment status detection.

[0008] The first aspect of this invention proposes a temporal collaborative calibration action detection method based on feature decoupling, comprising the following steps: Step 1: Use a fixed camera to collect monitoring video sample data of workers and equipment. Preprocess the monitoring video sample data to obtain a sample dataset containing the start and end times and categories of worker actions and equipment status. Divide the sample dataset into training set, validation set and test set according to the set ratio. Step 2: Construct a temporal collaborative calibration action detection model FDCC based on feature decoupling. The FDCC model detects the start and end times and categories of worker actions and equipment states from surveillance videos captured by fixed cameras. The structure of the FDCC model is as follows: starting from the input surveillance video, after obtaining enhanced temporal features through the backbone network and shared Transformer encoder, it is divided into two parallel branches, namely the worker branch and the equipment branch. The undecoupled features of the two branches are input into the feature decoupling module to obtain mutually independent clean features of workers and equipment. Both branches are connected to the adaptive segment generation module and the temporal memory network module to obtain proposed and enhanced clean features. Finally, the cross-subject temporal collaboration module and the time axis generator output a structured text string of the start and end times and categories of worker actions and equipment states. Step 3: Input the training set into the FDCC model for training to obtain the trained FDCC model; and use the validation set to validate the trained FDCC model to obtain the final FDCC model. Step 4: Input the test set into the final FDCC model to complete the detection of the start and end times and categories of worker actions and equipment status in the test set.

[0009] Furthermore, step 1 includes: Step 1-1: Use a fixed camera to collect surveillance video containing workers and equipment, and select several random time periods as sample data; Steps 1-2: For worker actions and equipment status appearing in the sample data, label their start and end timestamps and categories in the sample data to construct a sample dataset; Steps 1-3: Divide the sample dataset into training set, validation set and test set according to the set ratio.

[0010] Furthermore, in the FDCC model described in step 2, the backbone network is used to extract segment-level temporal features from the surveillance video; the shared Transformer encoder is used to perform temporal context enhancement on the temporal features output by the backbone network; the feature decoupling module decouples the features of the two branches into independent worker clean features and equipment clean features through adversarial training and backpropagation of the detection head; the adaptive segment generation module is used to output worker action proposals and equipment state proposals; the temporal memory network module is used to perform local and global temporal enhancement on the clean features and output the enhanced worker clean features and equipment clean features; the cross-subject temporal collaboration module is used to detect and calibrate the start and end times and categories of worker actions and equipment states; and the timeline generator is used to convert the output of the cross-subject temporal collaboration module into a structured text string.

[0011] Furthermore, the structure of the FDCC model also includes: the backbone network is composed of a pre-trained 3D convolutional neural network; the shared Transformer encoder is composed of a multi-head self-attention layer and a feedforward network connected in series; both branches are composed of stacked 1D dilated convolutional layers, feature extractors, and detector heads connected in sequence, and the convolutional kernel parameters of the two branches are not shared; the feature decoupling module includes a discriminator network, which forces the feature extractor to generate features that cannot distinguish between positive and negative samples through adversarial loss, thereby minimizing mutual information; the adaptive segment generation module consists of an adaptive receptive field, a continuous boundary prediction head, and a boundary attention refinement submodule; wherein, the adaptive receptive field is composed of parallel 1D convolutional layers and a gating network, and the gating network is composed of a global average pooling layer and two fully connected layers connected in series and connected to Sof. The system employs the following mechanisms: tmax activation function; continuous boundary prediction head consisting of 1D convolutional layers connected to a Sigmoid activation function; boundary attention refinement submodule consisting of a local feature pruning unit and an attention weighting unit, whereby the local feature pruning unit extracts local feature fragments near coarse boundaries, and the attention weighting unit weights each position within the local feature fragments and calculates the fine boundary offset; temporal memory network module consisting of a short-term memory queue and a long-term recurrent memory, where the short-term memory queue is a first-in-first-out queue and the long-term recurrent memory is a fixed-size recurrent queue, with the two memory modules adaptively fused with the current window features through a cross-attention mechanism; and cross-subject temporal collaboration module consisting of a temporal RoI alignment layer, a classification multilayer perceptron, a regression multilayer perceptron, and a temporal collaborative calibration submodule.

[0012] Furthermore, the temporal collaborative calibration submodule also includes: a structured rule base and a dynamic rule weight generation unit; the structured rule base is generated offline by a large language model and contains temporal constraint rules between worker actions and equipment states; the dynamic rule weight generation unit, based on the window summary vectors stored in the long-term circular memory, performs scene segmentation by calculating the cosine similarity of adjacent window summary vectors, divides the video into several scene segments, extracts the center vector of each scene segment, inputs the center vector into a multilayer perceptron network, and outputs the weight of each rule in the current scene; the temporal collaborative calibration submodule generates a soft label matrix based on the temporal constraint rules in the structured rule base and the dynamic rule weights, combined with the temporal overlap between the proposed actions of each subject, and constructs a collaborative loss using the soft label matrix as a supervision signal, the collaborative loss being used to force worker actions and equipment states to align on the time axis.

[0013] Furthermore, the process of constructing the FDCC model also includes: designing the kernel size, dilation rate, and number of input and output channels of the 1D dilated convolutional layer in the feature decoupling module based on the dimension of the segment-level temporal features input to the FDCC model; designing the kernel size of the parallel 1D convolutional layer and the input and output dimensions of the gating network in the adaptive segment generation module; designing the input and output feature dimensions of the cross-attention mechanism in the temporal memory network module; designing the output dimension of temporal RoI alignment in the cross-subject temporal collaboration module; and designing the input and output dimensions of each layer of the classification and regression multilayer perceptron.

[0014] Furthermore, step 2 includes: Step 2-1: Input the surveillance video into the backbone network and extract segment-level temporal features; Step 2-2: Input the fragment-level temporal features into the feature decoupling module. After temporal context enhancement by the shared Transformer encoder, the features are sent to the worker branch and the equipment branch respectively. Under the supervision of worker action labels and equipment status labels, the semantic anchoring of the branches is achieved through backpropagation of the detection head. At the same time, the mutual information between worker features and equipment features is minimized through adversarial training, and the mutually independent clean worker features and clean equipment features are output. Steps 2-3: Input the worker clean features and equipment clean features into the adaptive fragment generation module, and output worker action proposals and equipment status proposals; Steps 2-4: Input the worker purity features and equipment purity features into the temporal memory network module respectively, fuse local and global features through the memory attention mechanism, and output the enhanced worker purity features and equipment purity features; Steps 2-5: Input the enhanced worker clean features and equipment clean features, as well as the worker action proposals and equipment status proposals, into the cross-subject temporal collaboration module. Extract the feature vectors of each proposal through temporal RoI alignment, and perform classification and boundary regression by each subject's detection head. At the same time, generate a soft label matrix through the temporal collaboration calibration submodule to construct the collaboration loss, and jointly optimize it with the classification loss and regression loss to achieve temporal alignment and mutual calibration of worker actions and equipment status. Steps 2-6: Input the co-calibrated detection results into the timeline generator to generate a structured text string containing the start and end times and categories of worker actions and equipment status.

[0015] Furthermore, step 3 includes: Step 3-1: Input the training set samples into the FDCC model in batches, and follow steps 2-1 to 2-6 to obtain the detection results of worker actions and equipment status for each batch; Step 3-2: Calculate the classification loss, regression loss, and collaborative loss respectively using the detection results obtained in Step 3-1 and the labeled information in the training set. Sum the three losses by weight to obtain the total loss value. Use the adaptive moment estimation optimizer as the gradient descent algorithm to achieve backpropagation and iterative training of the FDCC model. Step 3-3: In each training round, train the model using the training set and validate it using the validation set. After completing all training rounds, save the weight file of the model that performs best on the validation set.

[0016] A second aspect of the present invention relates to an action detection apparatus for time-series co-calibration based on feature decoupling, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the action detection method for time-series co-calibration based on feature decoupling of the present invention.

[0017] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the action detection method of the present invention based on feature decoupling and temporal co-calibration.

[0018] The working principle of this invention is: After the surveillance video captured by the fixed camera is processed by the backbone network to extract segment-level temporal features, it is first enhanced with temporal context through a shared Transformer encoder, enabling the features to fuse global temporal information. The enhanced features are then input into the worker branch and the equipment branch, respectively. Both branches consist of stacked 1D dilated convolutional layers with non-shared kernel parameters, forcing the learning of discriminative feature representations for their respective subjects. Based on this, a discriminator network, using the concatenated worker and equipment feature pairs at each time step as input, forces the feature extractor to generate features that cannot distinguish between positive and negative samples through adversarial loss, minimizing mutual information. This results in the output of independent, clean worker and clean equipment features without the need for subject detection boxes or trajectory annotations. Subsequently, the clean worker and clean equipment features are processed by an adaptive segment generation module to output action proposals, and then enhanced by a temporal memory network module through short-term and long-term memory and cross-attention mechanisms. Finally, the enhanced features and action proposals are input into the cross-subject temporal collaboration module. The proposed features are extracted through temporal RoI alignment, and the start and end times and categories are predicted by classification and regression. At the same time, a structured rule base is generated offline using a large language model, and unsupervised scene segmentation is performed online based on long-term memory window summaries and the rule weights are dynamically predicted. A collaboration loss is constructed to force worker actions and equipment states to align on the timeline. Finally, the timeline generator outputs a structured text string.

[0019] The innovative aspects of this invention: (1) Under weak supervision conditions requiring only a small number of worker action labels and equipment status labels, by combining adversarial mutual information minimization with backpropagation of the detection head, the mutual independence and semantic anchoring of worker and equipment features are achieved, eliminating the need for explicit subject detection boxes or trajectory annotations and eliminating visual feature mixing. (2) A knowledge-driven temporal collaborative calibration mechanism is proposed. Offline, the operation specifications are automatically converted into a structured rule base using a large language model. Online, unsupervised scene segmentation is performed based on long-term memory window summaries and rule weights are dynamically predicted. Collaborative loss is constructed to force worker actions and equipment status to align on the time axis, so that the collaborative constraint strength can adapt to different production conditions, effectively improving cross-subject temporal consistency and detection robustness.

[0020] The present invention has the following advantages: (1) Feature decoupling capability without bounding boxes under weak supervision. This invention, using only a small number of worker action labels and equipment status labels, leverages the branching structure after a shared Transformer encoder, combined with adversarial mutual information minimization and backpropagation of the detection head, to achieve independent separation and semantic anchoring of worker and equipment features. Compared to traditional schemes requiring a large number of subject bounding boxes or trajectory annotations, this significantly reduces data annotation costs and effectively separates features even when workers and equipment occlude each other, fundamentally eliminating the problem of visual feature mixing.

[0021] (2) Knowledge-driven cross-subject temporal collaborative calibration capability. This invention automatically transforms operational specification documents into a structured temporal rule base offline using a large language model, upgrading the rule source from manual setting to knowledge-driven. Online, it performs unsupervised scene segmentation based on window summary vectors from a long-term memory network, automatically identifying different production conditions, and dynamically predicting the applicable weight of each rule in the current scenario through a lightweight multilayer perceptron network. By forcing worker actions and equipment states to align on the time axis through a collaborative loss function, different subjects provide calibration signals to each other, effectively suppressing unilateral noise false detections and improving cross-subject temporal consistency and detection robustness.

[0022] (3) End-to-end structured timeline output capability. This invention transforms the detection results of worker actions and equipment status into structured text strings containing subject categories and start and end times through a timeline generator after collaborative calibration. These strings can be directly embedded with large language model prompts, providing a directly usable data interface for advanced semantic analysis tasks such as operation specification verification, efficiency statistics, or anomaly warning, thus expanding the application depth of action detection technology. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the steps of the method of the present invention.

[0024] Figure 2 This is a flowchart of the method of the present invention.

[0025] Figure 3 This is a schematic diagram of the method of the present invention. Detailed Implementation

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0027] Example 1

[0028] like Figure 1 , Figure 2 As shown, this embodiment relates to an action detection method based on feature decoupling and temporal collaborative calibration, including the following steps: Step 1: Use a fixed camera to collect monitoring video sample data of workers and equipment. Preprocess the monitoring video sample data to obtain a sample dataset containing the start and end times and categories of worker actions and equipment status. Divide the sample dataset into training set, validation set and test set according to the set ratio. Step 2: Construct a temporal collaborative calibration action detection model FDCC based on feature decoupling. The FDCC model detects the start and end times and categories of worker actions and equipment states from surveillance videos captured by fixed cameras. The structure of the FDCC model is as follows: starting from the input surveillance video, after obtaining enhanced temporal features through the backbone network and shared Transformer encoder, it is divided into two parallel branches, namely the worker branch and the equipment branch. The undecoupled features of the two branches are input into the feature decoupling module to obtain mutually independent clean features of workers and equipment. Both branches are connected to the adaptive segment generation module and the temporal memory network module to obtain proposed and enhanced clean features. Finally, the cross-subject temporal collaboration module and the time axis generator output a structured text string of the start and end times and categories of worker actions and equipment states. Step 3: Input the training set into the FDCC model for training to obtain the trained FDCC model; and use the validation set to validate the trained FDCC model to obtain the final FDCC model. Step 4: Input the test set into the final FDCC model to complete the detection of the start and end times and categories of worker actions and equipment status in the test set.

[0029] The steps for constructing the sample dataset in step 1 are as follows: Step 1-1: Continuously collect surveillance video containing workers and equipment using fixed cameras, with the collection period covering different production stages to ensure that the sample data covers worker actions and equipment status under various working conditions. Select several random time periods from the collected surveillance video as sample data to ensure that it covers the complete process of worker actions and equipment status changes; Steps 1-2: For worker actions and equipment states appearing in the sample data, label their categories, start timestamps, and end timestamps respectively, forming labeling information containing the start and end times and categories of worker actions and equipment states. In this embodiment, only the start and end times and categories of worker actions and equipment states need to be labeled; it is not necessary to label the spatial position of workers and equipment in the image or the detection box information. Steps 1-3: Divide the sample dataset into training, validation, and test sets according to a set ratio. In this embodiment, the training, validation, and test sets are divided in a 7:2:1 ratio.

[0030] In the FDCC model described in step 2, the backbone network is used to extract segment-level temporal features from the surveillance video; the shared Transformer encoder is used to perform temporal context enhancement on the temporal features output by the backbone network; the feature decoupling module decouples the features of the two branches into independent worker clean features and equipment clean features through adversarial training and backpropagation of the detection head; the adaptive segment generation module is used to output worker action proposals and equipment state proposals; the temporal memory network module is used to perform local and global temporal enhancement on the clean features and output the enhanced worker clean features and equipment clean features; the cross-subject temporal collaboration module is used to detect and calibrate the start and end times and categories of worker actions and equipment states; and the timeline generator is used to convert the output of the cross-subject temporal collaboration module into a structured text string.

[0031] Furthermore, the structure of the FDCC model described in step 2 also includes: the backbone network is composed of a pre-trained 3D convolutional neural network, including 3D convolutional layer 1, 3D convolutional layer 2, 3D convolutional layer 3, and 3D convolutional layer 4; the shared Transformer encoder is composed of a multi-head self-attention layer and a feedforward network connected in series; the two branches are respectively composed of three stacked 1D dilated convolutional layers and a detection head connected in sequence, wherein the worker branch uses 1D dilated convolutional layer 1, 1D dilated convolutional layer 2, and 1D dilated convolutional layer 3, and the equipment branch uses 1D dilated convolutional layer 4, 1D dilated convolutional layer 5, and 1D dilated convolutional layer 6, and the convolutional kernel parameters of the two branches are not shared; the feature decoupling module includes a discriminator network, including fully connected layer 2 and fully connected layer 3, which forces the feature extractor to generate features that cannot distinguish between positive and negative samples through adversarial loss, thereby minimizing mutual information; the adaptive segment generation module consists of an adaptive receptive field, a continuous boundary prediction head, and a boundary attention refinement submodule; wherein the adaptive receptive field is composed of parallel 1D convolutional layers The system consists of three convolutional layers: 1, 2, and 3, and a gating network. The gating network is composed of a global average pooling layer and two fully connected layers connected in series, and then connected to a Softmax activation function. The continuous boundary prediction head consists of a boundary offset prediction layer and an action score prediction layer, where the action score prediction layer is connected to a Sigmoid activation function. The boundary attention refinement submodule consists of a local feature pruning unit and an attention weighting unit. The local feature pruning unit is used to extract local feature fragments near the coarse boundary, and the attention weighting unit is used to perform attention weighting on each position in the local feature fragment and calculate the fine boundary offset. The temporal memory network module consists of a short-term memory queue and a long-term circular memory bank. The short-term memory queue is a first-in-first-out queue, and the long-term circular memory bank is a fixed-size circular queue. The two memory modules are adaptively fused with the current window features through a cross-attention mechanism. The cross-subject temporal collaboration module consists of a temporal RoI alignment layer, a classification multilayer perceptron, a regression multilayer perceptron, and a temporal collaborative calibration submodule.

[0032] Furthermore, the temporal collaborative calibration submodule also includes: a structured rule base and a dynamic rule weight generation unit; the structured rule base is generated offline by a large language model and contains temporal constraint rules between worker actions and equipment states; the dynamic rule weight generation unit, based on the window summary vectors stored in the long-term circular memory, performs scene segmentation by calculating the cosine similarity of adjacent window summary vectors, divides the video into several scene segments, extracts the center vector of each scene segment, inputs the center vector into a multilayer perceptron network, and outputs the weight of each rule in the current scene; the temporal collaborative calibration submodule generates a soft label matrix based on the temporal constraint rules in the structured rule base and the dynamic rule weights, combined with the temporal overlap between the proposed actions of each subject, and constructs a collaborative loss using the soft label matrix as a supervision signal, the collaborative loss being used to force worker actions and equipment states to align on the time axis.

[0033] Furthermore, the process of constructing the FDCC model also includes: designing the kernel size, dilation rate, and number of input / output channels of the 1D dilated convolutional layer in the feature decoupling module based on the dimension of the fragment-level temporal features input to the FDCC model; designing the kernel size of the parallel 1D convolutional layer and the input / output dimension of the gating network in the adaptive fragment generation module; designing the input / output feature dimension of the cross-attention mechanism in the temporal memory network module; designing the output dimension of temporal RoI alignment in the cross-subject temporal collaboration module; and designing the input / output dimensions of each layer of the classification and regression multilayer perceptron. In this implementation scheme, the specific parameters involved in the FDCC model are shown in Table 1.

[0034] like Figure 3 As shown, step 2 specifically includes: Step 2-1: Input the surveillance video into the backbone network and extract segment-level temporal features. Specifically, input the original video stream and divide it into several non-overlapping video segments in chronological order. Let the number of segments obtained after division be... The feature vector extracted from each segment by the backbone network is: ,in Let be the feature dimension. Then, the segment-level temporal feature sequence of the entire video can be represented as a matrix: .

[0035] In this embodiment, the backbone network is pre-trained on a large video dataset and fine-tuned end-to-end on the task data to adapt to the action recognition scenario.

[0036] Step 2-2: Input the fragment-level temporal features into the feature decoupling module. After temporal context enhancement by the shared Transformer encoder, the features are sent to the worker branch and the equipment branch respectively. Under the supervision of worker action labels and equipment status labels, the semantic anchoring of the branches is achieved through backpropagation of the detection head. At the same time, the mutual information between worker features and equipment features is minimized through adversarial training, and the mutually independent clean worker features and clean equipment features are output.

[0037] Specifically, the feature decoupling module first feeds the fragment-level temporal feature sequence into a shared lightweight Transformer encoder, which includes a multi-head self-attention layer and a feedforward network. Its input is the backbone network output, and its output is the context-enhanced feature sequence. , Each of them It integrates global timing information. The feature dimension. In this embodiment, the feature dimension... The value is 256.

[0038] A self-attention mechanism is used to capture temporal dependencies between segments, enabling features at each time step to be fused with global contextual information. The enhanced features are then input into the worker branch and the equipment branch, respectively. Each branch consists of stacked 1D dilated convolutional layers, and the kernel parameters of the two branches are not shared, thus forcing each branch to learn discriminative feature representations specific to the target subject. Let the output of the worker branch be... The device branch output is .

[0039] Under the supervision of worker action labels and equipment status labels, the worker branch is followed by a worker action detection head, and the equipment branch is followed by an equipment status detection head. Through backpropagation of the detection heads, the worker branch is forced to learn features that are conducive to the differentiation of worker actions, and the equipment branch is forced to learn features that are conducive to the differentiation of equipment status, thereby achieving semantic anchoring of branches.

[0040] Furthermore, to minimize the mutual information between worker characteristics and equipment characteristics, a discriminator network is set up. The input is the worker and equipment feature pairs concatenated at each time step. That is, given a time step... The concatenated vector is .

[0041] The discriminator outputs a scalar. , which represents the probability of the feature pair coming from the same time.

[0042] Constructing positive samples involves constructing true pairings between worker characteristics and equipment characteristics at the same time. Negative samples are constructed by pairing the current-time features of the same worker with the equipment features at random times. And random pairing across videos to increase the difficulty of discrimination.

[0043] Define the adversarial loss function as follows: , The first term encourages the discriminator to correctly identify positive samples, and the second term encourages the discriminator to correctly identify negative samples. The feature extractor minimizes... This is used to deceive the discriminator, making it unable to distinguish between positive and negative samples, thus forcing worker features and equipment features to become independent. When the discriminator cannot distinguish between feature pairs aligned to real time and randomly shuffled feature pairs, it indicates that effective decoupling has been achieved between worker and equipment features.

[0044] Preferably, to ensure training stability in small data scenarios, a progressive startup strategy is adopted, with the adversarial loss weight increasing linearly from 0 to the target value. In this embodiment, only task-related losses are calculated for the first 5 training cycles, without introducing adversarial loss, and then the adversarial loss weight increases linearly from 0 to 0.1. Simultaneously, an alternating optimization approach is used, updating the discriminator parameters 3 to 5 steps in each iteration, followed by updating the feature extractor parameters 1 step. Dropout and weight decay can be added to the discriminator network to suppress overfitting. In this embodiment, the dropout rate of the discriminator network is set to 0.3.

[0045] After decoupling, the outputs are independent worker pure characteristics. and equipment purity characteristics Their characteristic shapes are all .

[0046] Steps 2-3: Input the worker purity features and equipment purity features into the adaptive fragment generation module, and output worker action proposals and equipment state proposals; using the worker purity features... For example, it is input into an adaptive fragment generator, which contains three sub-modules: an adaptive receptive field module, a continuous boundary prediction head, and a boundary attention refinement module.

[0047] In the adaptive receptive field module, multiple parallel 1D convolutional layers are used to extract multi-scale temporal information, and a gating network is used to dynamically weight and fuse the multi-scale features. In this embodiment, three parallel 1D convolutional layers are used, with kernel sizes of 3, 5, and 7, and both the input and output channels have 256 channels. Padding is used to maintain the time length. The gating network takes the global average pooling vector of the input features as input, and generates three fusion weights through a multilayer perceptron and a softmax function. The outputs of the three branches are then weighted and summed to obtain the fused features. .

[0048] In continuous boundary prediction heads, in fused features The algorithm predicts the continuous offset of each time point relative to the start and end of the actual action, and outputs an actionability score, representing the probability that the time point belongs to the action region. In this embodiment, the offset loss is calculated only at 5 time points near the actual boundary, using a smoothed L1 loss; the actionability score is supervised using a binary cross-entropy loss.

[0049] In the boundary attention refinement module, local feature fragments are cropped centered on the coarse boundary. The local positions are then weighted and summed using an attention mechanism to calculate the fine boundary offset, thereby outputting a high-precision action proposal. Each proposal includes a start time, an end time, and a corresponding confidence level.

[0050] Equipment purity characteristics Device state proposals are obtained through an adaptive fragment generator with the same structure.

[0051] Steps 2-4: Input the worker purity features and equipment purity features into the temporal memory network module respectively, fuse local and global features through the memory attention mechanism, and output the enhanced worker purity features and equipment purity features.

[0052] Specifically, the temporal memory network module includes a short-term memory module and a long-term memory module, employing a two-layer memory structure. Let the index of the current processing window be... Window length is The decoupled worker's pure characteristics Based on the window division, we get the first one. Window feature matrix In this embodiment, the window length... The value can be 16 segments.

[0053] The short-term memory module stores the decoupled features of the most recent windows in a queue to retain second-level detailed features. In this embodiment, the queue capacity is... The value is 16, which means storing the original features of the 16 most recent windows. These features are uncompressed and retain the complete temporal structure inside the window.

[0054] The long-term memory module uses a learnable encoder to compress each window into a summary vector and stores it in a fixed-size circular memory to maintain a minute-level global summary vector. Let the encoder be... Then the first The summary vector for each window is .

[0055] In this embodiment, the capacity of the circular memory bank The value is 100.

[0056] The memory attention reading module uses the current window features For querying, local and global features are obtained from the short-term memory module and the long-term memory module respectively through an attention mechanism. The obtained features are then adaptively fused with the current window features using learnable weights to output the enhanced worker features. .

[0057] Similarly, regarding the purity characteristics of the equipment Perform the same operations as described above to obtain the enhanced device features. Through this mechanism, the model can simultaneously consider instantaneous details and long-term process context when making detection decisions, effectively mitigating false positives and false negatives caused by occlusion or brief movement.

[0058] Steps 2-5: Input the enhanced worker clean features and equipment clean features, as well as the worker action proposals and equipment status proposals, into the cross-subject temporal collaboration module. Extract the feature vectors of each proposal through temporal RoI alignment, and perform classification and boundary regression by each subject's detection head. At the same time, generate a soft label matrix through the temporal collaboration calibration submodule to construct a collaborative loss, and jointly optimize it with the classification loss and regression loss to achieve temporal alignment and mutual calibration of worker actions and equipment status.

[0059] Specifically, for each action proposal, a fixed-length feature vector is extracted using linear interpolation on the enhanced feature sequence through temporal RoI alignment. Let the set of worker action proposals be... The device status proposal set is Each of the proposals Including start time End time Through temporal RoI alignment operations, in the enhanced worker feature sequence The above proposes extracting a fixed-length feature vector for each worker. In the enhanced device status characteristics The above proposes extracting a fixed-length feature vector for each device. Then, the feature vectors of different subjects are input into the corresponding detection heads. Each detection head contains a classification multilayer perceptron and a regression multilayer perceptron. The classification multilayer perceptron outputs an action / state category probability vector. and ,in For the number of worker action categories, The number of device state categories; the offset correction amount of the regression multilayer perceptron output to the proposed start and end boundaries. and .

[0060] To achieve cross-entity temporal collaboration, this step introduces a scenario-adaptive rule weighting mechanism, the specific implementation details of which are as follows: (1) Offline Rule Generation: The operation specification document is transformed into a structured temporal rule base using a large language model. In this embodiment, the large language model uses an offline deployed Llama-7B. Preferably, the structured rule base is stored in JSON format, and each rule includes fields such as worker action category, equipment status category, temporal relationship, and allowed time window. For example, a rule can be represented as: worker action is "place part", equipment status is "clamp workpiece", the temporal relationship is that the equipment status begins within 0 to 3 seconds after the worker action ends, and the default weight is 0.8. Assume there are a total of [number missing] rules in the rule base. Rule number 1 The rule stipulates: worker action categories With device status category The relationships between them satisfy a temporal sequence, and the allowed time window is [ , The default weight is... The rule base remains fixed during training and inference, and the large language model does not participate in real-time computation.

[0061] (2) Online Scene Segmentation and Dynamic Weight Prediction: In this step, rule weights are dynamically predicted online based on scene segmentation and a multilayer perceptron network. Specifically, this includes: Based on the window summary vector stored in the long-term memory module Each of them Unsupervised scene segmentation is performed by calculating the cosine similarity of adjacent window summary vectors. Adjacency similarity is defined as follows: .

[0062] when When a scene transition point is identified, the entire video is divided into several semantically coherent scene segments. In this embodiment, The value is set to 0.7. The average of all window summary vectors within each scene segment is calculated as the scene center vector for that scene segment. Let the... The window index range contained in each scene segment is [ , If ], then the center vector of this scene segment is: .

[0063] The scene center vector The input is fed into a lightweight multilayer perceptron network, which consists of two fully connected layers, with an input dimension of [dimensionality missing]. The hidden layer has a dimension of 128, and the output dimension is the number of rules. After activation by the Sigmoid algorithm, the applicable weights of each rule in the current scenario are output: , in , For learnable weight matrix, and This is used as a bias. The multilayer perceptron network is jointly trained with the main network, and learns a reasonable mapping from scene context to rule weights through backpropagation of cooperative loss.

[0064] (3) Soft label matrix generation and collaborative loss construction: In this step, the specific method for constructing the collaborative loss function is as follows: Obtain the foreground confidence vector output by each subject detection head. For the ... The rule extracts the worker's proposed centralized prediction category as follows: probability vector ,in Similarly, the extraction equipment proposes a centralized prediction category as follows: probability vector Predicting the correlation matrix The product of confidence levels for different actions represents the strength of the correlation between the worker's predicted actions and the equipment's state. The specific calculation method is as follows: , .

[0065] Based on the offline-generated structured rule base and the online predicted scene-adaptive rule weights, a soft-label association matrix is ​​generated by combining the overlap of each subject's proposals on the time axis. For the... The rule is to first calculate the proposal. and Time overlap (IoU): .

[0066] Simultaneously define rules that satisfy indicator functions: , Rule-based soft label matrix Generated based on rule relationships, temporal overlap, or time intervals, representing the expected association strength of the prior rules. The specific calculation method is as follows: , in Input scene segment for the current video Adaptive weight vector Corresponding rules The component. For subject category pairs with undefined rules, the soft tag is set to 0.

[0067] Calculate the difference loss between the predicted association matrix and the soft-label association matrix, preferably using the mean squared error (MSE) loss: .

[0068] The total collaborative loss is the weighted sum of the losses of all rules: .

[0069] Finally, the collaborative loss is weighted and summed with the classification loss and boundary regression loss of each subject, and joint backpropagation optimization is performed. Let the worker classification loss be as follows: (Cross-entropy), device classification loss is Boundary regression loss is and (Smoothed L1). The total loss function is then: , in As a balance coefficient, in this embodiment, The value is set to 0.1. Through this collaborative loss, the actions / states of different subjects with high confidence are forced to be aligned in time, unilateral noise false detections are suppressed, thereby improving the temporal consistency of joint detection.

[0070] Steps 2-6: Input the co-calibrated detection results into the timeline generator to generate a structured text string containing the start and end times and categories of worker actions and equipment status.

[0071] Specifically, let the set of all detection results after collaborative calibration be... Each test result Includes main identifier (Indicates worker or equipment), Action / Status Category Start time (seconds), end time (seconds) and confidence score .

[0072] The timeline generation module first uses a fixed confidence threshold. Filter out low-quality test results, that is, select those that meet the requirements. The valid entries constitute a set. In this embodiment, the confidence threshold... The value is 0.5.

[0073] Then, the start and end times of each valid entry are converted from seconds to a uniform "HH:MM:SS" format. The conversion function is denoted as... , convert seconds to Convert to hour, minute, and second strings.

[0074] Next, merge the action / state entries from different entities and sort them in ascending order by start time. For entries with the same start time, sort them in ascending order by end time.

[0075] Finally, a fixed-format text line is generated for each entry, containing the main category, action / state category, start time, and end time, in the following format: , Concatenate all the text lines in order to obtain a complete structured text string.

[0076] Preferably, the structured text string can be directly embedded in the prompt words of a large language model for advanced semantic analysis tasks such as operation standard checking, efficiency statistics, or anomaly warning.

[0077] Step 3 includes: Step 3-1: Input the training set samples into the FDCC model in batches, and follow steps 2-1 to 2-6 to obtain the detection results of worker actions and equipment status for each batch. In this embodiment, the training set, validation set, and test set are divided in a 7:2:1 ratio, and the hyperparameters batch size, initial learning rate, and training epochs are set during training.

[0078] Step 3-2: Using the detection results obtained in Step 3-1 and the labeled information in the training set, calculate the classification loss, regression loss, and collaborative loss respectively. Summate the three losses using weighted averages to obtain the total loss value. Use the adaptive moment estimation optimizer as the gradient descent algorithm to achieve backpropagation and iterative training of the FDCC model. Specifically, the classification loss uses cross-entropy loss, the regression loss uses smoothing L1 loss, and the collaborative loss is jointly optimized with the classification and regression losses. The total loss function is... , Where λ is the balance coefficient, and in this embodiment, λ is set to 0.1. In this embodiment, the training set, validation set, and test set are divided in a ratio of 7:2:1, the initial learning rate is set to 0.001, the batch size is set to 32, and the training epochs are set to 100.

[0079] Step 3-3: In each training epoch, train the model using the training set and validate it using the validation set. After completing all training epochs, save the weight file of the model that performs best on the validation set. In this embodiment, an adaptive moment estimator optimizer is used for gradient descent optimization.

[0080] Step 4: Input the test set into the final FDCC model to complete the detection of the start and end times and categories of worker actions and equipment status in the test set.

[0081] In this embodiment, the test set is input into the trained FDCC model to obtain the detection results for different worker actions and equipment states in the test set. The detection results are compared with the start and end times and categories of worker actions and equipment states labeled for each sample in the test set, and the detection accuracy of the FDCC model is calculated. Experimental results show that on the test set, the FDCC model achieves a level of detection accuracy comparable to existing methods, with an average detection accuracy of 63.5%. Simultaneously, it significantly outperforms existing methods in cross-subject temporal consistency, achieving a cross-subject rule satisfaction rate of 87.6%, which is approximately 20% higher than the baseline model without collaborative calibration. This demonstrates that the present invention, through joint optimization of feature decoupling and temporal collaboration, can effectively improve the temporal alignment capability of multi-subject detection results while maintaining high-precision detection.

[0082] The working principle of this invention is: After the surveillance video captured by the fixed camera is processed by the backbone network to extract segment-level temporal features, it is first enhanced with temporal context through a shared Transformer encoder, enabling the features to fuse global temporal information. The enhanced features are then input into the worker branch and the equipment branch, respectively. Both branches consist of stacked 1D dilated convolutional layers with non-shared kernel parameters, forcing the learning of discriminative feature representations for their respective subjects. Based on this, a discriminator network, using the concatenated worker and equipment feature pairs at each time step as input, forces the feature extractor to generate features that cannot distinguish between positive and negative samples through adversarial loss, minimizing mutual information. This results in the output of independent, clean worker and clean equipment features without the need for subject detection boxes or trajectory annotations. Subsequently, the clean worker and clean equipment features are processed by an adaptive segment generation module to output action proposals, and then enhanced by a temporal memory network module through short-term and long-term memory and cross-attention mechanisms. Finally, the enhanced features and action proposals are input into the cross-subject temporal collaboration module. The proposed features are extracted through temporal RoI alignment, and the start and end times and categories are predicted by classification and regression. At the same time, a structured rule base is generated offline using a large language model, and unsupervised scene segmentation is performed online based on long-term memory window summaries and the rule weights are dynamically predicted. A collaboration loss is constructed to force worker actions and equipment states to align on the timeline. Finally, the timeline generator outputs a structured text string.

[0083] Example 2

[0084] This embodiment relates to an action detection device based on feature decoupling and time-series co-calibration, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the action detection method based on feature decoupling and time-series co-calibration of Embodiment 1.

[0085] Example 3

[0086] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the action detection method based on feature decoupling and temporal collaborative calibration of Embodiment 1. The embodiments described in this specification are merely examples of implementations of the inventive concept, and the scope of protection of this invention should not be considered limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A temporal collaborative calibration action detection method based on feature decoupling, characterized in that, Includes the following steps: Step 1: Use a fixed camera to collect monitoring video sample data of workers and equipment. Preprocess the monitoring video sample data to obtain a sample dataset containing the start and end times and categories of worker actions and equipment status. Divide the sample dataset into training set, validation set and test set according to the set ratio. Step 2: Construct a temporal collaborative calibration action detection model FDCC based on feature decoupling. The FDCC model detects the start and end times and categories of worker actions and equipment states from surveillance videos captured by fixed cameras. The structure of the FDCC model is as follows: starting from the input surveillance video, after obtaining enhanced temporal features through the backbone network and shared Transformer encoder, it is divided into two parallel branches, namely the worker branch and the equipment branch. The undecoupled features of the two branches are input into the feature decoupling module to obtain mutually independent clean features of workers and equipment. Both branches are connected to the adaptive segment generation module and the temporal memory network module to obtain proposed and enhanced clean features. Finally, the cross-subject temporal collaboration module and the time axis generator output a structured text string of the start and end times and categories of worker actions and equipment states. Step 3: Input the training set into the FDCC model for training to obtain the trained FDCC model; and use the validation set to validate the trained FDCC model to obtain the final FDCC model. Step 4: Input the test set into the final FDCC model to complete the detection of the start and end times and categories of worker actions and equipment status in the test set.

2. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 1, characterized in that, The dataset construction steps in step 1 are as follows: Step 1-1: Use a fixed camera to collect surveillance video containing workers and equipment, and select several random time periods as sample data; Steps 1-2: For worker actions and equipment status appearing in the sample data, label their start and end timestamps and categories in the sample data to construct a sample dataset; Steps 1-3: Divide the sample dataset into training set, validation set and test set according to the set ratio.

3. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 1, characterized in that, In the FDCC model described in step 2, the backbone network is used to extract segment-level temporal features from the surveillance video; the shared Transformer encoder is used to perform temporal context enhancement on the temporal features output by the backbone network; the feature decoupling module decouples the features of the two branches into independent worker clean features and equipment clean features through adversarial training and backpropagation of the detection head; the adaptive segment generation module is used to output worker action proposals and equipment state proposals; the temporal memory network module is used to perform local and global temporal enhancement on the clean features and output the enhanced worker clean features and equipment clean features; the cross-subject temporal collaboration module is used to detect and calibrate the start and end times and categories of worker actions and equipment states; and the timeline generator is used to convert the output of the cross-subject temporal collaboration module into a structured text string.

4. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 3, characterized in that, The structure of the FDCC model described in step 2 further includes: the backbone network is composed of a pre-trained 3D convolutional neural network; the shared Transformer encoder is composed of a multi-head self-attention layer and a feedforward network connected in series; both branches are composed of stacked 1D dilated convolutional layers, feature extractors, and detectors connected in sequence, and the convolutional kernel parameters of the two branches are not shared; the feature decoupling module includes a discriminator network, which forces the feature extractor to generate features that cannot distinguish between positive and negative samples through adversarial loss, thereby minimizing mutual information; the adaptive segment generation module consists of an adaptive receptive field, a continuous boundary prediction head, and a boundary attention refinement submodule; wherein, the adaptive receptive field is composed of parallel 1D convolutional layers and a gating network, and the gating network is composed of a global average pooling layer and two fully connected layers connected in series and then connected to a soft... The continuous boundary prediction head consists of 1D convolutional layers connected to a sigmoid activation function; the boundary attention refinement submodule consists of a local feature pruning unit and an attention weighting unit. The local feature pruning unit is used to extract local feature fragments near the coarse boundary, and the attention weighting unit is used to perform attention weighting on each position in the local feature fragment and calculate the fine boundary offset; the temporal memory network module consists of a short-term memory queue and a long-term circular memory bank. The short-term memory queue is a first-in-first-out queue, and the long-term circular memory bank is a fixed-size circular queue. The two memory modules are adaptively fused with the current window features through a cross-attention mechanism; the cross-subject temporal collaboration module consists of a temporal RoI alignment layer, a classification multilayer perceptron, a regression multilayer perceptron, and a temporal collaborative calibration submodule.

5. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 4, characterized in that, The temporal collaborative calibration submodule further includes: a structured rule base and a dynamic rule weight generation unit; the structured rule base is generated offline by a large language model and contains temporal constraint rules between worker actions and equipment states; the dynamic rule weight generation unit, based on the window summary vectors stored in the long-term circular memory, performs scene segmentation by calculating the cosine similarity of adjacent window summary vectors, divides the video into several scene segments, extracts the center vector of each scene segment, inputs the center vector into a multilayer perceptron network, and outputs the weight of each rule in the current scene; the temporal collaborative calibration submodule generates a soft label matrix based on the temporal constraint rules in the structured rule base and the dynamic rule weights, combined with the temporal overlap between the proposed actions of each subject, and constructs a collaborative loss using the soft label matrix as a supervision signal, the collaborative loss being used to force worker actions and equipment states to align on the time axis.

6. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 5, characterized in that, The process of constructing the FDCC model also includes: designing the kernel size, dilation rate, and number of input and output channels of the 1D dilated convolutional layer in the feature decoupling module based on the dimension of the segment-level temporal features input to the FDCC model; designing the kernel size of the parallel 1D convolutional layer and the input and output dimensions of the gating network in the adaptive segment generation module; designing the input and output feature dimensions of the cross-attention mechanism in the temporal memory network module; designing the output dimension of temporal RoI alignment in the cross-subject temporal collaboration module; and designing the input and output dimensions of each layer of the classification and regression multilayer perceptron.

7. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 6, characterized in that, Step 2 specifically includes: Step 2-1: Input the surveillance video into the backbone network and extract segment-level temporal features; Step 2-2: Input the fragment-level temporal features into the feature decoupling module. After temporal context enhancement by the shared Transformer encoder, the features are sent to the worker branch and the equipment branch respectively. Under the supervision of worker action labels and equipment status labels, the semantic anchoring of the branches is achieved through backpropagation of the detection head. At the same time, the mutual information between worker features and equipment features is minimized through adversarial training, and the mutually independent clean worker features and clean equipment features are output. Steps 2-3: Input the worker clean features and equipment clean features into the adaptive fragment generation module, and output worker action proposals and equipment status proposals; Steps 2-4: Input the worker purity features and equipment purity features into the temporal memory network module respectively, fuse local and global features through the memory attention mechanism, and output the enhanced worker purity features and equipment purity features; Steps 2-5: Input the enhanced worker clean features and equipment clean features, as well as the worker action proposals and equipment status proposals, into the cross-subject temporal collaboration module. Extract the feature vectors of each proposal through temporal RoI alignment, and perform classification and boundary regression by each subject's detection head. At the same time, generate a soft label matrix through the temporal collaboration calibration submodule to construct the collaboration loss, and jointly optimize it with the classification loss and regression loss to achieve temporal alignment and mutual calibration of worker actions and equipment status. Steps 2-6: Input the co-calibrated detection results into the timeline generator to generate a structured text string containing the start and end times and categories of worker actions and equipment status.

8. The action detection method based on feature decoupling and temporal collaborative calibration according to claim 1, characterized in that, Step 3 specifically includes: Step 3-1: Input the training set samples into the FDCC model in batches, and follow steps 2-1 to 2-6 to obtain the detection results of worker actions and equipment status for each batch; Step 3-2: Calculate the classification loss, regression loss, and collaborative loss respectively using the detection results obtained in Step 3-1 and the labeled information in the training set. Sum the three losses by weight to obtain the total loss value. Use the adaptive moment estimation optimizer as the gradient descent algorithm to achieve backpropagation and iterative training of the FDCC model. Step 3-3: In each training round, train the model using the training set and validate it using the validation set. After completing all training rounds, save the weight file of the model that performs best on the validation set.

9. An action detection device based on feature decoupling and temporal collaborative calibration, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the action detection method based on feature decoupling and time-series collaborative calibration as described in any one of claims 1-8.

Images