An assembly video real-time action segmentation method based on double-flow neural architecture search

Abstract

The application discloses an assembled video real-time action segmentation method based on a double-flow neural architecture search, and belongs to the technical field of video recognition. In view of the problems that an existing assembled action segmentation model is difficult to consider real-time performance and segmentation accuracy, double-flow feature fusion effect is poor, and boundary recognition is fuzzy, two independent neural architecture search spaces are constructed for RGB images and optical flow fields, and the optimal network topological structure is adaptively searched for the RGB and optical flow field branches; a double-flow interactive guiding module containing frame difference attention is designed, frame difference features and double-flow features are cross-modality channel weighted fusion; a metric learning with interval constraint is used to construct an action category prototype, and a time sequence focal loss function is combined to optimize feature distribution, so that features of a real category are gathered to the corresponding prototype, and features of other categories maintain intervals between the feature distributions. The application effectively improves the precision and real-time performance of assembled video action segmentation, and is suitable for online anomaly detection of a discrete manufacturing industry assembly process.

Description

Technical Field

[0001] This invention belongs to the field of video recognition technology, specifically relating to a real-time motion segmentation method for assembled videos based on dual-stream neural architecture search. Background Technology

[0002] Discrete manufacturing is a crucial component of my country's real economy, widely applied in fields such as automotive, precision electronics, and aerospace. Its production process consists of multiple discrete steps, characterized by complex product structures, diverse production batches, and strict logical sequences between steps. Although automated equipment is widely used in discrete manufacturing, some highly complex processes still rely on manual labor. Manual operation is prone to problems such as disordered assembly sequences, operational errors, and omissions of critical steps. If not detected in time, these defects can cascade down the chain, leading to finished product failures. Traditional manual inspection methods suffer from poor real-time performance, high missed detection rates, and continuously rising labor costs, making them unsuitable for the development requirements of intelligent manufacturing.

[0003] Currently, the detection of abnormal action sequences in the assembly process largely relies on action segmentation technology. Action segmentation technology provides core intelligent support for detecting out-of-order assembly processes by accurately identifying and understanding complex action sequences in long videos. While traditional action segmentation models have high accuracy, they are limited by massive computational demands and memory usage, typically requiring offline processing and failing to meet the real-time response requirements of industrial assembly inspection tasks. To achieve real-time segmentation, current mainstream real-time processing models often adopt a compromise, extracting only RGB features while ignoring optical flow field features. Compared to the dual-stream features extracted by offline detection models, the model's accuracy inevitably decreases. Furthermore, existing dual-stream processing mechanisms rely heavily on simple feature concatenation operations, making it difficult to effectively capture the inherent logical relationship between RGB features and motion states in the video stream, limiting further performance improvements. In addition, due to the significant differences in the duration of different actions in assembly videos, a fixed feature extraction architecture can lead to a mismatch between the temporal receptive field and the action scale. If the receptive field is too small, the global contextual information of long-duration actions is easily truncated; if the receptive field is too large, the fine-grained features of short-duration actions are easily over-smoothed. Existing action segmentation models generally use the Softmax function for classification. However, the traditional Softmax classification mechanism fails to effectively constrain inter-class distances, resulting in insufficient safety margins for the feature distributions of different actions. This feature space with unclear boundaries makes the model highly susceptible to interference from weak noise when processing action transition frames, leading to blurred boundary recognition. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, the present invention aims to provide a real-time action segmentation method for assembly videos based on dual-stream neural architecture search. This method effectively solves the challenge of extracting complex temporal features in real-time segmentation scenarios, improves the segmentation accuracy of assembly videos, and provides high-precision state input for out-of-order detection in assembly videos.

[0005] To achieve the above objectives, this invention provides a real-time motion segmentation method for assembled videos based on dual-stream neural architecture search, comprising the following steps: S1. Obtain the RGB image sequence of the assembly video; input the RGB image sequence into the lightweight optical flow field extraction model to extract the optical flow field; S2. Input the RGB image sequence and the optical flow field into the corresponding branches of the optimal backbone network for feature extraction to obtain RGB features and optical flow field features. The optimal backbone network is constructed based on neural architecture search. It is obtained by establishing independent search spaces for the RGB image and the optical flow field respectively and training the architecture parameters. After the training converges, the candidate convolution operation with the largest architecture parameters of each layer is retained. S3. Input the RGB features and the optical flow field features into the interactive guidance module. Calculate the frame difference features between adjacent frames through the interactive guidance module. Perform cross-modal concatenation of the frame difference features of different modalities with the original features of another modality. Generate channel-level mixing weights through the channel attention mechanism. Perform weighted fusion of the channel-level mixing weights with the original features of the corresponding modality to obtain the fused features. S4. The fusion features are matched with the pre-trained category prototypes to obtain the action classification results of each frame, thereby realizing real-time action segmentation of the assembly video. The category prototypes are obtained through metric learning training. The fusion features of the real category are clustered towards the corresponding prototype, while the fusion features of other categories maintain a gap with the corresponding prototype.

[0006] As a further improvement of the present invention, the optimal backbone network is obtained in the following manner: Independent neural architecture search spaces are constructed for RGB images and optical flow fields, respectively. Each search space consists of multiple convolutional layers, each configured with multiple candidate convolutional operations. Optimizable architecture parameters are bound to each candidate convolutional operation. Classification loss is calculated using an auxiliary classification head, and model weights and architecture parameters are updated based on backpropagation. After training convergence, candidate operations in each layer are pruned, retaining the candidate convolutional operation with the largest architecture parameter value in each layer, thus obtaining the optimal backbone network.

[0007] As a further improvement of the present invention, the candidate convolution operation adopts temporal causal convolution; wherein, the temporal causal convolution is configured with a cache space, the cache space is used to store historical frame features, and its capacity is one less than the temporal convolution kernel size; the feature extraction of temporal causal convolution depends only on the information of the current frame and the historical frames, and does not depend on the information of future frames.

[0008] As a further improvement of the present invention, the interactive guidance module calculates the frame difference features between adjacent frames, including: calculating the difference between adjacent frames of the RGB features to obtain RGB frame difference features, and calculating the difference between adjacent frames of the optical flow field features to obtain optical flow field frame difference features.

[0009]

[0010] in, For the first RGB feature difference results of the frame For the first RGB features of the frame For the first RGB characteristics of the frame; For the first The optical flow field characteristic difference results of the frame, For the first Optical flow data corresponding to the frame, For the first Optical flow data corresponding to the frame; The optical flow field frame difference feature and the RGB feature are concatenated along the channel dimension to generate the cross-modal input of the RGB branch. The RGB frame difference features and the optical flow field features are concatenated along the channel dimension to generate the cross-modal input of the optical flow field branch. .

[0011] As a further improvement of the present invention, the interactive guidance module generates channel-level hybrid weights and performs weighted fusion, specifically including:

[0012]

[0013] in, This is the first fully connected layer, acting as a bottleneck layer to perform channel dimensionality reduction on the input features; For the second fully connected layer, feature dimensionality is increased; To generate the attention weights for the RGB branch channels, Attention weights for optical flow field branch channels; For RGB modal input features, The input features of the optical flow field mode are represented by R, which represents the set of real numbers, indicating that all elements in the tensor are real numbers. Will Multiply the RGB features element by element to obtain the result. The optical flow field feature is multiplied element-wise, and the result is added to the original RGB feature and optical flow field feature to obtain the joint residual feature. The joint residual feature is then concatenated with the RGB feature and the optical flow field feature along the channel dimension and reconstructed by a fully connected layer to obtain the fused feature.

[0014] As a further improvement of the present invention, the training process of the pre-trained category prototypes includes: randomly initializing a set of learnable category prototypes, and performing the fusion feature with the category prototypes. Normalization operation; calculate the cosine similarity between the normalized fusion features and the prototypes of each category; apply an interval constraint to the cosine similarity of the real category and then map it to the prediction probability using Softmax; use the prediction probability to calculate the temporal focus loss as the optimization objective, so that the cosine similarity between the fusion features of the real category and the corresponding category prototype increases, and the cosine similarity between the fusion features of other categories and the corresponding category prototype decreases.

[0015] As a further improvement to this invention, an interval constraint mechanism is introduced, and the formula for calculating the prediction probability is:

[0016] in, For the predicted probability with interval constraints, For hyperparameters, This is the cosine interval penalty parameter.

[0017] As a further improvement of the present invention, in the training of the neural architecture search phase, temporal focus loss is used as the classification loss:

[0018] in, Represents a valid set, Represents the total number of frames in the valid set. Indicates the first The actual action category label corresponding to the frame. It is a category balancing weight. It is the model's predicted probability of the true label. It is a regulating factor.

[0019] As a further improvement of the present invention, a NAS resource loss is introduced during the training phase of the neural architecture search to control the computational overhead of the model:

[0020] in, This indicates the total number of network layers involved in the search. This represents the total number of candidate operations. This represents the probability of selection. Indicates the first The inherent computational cost of this operation, This is a penalty coefficient used to adjust the strength of resource constraints.

[0021] The beneficial effects of the above technical solution are as follows: 1. A dual-stream search space is constructed by introducing temporal causal convolution to ensure that feature extraction of the current frame depends only on the current and historical frames, thus satisfying the causal constraints of real-time inference. Independent RGB feature extraction and optical flow field feature extraction branches are set up in the search space to adaptively search for the optimal assembly video feature extraction architecture.

[0022] 2. An interactive guidance module is designed, and a frame difference attention mechanism is introduced. This module calculates the differences between adjacent video frames to explicitly extract temporally significant features that reflect changes in action and motion state, namely frame difference features. Subsequently, the dual-stream features and frame difference features are fused across modalities, and attention weights are calculated channel by channel to promote deep coupling between features. Combined with a multimodal joint residual strategy, the model's representation ability and detection performance are further improved.

[0023] 3. A metric learning approach is employed to construct highly discriminative category prototypes for each assembly action. By optimizing feature distribution, the feature vectors of the same action are forced to cluster spatially towards their prototypes, while simultaneously maintaining a clear safety margin between the feature distributions of different actions. This highly cohesive and widely spaced feature representation mechanism significantly improves the accuracy of action boundary recognition in assembly scenarios. Attached Figure Description

[0024] Figure 1 This is a diagram of the overall architecture of the present invention; Figure 2 This is a schematic diagram of optical flow field extraction according to the present invention; Figure 3 This is a flowchart of the optimal backbone network acquisition process of this invention; Figure 4 This is a flowchart of the interactive guidance module of the present invention; Figure 5 This is a prototype training diagram for the present invention category; Figure 6 This is a framework diagram of the neural architecture search space of the present invention; Figure 7 This is an architecture diagram of the interactive guidance module of the present invention; Figure 8 This is a prototype training framework diagram for this invention. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] It should be noted that, unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0027] like Figure 1 As shown, the real-time action segmentation framework for assembly videos based on dual-stream neural architecture search consists of four modules: an input network, an optimal backbone network, a neck network, and a head network. In the input network, the original RGB image is processed by a lightweight optical flow extraction model to obtain the optical flow field. The RGB image and the optical flow field are then fed into the optimal backbone network constructed based on neural architecture search technology for feature processing. This backbone network establishes independent neural architecture search spaces for the RGB image and the optical flow field, respectively. By updating the architecture parameters through an auxiliary classification head, the candidate operations with the largest architecture parameters at each layer are retained to establish the optimal network architecture. The neck network uses an interactive guidance mechanism to deeply fuse the dual-stream features. The head network uses a cosine similarity module to calculate the cosine similarity between the fused features and the prototypes of each category in a hypersphere. This similarity is optimized to train action category prototypes that are compact within each category and far apart between categories, ultimately achieving real-time action segmentation of the assembly video.

[0028] like Figure 2 As shown, the input network is mainly responsible for preprocessing the original assembly video stream with multimodal data, aiming to provide subsequent networks with dual-branch input data containing both static appearance information and temporal dynamic information. First, the system retains the continuous RGB image sequence as the spatial appearance branch and feeds it into the pre-trained lightweight optical flow extraction model. By calculating the displacement of each pixel in the horizontal and vertical directions between two adjacent frames, the system outputs the optical flow field corresponding to the original RGB image, thereby explicitly encoding the coherent motion trajectory during the assembly process.

[0029] like Figure 3As shown, this invention constructs independent neural architecture search spaces for RGB images and optical flow fields, respectively, and introduces a temporary auxiliary classification head to drive the architecture search. This search space consists of multiple convolutional layers, each with n pre-defined candidate convolutional operations, and each operation is bound to an optimizable architecture parameter. Features extracted from the dual-stream search space are fed into an interactive guidance module for deep fusion, and then processed by fully connected layers and a Softmax function to output classification prediction results. The model calculates the loss based on these prediction results and updates the model weights and architecture parameters through backpropagation. After the model converges, a pruning operation is performed on the NAS search space in the backbone network, retaining only the candidate convolutional operations with the largest architecture parameter values ​​in each layer, thereby obtaining the optimal backbone network and fixing it as the feature extraction network architecture best suited for the assembly task.

[0030] like Figure 4 As shown, this invention designs an interactive guidance module in the neck network to obtain frame difference attention: the frame difference information is interactively concatenated with the dual-stream features, and the feature weights of each channel are calculated by combining the channel attention mechanism. The feature weighting is completed by multiplying each channel. A multimodal joint residual mechanism is introduced to ensure stable gradient backpropagation during network training. Finally, the dual-stream features and fused features are concatenated, and the deep coupling of the dual-stream information is achieved through a fully connected layer.

[0031] like Figure 5 As shown, based on the optimal backbone network architecture, a metric learning method is introduced to learn a class prototype with intra-class compactness and inter-class disparity for each action category. A set of learnable class prototypes is randomly initialized, and the prototypes and the interaction-guided features are subjected to L2 normalization, and their cosine similarity is calculated. By optimizing this similarity, the feature vectors of the real category and the corresponding prototype vectors are made to align in the hyperspherical space, training highly discriminative class prototypes. Accurate real-time action segmentation is then achieved based on the similarity matching between features and prototypes.

[0032] The specific implementation steps of a real-time motion segmentation method for assembly videos based on dual-stream neural architecture search are as follows: (a) Input video data and initialize model parameters First, video stream data of the discrete manufacturing assembly process is acquired, sourced from production line vision sensors and industrial cameras. This invention uses continuous frame-by-frame images from the video stream as the basic processing unit. After acquiring the original RGB image sequence, it is input into a lightweight optical flow field extraction model to extract the optical flow field, thereby constructing dual-modal basic input data encompassing both spatial appearance and temporal motion.

[0033] The initialization parameter settings for the two-stream neural architecture search model include: the number of action categories. Category prototypes for each action category Architecture weight parameters for controlling network topology Feature scaling factor in metric learning classifier head Cosine interval penalty parameter Focusing parameters of timing focus loss And the computational resource penalty coefficient during the neural architecture search phase. .

[0034] (II) Constructing a neural architecture search space This invention employs an offline, fully supervised training method, but the feature processing flow strictly meets real-time requirements. First, the RGB image sequence and optical flow field data are... The input is in the form of the neural architecture search space. Here, B is the batch size, T represents the number of sequence frames per batch, C is the number of channels, and H and W represent the height and width of the image, respectively. Figure 6 As shown.

[0035] Each layer of the search space is configured with n candidate convolutional operations, and each operation is assigned a learnable architectural parameter. During the search phase, the candidate convolutional branches within a layer are specifically configured in this embodiment as three-dimensional depth-separable causal convolutions, including... Equal size; during the search phase, the same input features are received in parallel. After performing forward computation, the final output of this layer is a linear weighted sum of the computation results of these candidate convolutional branches, as shown in Equation (1):

[0036] in, Representing input features After the first The feature output results calculated by each candidate convolutional branch Architecture parameters configured internally All passed The function is transformed into a normalized multinomial distribution probability weight.

[0037] During training, the model updates these parameters using backpropagation. If a convolutional operation effectively reduces the model's loss, its corresponding architectural parameter weights are amplified; conversely, the weights are reduced. After the dual-stream data enters the search space, features are extracted in parallel through the aforementioned n candidate convolutions in each layer.

[0038] Furthermore, to meet the stringent real-time inference requirements, this invention employs temporal causal convolution to extract features, and establishes a dedicated cache space for each candidate temporal convolution, with its capacity set to one less than the temporal convolution kernel size. The characteristic of causal convolution is that its computation relies solely on information from the current and historical frames, strictly avoiding dependence on future frames, thereby ensuring the real-time nature of the inference process.

[0039] Specifically, the feature values ​​in the cache space are first initialized to 0, and then concatenated with the T-frame RGB image sequence along the time dimension. Subsequently, a causal convolution is slid along the time dimension to complete feature extraction. During convolution operations, this invention introduces a depthwise separable convolution mechanism, significantly reducing the computational complexity of the convolution operation.

[0040] (III) Dual-stream feature fusion based on interactive guidance module The two-stream features extracted via the NAS search space have a tensor dimension of Subsequently, the dual-stream features are processed through an interactive guidance module, deeply fusing the intrinsic correlation between RGB appearance features and motion states. The specific network structure of the interactive guidance module is as follows: Figure 7 As shown.

[0041] First, to capture temporal saliency, the differences between adjacent frames of RGB features and optical flow field features need to be calculated frame by frame. Specifically, the dynamic changes of features are quantified by calculating the L1 distance between features in two adjacent frames, and the frame difference of the initial frame is set to 0 by default, as shown in formulas (2) and (3):

[0042]

[0043] After obtaining the frame difference features of the two branches, cross-modal feature concatenation is performed. The frame difference of the optical flow field features is concatenated with the RGB features along the channel dimension, and the frame difference of the RGB features is concatenated with the optical flow field features. This interactive guidance module aims to enable the subsequent fully connected layers to effectively utilize the motion change information of the optical flow field to enhance the appearance features of RGB when performing dimensionality reduction and feature fusion, and to utilize the appearance change information of RGB to optimize the motion features of the optical flow field, as shown in formulas (4) and (5):

[0044]

[0045] Based on the concatenated cross-modal features, the network further generates channel-level hybrid weights, as shown in Equations (6) and (7):

[0046]

[0047] In the weight generation module, The first fully connected layer acts as the bottleneck layer, with 256 neurons, used for feature dimensionality reduction, reducing the input 2048-dimensional features to 256 dimensions. This operation not only achieves preliminary aggregation of cross-modal features but also significantly compresses the number of model parameters, thereby significantly improving computational efficiency.

[0048] Subsequently, The introduction of functions endows the model with nonlinear expressive power, enhancing the network's ability to fit and model complex temporal action changes.

[0049] Finally, through (The second fully connected layer) performs feature upsizing. Configured with 1024 neurons, it upscales the 256-dimensional features to 1024 dimensions, realigning the number of output channels with the original two-stream feature dimensions; and combines... The activation function generates channel-wise attention weight scores. This mechanism adaptively evaluates the information value of different channels, giving higher weights to key channels that contribute more to minimizing model loss.

[0050] After obtaining the channel weights, they are multiplied element-wise with the original features of the corresponding branches. Subsequently, a multimodal joint residual structure is constructed to ensure stable gradient backpropagation during network training while fusing dual-stream information, as shown in Equation (8):

[0051] Although the above residual addition operation achieves preliminary fusion, simple direct addition can easily lead to feature aliasing between different modalities. To solve this problem and preserve the unique expression of each modality, this invention combines residual features. With the original dual-flow characteristics , The concatenation is performed along the channel dimension, followed by dimensionality reduction and feature reconstruction using a fully connected layer, as shown in Equation (9):

[0052] The features output by the interactive guidance module can effectively uncover the intrinsic relationship between the two streams of information, enhance the model's ability to discriminate complex action changes, and provide a highly robust feature representation for real-time action segmentation tasks.

[0053] (iv) Obtaining the optimal backbone network architecture The fused features after interactive guidance are input into the fully connected classification head, mapping its feature dimensions to the target action category space. Then, the Softmax function is used to calculate the probability distribution of each video frame belonging to different action categories, and the weights and architecture parameters are jointly updated through backpropagation based on the classification loss. Specifically, the training in this stage uses temporal focus loss, which introduces a focus loss mechanism on top of cross-entropy loss, aiming to guide the model to pay more attention to samples that are difficult to classify, as shown in formula (10):

[0054] in The effective set represents the initial frame in the model's initial stages. Because causal convolution is used when processing time-series data, some frames haven't yet established sufficient historical memory, leading to unreliable predictions. To avoid these early unstable predictions interfering with the model's optimization direction, this invention introduces the concept of an effective set, discarding these earlier frames when calculating the loss. Represents the total number of frames in the valid set; Indicates the first The actual action category label corresponding to the frame; It is a class-balanced weighting system that addresses the problem of long-tailed data distribution (i.e., some action categories have few samples), by assigning larger weights to minority classes. Values ​​that mitigate the impact of class imbalance; It is the model's predicted probability of the true label; It is a moderating factor used to reduce the proportion of easily classifiable and simple samples in the total loss, prompting the model to retain and focus on learning difficult samples.

[0055] To effectively control the computational cost of the model during neural architecture search, this invention introduces a NAS resource loss function. Let the architecture parameters of the i-th candidate operation in the m-th layer of the network be... By applying the Softmax function to all architectural parameters of this layer, they can be converted into the selection probability distribution of each operation, as shown in Equation (11):

[0056] Based on this, by calculating the mathematical expectation of the cost of each candidate operation at each layer, the final NAS resource loss of the entire network is obtained, as shown in formula (12):

[0057] in, Indicates the total number of network layers involved in the search; Indicates the total number of candidate operations; Indicates the probability of selection; Indicates the first The inherent computational cost of each operation; the more complex the operation, the higher the cost. This is a penalty coefficient used to adjust the strength of resource constraints.

[0058] After the training in this stage converges, a pruning operation is performed on the neural architecture search space. In each layer of the network, only the candidate convolution operation with the largest architectural parameter weight is retained, and the rest of the branches are removed, thus obtaining the final optimal backbone network structure. This network architecture still uses the mechanism of causal convolution and depthwise separable convolution, and uses this to start the training of the category prototype.

[0059] (v) Training Category Prototype The training objective at this stage is to learn a set of class prototypes that are compact within classes and far apart between classes. The specific process is as follows: Figure 8 As shown.

[0060] First, the optimal network architecture after pruning is used to extract features from the bimodal input data; then, the deep information coupling of the bimodal features is achieved through the interactive guidance module.

[0061] Next, a set of learnable category prototypes is randomly initialized based on the dimensions of the coupled features, and the feature vectors of the interactive output are executed simultaneously with the category prototypes. A normalization operation maps the features to a unit hypersphere (i.e., all features have a magnitude of 1). Based on this, the cosine similarity between each frame's features and its class prototypes is calculated. Since normalization has already been performed, calculating the cosine similarity is equivalent to calculating the inner product of two vectors. Finally, backpropagation optimizes the network parameters to maximize the cosine similarity between features and their true class prototypes, while minimizing the cosine similarity between features and other class prototypes.

[0062] The total loss function in this stage is composed of the cosine interval loss and the temporal focus loss. First, the extracted temporal features... Weights of prototypes for each category Execute separately After normalization, the cosine similarity between the two in the feature space is calculated as shown in formula (13).

[0063]

[0064] To enhance the separability and discriminative power of different assembly action category prototypes in the feature space, this invention introduces a margin constraint mechanism, applying a cosine margin penalty parameter to the cosine similarity of the true target category. And through hyperparameters The feature scale is uniformly scaled. After this constraint processing, the Logit representation of the target class is: Logit for non-target categories is represented as .

[0065] Subsequently, the predicted probability of the current frame belonging to the true class is calculated using the Softmax function. As shown in formula (14).

[0066]

[0067] Finally, the predicted probabilities with interval constraints will be... Substitute these values ​​into the temporal focus loss function described in formula (10) to complete the iteration and optimization of the category prototype.

[0068] (vi) Real-time motion segmentation First, the streaming video is input frame by frame into the RAFT-Small lightweight optical flow network to extract the optical flow field. After T frames of data are buffered, the RGB data and optical flow field data are respectively input into their respective pruned feature extraction networks to extract features. Since the processing speed of the subsequent networks is higher than the cumulative time of the video frame buffer, this model can stably maintain a fixed delay of T frames during the real-time processing stage.

[0069] Subsequently, the extracted dual-stream features are deeply coupled using an interactive guidance module, and the coupled features are then processed. Normalization is performed, and the cosine similarity between the features of each frame and the prototype of each category is calculated. The category corresponding to the maximum cosine similarity is taken as the action classification result of that frame. This enables real-time action segmentation of assembly videos, providing core intelligent support for out-of-order detection in assembly videos.

[0070] Comparative experiment This embodiment was quantitatively evaluated against current mainstream methods, and the results are shown in Table 1.

[0071] Table 1. Comparison of quantitative results with mainstream methods on the GTEA dataset.

[0072] Here, Acc represents frame-level accuracy, which only cares about whether the action category prediction is correct for each frame, and does not care about the action boundaries or segmentation structure. Characteristics: It is easily dominated by the mainstream action categories that occupy a lot of time, and is not sensitive to short actions; the higher the value, the more accurate the overall frame-level classification.

[0073] Edit represents the edit distance score, which focuses more on the overall order of actions, the number of segments, and the structure of the category sequence. It can reflect the model's ability to model action boundaries and the logic of action sequence. The higher the score, the better the predicted overall sequence of actions matches the real situation.

[0074] F1@{10,25,50} represents the segmented F1 value, corresponding to different IoU thresholds. The core function is to determine whether the predicted action segment matches the actual action segment using the Intersection over Union (IoU). A higher value indicates a higher degree of matching.

[0075] This method outperforms Causal TCN in frame-level accuracy (Acc) and edit distance score (Edit), indicating better performance in single-frame classification and overall sequence structure. It also performs on par with Causal TCN in the three F1 scores, maintaining high segment matching accuracy.

[0076] In summary, this method can significantly improve the real-time performance and accuracy of action segmentation while ensuring the consistency of action timing. It is applicable to discrete manufacturing and assembly scenarios with complex actions and strong timing dependencies.

[0077] Finally, it should be noted that all parts of this invention not described in detail are prior art. Those skilled in the art will understand that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A real-time motion segmentation method for assembly videos based on dual-stream neural architecture search, characterized in that, Includes the following steps: S1. Obtain the RGB image sequence of the assembly video; input the RGB image sequence into the lightweight optical flow field extraction model to extract the optical flow field; S2. Input the RGB image sequence and the optical flow field into the corresponding branches of the optimal backbone network for feature extraction to obtain RGB features and optical flow field features. The optimal backbone network is constructed based on neural architecture search. It is obtained by establishing independent search spaces for the RGB image and the optical flow field respectively and training the architecture parameters. After the training converges, the candidate convolution operation with the largest architecture parameters of each layer is retained. S3. Input the RGB features and the optical flow field features into the interactive guidance module. Calculate the frame difference features between adjacent frames through the interactive guidance module. Perform cross-modal concatenation of the frame difference features of different modalities with the original features of another modality. Generate channel-level mixing weights through the channel attention mechanism. Perform weighted fusion of the channel-level mixing weights with the original features of the corresponding modality to obtain the fused features. S4. The fusion features are matched with the pre-trained category prototypes to obtain the action classification results of each frame, thereby realizing real-time action segmentation of the assembly video. The category prototypes are obtained through metric learning training. The fusion features of the real category are clustered towards the corresponding prototype, while the fusion features of other categories maintain a gap with the corresponding prototype.

2. The method according to claim 1, characterized in that, The optimal backbone network is obtained by constructing independent neural architecture search spaces for RGB images and optical flow fields, respectively. Each search space consists of multiple convolutional layers, each configured with multiple candidate convolutional operations. Optimizable architecture parameters are bound to each candidate convolutional operation. The classification loss is calculated using an auxiliary classification head, and the model weights and architecture parameters are updated based on backpropagation. After training converges, the candidate operations of each layer are pruned, and the candidate convolutional operation with the largest architectural parameter value in each layer is retained to obtain the optimal backbone network.

3. The method according to claim 2, characterized in that, The candidate convolution operation employs temporal causal convolution; wherein, the temporal causal convolution is configured with a cache space, which is used to store historical frame features, and its capacity is one less than the temporal convolution kernel size; the feature extraction of temporal causal convolution depends only on the information of the current frame and historical frames, and does not depend on the information of future frames.

4. The method according to claim 1, characterized in that, The interactive guidance module calculates the frame difference features between adjacent frames, including: calculating the difference between adjacent frames of the RGB features to obtain RGB frame difference features, and calculating the difference between adjacent frames of the optical flow field features to obtain optical flow field frame difference features. in, For the first RGB feature difference results of the frame For the first RGB features of the frame For the first RGB characteristics of the frame; For the first The optical flow field characteristic difference results of the frame, For the first Optical flow data corresponding to the frame, For the first Optical flow data corresponding to the frame; The optical flow field frame difference feature and the RGB feature are concatenated along the channel dimension to generate the cross-modal input of the RGB branch. The RGB frame difference features and the optical flow field features are concatenated along the channel dimension to generate the cross-modal input of the optical flow field branch. .

5. The method according to claim 4, characterized in that, The interactive guidance module generates channel-level hybrid weights and performs weighted fusion, specifically including: in, This is the first fully connected layer, acting as a bottleneck layer to perform channel dimensionality reduction on the input features; For the second fully connected layer, feature dimensionality is increased; To generate the attention weights for the RGB branch channels, Attention weights for optical flow field branch channels; For RGB modal input features, The input features of the optical flow field mode are represented by R, which represents the set of real numbers, indicating that all elements in the tensor are real numbers. Will Multiply the RGB features element by element to obtain the result. The optical flow field feature is multiplied element-wise, and the result is added to the original RGB feature and optical flow field feature to obtain the joint residual feature. The joint residual feature is then concatenated with the RGB feature and the optical flow field feature along the channel dimension and reconstructed by a fully connected layer to obtain the fused feature.

6. The method according to claim 1, characterized in that, The training process of the pre-trained category prototypes includes: randomly initializing a set of learnable category prototypes, and performing the fusion feature with the category prototypes. Normalization operation; calculate the cosine similarity between the normalized fusion features and the prototypes of each category; apply an interval constraint to the cosine similarity of the real category and then map it to the prediction probability using Softmax; use the prediction probability to calculate the temporal focus loss as the optimization objective, so that the cosine similarity between the fusion features of the real category and the corresponding category prototype increases, and the cosine similarity between the fusion features of other categories and the corresponding category prototype decreases.

7. The method according to claim 6, characterized in that, An interval constraint mechanism was also introduced, and the formula for calculating the prediction probability is as follows: in, For the predicted probability with interval constraints, For hyperparameters, This is the cosine interval penalty parameter.

8. The method according to claim 2, characterized in that, During the training phase of neural architecture search, temporal focus loss is used as the classification loss: in, Represents a valid set, Represents the total number of frames in the valid set. Indicates the first The actual action category label corresponding to the frame. It is a category balancing weight. It is the model's predicted probability of the true label. It is a regulating factor.

9. The method according to claim 2, characterized in that, During the training phase of the neural architecture search, a NAS resource loss is introduced to control the computational overhead of the model: in, This indicates the total number of network layers involved in the search. This represents the total number of candidate operations. This represents the probability of selection. Indicates the first The inherent computational cost of this operation, This is a penalty coefficient used to adjust the strength of resource constraints.

Images