An assembly operation quality monitoring method based on multi-source feature fusion

Abstract

The application discloses an assembly operation quality monitoring method based on multi-source feature fusion, and belongs to the technical field of industrial assembly action recognition and quality monitoring. The assembly operation video stream and acceleration and angular velocity time series data of a wearable inertial measurement unit are synchronously collected, time alignment, fixed-length interception and standardization are completed; features are extracted by using a visual branch and an inertial measurement branch respectively, and after fusion, an assembly action type is recognized; according to a corresponding standard action reference, evaluation weights are configured according to the action type, skeleton sequence differences, acceleration time series differences and action time length deviations are fused, an assembly action standardized comprehensive score is obtained, and standardized degrees and difference information are output. The method is complementary in representation through visual data and inertial measurement data, is helpful to enhance the stability of assembly action recognition, and is used for quantitative evaluation of assembly operation standardization.

Description

Technical Field

[0001] This invention relates to the field of industrial assembly action recognition and quality monitoring technology, and in particular to an assembly operation quality monitoring method based on multi-source feature fusion. Background Technology

[0002] In the assembly and production of automotive parts, electronic components, and precision machinery, some precision and delicate operations are still performed manually. The sequence of actions, posture, force of movement, and execution time of operators can affect the consistency of the assembly process. Current production sites typically use work instructions, operation videos, and manual inspections to standardize and check assembly operations. This method relies on human judgment, making it difficult to continuously record the differences in actions among different operators, and also hinders the objective quantification of assembly operations.

[0003] Chinese invention patent application CN120597043A (publication date: September 5, 2025) discloses a deep learning-based method for industrial assembly motion recognition and error correction training. This method utilizes video images, motion sensing data, and mechanical data to recognize assembly motions. For example, it simultaneously acquires skeletal data, tool trajectories, and mechanical signals using an RGB-D camera, inertial measurement unit, and force or torque sensors. Different modal features are then extracted using graph convolutional networks, long short-term memory networks, and one-dimensional convolutional networks, respectively. The fused features are used for motion classification, anomaly detection, and keyframe localization, and a quality score is generated based on trajectory consistency, posture deviation, and mechanical anomaly index. This method can analyze assembly motions from multiple dimensions, but it requires simultaneous processing of skeletal data, tool trajectories, and mechanical signals. Its quality evaluation indicators are mainly established around tool trajectory, human posture, and mechanical anomalies.

[0004] Chinese invention patent application CN120765120A (publication date October 10, 2025) discloses an artificial intelligence-based production data monitoring method and system. This solution collects visual images, sensor time-series data, equipment control signals, and process parameter logs during the assembly production process. Data from different sources is time-aligned, preprocessed, feature-extracted, and fused to form a quality feature vector. The system compares the quality feature vector with a quality judgment threshold and determines the defect type and risk level through a quality prediction model, then adjusts the monitoring strategy based on the risk level. This solution focuses on monitoring assembly line status, process parameters, and product defects, with relatively limited evaluation of the specific types of operator actions and their level of standardization.

[0005] Existing solutions also use video and voice data to simultaneously evaluate inspection behavior. However, industrial assembly sites are subject to equipment operation, personnel communication, and environmental noise, making stable voice data acquisition susceptible to environmental conditions. Furthermore, relying solely on video recognition makes it difficult to simultaneously reflect the operator's activity intensity, joint movement process, and rhythm.

[0006] Industrial cameras can record the spatial relationships between operators, tools, and assembly objects, but visual data is easily affected by occlusion and changes in lighting. Wearable inertial measurement units can record changes in acceleration and angular velocity, but their output may be affected by sensor noise and drift. Different assembly actions have different requirements for human posture, wrist movements, and action duration, and using a fixed evaluation focus is insufficient to fully reflect the standardized differences of various assembly actions.

[0007] Therefore, how to improve the stability of assembly action recognition in complex industrial assembly environments and achieve interpretable, standardized, and quantitative evaluation for different action types has become an urgent technical problem to be solved. Summary of the Invention

[0008] The main objective of this invention is to provide a method for monitoring the quality of assembly operations based on multi-source feature fusion, which aims to improve the stability of assembly action recognition in complex industrial assembly environments and to achieve interpretable, standardized, and quantitative evaluation for different action types.

[0009] To achieve the above objectives, this invention proposes a method for monitoring assembly operation quality based on multi-source feature fusion, comprising: Simultaneously acquire video streams from industrial cameras and acceleration and angular velocity timing data from wearable inertial measurement units, and align them according to a unified time reference; The video stream is converted into a sequence of image frames, and the temporal data is filtered, denoised, and normalized in dimension. Simultaneously, fixed-length truncation and standardization of the two types of data are performed to obtain assembly action samples. It also includes: inputting samples into a dual-branch assembly operation recognition network; the visual branch extracts visual features through a 3D convolutional module, a convolutional compression layer, and a Transformer encoder; the inertial measurement branch inputs acceleration and angular velocity data into a long short-term memory network, and then extracts inertial measurement features through a fully connected layer and a Transformer encoder; the two types of features are fused and classified to obtain the assembly action type; The standard skeleton sequence features, standard acceleration timing features, and standard action duration of the corresponding types are retrieved from the standard action benchmark library established by the standardized assembly operation data; the human skeleton sequence features, acceleration timing features, and action duration features of the samples are extracted, and evaluation weights are assigned to the skeleton and acceleration dimensions according to the action type. The timing similarity measure values ​​between the two types of actual features and their respective standard features are calculated. The weighted measure values ​​are fused with the action duration deviation to obtain the comprehensive score of assembly action standardization. The score is compared with a preset threshold to output the standardization of assembly actions and the characteristic differences of each evaluation dimension.

[0010] Preferably, the timestamp alignment includes: using microsecond-level timestamps as the unified time reference, obtaining the timestamps of each video frame and each inertial measurement sampling point respectively, and determining the inertial measurement sampling point with the smallest timestamp deviation for each video frame, so as to complete the matching of the video stream and the timing data.

[0011] Preferably, the industrial camera is fixed directly above the assembly station, and the two wearable inertial measurement units are worn on the operator's left and right wrists, respectively.

[0012] Preferably, the time-series data is filtered, denoised, and dimensionally normalized, including: smoothing the acceleration and angular velocity time-series data using a sliding window filter; removing data points whose deviation from the mean is greater than three times the standard deviation based on the mean and standard deviation of the processed time-series data; and converting the data collected by different inertial measurement units into a time-series matrix of uniform dimension.

[0013] Preferably, the fixed-length truncation includes: synchronously trunculating the image frame sequence and time-series data according to a preset window length and sliding step size, wherein the overlap rate of the sliding window is 50%; the standardization includes: mapping the pixel values ​​of the image frames to the range of 0 to 1 using max-min normalization, and processing the acceleration and angular velocity time-series data using z-score normalization.

[0014] Preferably, the visual branch includes three sequentially connected 3D convolutional modules, the convolutional compression layer, and the Transformer encoder with a depth of 2; each 3D convolutional module uses a 3×3×3 convolutional kernel and a ReLU activation function, the convolutional compression layer compresses the temporal dimension of the output features of the 3D convolutional modules to 1, and the Transformer encoder uses an 8-head self-attention and has a hidden layer dimension of 256.

[0015] Preferably, the inertial measurement branch inputs the acceleration time-series data and angular velocity time-series data into two layers of long short-term memory networks, each with a hidden dimension of 128 and a dropout of 0.2; the outputs of the two long short-term memory networks are normalized into 256-dimensional features through the fully connected layer and input into the Transformer encoder with a depth of 6, using 8-head self-attention and a hidden layer dimension of 256.

[0016] Preferably, both the visual features and the inertial measurement features are 256-dimensional features; the fusion includes splicing the visual features and the inertial measurement features along the channel direction into a 512-dimensional fused feature, and then performing feature enhancement sequentially through a fully connected layer and a ReLU activation function, and finally outputting the category probability distribution of the assembly action type through a multilayer perceptron classification head and a Softmax activation function.

[0017] Preferably, extracting the human skeleton sequence features includes: using a lightweight OpenPose model to extract 18 key joints of the human body from the image frame sequence to obtain the original skeleton coordinate sequence, performing coordinate normalization and noise reduction on the original skeleton coordinate sequence, and performing temporal encoding on the processed skeleton coordinate sequence.

[0018] Preferably, a dynamic time warping algorithm is used to calculate the first difference value between the human skeleton sequence features and the standard skeleton sequence features, and the second difference value between the acceleration time series features and the standard acceleration time series features. Evaluation weights are assigned to the first difference value and the second difference value according to the assembly action type. The weighted first difference value, the second difference value and the deviation of the actual action duration from the standard action duration are then fused to obtain the standardized comprehensive score of the assembly action.

[0019] Preferably, a detailed breakdown of differences for each evaluation dimension is formed based on the first difference value, the second difference value, and the deviation in action duration. Based on the detailed breakdown of differences, deviation points, force application method deviations, or action duration problems are identified, and operation optimization suggestions corresponding to the identification results are generated.

[0020] Preferably, the assembly action samples are classified and stored according to a directory structure consisting of operation type, worker number and acquisition timestamp, and a metadata file corresponding to the assembly action sample is generated. The metadata file includes camera frame rate, inertial measurement sampling rate, operation type label, acquisition timestamp and data check code.

[0021] The above technical solution has the following advantages: This application aligns the assembly operation video stream and inertial measurement timing data with a unified time reference, and forms corresponding assembly action samples through fixed-length truncation and standardization. This reduces the impact of different sensor acquisition delays, data lengths, and dimensional differences on feature fusion. The visual branch extracts spatiotemporal features between operators, tools, and assembly objects, while the inertial measurement branch extracts temporal features of acceleration and angular velocity. These two types of features complement each other, helping to enhance the stability of action classification in complex assembly scenarios. The identified assembly action types are used to configure the evaluation weights for the skeleton sequence dimension and the acceleration timing dimension, allowing different assembly actions to be evaluated with a focus corresponding to their action characteristics. By fusing skeleton sequence differences, acceleration timing differences, and action duration deviations, a standardized comprehensive score for assembly actions can be formed from multiple dimensions, including action form, motion process, and execution rhythm. The difference information of each evaluation dimension is output, improving the objectivity and interpretability of assembly operation quality evaluation. Compared with the scheme in CN120597043A, which mainly uses skeletons, tool trajectories, and mechanical features for action recognition and quality scoring, this application adopts a dual-branch feature extraction method using assembly operation videos and wearable inertial measurement data, and configures evaluation weights according to action type. Compared with the scheme in CN120765120A, which focuses on assembly line status, process parameters, and product defect monitoring, this application evaluates the specific action types of operators and their degree of standardization. Attached Figure Description

[0022] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein: Figure 1 The overall flowchart of the assembly operation quality monitoring method based on multi-source feature fusion provided in the embodiments of this application is shown.

[0023] Figure 2 This is a schematic diagram of a multi-source data acquisition framework for assembly operation quality monitoring provided in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram of the multi-source feature dual-branch action recognition network structure provided in the embodiments of this application.

[0025] Figure 4 A flowchart for the standardized evaluation of assembly actions provided in this application embodiment. Detailed Implementation

[0026] The technical solution of this application will be described below with reference to the accompanying drawings and specific embodiments. The described embodiments are used to explain the technical solution of this application and are not intended to limit the scope of protection of this application. Where there is no conflict, the technical features in this embodiment can be combined with each other.

[0027] like Figure 1As shown, this method includes steps S1 of multi-source data acquisition, annotation and preprocessing, step S2 of building a multi-source feature dual-branch action recognition network, step S3 of real-time acquisition of assembly operation video data and inertial measurement data, step S4 of assembly operation recognition and classification, and step S5 of standardized evaluation of assembly actions.

[0028] In this manual, the English name for the inertial measurement unit is Inertial Measurement Unit, abbreviated as IMU; RGB represents the red, green, and blue three-channel image; LSTM represents Long Short-Term Memory Network; ReLU represents Modified Linear Unit Activation Function; Transformer represents a coding structure based on self-attention mechanism; OpenPose represents a human pose keypoint detection model; and z-score represents standard score normalization.

[0029] Example 1 This embodiment provides a method for monitoring assembly operation quality based on multi-source feature fusion. This method is applied to manually participated assembly production processes, using video data and motion timing data collected from operators performing assembly actions to identify the types of assembly actions and to perform standardized evaluations of the actual assembly actions based on corresponding standard actions.

[0030] This embodiment utilizes an industrial camera, wearable inertial measurement units (IMUs), and a server. The industrial camera is positioned above the assembly station to capture video streams of operators performing assembly tasks. Two wearable IMUs are worn on the operator's left and right wrists respectively to collect triaxial acceleration and triaxial angular velocity data from both wrists. The server communicates with the industrial camera and wearable IMUs, and invokes pre-stored programs to perform data synchronization, data preprocessing, motion recognition, and quality evaluation.

[0031] The industrial camera's acquisition area covers the operator's upper body, assembly tools, and the assembly object. A wearable inertial measurement unit moves with the operator's wrist, reflecting changes in movement direction, speed, and wrist posture during the assembly process. The acquired assembly actions can include screwdriver operation, wrench operation, measuring tool use, component assembly, switch pressing, and electric screwdriver operation.

[0032] The server first performs a multi-source data synchronous acquisition step. The industrial camera outputs a continuous video stream of assembly operations. The wearable inertial measurement unit outputs acceleration and angular velocity timing data. The server adds microsecond-level timestamps to the video frames and inertial measurement sampling points according to a unified time base.

[0033] Let the total number of video frames be , No. The timestamp of each video frame is ,in The total number of inertial measurement sampling points is , No. The timestamps of each inertial measurement sampling point are: ,in . and Microseconds are used as the unified unit of measurement. The server searches for the value of the first inertial measurement sampling point among all sampling points. The index of the sampling point with the smallest time deviation in a video frame. Represented as: (1) The server timestamps the video data and inertial measurement data based on this matching relationship. This process reduces the acquisition latency and clock skew between the industrial camera and the wearable inertial measurement unit, ensuring that visual and motion information at the same assembly moment correspond.

[0034] The server converts the synchronized video stream into a sequence of continuous image frames according to a preset frame rate. The server then normalizes the size of each image frame, processing them into images of a fixed resolution and uniformly converting them to 3-channel RGB format. This normalized image frame sequence serves as the input data for the visual branch.

[0035] In a common configuration in this field, the frame rate of the industrial camera can be set to 25 frames per second to 30 frames per second, the image can be normalized to 224×224 pixels, and the inertial measurement sampling rate can be set to 50 Hz to 200 Hz. The specific parameters are selected according to the shortest duration of the action to be identified, the speed of the action, and the capabilities of the acquisition device, so that an action window contains enough continuous sampling points to characterize the change of the action.

[0036] The server filters, denoises, and normalizes the synchronized acceleration and angular velocity time-series data. For any inertial measurement channel, the sliding filter window length is assumed to be a positive integer. ,time The raw channel data is The filtered data is ,but: (2) The server further analyzes the mean of the filtered inertial measurement data. and standard deviation Identify outliers. Data points are identified as outliers when they satisfy the following formula: (3) The server identified the data point as an outlier and removed it from the current time series data. The server then normalized the acceleration and angular velocity data of the left and right wrists into a time series matrix of uniform dimensions to adapt to the input requirements of subsequent networks.

[0037] When the sliding window is not yet full at the beginning of the sequence, the average value is calculated using the currently obtained valid historical sampling points. For a small number of consecutively missing inertial measurement sampling points or human key points, linear interpolation of adjacent valid sampling points can be used. Action windows with a missing proportion exceeding a preset validity threshold are not included in training and evaluation. The validity threshold is jointly determined by the data integrity and recognition results on the validation set.

[0038] After filtering and dimension normalization, the server uses a sliding window strategy to simultaneously capture image frame sequences and inertial measurement time-series data. Each sliding window corresponds to a fixed-duration assembly motion segment. The window length is denoted as a positive integer. The sliding step size is denoted as a positive integer. ; Inertial measurement data is extracted using the same window start and end times, expressed as video frame counts. In this embodiment, the window overlap rate is set to 50%, i.e. ;when When it is an odd number, Take and The closest positive integer. This processing reduces the probability that the assembly cycle is truncated by the window boundary and increases the number of effective samples available for model training and recognition.

[0039] The server normalizes the maximum and minimum values ​​of the captured image frame data. Let the original pixel value be... The minimum pixel value within the current normalization range is The maximum pixel value is The normalized dimensionless pixel value ,but: (4) when At that time, the normalized pixel value Located within the closed interval [0,1] including the endpoints; when At that time, within the normalization range Set it to 0 to avoid the denominator being zero.

[0040] The server performs z-score normalization on the inertial measurement time series data. Let the original inertial measurement channel data be... The mean of the corresponding data group is The standard deviation is If the standardized dimensionless data is z, then: (5) when At that time, the corresponding data group Set to 0 to avoid the denominator being zero. The raw acceleration data is in meters per second squared, and the raw angular velocity data is in radians per second; the standardized z is a dimensionless quantity.

[0041] After the above processing, the image frame sequence and inertial measurement time series data form an assembly action sample with corresponding length, time alignment and dimensional processing.

[0042] The server inputs assembly action samples into a pre-trained two-branch assembly operation recognition network. This two-branch assembly operation recognition network includes a vision branch, an inertial measurement branch, a feature fusion part, and a classification part.

[0043] The input to the visual branch is a sequence of video frames from the assembly operation. Let the visual input be: (6) in, Indicates the number of video frames. Indicates the height of a single frame image. This indicates the width of a single frame image, and 3 indicates the number of RGB channels. , and All are positive integers.

[0044] The vision branch consists of three sequentially connected 3D convolutional modules. Each 3D convolutional module uses a size of [size missing]. The convolution kernel and ReLU activation function are used. The 3D convolution module performs convolution processing on the video frame sequence in the temporal and spatial dimensions to extract the spatiotemporal features formed by limb movements, tool movements, and changes in the assembled objects during the assembly operation.

[0045] Let the input features of the 3D convolution module be... The convolution parameters are The bias parameter is Then, the three-dimensional convolution process is represented as: (7) The symbol “*” represents a three-dimensional convolution operation.

[0046] The high-dimensional feature maps output by the three 3D convolutional modules are fed into the convolutional compression layer. The convolutional compression layer uses 3D convolution operations to normalize the dimensions of the feature maps and compress the temporal dimension to 1, resulting in a 2D visual feature map adapted to the subsequent Transformer encoder. Since the 3D convolutional modules have already spatiotemporally encoded the changes between consecutive frames before compression, the compressed feature map retains the temporal representation of the assembly action and is input into the Transformer encoder as a 2D region feature map.

[0047] The 2D visual feature map, after being flattened and serialized, is input to a Transformer encoder with a depth of 2. This Transformer encoder employs an 8-head self-attention mechanism with a hidden layer dimension of 256. The self-attention calculation is represented as follows: (8) in, Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the feature dimension of a single attention head.

[0048] are positive integers and ,therefore Not zero; The output probabilities for each category are all greater than 0 and their sum is 1.

[0049] The Transformer encoder learns the region associations between the operator's limbs, assembly tools, and assembly objects in an image, outputting encoded visual features. These visual features are then normalized into a 256-dimensional visual feature vector by a fully connected layer. .

[0050] The input to the inertial measurement branch includes acceleration time-series data and angular velocity time-series data. Let the triaxial acceleration at any time t be: (9) The triaxial angular velocities at time t are: (10) The inertial measurement input sequence can be represented as: (11) in: (12) This is the number of inertial measurement sampling points within the action window, and it is a positive integer. The unit of measurement is meters per second squared. The unit of measurement is radians per second. This represents a six-dimensional inertial measurement vector at the same sampling time.

[0051] The inertial measurement branch processes acceleration and angular velocity time-series data separately. Acceleration time-series data is input into the first long short-term memory network, and angular velocity time-series data is input into the second long short-term memory network. Both the first and second long short-term memory networks consist of two long short-term memory layers with a hidden dimension of 128 and a dropout parameter of 0.2.

[0052] The first Long Short-Term Memory (LSTM) network is used to extract the acceleration and force changes of the assembly motion. The second LSM network is used to extract the wrist rotation process and posture changes. The two LSM networks output the temporal features of acceleration and the temporal features of angular velocity, respectively.

[0053] The server concatenates the time-domain features of acceleration and the time-domain features of angular velocity, and then normalizes them into 256-dimensional preliminary inertial measurement features through a fully connected layer. These preliminary inertial measurement features are input to a Transformer encoder with a depth of 6. This Transformer encoder employs an 8-head self-attention mechanism with 256 hidden layers to learn the correlation between changes in acceleration and angular velocity. The features output from the Transformer encoder are then linearly transformed through a fully connected layer to obtain a 256-dimensional inertial measurement feature vector. .

[0054] The feature fusion part combines visual feature vectors With inertial measurement eigenvectors By concatenating along the feature channel direction, a fused feature vector is obtained: (13) The fused feature vectors are passed through a fully connected layer and Activation functions are used to enhance features, resulting in enhanced fused features: (14) in, This represents the weight parameters of the fused fully connected layer. This represents the bias parameters for fusing fully connected layers.

[0055] Enhanced fusion features are input to the multilayer perceptron classification head. The multilayer perceptron classification head is then processed through... The activation function outputs the category probability distribution for each assembly action type: (15) in, The weight parameters represent the classification heads. This represents the bias parameter of the classification head. The assembly action type corresponding to the dimension with the highest class probability is used as the recognition result of the current assembly action sample.

[0056] The dual-branch assembly operation recognition network is trained using data containing both standard assembly actions and actual assembly actions. Training samples are labeled with corresponding action types. The server uses cross-entropy loss to optimize the classification results and incorporates L2 regularization to suppress overfitting. Let the number of training samples be a positive integer M, and the number of assembly action types be a positive integer. , No. For the nth sample The real labels for each type are The predicted probability is Then the cross-entropy loss is: (16) The total network loss is: (17) in, This represents all learnable parameters of the dual-branch assembly operation recognition network. This represents the sum of squares of each learnable parameter. The regularization coefficient is dimensionless; in this embodiment, it is taken as... .

[0057] Training data is divided into training, validation, and test sets based on operator grouping, with samples from the same operator not crossing sets. The training set is used to update network parameters, the validation set is used to select hyperparameters, determine early stop timing, and calibrate evaluation parameters, and the test set is used only to evaluate the model after training. Model performance can be evaluated through action classification accuracy, macro-average F1 score, and confusion matrix; standardized evaluation results can be compared with annotations made by experienced operators according to standard operating procedures, and the consistency rate can be calculated.

[0058] After identifying the action type, the server performs a standardized evaluation of the current assembly action. This standardized evaluation uses a standard action benchmark library as a reference. The standard action benchmark library is constructed from standardized assembly operation data. This standardized assembly operation data is obtained by collecting video data and inertial measurement data from experienced operators performing various assembly actions.

[0059] For each type of assembly action, the standard action benchmark library stores at least the standard skeleton sequence features, standard acceleration timing features, and standard action duration. The data in the standard action benchmark library is linked according to the assembly action type, enabling the server to identify the assembly action type output by the network based on the two-branch assembly operation and retrieve standard action data for the same action type.

[0060] The server uses a lightweight OpenPose model to extract 18 key joints of the human body from the image frame sequence of the current assembly action sample, obtaining the original skeleton coordinate sequence. The server performs coordinate normalization, denoising, and coordinate regularization on the original skeleton coordinate sequence, and performs temporal encoding on the processed joint sequence to obtain the human skeleton sequence features of the current assembly action.

[0061] The server extracts acceleration timing features from the inertial measurement data of the current assembly motion sample. The extraction of acceleration timing features includes filtering preprocessing, long short-term memory network temporal encoding, and feature mapping. The server also determines the actual motion duration based on the start and end times of the current assembly motion sample and calculates the deviation of the actual motion duration from the corresponding standard motion duration.

[0062] The server calculates the differences between the current human skeleton sequence features and the corresponding standard skeleton sequence features, as well as the differences between the current acceleration time series features and the corresponding standard acceleration time series features. Since different operators may have different speeds when performing the same assembly action, this embodiment uses a dynamic time warping algorithm for time series matching.

[0063] Let the actual feature sequence be: (18) The standard feature sequence is: (19) (20) (twenty one) in, , ; For local Euclidean distance, To accumulate distance, The cumulative distance to out-of-bounds positions is set to positive infinity. Matching paths are only allowed to use... , and Three local steps are used to satisfy continuity and monotonicity and to define stretching or compression on the time axis. The optimal matching path is assumed to contain... If there are matching points, then... As the time difference value after length normalization.

[0064] The server searches for the matching path with the minimum cumulative distance according to the recursive relationship between local distance and cumulative distance, and uses the length of the optimal path to normalize the cumulative distance as the difference value between the actual feature sequence and the standard feature sequence. Through dynamic time warping, the impact of differences in action execution speed and sequence length on the evaluation results can be reduced while preserving the order of actions.

[0065] For the skeleton sequence, the server obtains the first difference value. For the acceleration time series, the server obtains the second difference value. The server retrieves the corresponding evaluation weights based on the identified assembly action types. Different assembly action types can use different skeleton dimension weights and acceleration dimension weights.

[0066] During the model calibration phase, the server performs multi-dimensional feature difference analysis using standardized assembly action data and actual assembly action data, and determines the evaluation weight corresponding to each assembly action type. The determined evaluation weights are stored in association with the assembly action type. During the actual evaluation phase, the server directly retrieves the corresponding evaluation weight based on the current assembly action type.

[0067] The server weights the first and second difference values ​​according to the invoked evaluation weights, and then integrates the weighted result with the action duration deviation to obtain the standardized comprehensive score of the current assembly action. This evaluation process simultaneously utilizes human motion morphology, wrist movement changes, and action execution duration, allowing motion deviations from different dimensions to participate in the standardized evaluation.

[0068] The server compares the standardized comprehensive score of the assembly actions with a preset threshold to determine the standardization level of the current assembly actions. The server also outputs the feature differences in the skeleton sequence dimension, acceleration timing dimension, and action duration dimension. The output results can be displayed on a display terminal that communicates with the server, or written to the assembly operation quality monitoring record.

[0069] The server can also save assembly action samples and their processing results. Assembly action samples are categorized and stored according to operation type, worker number, and acquisition timestamp. The server generates a corresponding metadata file for each assembly action sample. The metadata file includes camera frame rate, inertial measurement sampling rate, operation type label, acquisition timestamp, and data checksum. This data organization method enables the establishment of a correspondence between training data, recognition results, and evaluation results.

[0070] This embodiment synchronizes video and inertial measurement data using a unified time reference, reducing the impact of heterogeneous sensor acquisition delays on feature fusion. The visual branch extracts the spatiotemporal relationships between the operator, tools, and assembly objects, while the inertial measurement branch extracts the temporal changes in acceleration and angular velocity; these two types of features complement each other. The action recognition results further serve as the basis for selecting evaluation weights, allowing different assembly actions to be evaluated according to their corresponding action characteristics. By fusing differences in skeleton sequences, acceleration temporal differences, and action duration deviations, the standardization degree of the assembly actions and the differences in each evaluation dimension can be output.

[0071] Example 2 This embodiment, based on Embodiment 1, further explains the data acquisition framework, training data construction method, and model training process used for assembly operation quality monitoring.

[0072] like Figure 2 As shown, the multi-source data acquisition framework for assembly operation quality monitoring includes a data perception layer, a fusion processing layer, and an intelligent service layer. Data output from the data perception layer is first processed through time synchronization before entering the fusion processing layer. Each layer and processing function is sequentially connected through data transmission relationships. This framework can be used not only in the training phase of a dual-branch assembly operation recognition network but also in the action recognition and quality evaluation phases on the production floor.

[0073] The data perception layer is used to acquire raw, multi-source data generated by operators performing assembly actions. The data perception layer includes industrial cameras and multiple wearable inertial measurement units (IMUs). The industrial cameras acquire video streams of the assembly operations, recording the operator's motion contours, body posture, tool positions, and the state of the assembled object. The wearable IMUs acquire triaxial acceleration and triaxial angular velocity data, recording the motion changes of the operator's upper limbs.

[0074] In one implementation, an industrial camera is fixed directly above the assembly station, its acquisition area covering the operator, assembly tools, and the object being assembled. Two wearable inertial measurement units are worn on the operator's left and right wrists, respectively, to acquire acceleration and angular velocity data during assembly movements.

[0075] In another implementation, the wearable inertial measurement unit is positioned at other key points on the operator's upper body. The specific placement is determined based on the main moving parts involved in the assembly action to be identified. Even with different wearing positions, industrial camera data and inertial measurement data are still synchronized using a unified time reference, without altering the subsequent dual-branch feature extraction and standardized evaluation process.

[0076] The time synchronization access process receives data streams from industrial cameras and wearable inertial measurement units (IMUs) and adds timestamps to the received data. This process uses a unified time reference to determine the correspondence between video frames and inertial measurement sampling points. For each video frame, the sampling point with the smallest time deviation is selected from the inertial measurement sampling points to generate time-aligned visual data and inertial measurement data.

[0077] The fusion processing layer preprocesses and constructs samples from the time-aligned multi-source data. Visual data is converted into a continuous sequence of image frames according to a preset frame rate. Each image frame is then normalized in size and channel to form visual input data in a unified format. Inertial measurement data is then processed sequentially through sliding window filtering, outlier removal, and dimension normalization to form a time-series matrix in a unified format.

[0078] The fusion processing layer uses a sliding window to simultaneously capture visual and inertial measurement data. Each window corresponds to one assembly action segment. The visual and inertial measurement data use the same window start and end times to ensure that the two types of data in an assembly action sample have the same action semantics.

[0079] In this embodiment, the overlap rate of adjacent windows is set to 50%. Using overlapping windows can increase the number of effective training samples and reduce the probability of complete actions being truncated by window boundaries. For samples with different action durations, subsequent quality assessment can also perform temporal matching using a dynamic time warping algorithm.

[0080] Image frame data is normalized using maximum and minimum values, mapping pixel values ​​to the range of 0 to 1. Acceleration and angular velocity data are normalized using z-score, converting the data from each inertial measurement channel into standardized data with a mean of 0 and a standard deviation of 1.

[0081] After preprocessing, the fusion processing layer organizes the visual data, acceleration data, angular velocity data, and corresponding labels into assembly action samples. Each assembly action sample includes at least an image frame sequence, an acceleration time sequence, an angular velocity time sequence, an operation type label, and a data acquisition timestamp.

[0082] Assembly action samples are categorized and stored according to operation type, worker number, and collection timestamp. Operation type distinguishes different assembly actions such as screwdriver operation, wrench operation, measuring tool use, accessory assembly, switch pressing, and electric screwdriver operation. Worker number identifies different operators. Collection timestamp indicates the collection order and time of the samples.

[0083] The server generates a corresponding metadata file for each assembly action sample. The metadata file includes camera frame rate, inertial measurement sampling rate, operation type label, acquisition timestamp, and data checksum. The metadata file is stored in association with the assembly action sample to support training data management, traceability of quality evaluation results, and subsequent model training.

[0084] When ensuring the security of production data is required, assembly action samples and metadata files can be stored using conventional encrypted storage methods. Encrypted storage does not alter the data structure of the assembly action samples or the subsequent feature extraction process.

[0085] The intelligent service layer uses stored assembly action samples to train, infer, and evaluate the dual-branch assembly operation recognition network. Training data includes standardized assembly action data and actual assembly action data. Standardized assembly action data is collected by experienced operators performing assembly actions according to standard operating procedures. Actual assembly action data is collected by different operators during actual assembly processes.

[0086] When constructing the training set, the server classifies assembly action samples based on operation type labels. Standard assembly action data and actual assembly action data of the same operation type are grouped into the same action category. The server can also reduce the impact of inconsistent category classification on model training by comparing the similarity of action features between different samples to check category boundaries and sample labels.

[0087] To reduce data leakage caused by the same operator's movement habits, the training, validation, and test sets are grouped according to operator ID; each action category remains representative across all datasets. Network parameters are updated using the training set, and the learning rate, batch size, and number of training epochs are selected based on the validation set loss. Training stops when the validation set loss no longer decreases for a preset number of consecutive epochs.

[0088] like Figure 3 As shown, the dual-branch assembly operation recognition network's visual branch and inertial measurement branch process visual data and inertial measurement data, respectively. The visual branch extracts spatiotemporal variation features from continuous video frames using a 3D convolutional module. The inertial measurement branch extracts temporal features of acceleration and angular velocity using a Long Short-Term Memory network. The two feature streams are then fused after processing by a Transformer encoder.

[0089] The vision branch includes three cascaded 3D convolutional modules. Each 3D convolutional module uses a size of [size missing]. The convolutional kernels are optimized and activated using the ReLU function. The output of the 3D convolutional module is then normalized by a convolutional compression layer, compressing the temporal dimension to 1.

[0090] The compressed visual feature map is flattened into a feature sequence and input into a Transformer encoder of depth 2. This Transformer encoder employs an 8-head self-attention mechanism with a hidden layer dimension of 256. The visual features output by the Transformer encoder are normalized into a 256-dimensional visual feature vector through a fully connected layer.

[0091] The inertial measurement branch inputs acceleration and angular velocity time-series data into two-layer Long Short-Term Memory (LSTM) networks, respectively. Each LSM network has a hidden dimension of 128 and a dropout parameter of 0.2. The acceleration LSM network outputs acceleration time-domain features. The angular velocity LSM network outputs angular velocity time-domain features.

[0092] The temporal features of acceleration and angular velocity are concatenated and input into a fully connected layer, then normalized to 256-dimensional preliminary inertial measurement features. These preliminary inertial measurement features are then input into a Transformer encoder with a depth of 6. This Transformer encoder employs an 8-head self-attention mechanism with 256 hidden layers to learn the correlation between the acceleration and angular velocity features.

[0093] The inertial measurement features output by the Transformer encoder are normalized into a 256-dimensional inertial measurement feature vector by a fully connected layer. The server concatenates the 256-dimensional visual feature vector and the 256-dimensional inertial measurement feature vector along the feature channel direction to obtain a 512-dimensional fused feature vector.

[0094] The fused feature vector is enhanced sequentially through a fully connected layer and a ReLU activation function. The enhanced fused features are then input into a multilayer perceptron classifier head, and the softmax activation function outputs the class probability distribution of the assembly action type.

[0095] During network training, the server calculates the cross-entropy loss based on the actual operation type labels and predicted class probabilities of the training samples. The server also adds an L2 regularization term to the cross-entropy loss to reduce network overfitting. The regularization coefficient is set to... .

[0096] After completing one round of parameter updates, the server continues to read training samples and repeat the forward computation, loss calculation, and parameter updates until the preset training process is completed. The trained bi-branch assembly operation recognition network is saved to the server and used for subsequent assembly action recognition.

[0097] After training, classification accuracy, macro-average F1 score, and confusion matrix for each class are calculated on an independent test set. These metrics are then statistically analyzed under different lighting conditions, partial occlusion, and operating speeds to evaluate the model's stability in response to changes in the field. This evaluation method is used to validate the model and does not pre-determine specific performance values.

[0098] During the model application phase, the intelligent service layer receives assembly action samples generated on the production floor. The server uses the trained dual-branch assembly operation recognition network for forward computation and outputs the assembly action type corresponding to the current sample. This assembly action type is used to select the corresponding standard action data and evaluation weights, thus establishing a continuous data processing relationship between the action recognition results and subsequent standardized evaluations.

[0099] Example 3 This embodiment, based on Embodiment 1 and Embodiment 2, further explains the construction of the standard action benchmark library and the standardization evaluation process of assembly actions.

[0100] like Figure 4 As shown, the assembly action quality evaluation process includes multi-source data acquisition, skeleton and action feature extraction, standard action benchmark construction, multi-dimensional similarity measurement, comprehensive evaluation, and standardized judgment.

[0101] The server first constructs a standard motion benchmark library according to the assembly action type. The data in the standard motion benchmark library comes from video data and inertial measurement data collected by experienced operators when performing standardized assembly operations. The data acquisition process uses the same data types and preprocessing procedures as the actual assembly action acquisition process to ensure that the standard motion data and the actual motion data can be compared and correspond.

[0102] For each type of assembly action, the server extracts a human skeleton sequence from the standardized assembly operation video. The server uses a lightweight OpenPose model to detect key human joints in each image frame and extracts the coordinates of 18 key joints in each frame.

[0103] The server organizes the extracted coordinates according to the keypoint numbers and video frame order to obtain the original skeleton coordinate sequence. The server then performs coordinate normalization, denoising, and other processing on the original skeleton coordinate sequence to reduce the impact of image size, person position, and local detection fluctuations on skeleton comparison.

[0104] Coordinate normalization can be achieved by translating the center of the human torso and normalizing the scale: the shoulder or torso center is used as the origin of the coordinates, and the shoulder width or torso length is used as the scale factor; a moving average filter is used on the keypoint coordinate sequence to suppress local detection jitter. The coordinates of 18 key points in each frame after normalization are arranged according to the key point number to form a temporal vector, and then input into a long short-term memory network or a Transformer encoder for temporal encoding.

[0105] The normalized skeleton coordinate sequence is time-encoded to form human skeleton sequence features; for standardized assembly operations, the resulting features serve as standard skeleton sequence features. The server stores the standard skeleton sequence features corresponding to the same assembly action type in the standard action benchmark library.

[0106] The server extracts standard acceleration timing features from the inertial measurement data corresponding to the standardized assembly operations. The inertial measurement data is first preprocessed by filtering to reduce random noise. The filtered acceleration timing data is then input into a long short-term memory network to extract the acceleration variation patterns during the motion. The time-domain features output by the long short-term memory network are then processed by feature mapping to form the standard acceleration timing features.

[0107] The server also determines the standard action duration based on the start and end times of the standardized assembly actions. Standard skeleton sequence features, standard acceleration timing features, and standard action durations are stored in association according to the assembly action type, forming a standard action benchmark template for the corresponding action type.

[0108] When the server receives an assembly action sample to be evaluated, it first determines the type of assembly action using a two-branch assembly operation recognition network. Based on the identified assembly action type, the server retrieves the corresponding standard skeleton sequence features, standard acceleration timing features, and standard action duration from the standard action benchmark library.

[0109] The server uses the same feature extraction method as the standard action to extract human skeleton sequence features and acceleration timing features from the assembly action sample to be evaluated, and determines the actual action duration of the assembly action to be evaluated.

[0110] Actual assembly actions and standard assembly actions may have different execution speeds and sequence lengths. To compare the two while maintaining the order of actions, this embodiment employs a dynamic time warping algorithm for both the skeleton sequence and the acceleration timing sequence.

[0111] For the skeleton sequence, the server constructs a distance matrix using human skeleton sequence features and standard skeleton sequence features. Each element in the distance matrix represents the local distance between the skeleton features at a given time step in the actual sequence and the skeleton features at a given time step in the standard sequence.

[0112] The server searches for matching paths that satisfy the continuity and monotonicity constraints in the distance matrix according to formulas (20) and (21), and calculates the cumulative distance of each candidate matching path. The path with the smallest cumulative distance is determined as the matching path of the skeleton sequence, and its length is normalized to the cumulative distance as the first difference value.

[0113] For acceleration timing, the server uses the same dynamic time warping process to calculate the second difference value between the acceleration timing features and the standard acceleration timing features. Dynamic time warping uses a local step defined by formula (21) to match the time axes of adjacent feature points, thereby reducing the impact of motion speed differences on similarity evaluation.

[0114] Different assembly actions depend to varying degrees on human posture and wrist movement. Compared to wrench and screwdriver operations, the use of measuring tools can be assigned a larger skeletal dimension evaluation weight during model calibration; wrench and screwdriver operations can be assigned a larger acceleration dimension evaluation weight based on changes in wrist acceleration. The server retrieves the corresponding skeletal dimension evaluation weight and acceleration dimension evaluation weight based on the type of assembly action.

[0115] Evaluation weights are determined during the model calibration phase. During calibration, the server calculates the differences between standardized assembly action samples and actual assembly action samples of the same action type in the skeleton dimension, acceleration dimension, and action duration dimension. For each action type, a grid search is performed among candidate weight combinations that are non-negative and sum to 1. The skeleton dimension evaluation weight is determined based on the selection criterion of the highest consistency rate between the system's judgment and the standard operation annotation on the validation set. Acceleration dimension evaluation weight Weighting of action duration dimension The evaluation weights are then associated with and stored in relation to the action type.

[0116] (twenty two) During the actual evaluation, the server reads the corresponding evaluation weights based on the identified assembly action types. The server uses the skeleton dimension to evaluate the weights. The first difference value is weighted, and the weights are evaluated using the acceleration dimension. The second difference value is weighted.

[0117] The server calculates the actual action duration. Compared with standard action duration The relative motion duration deviation between ,in The first and second difference values ​​were respectively subjected to max-min normalization using the minimum and maximum values ​​in the calibration data to obtain the skeleton normalized difference. and acceleration normalization difference When the maximum value of a certain dimension equals the minimum value, the normalization difference for that dimension is set to 0. The server integrates the three dimensions according to the evaluation weights of the corresponding action type to obtain the standardized comprehensive score of the assembly action with values ​​in the range [0,1]. , The larger the value, the more consistent the actual movement is with the standard movement.

[0118] (twenty three) (twenty four) The server will standardize the assembly process and score the overall results. Standardized thresholds for corresponding action types Compare; when The output conforms to the standard action requirements. The output does not meet the standard action requirements. Threshold. During the model calibration phase, samples are selected from the score distributions of standardized assembly action samples and actual assembly action samples, with the highest consistency rate in the validation set as the selection criterion, and stored in association with the action type.

[0119] In addition to outputting the degree of standardization, the server also retains the first difference value, the second difference value, and the action duration deviation, forming a detailed feature difference profile for each evaluation dimension. The feature difference profile includes differences in the skeleton sequence dimension, acceleration timing dimension, and action duration dimension.

[0120] The server identifies joints whose local differences exceed the corresponding skeleton deviation threshold based on the local differences of each joint in the skeleton sequence; it determines the motion intervals where the motion changes are inconsistent with the standard motion based on the periods when the local differences in the acceleration time sequence exceed the acceleration deviation threshold; and it determines whether the actual motion is executed too fast or too slow based on the motion duration deviation threshold. Each deviation threshold is determined using a validation set during the model calibration phase.

[0121] For assembly actions where the deviation type can be determined based on the difference details, the server outputs operation optimization suggestions corresponding to the deviation dimension. These suggestions instruct operators to focus on the deviation points, force application methods, or action rhythm. The correspondence between difference type, threshold dimension, and guidance content is pre-established and stored based on the standard operating procedures for the corresponding assembly actions. The server retrieves the corresponding guidance content according to the identification results.

[0122] This embodiment does not replace the analysis of each evaluation dimension with a single comprehensive score, but retains the differences of each dimension while generating a comprehensive score for the standardization of assembly actions. Therefore, the quality evaluation results can reflect both the overall standardization level of the assembly actions and the main sources of deviation affecting the standardization level.

[0123] The server associates and saves the assembly action type, the standardized comprehensive score of the assembly action, the degree of standardization, and the differences in characteristics of each evaluation dimension. The saved results can be accessed by the display terminal and can also be used for subsequent quality record queries and training sample organization.

[0124] Example 4 This embodiment provides both real-time monitoring and offline evaluation methods for the above-described assembly operation quality monitoring method.

[0125] In real-time monitoring, industrial cameras and wearable inertial measurement units continuously collect data as operators perform assembly actions. The server generates assembly action samples in real time according to a sliding window. For each complete window, the server calls the dual-branch assembly operation recognition network to output the assembly action type and then uses the corresponding standard action benchmark data for standardized evaluation based on that action type.

[0126] After the server completes the evaluation of a window, it outputs the standardization level and differences in each evaluation dimension corresponding to the current action window. As the sliding window continues to move, the server processes subsequent assembly action data sequentially, thus forming continuous assembly operation quality monitoring results.

[0127] In the offline evaluation method, industrial camera video data and inertial measurement time-series data are first stored according to the acquisition timestamp. During evaluation, the server reads the stored data to complete timestamp alignment, motion sample extraction, motion type identification, and standardized evaluation. The offline evaluation method is suitable for training data inspection, historical operation verification, and assembly quality traceability.

[0128] Both real-time monitoring and offline evaluation methods employ the same data preprocessing, feature extraction, and evaluation logic. The difference lies in the timing of data input and result output; the fusion method of visual and inertial measurement features remains unchanged, as does the processing relationship of configuring evaluation weights based on assembly action type.

[0129] The above method can be applied to assembly production fields involving manual participation, such as automotive parts assembly, electronic device assembly, and precision machinery manufacturing. In different production fields, the industrial camera and wearable inertial measurement unit collect data on the assembly actions of operators, and the specific action categories and standard action reference data are established according to the corresponding assembly process.

[0130] For automotive parts assembly, actions such as wrench operation, electric screwdriver operation, and component assembly can be collected. For electronic component assembly, actions such as measuring tool use, component assembly, and switch pressing can be collected. For precision machinery manufacturing, corresponding standardized assembly action data and action type labels can be established according to specific processes.

[0131] When the application scenario changes, the server can re-collect the standardized assembly operation data and actual assembly operation data for the corresponding scenario, and train the dual-branch assembly operation recognition network as described in Example 2. The server also establishes standard action benchmark templates and evaluation weights for each action type as described in Example 3.

[0132] Different application scenarios can use the same vision branch, inertial measurement branch, feature fusion, and standardized evaluation process. The main adjustments needed include operation type labels, training data, standard action benchmark data, and corresponding evaluation weights. This approach can adapt to different assembly processes without changing the overall processing architecture.

[0133] Before applying this method to new assembly processes, sensor placement locations, or production environments, the model should be retrained or calibrated using standard assembly operation data and actual assembly operation data for that scenario, and the action categories, evaluation weights, and thresholds should be verified to be suitable for that scenario on independent validation and test sets.

[0134] During production line deployment, industrial cameras, wearable inertial measurement units, and servers can transmit data via conventional communication methods. The server can interact with existing display terminals or industrial monitoring platforms on the production floor. These connections are solely for data transmission, result display, and recording / storage, and do not alter the motion recognition and quality evaluation methods described in this application.

[0135] By deploying industrial cameras and wearable inertial measurement units at existing assembly stations, operator motion data can be collected without altering assembly process steps. A dual-branch assembly operation recognition network is used to determine the type of assembly action, and a standardized evaluation process is used to quantify the differences between actual actions and standard actions, thus providing a continuous data foundation for assembly operation quality monitoring.

[0136] The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit the scope of protection of this application. After reading this application, those skilled in the art can adjust the data acquisition method, network training process, standard action reference data, and result output method according to the actual assembly process, action type, and production environment. Equivalent substitutions, conventional modifications, or combinations of technical features made without departing from the technical concept of this application shall fall within the scope of protection of this application.

Claims

1. A method for monitoring assembly operation quality based on multi-source feature fusion, comprising: Simultaneously acquire video streams from industrial cameras and acceleration and angular velocity timing data from wearable inertial measurement units, and align them according to a unified time reference; The video stream is converted into a sequence of image frames, and the time-series data is filtered, denoised, and dimensionally normalized. Simultaneously extract and standardize the two types of data by fixed length to obtain assembly action samples; Its characteristic is that it further includes: The samples are input into a dual-branch assembly operation recognition network. The visual branch extracts visual features through a 3D convolutional module, a convolutional compression layer, and a Transformer encoder. The inertial measurement branch inputs acceleration and angular velocity data into a long short-term memory network, and then extracts inertial measurement features through a fully connected layer and a Transformer encoder. The two types of features are fused and classified to obtain the assembly action type. The standard skeleton sequence features, standard acceleration timing features, and standard action duration of the corresponding types are retrieved from the standard action benchmark library established by the standardized assembly operation data; the human skeleton sequence features, acceleration timing features, and action duration features of the samples are extracted, and evaluation weights are assigned to the skeleton and acceleration dimensions according to the action type. The timing similarity measure values ​​between the two types of actual features and their respective standard features are calculated. The weighted measure values ​​are fused with the action duration deviation to obtain the comprehensive score of assembly action standardization. The score is compared with a preset threshold to output the standardization of assembly actions and the characteristic differences of each evaluation dimension.

2. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, The timestamp alignment includes: using microsecond-level timestamps as the unified time reference, obtaining the timestamps of each video frame and each inertial measurement sampling point, and determining the inertial measurement sampling point with the smallest timestamp deviation for each video frame, so as to complete the matching of the video stream and the timing data.

3. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 2, characterized in that, The industrial camera is fixed directly above the assembly station, and the two wearable inertial measurement units are worn on the operator's left and right wrists, respectively.

4. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, The time-series data is filtered, denoised, and dimensionally normalized, including: smoothing the acceleration and angular velocity time-series data using a sliding window filter; removing data points whose deviation from the mean is greater than three times the standard deviation based on the mean and standard deviation of the processed time-series data; and converting the data collected by different inertial measurement units into a time-series matrix of a unified dimension.

5. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 4, characterized in that, The fixed-length cropping includes: synchronously cropping the image frame sequence and time-series data according to a preset window length and sliding step size, wherein the overlap rate of the sliding window is 50%; the standardization includes: using max-min normalization to map the pixel values ​​of the image frames to the range of 0 to 1, and using z-score normalization to process the acceleration and angular velocity time-series data.

6. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, The visual branch includes three sequentially connected 3D convolutional modules, the convolutional compression layer, and the Transformer encoder with a depth of 2. Each 3D convolutional module uses a 3×3×3 convolutional kernel and a ReLU activation function. The convolutional compression layer compresses the temporal dimension of the output features of the 3D convolutional modules to 1. The Transformer encoder uses 8-head self-attention and has a hidden layer dimension of 256.

7. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, The inertial measurement branch inputs the acceleration time-series data and angular velocity time-series data into two layers of long short-term memory networks, each with a hidden dimension of 128 and a dropout of 0.

2. The outputs of the two long short-term memory networks are normalized into 256-dimensional features through the fully connected layer and input into the Transformer encoder with a depth of 6, using 8-head self-attention and a hidden layer dimension of 256.

8. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, Both the visual features and the inertial measurement features are 256-dimensional features; the fusion includes splicing the visual features and the inertial measurement features along the channel direction to form a 512-dimensional fused feature, and then performing feature enhancement through a fully connected layer and a ReLU activation function in sequence, and finally outputting the category probability distribution of the assembly action type through a multilayer perceptron classification head and a Softmax activation function.

9. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, Extracting the human skeleton sequence features includes: using a lightweight OpenPose model to extract 18 key joints of the human body from the image frame sequence to obtain the original skeleton coordinate sequence; performing coordinate normalization and noise reduction on the original skeleton coordinate sequence; and performing temporal encoding on the processed skeleton coordinate sequence.

10. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 9, characterized in that, The dynamic time warping algorithm is used to calculate the first difference value between the human skeleton sequence features and the standard skeleton sequence features, and the second difference value between the acceleration time series features and the standard acceleration time series features. Evaluation weights are assigned to the first difference value and the second difference value according to the assembly action type. The weighted first difference value, the second difference value and the deviation of the actual action duration from the standard action duration are fused to obtain the standardized comprehensive score of the assembly action.

11. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 10, characterized in that, Based on the first difference value, the second difference value, and the action duration deviation, a difference detail is formed for each evaluation dimension. Based on the difference detail, deviation key points, force application method deviations, or action duration problems are identified, and operation optimization suggestions corresponding to the identification results are generated.

12. The assembly operation quality monitoring method based on multi-source feature fusion according to claim 1, characterized in that, The assembly action samples are classified and stored according to a directory structure consisting of operation type, worker number and acquisition timestamp, and a metadata file corresponding to the assembly action sample is generated. The metadata file includes camera frame rate, inertial measurement sampling rate, operation type label, acquisition timestamp and data check code.

Images