Real-time assembly error recognition and early warning method based on multi-dimensional dynamic threshold adjustment

Abstract

The invention discloses a real-time assembly error recognition and early warning method based on multi-dimensional dynamic threshold adjustment. The method comprises the following steps: extracting a multi-modal assembly state descriptor and a field assembly information sequence; field assembly information fragments are intercepted for similarity calculation; constructing a time sequence prediction module, and taking the field assembly information sequence as input to predict the assembly state of the next frame; constructing a weighted Euclidean distance matrix of the field fragment and the template fragment, and calculating a minimum cumulative distance to quantify the difference between the two sequences; a dynamic threshold model is established based on statistical process control, and whether errors exist in the assembly process or not is judged according to the model; based on an incremental dynamic time planning algorithm and a real-time data acquisition module, a real-time identification system is constructed to realize early warning and standard step pushing, and a real-time adaptive quality control closed loop is formed. According to the method, a real-time and self-adaptive quality control closed-loop system is formed through integration with multi-mode sensing and AR guide technologies, and an effective solution is provided for a complex industrial assembly scene.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality-assisted product assembly, specifically relating to a real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment. Background Technology

[0002] With the rapid development of intelligent manufacturing technologies, the assembly process of complex products is becoming increasingly challenging. Currently, research on complex product assembly is deeply integrated with digital and intelligent technologies, shifting the focus from traditional process planning to dynamic optimization and decision-making based on real-time data. In this process, augmented reality (AR) technology provides intuitive virtual-real fusion guidance for complex product assembly, evolving from a one-way guidance tool into an integrated intelligent system that combines guidance, real-time detection, and real-time feedback. By comprehensively analyzing collaborative information and real-time data from the assembly process and integrating multimodal data with attention mechanisms and other strategies, the system's ability to automatically identify assembly errors is enhanced. This has significant engineering application value and guiding significance for improving the overall efficiency and quality of complex product assembly.

[0003] Several methods for assembly status recognition and error detection have emerged. One study proposed using digital models to calculate the optimal viewing angle for each adjacent sequence, thereby constructing a product assembly recognition template. This method analyzes the viewing angle recognition capability, collects point cloud data from several different perspectives, constructs shape vectors, and ranks the similarity between models using cosine similarity to obtain the optimal viewing angle, thus improving the accuracy of the matching results. However, further research revealed that this method cannot effectively address the frame rate deviation problem caused by assembly speed variations, leading to difficulty in identifying erroneous data due to low real-time performance. Other researchers have proposed a mechanical assembly monitoring method based on multi-view change detection of depth images, which analyzes changes in depth images from different perspectives to determine errors during the assembly process. However, its model generalization ability is insufficient, and it struggles to meet strict real-time requirements when handling assembly tasks with extremely high frame rates. Therefore, a method for assembly error recognition and early warning that can solve the above technical problems is urgently needed. Summary of the Invention

[0004] Purpose of the invention: In order to solve the problems of poor flexibility, strong lag and insufficient reasoning ability of existing assembly progress recognition methods, as well as the large error of traditional similarity calculation caused by the difference in assembly operation speed, this invention proposes a real-time assembly error recognition and early warning method based on multi-dimensional dynamic threshold adjustment, so as to realize error recognition and real-time early warning of assembly tasks.

[0005] Technical solution: The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment described in this invention is implemented as follows:

[0006] S1. Record standard assembly videos, sample key frames from continuous video streams, extract assembly status descriptors from multiple dimensions to form assembly sequence templates, and extract on-site sequence information during actual assembly.

[0007] S2. Extract fragments of on-site assembly information, adjust the comparison template range according to the assembly stage, and determine the current assembly progress.

[0008] S3. Construct a time series forecasting model, using on-site assembly information as input, to predict the next state of the assembly process;

[0009] S4. Construct a weighted distance matrix, use dynamic programming algorithm to solve for the optimal path, and calculate the cumulative Euclidean distance of the optimal path;

[0010] S5. Establish a dynamic threshold model, set an adaptive threshold, and determine that the cumulative distance exceeding the threshold is a possible assembly process error;

[0011] S6. Build a real-time identification system to provide immediate warnings of assembly errors and call standard assembly series templates to push standard operating procedures.

[0012] Furthermore, the multi-dimensional extraction of assembly state descriptors includes extracting hand posture state descriptors, relative pose state descriptors between parts, and distance state descriptors between the hand and the parts.

[0013] Furthermore, the process of extracting the hand pose state descriptor is as follows:

[0014] Based on the MediaPipe algorithm, vectors of the orientation of the main hand bones, including the thumb pointing vector, were calculated from 21 hand joints. The index finger points to the vector and palm direction vector ;

[0015] To measure the stability of operations during assembly, the normal vector of the palm plane is calculated, and its deviation angle variance between consecutive frames is calculated as an indicator of hand stability. First, five key points in the palm region are selected to construct a plane, and the least squares method is used. The optimal plane equation is used to extract the normal vector. In a continuous time window The angle between the normal vectors of adjacent frames is calculated using the following formula:

[0016]

[0017] Calculate the statistical characteristics of the angle change, and let the stability index be:

[0018]

[0019] in, The standard deviation of the angle change It is the average change angle; combining the skeletal feature vector and planar operational stability, the comprehensive state descriptor of hand posture is obtained as follows:

[0020] .

[0021] Furthermore, the process of extracting the relative pose state descriptor between parts is as follows:

[0022] A graph model is introduced to represent the spatial logical relationships between parts during the assembly process, treating the assembly process as a dynamically evolving relational network; an assembly relational graph is constructed for each frame. , where nodes Represents all assembled parts, edge Connect pairs of parts that have an assembly relationship; assign a closeness value to each edge, calculated based on the relative poses between the parts and the preset assembly constraint type, using the following formula:

[0023]

[0024] in, The difference between the current relative rotation of the part and the target assembly state (identity matrix I) These are the weighting coefficients.

[0025] Furthermore, step S2 is implemented as follows:

[0026] Extract the on-site assembly sequence information centered on the current frame and of length L. Then, a window of the same length is slid across the template sequence T to extract segments sequentially, and the matching degree between the two is calculated using the following weighted similarity formula:

[0027]

[0028] in, The value range of the k-th dimension feature of the template is used to normalize the units; after traversing all feasible positions in the template, the index corresponding to the maximum similarity is taken. As the matching position of the current frame, and then according to The assembly schedule is determined by the proportion of the total length in the template.

[0029] Furthermore, the construction of the time series prediction model in step S3 involves introducing a trainable memory matrix on top of the existing multi-head attention module. , The number of typical assembly patterns to be stored; the formula for calculating the memory matrix is:

[0030]

[0031]

[0032] in, For a new field sequence fragment as input, the model calculates the similarity weight w between this fragment and each typical assembly pattern in the memory matrix M; a weighted sum is then performed on all typical assembly patterns in the memory matrix to obtain the reference summary. It includes information retrieved from historical typical patterns that is relevant to the current context, and combines summary information and original sequence information for subsequent processing and calculation.

[0033] Furthermore, the process of calculating the cumulative Euclidean distance of the optimal path in step S4 is as follows:

[0034] Construct a multidimensional weighted Euclidean distance matrix, and the on-site assembly information sequence fragments are as follows: The template fragment is , build a The local distance matrix D, where each element of the matrix... Point and The weighted Euclidean distance between them is calculated using the following formula:

[0035]

[0036] in, and These represent the values ​​of the sequence in the k-th information dimension, To assess the statistical stability of the machine in the normal assembly process based on data from various dimensions at time t, differential weights are automatically assigned to determine its criticality in the assembly process.

[0037] The optimal curved path between the field data sequence fragment and the corresponding template data is searched based on the distance matrix, and the minimum Euclidean distance between corresponding frames of unequal length is calculated. The distance matrix is ​​filled using dynamic recursion. For each point (i, j) in the matrix, the recursive formula is:

[0038]

[0039] This recursive process starts from Start traversing to ,final The value stored in the middle is the minimum cumulative Euclidean distance between the two sequences under the optimal curved path, which is used to quantify the difference between the two sequences.

[0040] Furthermore, the automatic allocation of differentiated weights is implemented as follows:

[0041] Based on the dynamic time planning method, we first calculate the statistical stability index of each dimension during the complete assembly process using historical normal assembly data. Basic weight It is directly proportional to the stability index; considering the assembly characteristics of complex products, assembly criticality is introduced based on failure mode and effects analysis. The severity of the failure, the frequency of failure, and the difficulty of failure detection are all factors to consider.

[0042]

[0043] Considering the time-varying characteristics of sensitivity in various dimensions during assembly, a real-time sensitivity factor is introduced. :

[0044]

[0045] in, Let be the distance gradient of the k-th information dimension at time t. When an anomalous change occurs in a certain dimension, its weight is temporarily boosted. Combining statistical stability and assembly criticality, the formula for calculating the comprehensive differentiated weight is:

[0046] .

[0047] Furthermore, the implementation process of step S5 is as follows:

[0048] A criticality threshold model based on assembly stage is established, and a stage criticality factor is introduced for assembly stage k according to the differences in accuracy requirements at different assembly stages. For critical stages such as precision alignment and final assembly The value will be reduced accordingly; the basic threshold calculation formula is as follows:

[0049]

[0050] in, and These are the sample mean and standard deviation, respectively. This is the sensitivity coefficient, corresponding to a certain confidence level;

[0051] To accommodate differences in proficiency and skill levels among operators, a real-time process capability index is introduced for dynamic threshold adjustment. Its calculation formula is as follows:

[0052]

[0053] in, and These are upper and lower limit specifications set based on historical data. , The mean and standard deviation are calculated in real time.

[0054] To establish a joint monitoring threshold that considers the correlation between dimensions, first calculate the correlation coefficient matrix between each dimension, and then construct a multivariate statistic based on Mahalanobis distance:

[0055]

[0056] The final comprehensive dynamic threshold is defined as follows:

[0057]

[0058] in, As the benchmark adjustment factor, This is the Mahalanobis distance influence factor.

[0059] Furthermore, step S6 is implemented as follows:

[0060] Incremental dynamic time programming algorithm is used to achieve real-time sequence matching, and based on the comprehensive dynamic threshold obtained in step 5. A multi-level confidence assessment mechanism is established. When a potential error is detected, the error confidence score is calculated using the following formula:

[0061]

[0062] in, The number of consecutive frames exceeding the limit is M, which is the preset minimum continuous frame count. Different levels of warnings are issued based on the confidence score. If the score exceeds the preset range, the assembly process is stopped immediately and a correction plan is pushed based on the template assembly information.

[0063] Beneficial Effects: Compared with the prior art, the beneficial effects of this invention are as follows: This invention constructs an assembly information sequence template as a comparison benchmark, achieving nonlinear temporal alignment between on-site operations and standard templates within dynamically defined sequence segments; it introduces a deep neural network to preprocess the operation sequence to amplify feature differences, and combines statistical process control to set dynamic thresholds, effectively distinguishing between normal fluctuations and real errors; it fully utilizes the mathematical properties of the time-scale invariance of dynamic time planning algorithms, and through integration with multimodal perception and AR guidance technologies, forms a real-time, adaptive quality control closed-loop system, providing an effective solution for complex industrial assembly scenarios. Attached Figure Description

[0064] Figure 1 This is a flowchart of the present invention;

[0065] Figure 2 A schematic diagram of finger joint extraction;

[0066] Figure 3 Diagram of a deep neural network time series prediction module;

[0067] Figure 4 A schematic diagram illustrating the solution of the optimal path using a dynamic time planning algorithm;

[0068] Figure 5 This diagram illustrates error warnings and standard procedure push notifications in a real-world assembly scenario. Detailed Implementation

[0069] The invention will now be further described with reference to the accompanying drawings.

[0070] like Figure 1 As shown, this invention proposes a real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment, specifically including the following steps:

[0071] Step S1: Record standard assembly video, sample key frames from continuous video streams, calculate assembly status descriptors to form assembly sequence templates, and extract on-site sequence information during actual assembly.

[0072] Skilled workers perform assembly tasks according to standard assembly procedures and speeds, wearing lightweight devices such as AR glasses to record the complete assembly process from a first-person perspective, ensuring the capture of every critical step. To improve process efficiency, key frames are sampled from the continuous video stream and discretized into a compact assembly process sequence. By extracting representative frames at fixed intervals, data redundancy and system computational load are reduced while ensuring the integrity of process information. Assembly process features are captured from multiple dimensions and combined to form an assembly state descriptor that collectively describes the assembly process. Hand posture parameters, relative poses between parts, and distances between hands and parts are the main information extracted and calculated for the assembly process. The assembly process state descriptor, composed of multiple frames of continuous information, reflects the dynamics and continuity of the assembly process.

[0073] Multi-dimensional calculation of standard assembly process sequence assembly state descriptor:

[0074] 1) such as Figure 2 As shown, based on the 3D coordinates of 21 key hand points extracted using the MediaPipe algorithm, the vectors of the orientation of the main hand bones are first calculated, and the thumb is then positioned relative to the vector. The index finger points to the vector and palm direction vector Defined as:

[0075]

[0076]

[0077]

[0078] To eliminate the influence of hand size differences, each bone vector is normalized. Considering that operational stability in a real-world scenario determines the quality of the entire assembly process, the normal vector of the palm plane is calculated, and its deflection angle variance across consecutive frames is calculated as an indicator of hand stability. First, five key points in the palm region are selected to construct a plane, and the least squares method is used. The optimal plane equation is used to extract the normal vector. In a continuous time window The angle between the normal vectors of adjacent frames is calculated using the following formula:

[0079]

[0080] Further calculations are made of the statistical characteristics of the angle change, and the stability index is set as follows: The calculation formula is as follows:

[0081]

[0082] in, The standard deviation of the angle change This is the average angle of change. Combining the skeletal feature vector and planar operational stability, the comprehensive state descriptor for hand pose is obtained as follows:

[0083]

[0084] 2) Calculate the distance between the hand and the part, defining this distance as the minimum distance between the palm coordinates and the bounding box of the part. Based on the MediaPipe algorithm's recognition results of the main operating finger joints, select the wrist base point (point 0), the index finger metacarpophalangeal joint (point 5), and the middle finger metacarpophalangeal joint (point 9) to calculate the palm coordinates. Based on the CAD model of the part, its axially aligned bounding box is predefined in its local coordinate system. The resulting part bounding box consists of a pair of minimum-maximum vertices. as well as The key is to determine the spatial relationship between the palm and the bounding box of the part, identifying whether the palm is inside, on the surface, or outside the bounding box. If the palm point is inside the bounding box, the minimum distance is 0; if the palm point is outside the bounding box, the distances from the palm point to the nearest surface of the bounding box along the X, Y, and Z axes are calculated as follows:

[0085]

[0086]

[0087]

[0088] The final minimum distance between the palm center coordinates and the bounding box of the part is the magnitude of the vector formed by the above three distance components:

[0089]

[0090] 3) Introduce a graphical model to represent the spatial logical relationships between parts during the assembly process, viewing the assembly process as a dynamically evolving relational network. Construct an assembly relationship diagram for each frame. , where nodes Represents all parts to be assembled, edge Connect pairs of parts that have an assembly relationship. Assign a closeness value to each edge. The calculation is based on the relative poses between parts and the preset assembly constraint types. The calculation formula is as follows:

[0091]

[0092] in, The difference between the current relative rotation of the part and the target assembly state (identity matrix I) These are the weighting coefficients.

[0093] Extract on-site sequence information during actual assembly to prepare for subsequent comparison between on-site data and template data.

[0094] Step S2: Extract fragments of on-site assembly information, adjust the comparison template range according to the assembly stage, and determine the current assembly progress.

[0095] First, extract the on-site assembly sequence information centered on the current frame and of length L. Then, a window of the same length is slid across the template sequence T to extract segments sequentially, and the matching degree between the two is calculated using the following weighted similarity formula:

[0096]

[0097] in, The value range of the k-th dimension feature of the template is used for normalization of dimensions. After traversing all feasible positions in the template, the index corresponding to the maximum similarity is taken. As the matching position of the current frame, and then according to The assembly schedule is determined by the proportion of the total length in the template.

[0098] Step S3, construct as follows Figure 3 The time series prediction model shown uses on-site assembly information as input to predict the next state of the assembly process. This model integrates a deep learning framework consisting of a convolutional neural network (CNN), a multi-head attention model, and a gated recurrent unit (GRU), and is trained using standard assembly information.

[0099] The convolutional neural network module uses two layers of one-dimensional convolutional networks to process the input sequence. Local temporal patterns are extracted, and short-term patterns in the time dimension and local dependencies in the variable dimension are extracted simultaneously. The calculation formula is as follows:

[0100]

[0101] Here, * represents a one-dimensional convolution operation, which captures features at multiple time scales in parallel using convolution kernels of different sizes.

[0102] A multi-head attention model consists of multiple single-head attention models. Each single-head attention model learns different features of the input information, and finally, the outputs of all single heads are combined to extract global features. The calculation process of single-head attention can be represented by the following formula:

[0103]

[0104] Here, Q is the query matrix, K is the key matrix, and their similarity is calculated through a dot product to determine the weights assigned to each value vector during information aggregation; V is the value matrix, and the final output is a weighted sum of value vectors based on attention weights, directly constituting the output representation of the attention layer. Additionally, the softmax function is used to obtain the attention weights and introduces a scaling factor. To maintain the stability of the variance of the dot product, and to further enhance the model's generalization ability and its ability to identify key steps, a trainable memory matrix is ​​constructed. ,in The number of typical assembly patterns to be stored is used to store these patterns. The formula for calculating the memory matrix is:

[0105]

[0106]

[0107] in, Given a new field sequence fragment as input, the model calculates the similarity weight w between this fragment and each typical assembly pattern in the memory matrix M. Further, a weighted summation is performed on all typical assembly patterns in the memory matrix to obtain the reference summary. It contains information retrieved from historical typical patterns that is relevant to the current context. The summary information and the original sequence information are combined before subsequent processing and calculation.

[0108] A gated cyclic unit module is constructed to further improve the accuracy of time series forecasting. The gated cyclic unit module includes a reset gate. With the update gate The hidden state at the previous time step is now known to be: The input information at the current moment is The calculation process for both is as follows:

[0109]

[0110]

[0111] in, Let W be the sigmoid function and W be the weight matrix. Based on the above calculation process, the candidate activation states can be obtained. and final hidden state The calculation formula is as follows:

[0112]

[0113]

[0114] Then the output at that time node and Similarly, the final predicted time series is as follows:

[0115]

[0116] Based on the above predicted time series, a dynamic comparison is made with the standard assembly information template to initially predict assembly errors and enhance the difference.

[0117] Step S4, as follows Figure 4 As shown, a weighted distance matrix is ​​constructed, and the optimal path is solved using a dynamic programming algorithm. The cumulative Euclidean distance of the optimal path is then calculated.

[0118] In the actual assembly process, the current frame number of the field data is first obtained, and the next frame data is obtained using the time series prediction model in step S2. The above several frames of data are combined to obtain a field data sequence segment. Using the template data comparison method in step S2, the corresponding frames of each frame in the field data sequence segment in the template sequence are obtained, where the dimension l of each data point is determined by the dimension of the assembly state descriptor in step 1.

[0119] Construct a multidimensional weighted Euclidean distance matrix, and let the on-site assembly information sequence segment be . The template fragment is First, build a The local distance matrix D, where each element of the matrix... Point and The weighted Euclidean distance between them is calculated using the following formula:

[0120]

[0121] in, and These represent the values ​​of the sequence in the k-th information dimension, To assess the statistical stability of the machine in the normal assembly process based on data from each dimension at time t, differentiated weights are automatically assigned to determine its criticality in the assembly process.

[0122] Calculating the differential weights involves the following steps:

[0123] 1) First, based on historical normal assembly data, calculate the statistical stability index of each dimension during the complete assembly process. The calculation formula is as follows:

[0124]

[0125] in, and Let be the mean and standard deviation of the k-th information dimension during the normal assembly process. The coefficient of variation reflects the degree of relative volatility. Its basic weight is proportional to the stability index, and its calculation formula is:

[0126]

[0127] in Sum the stability indices for all information dimensions.

[0128] 2) Considering the assembly characteristics of complex products, assembly criticality is introduced based on failure mode and effects analysis. The criticality weight is determined by the severity of the failure, the frequency of failure, and the difficulty of failure detection. Therefore, the formula for calculating the assembly criticality weight is:

[0129]

[0130] in Assemble keyness summation for all dimensions of information.

[0131] 3) Considering the time-varying characteristics of sensitivity in each dimension during assembly, a real-time sensitivity factor is introduced, the calculation formula of which is:

[0132]

[0133] in, Let be the distance gradient of the k-th information dimension at time t. When an anomalous change occurs in a certain dimension, its weight is temporarily boosted. Combining statistical stability and assembly criticality, the formula for calculating the comprehensive differential weight is:

[0134]

[0135] Based on the distance matrix constructed above, the optimal curved path between the field data sequence fragments and the corresponding template data is searched, and the minimum Euclidean distance between corresponding frames of unequal length is calculated. This optimal path must start from the lower left corner (1, 1) of the matrix and end at the upper right corner (m, n), meaning that the starting and ending points of both sequences must be fully considered to ensure global matching. Furthermore, consecutive points on the path must be adjacent. For example, if the current point is at position (i, j) in the distance matrix, the next point can only be (i+1, j), (i, j+1), or (i+1, j+1), thus ensuring that no frame in the sequence is skipped and guaranteeing the continuity of the matching. Additionally, the path must monotonically advance over time to ensure causal consistency in the temporal order.

[0136] Under the premise of satisfying the above constraints, a dynamic recursive approach is used to fill the distance matrix. For each point (i, j) in the matrix, the recursive formula is:

[0137]

[0138] This recursive process starts from Start traversing to ,final The value stored in the middle is the minimum cumulative Euclidean distance between the two sequences under the optimal curved path, which is used to quantify the difference between the two sequences.

[0139] Step S5: Establish a dynamic threshold model, set an adaptive threshold, and determine that the cumulative distance exceeding the threshold is a possible assembly process error.

[0140] First, a criticality threshold model based on the assembly stage is established. Then, according to the different accuracy requirements of different assembly stages, a stage criticality factor is introduced for assembly stage k. For critical stages such as precision alignment and final assembly The value will be reduced accordingly. The formula for calculating the basic threshold is as follows:

[0141]

[0142] in, and These are the sample mean and standard deviation, respectively. This is the sensitivity coefficient, which is usually set between 2.0 and 3.0 according to actual needs, corresponding to a certain confidence level.

[0143] To accommodate differences in proficiency and skill levels among operators, a real-time process capability index is introduced for dynamic threshold adjustment. Its calculation formula is as follows:

[0144]

[0145] in, and These are upper and lower limit specifications set based on historical data. , The mean and standard deviation are calculated in real time.

[0146] To establish a joint monitoring threshold that considers the correlation between dimensions, first calculate the correlation coefficient matrix between each dimension, and then construct a multivariate statistic based on Mahalanobis distance:

[0147]

[0148] Where W is the observation vector representing the current assembly state to be evaluated, and μ is the mean vector of the weighted distances of each dimension during the assembly process.

[0149] The final comprehensive dynamic threshold is defined as follows:

[0150]

[0151] in, As the benchmark adjustment factor, This is the Mahalanobis distance influence factor.

[0152] Step S6: Build a real-time identification system to provide immediate warnings of assembly errors and call standard assembly series templates to push standard operation solutions.

[0153] like Figure 5 As shown, step S6 uses an incremental dynamic time programming algorithm to achieve real-time sequence matching, and is based on the comprehensive dynamic threshold obtained in step S5. A multi-level confidence assessment mechanism is established. When a potential error is detected, the error confidence score is calculated using the following formula:

[0154]

[0155] in, The maximum number of consecutive frames exceeding the limit is M, where M is the preset minimum continuous frame count. Different levels of warnings are issued based on the confidence score; if the score exceeds a certain range, the assembly process is immediately stopped and a corrective plan is pushed out based on the template assembly information.

[0156] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment, characterized in that, Includes the following steps: S1. Record standard assembly videos, sample key frames from continuous video streams, extract assembly status descriptors from multiple dimensions to form assembly sequence templates, and extract on-site sequence information during actual assembly. S2. Extract fragments of on-site assembly information, adjust the comparison template range according to the assembly stage, and determine the current assembly progress. S3. Construct a time series forecasting model, using on-site assembly information as input, to predict the next state of the assembly process; S4. Construct a weighted distance matrix, use dynamic programming algorithm to solve for the optimal path, and calculate the cumulative Euclidean distance of the optimal path; S5. Establish a dynamic threshold model, set an adaptive threshold, and determine that the cumulative distance exceeding the threshold is a possible assembly process error; S6. Build a real-time identification system to provide immediate warnings of assembly errors and call standard assembly series templates to push standard operating procedures.

2. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 1, characterized in that, The multi-dimensional extraction of assembly state descriptors includes extracting hand posture state descriptors, relative pose state descriptors between parts, and distance state descriptors between the hand and parts.

3. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 2, characterized in that, The process of extracting the hand pose state descriptor is as follows: Based on the MediaPipe algorithm, vectors of the orientation of the main hand bones, including the thumb pointing vector, were calculated from 21 hand joints. The index finger points to the vector and palm direction vector ; To measure the stability of operations during assembly, the normal vector of the palm plane is calculated, and its deviation angle variance between consecutive frames is calculated as an indicator of hand stability. First, five key points in the palm region are selected to construct a plane, and the least squares method is used. The optimal plane equation is used to extract the normal vector. In a continuous time window The angle between the normal vectors of adjacent frames is calculated using the following formula: Calculate the statistical characteristics of the angle change, and let the stability index be: in, The standard deviation of the angle change It is the average change angle; combining the skeletal feature vector and planar operational stability, the comprehensive state descriptor of hand posture is obtained as follows: 。 4. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 2, characterized in that, The process of extracting the relative pose state descriptor between the parts is as follows: A graph model is introduced to represent the spatial logical relationships between parts during the assembly process, treating the assembly process as a dynamically evolving relational network; an assembly relational graph is constructed for each frame. , where nodes Represents all assembled parts, edge Connect pairs of parts that have an assembly relationship; assign a closeness value to each edge, calculated based on the relative poses between the parts and the preset assembly constraint type, using the following formula: in, The difference between the current relative rotation of the part and the target assembly state (identity matrix I) These are the weighting coefficients.

5. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 1, characterized in that, The implementation process of step S2 is as follows: Extract the on-site assembly sequence information centered on the current frame and of length L. Then, a window of the same length is slid across the template sequence T to extract segments sequentially, and the matching degree between the two is calculated using the following weighted similarity formula: in, The value range of the k-th dimension feature of the template is used to normalize the units; after traversing all feasible positions in the template, the index corresponding to the maximum similarity is taken. As the matching position of the current frame, and then according to The assembly schedule is determined by the proportion of the total length in the template.

6. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 1, characterized in that, Step S3 describes constructing a time series prediction model by introducing a trainable memory matrix on top of the existing multi-head attention module. , The number of typical assembly patterns to be stored; the formula for calculating the memory matrix is: in, For a new field sequence fragment as input, the model calculates the similarity weight w between this fragment and each typical assembly pattern in the memory matrix M; a weighted sum is then performed on all typical assembly patterns in the memory matrix to obtain the reference summary. It includes information retrieved from historical typical patterns that is relevant to the current context, and combines summary information and original sequence information for subsequent processing and calculation.

7. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 1, characterized in that, The process of calculating the cumulative Euclidean distance of the optimal path in step S4 is as follows: Construct a multidimensional weighted Euclidean distance matrix, and the on-site assembly information sequence fragments are as follows: The template fragment is , build a The local distance matrix D, where each element of the matrix... Point and The weighted Euclidean distance between them is calculated using the following formula: in, and These represent the values ​​of the sequence in the k-th information dimension, To assess the statistical stability of the machine in the normal assembly process based on data from various dimensions at time t, differential weights are automatically assigned to determine its criticality in the assembly process. The optimal curved path between the field data sequence fragment and the corresponding template data is searched based on the distance matrix, and the minimum Euclidean distance between corresponding frames of unequal length is calculated. The distance matrix is ​​filled using dynamic recursion. For each point (i, j) in the matrix, the recursive formula is: This recursive process starts from Start traversing to ,final The value stored in the middle is the minimum cumulative Euclidean distance between the two sequences under the optimal curved path, which is used to quantify the difference between the two sequences.

8. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 7, characterized in that, The automatic allocation of differentiated weights is implemented as follows: Based on the dynamic time planning method, we first calculate the statistical stability index of each dimension during the complete assembly process using historical normal assembly data. Basic weight It is directly proportional to the stability index; considering the assembly characteristics of complex products, assembly criticality is introduced based on failure mode and effects analysis. The severity of the failure, the frequency of failure, and the difficulty of failure detection are all factors to consider. Considering the time-varying characteristics of sensitivity in various dimensions during assembly, a real-time sensitivity factor is introduced. : in, Let be the distance gradient of the k-th information dimension at time t. When an anomalous change occurs in a certain dimension, its weight is temporarily boosted. Combining statistical stability and assembly criticality, the formula for calculating the comprehensive differentiated weight is: 。 9. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 1, characterized in that, The implementation process of step S5 is as follows: A criticality threshold model based on assembly stage is established, and a stage criticality factor is introduced for assembly stage k according to the differences in accuracy requirements at different assembly stages. For critical stages such as precision alignment and final assembly The value will be reduced accordingly; the basic threshold calculation formula is as follows: in, and These are the sample mean and standard deviation, respectively. This is the sensitivity coefficient, corresponding to a certain confidence level; To accommodate differences in proficiency and skill levels among operators, a real-time process capability index is introduced for dynamic threshold adjustment. Its calculation formula is as follows: in, and These are upper and lower limit specifications set based on historical data. , The mean and standard deviation are calculated in real time. To establish a joint monitoring threshold that considers the correlation between dimensions, first calculate the correlation coefficient matrix between each dimension, and then construct a multivariate statistic based on Mahalanobis distance: The final comprehensive dynamic threshold is defined as follows: in, As the benchmark adjustment factor, This is the Mahalanobis distance influence factor.

10. The real-time assembly error identification and early warning method based on multi-dimensional dynamic threshold adjustment according to claim 1, characterized in that, The implementation process of step S6 is as follows: Incremental dynamic time programming algorithm is used to achieve real-time sequence matching, and based on the comprehensive dynamic threshold obtained in step 5. A multi-level confidence assessment mechanism is established. When a potential error is detected, the error confidence score is calculated using the following formula: in, The number of consecutive frames exceeding the limit is M, which is the preset minimum continuous frame count. Different levels of warnings are issued based on the confidence score. If the score exceeds the preset range, the assembly process is stopped immediately and a correction plan is pushed based on the template assembly information.

Images