A physical constraint multi-modal work time prediction method for small sample assembly scene

Abstract

The present application relates to the cross field of intelligent manufacturing and industrial big data analysis, and in particular to a physical constraint multi-modal work hour prediction method for small sample assembly scenarios, comprising: S1 collecting multi-modal assembly work hour data, constructing a core physical feature subset based on a triangular fuzzy analytic hierarchy process, and completing coding, normalization and vector reconstruction, while extracting assembly process text semantic features; S2 performing multi-modal data augmentation based on a physical constraint correction strategy, protecting key process entities without mask, and performing double boundary constraint correction on augmented numerical features; S3 constructing a deep hybrid attention network optimized by a sparrow search algorithm, realizing cross-modal fusion of text semantic features and structured physical features, and outputting work hour prediction values; and S4 verifying and adjusting augmented samples and the trained model. The present application can solve the problems of data augmentation damaging process logic, insufficient multi-modal feature coupling and low prediction accuracy in small sample assembly scenarios.

Description

Technical Field

[0001] This invention relates to the intersection of intelligent manufacturing and industrial big data analysis technology, specifically to a physical constraint multimodal time prediction method for small sample assembly scenarios. It is applicable to multimodal sample augmentation, cross-modal feature fusion and time prediction modeling of small sample assembly time data, and can be applied to time prediction and production scheduling auxiliary decision-making scenarios in multi-variety small-batch assembly production of special vehicles and other products. Background Technology

[0002] In the manufacturing of complex equipment, assembly time quotas are crucial for enterprises to formulate production plans, organize resource scheduling, conduct cost accounting, and provide scientific pricing. Their accuracy directly impacts production rhythm, capacity assessment, and market competitiveness. Currently, the production of complex equipment such as special vehicles is increasingly characterized by multi-variety, small-batch, and discrete manufacturing. The assembly objects have complex structures, significantly different assembly steps, and dense process constraints, resulting in a limited number of effective historical time samples available to enterprises. Under these small sample conditions, traditional prediction models relying on large-scale sample training are prone to overfitting and struggle to meet the actual needs of workshop scheduling and time quota management.

[0003] Assembly time data comes from complex sources, typically involving various documents such as two-dimensional drawings, assembly process cards, assembly lists, and historical measured time records. These documents include both structured physical parameters and unstructured assembly process text. Among these, assembly process cards, which meticulously record assembly objects, operating steps, tooling usage, process parameters, and quality requirements, are crucial data carriers for predicting assembly time. This type of data includes structured physical characteristics such as component specifications, number of components, number of fasteners, component weight, equipment type, and precision requirements; and unstructured process text information such as assembly actions ("assembly," "adjustment," "press-fitting," "tightening," "inspection"), their relationships, sequence constraints, and operational space constraints. Relying solely on structured physical parameters makes it difficult to accurately describe the semantic differences in the actual assembly process; conversely, relying solely on process text fails to express the impact of underlying physical parameters on time consumption.

[0004] Existing assembly time prediction methods can be mainly divided into three categories: manual experience estimation, traditional machine learning methods, and deep learning methods. Manual experience estimation relies on the experience of process engineers to determine time, resulting in problems such as strong subjectivity, low efficiency, and insufficient reusability. Traditional machine learning methods typically use structured process parameters as input and employ models such as support vector regression, random forests, and backpropagation neural networks to establish a mapping relationship between features and time. While these methods have a certain fitting ability for physical parameters, they fail to adequately utilize the implicit semantics in the assembly process text, making it difficult to express the sequence of assembly actions, process constraints, and operational complexity. Deep learning methods extract textual semantic features through structures such as BERT, TextCNN, or attention mechanisms, offering advantages in complex semantic modeling. However, most studies simply concatenate textual semantic features with structured physical features, lacking in-depth exploration of the deep correlation between the two modalities, and cannot effectively characterize the dynamic coupling relationship between assembly process semantics and underlying physical parameters.

[0005] Furthermore, under small sample conditions, existing data augmentation methods such as variational autoencoders, generative adversarial networks, TimeGAN, or ordinary masking language model replacement methods, while able to expand the number of training samples, struggle to simultaneously maintain textual semantic integrity and assembly physical constraints. Random masking or replacing key process entities may disrupt the engineering logic between component specifications, connector models, mating relationships, and assembly actions; unconstrained perturbation of numerical features may generate anomalous samples below physical limits or exceeding reasonable assembly ranges, causing augmented samples to deviate from the actual assembly process. Therefore, it is necessary to propose an assembly time prediction method that can simultaneously satisfy physical constraint consistency, deep fusion of multimodal features, and global model optimization requirements in small sample assembly scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a physical constraint multimodal time prediction method for small-sample assembly scenarios. Addressing the problems of limited effective sample numbers, strong heterogeneity in assembly data, insufficient integration of process text semantics and physical parameters, and the tendency of augmented samples to violate engineering physical constraints in complex equipment multi-variety small-batch assembly production, which lead to weak generalization ability and low prediction accuracy of time prediction models, this invention constructs a time prediction framework consisting of "multi-source data acquisition - core physical feature screening - physical constraint sample augmentation - deep hybrid attention network prediction optimized by sparrow search algorithm - result verification and parameter adjustment".

[0007] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0008] A physical constraint multimodal time prediction method for small-sample assembly scenarios includes the following steps:

[0009] S1: Collect multimodal data of assembly time, construct a subset of core physical features of assembly time using triangular fuzzy hierarchical analysis, and encode, normalize and reconstruct the structured physical features into vectors; and perform semantic encoding on assembly process text to extract semantic features of assembly process text.

[0010] S2: Based on the physical constraint correction strategy, small sample multimodal data augmentation is performed. Key process entities in the assembly process text are protected without masking. The physical numerical features generated by augmentation are corrected by double boundary constraints to obtain multimodal augmented samples that meet the assembly physical constraints.

[0011] S3: Construct a deep hybrid attention network (SSA-DHAN) optimized by the sparrow search algorithm. Input the semantic features of the assembly process text and the structured physical feature vector into the deep hybrid attention network. Dynamically weight the structured physical features guided by the text semantics to generate cross-modal fusion features. Use the sparrow search algorithm to globally optimize the key hyperparameters of the deep hybrid attention network. Substitute the obtained optimal hyperparameter combination into the deep hybrid attention network for training. Output the assembly time prediction value through the regression prediction layer.

[0012] S4: Verify and adjust the models and methods constructed in S2 and S3.

[0013] Furthermore, step S1 specifically includes: collecting assembly time data from process documents such as two-dimensional drawings, assembly process cards, and assembly detail sheets, as well as historical measured time records; cleaning missing fields, abnormal records, and duplicate samples; establishing a candidate time influencing factor system around assembly objects, assembly actions, assembly resources, and work environment constraints; converting expert evaluation results into a triangular fuzzy judgment matrix using a triangular fuzzy scale; obtaining the non-fuzzy weights of each candidate influencing factor through defuzzification and probability calculation; screening core physical features according to weight ranking; performing one-hot encoding on discrete features and Min-Max normalization on continuous features, concatenating them to generate a structured physical feature vector; and inputting the assembly process text into a fine-tuned BERT model to extract textual semantic features used to characterize the global semantic information of the assembly process text.

[0014] Furthermore, step S2 specifically includes: constructing an assembly process entity dictionary by combining standard operating procedures, historical process documents, parts lists, and assembly technology manuals, and incorporating domain entities into the BERT underlying vocabulary; setting mask-free protection for words belonging to the assembly process entity dictionary, dynamically calculating mask probabilities for non-entity words based on word frequency, and performing mask language model augmentation; generating candidate augmented values ​​for numerical features, and using extreme value relaxation boundaries and physical bottom line constraints to truncate and correct the candidate augmented values, thereby obtaining multimodal augmented samples that satisfy assembly physical logic.

[0015] Furthermore, step S3 specifically includes: extracting text semantic features As query information, the structured physical feature vector As guided information, it is mapped to a query matrix Q, a key matrix K, and a value matrix V, respectively. Cross-modal weights are calculated using scaled dot product attention to dynamically weight the importance of process semantics to different physical parameters. The weighted physical features are concatenated with the original text semantic features, and the original text semantic features are preserved through residual connections to obtain cross-modal fusion features. The initial learning rate, the number of hidden nodes in the fusion layer, the Dropout inactivation rate, and the number of BERT frozen layers are used as parameters to be optimized and encoded as sparrow individual positions. The validation set mean squared error is used as the fitness function, and the population positions are iteratively updated through the sparrow search algorithm to obtain the globally optimal hyperparameter combination. Under the globally optimal hyperparameter combination, the cross-modal fusion features are input into the regression layer of the multilayer perceptron, and a deep hybrid attention network is trained using the mean squared error loss function combined with the L2 regularization term to output the assembly time prediction value.

[0016] Further, step S4 specifically includes: performing distribution consistency verification and physical constraint consistency verification on the multimodal augmented samples obtained in step S2 to determine whether the multimodal augmented samples meet the preset augmentation quality requirements; performing prediction performance verification on the prediction model trained in step S3, wherein the prediction performance verification is based on the validation set and independent test samples to calculate preset prediction performance evaluation indicators, wherein the preset prediction performance evaluation indicators include at least one of mean square error, mean absolute error, root mean square error, mean absolute percentage error, and coefficient of determination; when the multimodal augmented samples do not meet the preset augmentation quality requirements or the prediction model does not meet the preset error requirements, adjusting the data augmentation ratio, numerical feature perturbation coefficient, hyperparameter optimization range, or model training parameters, and re-executing the corresponding data augmentation, model training, or parameter optimization process; when the multimodal augmented samples meet the preset augmentation quality requirements and the prediction model meets the preset error requirements, outputting the final assembly time prediction model and assembly time prediction results.

[0017] The beneficial effects of this invention are:

[0018] In the scenario of discrete assembly production of complex equipment such as special vehicles, with multiple varieties and small batches, assembly time data mainly comes from process documents such as two-dimensional drawings, assembly process cards, and assembly detail sheets. This presents problems such as a small effective sample size, the coexistence of structured physical features and unstructured process text, a high density of professional assembly terminology, complex process constraints, and difficulty in extracting key time-influencing factors. These issues lead to existing time prediction models being prone to overfitting, inaccurate feature matching, and insufficient engineering generalization ability. Among existing time prediction methods, traditional machine learning models mainly rely on structured physical parameters and lack the ability to understand the professional semantics of assembly action sequence, mating relationships, torque constraints, and tooling usage. Single-text deep learning models struggle to simultaneously characterize the dynamic coupling relationship between assembly process text and physical parameters. Ordinary Early Fusion methods cannot adaptively adjust physical parameter weights according to different process semantics. Conventional VAE, GAN, or randomized MLM augmentation methods struggle to balance the integrity of textual semantics with assembly physical constraints, easily generating samples that deviate from the actual assembly logic.

[0019] The proposed method for predicting physical constraints and multimodal work time in small-sample assembly scenarios effectively solves the above problems through a technical framework of "triangular fuzzy core physical feature screening - physical constraint multimodal data augmentation - deep hybrid attention network prediction optimized by sparrow search algorithm - result verification and adjustment". Its core advantages are as follows:

[0020] This invention constructs a subset of core physical features using triangular fuzzy hierarchical analysis, establishes a system of candidate time-influencing factors around assembly objects, assembly actions, assembly resources, and work environment constraints, and utilizes expert group decision-making to transform assembly process experience into quantifiable feature weights. This method can eliminate marginal features with low contribution under small sample conditions, retaining key physical features such as the number of components, part weight, number of fasteners, precision requirements, assembly methods, tooling equipment, and work space constraints, avoiding redundant noise interference caused by directly inputting all candidate physical parameters into the model.

[0021] This invention proposes a multimodal sample augmentation mechanism oriented towards physical constraints. At the text augmentation end, an assembly process entity dictionary is constructed by combining standard operating procedures, historical process documents, parts lists, and assembly technical manuals. Key process entities such as "M8 bolt," "20-30 N·m," "locating pin," and "interference fit" are protected against masking. For non-entity, ordinary process terms, masking probabilities are dynamically calculated based on word frequency, and augmented text is generated using the BERT masking language model. At the numerical augmentation end, the augmented numerical features are truncated and corrected by using an upper boundary of extreme value relaxation and a lower boundary of physical constraints to avoid generating negative weights, negative quantities, abnormal torques, or physical parameters exceeding reasonable assembly ranges, thereby ensuring the augmented samples possess engineering feasibility.

[0022] This invention constructs a deep hybrid attention network (SSA-DHAN) optimized by the Sparrow Search algorithm. It employs a "dual-channel input - cross-attention fusion - regression mapping" structure, using the semantic features of the assembly process text as the query matrix and structured physical features as the key and value matrices. This allows the text semantics to dynamically guide the weighting of physical parameters. The model can adaptively adjust the weights of physical features such as fit accuracy, number of fasteners, torque range, tooling resources, or inspection requirements according to different assembly processes, thus overcoming the shortcomings of traditional simple splicing and fusion methods that struggle to depict the deep coupling between process semantics and physical parameters. Simultaneously, the Sparrow Search algorithm is used to globally optimize key hyperparameters such as learning rate, Dropout inactivation rate, number of hidden nodes in the fusion layer, and number of BERT frozen layers. This reduces the risk of instability from manual parameter tuning and the model getting trapped in local optima, improving the stability of model training under small sample conditions.

[0023] Experimental results show that, under the same augmented training set and five-fold cross-validation conditions, the mean absolute error (MAE) of the SSA-DHAN model of this invention is 3.34, the root mean square error (RMSE) is 4.29, the mean absolute percentage error (MAPE) is 15.26%, and the coefficient of determination is [missing data]. The accuracy is 0.85, outperforming comparable models such as Random Forest, Support Vector Regression, Backpropagation Neural Network, BERT, TextCNN, and Early Fusion. It maintains a low error on independent test sets, with 80% of the test sample prediction errors within the industrial tolerance range, demonstrating good engineering generalization ability. Ablation experiments further show that removing the physical constraint multimodal augmentation module or the physical constraint masking mechanism significantly degrades the model's prediction performance, indicating that both play a crucial role in mitigating overfitting and maintaining the consistency of the augmented text's process logic under small sample conditions. In summary, this invention can provide accurate, stable, and engineering-deployable data support for time assessment, production scheduling, cost accounting, and time quota management in the discrete assembly process of complex equipment. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 The following are schematic diagrams of the assembly process: (a) Assembly process card, (b) Assembly list;

[0026] Figure 2 This is a schematic diagram of the core physical feature screening process based on the triangular fuzzy hierarchical analysis method in this invention;

[0027] Figure 3 This is a schematic diagram of text feature extraction for assembly process in this invention;

[0028] Figure 4 This is a schematic diagram of the assembly data augmentation and augmented numerical feature dual boundary constraint correction process in this invention;

[0029] Figure 5 This is a schematic diagram of the cross-modal fusion structure of the deep hybrid attention network in this invention;

[0030] Figure 6 This is a schematic diagram of the hyperparameter optimization and time prediction process based on the sparrow search algorithm in this invention;

[0031] Figure 7 This is a visualization of the t-SNE latent semantic space of the original samples and augmented samples in this invention;

[0032] Figure 8 The assembly time prediction results in this invention are as follows: (a) Comparison of time prediction results, (b) Prediction error distribution diagram, and (c) Overall regression diagram of the full sample of OOF. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1

[0035] This embodiment focuses on the analysis and extraction of assembly time data and the construction of core physical features.

[0036] Assembly process data was collected from the historical assembly data of a company's special vehicles. Data sources included two-dimensional drawings, assembly process cards, assembly detail sheets, and other process documents, as well as historical measured time records. Figure 1 As shown in the figure. The assembly process card records in detail the assembly objects, operation steps, tooling usage and process requirements, including both structured physical characteristics and unstructured assembly process text; the assembly list details all component information in the assembly process, including component name, code, and required quantity.

[0037] This embodiment uses an assembly process card as an example to illustrate feature analysis and extraction. It cleans the extracted text of missing fields, abnormal records, and duplicate samples, preserving process information and semantic relationships between processes. Each processed sample consists of unstructured process text, structured physical features, and measured time. The unstructured process text includes descriptions such as "press the bearing into the left arm hole" and "tighten the M8 bolt group with a torque of 20-30 N·m." The structured physical features may include the number of components, the number of fasteners, and accuracy requirements.

[0038] To address the challenges of numerous components, complex process dependencies, and numerous factors influencing labor time in the final assembly of special vehicles, this embodiment employs the triangular fuzzy hierarchical analysis method for core physical feature screening. The process is as follows: Figure 2 As shown. The specific steps are as follows:

[0039] Step 1: Establish a hierarchical structure including a target layer, a criterion layer, and an indicator layer. The criterion layer includes assembly objects, assembly actions, assembly resources, and work environment constraints. The indicator layer includes candidate influencing factors such as the number of components, component weight, number of fasteners, mating methods, accuracy requirements, tooling and equipment, work space constraints, and collaborative work requirements.

[0040] Step 2: Constructing a triangular fuzzy scale for feature evaluation. To reduce the impact of differences in expert evaluations on the results, a compressed triangular fuzzy scale is used to quantify and map discrete linguistic variables. The triangular fuzzy scales corresponding to each linguistic description are shown in Table 1.

[0041]

[0042] Step 3: Construct the fuzzy judgment matrix and calculate the fuzzy comprehensive value. Let a certain scale be the triangular fuzzy number. ,in , , Let represent the lower bound, most likely value, and upper bound of the distribution of expert evaluation results, respectively, and strictly satisfy . Calculate the first from equation (1). Fuzzy comprehensive evaluation value of each feature index :

[0043]

[0044] In the formula, The number of evaluation indicators; To determine the first element in the matrix The first indicator is relative to the first The triangular fuzzy evaluation value of each indicator; It is a fuzzy multiplication operator.

[0045] Step 4: Defuzzification mapping. Define the triangular fuzzy number. Greater than The probability can be calculated using equation (2). :

[0046]

[0047] Step 5: Calculate the unfuzzy weights. Further calculate the minimum probability that a certain feature fuzzy value is greater than all other fuzzy values. The result of equation (3) is the value of the feature. Nonnormalized fuzzy weights :

[0048]

[0049] After normalization, the final non-fuzzy weights of each physical feature can be calculated using equation (4):

[0050]

[0051] Based on the global non-fuzzy weight results of each feature, edge features with low contribution are removed according to the significance weight threshold to form a core physical feature set for subsequent work hour prediction.

[0052] After screening the core physical features, different types of features are uniformly vectorized. For discrete features such as assembly method, equipment type, and tooling category, one-hot encoding is used to obtain discrete feature vectors. For continuous or hierarchical numerical features such as the number of components, part weight, number of fasteners, and precision level, Min-Max normalization is used to obtain continuous feature vectors. By concatenating the two along the feature dimension, a unified structured physical feature vector is generated using equation (5):

[0053]

[0054] In the formula, This is a vector concatenation operation; This represents the total dimension of the reconstructed physical features; This is the final structured physical feature vector.

[0055] After constructing the structured physical feature vectors, semantic features are extracted and encoded for each sample's assembly process text. The technical route for assembly process text feature extraction is as follows: Figure 3 As shown, a fine-tuned BERT model is used, employing a bidirectional Transformer encoding structure to extract the contextual semantics of the process text, and the hidden state vector at the [CLS] position is used as the global semantic feature of the text. Compared to traditional TF-IDF or word frequency statistics methods, BERT can better express the contextual logic of action sequence, coordination relationships, and process constraints in assembly statements. The textual semantic features and the structured physical feature vectors together serve as input to the subsequent time prediction model.

[0056] Example 2

[0057] This embodiment focuses on augmenting small-sample multimodal data with physical constraints.

[0058] Building upon the text semantic features extracted in Example 1, this example focuses on augmenting assembly process text under small sample conditions. Since assembly process text is not ordinary natural language text, but contains key engineering entities such as component names, specifications, connector models, fit relationships, assembly actions, torque ranges, and specialized tooling, directly using a standard masked language model for random replacement can easily disrupt the physical relationships between these key entities. Therefore, this example constructs an assembly process entity dictionary by combining enterprise standard operating procedures, historical process documents, component lists, and assembly technical manuals, and extends the domain of the BERT underlying vocabulary. The assembly process entity dictionary may include entities such as "M8 bolt," "locating pin," and "interference fit."

[0059] In the mask prediction stage, the first... each word element The mask probability is calculated as shown in equation (6):

[0060]

[0061] In the formula, as a word element Word frequency statistics in the training corpus λ is the base mask rate, and λ is the word frequency attenuation coefficient. This strategy implements mask-free protection for key process entities and dynamically replaces non-entity ordinary process terms with masks, thereby expanding the text sample while maintaining the consistency of the assembly process logic.

[0062] For structured numerical characteristics, this embodiment constructs a dual-boundary constraint correction model combining extreme value relaxation boundaries and physical bottom lines. Its augmented correction process is as follows: Figure 4 As shown. Let For the first One candidate augmented numerical feature The first in the original training samples A set of physical numerical characteristics, where α is the flexible exploration coefficient allowing for distribution fluctuations, can be obtained from equation (7) as the corrected numerical characteristics:

[0063]

[0064] In the formula, and These represent the physical maximum and minimum values ​​of this feature in the real training set, respectively. The threshold for allowable fluctuations in the characteristic distribution.

[0065] The modified model allows for a limited distribution of exploration at the upper boundary and truncates when candidate values ​​significantly exceed a reasonable range; it introduces a non-negative physical baseline at the lower boundary to avoid generating samples with negative physical quantities such as weight, quantity, and time, or samples that do not conform to common sense in engineering.

[0066] Example 3

[0067] This embodiment focuses on the construction of a deep hybrid attention network optimized by the sparrow search algorithm and the prediction of assembly time.

[0068] Traditional Early Fusion methods typically concatenate textual semantic features with structured physical features directly, making it difficult to characterize the changing importance of physical parameters under different process semantic conditions. For example, in processes like "press-fitting bearings," fitting accuracy and pressure parameters significantly impact assembly time; in processes like "tightening bolts by torque," the number of fasteners and torque range have a more pronounced impact on assembly time. Therefore, this embodiment constructs a deep hybrid attention network optimized with a sparrow search algorithm. This network dynamically guides the weighting of structured physical features through process text semantics and utilizes the sparrow search algorithm to globally optimize key hyperparameters, thereby improving the stability and prediction accuracy of the model training in small-sample assembly time prediction scenarios. The network architecture is as follows: Figure 5 As shown, the hyperparameter optimization process is as follows: Figure 6 As shown.

[0069] Specifically, the text semantic features obtained in Example 1 With structured physical eigenvectors Input a deep hybrid attention network. Map the text semantic features to a query matrix Q, and the structured physical features to a key matrix K and a value matrix V, as shown in equation (8):

[0070]

[0071] In the formula, , , These represent the linear mapping matrices corresponding to the query, key, and value, respectively.

[0072] Subsequently, attention weights are calculated by querying the correlation between the query matrix Q and the key matrix K, and these weights guide the weighted aggregation of physical features. Further, the weighted physical features are concatenated with the original text semantic features, and the original text semantic features are preserved through residual connections. The cross-modal fusion features are then calculated using equation (9):

[0073]

[0074] In the formula, This is a scaling factor used to prevent the softmax gradient from vanishing due to an excessively large dot product result.

[0075] Cross-modal fusion features The input is a multilayer perceptron regression layer, and the output is the predicted working time ŷ. During the model training phase, the AdamW optimizer is used to update the network parameters, and the loss function uses mean squared error combined with an L2 regularization term to improve training stability under small sample conditions.

[0076] Since the prediction performance of deep hybrid attention networks is closely related to the learning rate, Dropout inactivation rate, number of hidden nodes in the fusion layer, cross-attention shared dimension, regularization coefficient, and number of BERT frozen layers, there are coupling relationships between different hyperparameters. Relying solely on human experience or grid search can easily lead to local optima and reduce the model's generalization ability. Therefore, this embodiment further introduces the Sparrow Search algorithm to globally optimize the key hyperparameters of the deep hybrid attention network.

[0077] The hyperparameters to be optimized are encoded as the locations of individual sparrows. ,in , Indicates the learning rate. Indicates the number of hidden nodes in the fusion layer. This indicates the Dropout failure rate. This indicates the number of layers to freeze in the pre-trained model. The learning rate search range is set to [2x10]. -5 1x10 -4 The candidate set for the number of hidden nodes in the fusion layer is set to {32, 64, 128, 256}. The values ​​of Dropout inactivation rate and BERT frozen layer number are optimized by the sparrow search algorithm within their respective preset ranges. The cross-attention shared dimension and regularization coefficient are fixed at 64 and 0.025 respectively, and do not participate in the optimization. The validation set mean squared error is used as the fitness function, and the population position is iteratively updated according to the position update rules of discoverers, joiners, and watchers until the maximum number of iterations or the fitness value converges, obtaining the globally optimal hyperparameter combination.

[0078] Finally, the optimal hyperparameter combination is substituted into a deep hybrid attention network, and the model is trained using physical constraint multimodal augmented samples. The assembly process text to be tested and its corresponding structured physical features are input into the trained model, and the assembly time prediction value is output through the regression prediction layer. After subsequent verification and parameter adjustment, the assembly time prediction value is used to obtain the assembly time prediction model and the final assembly time prediction result that meet the preset verification requirements.

[0079] Example 4

[0080] This embodiment focuses on validating the effectiveness of the multimodal augmented samples and the SSA-DHAN prediction model. Through distribution consistency verification, physical constraint consistency verification, model comparison experiments, and ablation experiments, the performance of the method in small-sample assembly time prediction tasks is evaluated. Specifically, physical constraint consistency verification ensures that the augmented numerical features, after correction by the dual boundary constraints described in step S2, all fall within a reasonable range defined by the upper boundary of extreme value relaxation and the lower boundary of the physical constraint, thereby guaranteeing that the augmented samples satisfy the assembly physical constraints. The specific operations are as follows:

[0081] Experimental data

[0082] The experimental data used in this embodiment comes from the actual final assembly line of a special vehicle manufacturer. Assembly time tags are extracted from the company's historical assembly process cards and completion records archived in the Manufacturing Execution System (MES). Corresponding input feature data are simultaneously extracted from process documents such as two-dimensional drawings, assembly process cards, and assembly detail sheets, as well as historical measured time records, including multimodal information such as assembly process text, structured physical parameters, and on-site operation records. After cleaning and manual verification of missing fields, abnormal records, and duplicate samples in the original data, 123 valid assembly process samples were finally obtained. Among them, 103 samples were used as training samples and multimodal data augmentation was performed according to the physical constraint correction strategy described in step S2 at an augmentation ratio of 1:3 (in this article, "1:n" means that each original training sample generates n additional augmented samples, so the total number of augmented samples is (1+n) times the original sample size). 309 augmented samples were generated from the 103 original training samples, resulting in a total of 412 training samples. The remaining 20 samples were used as independent test samples to verify the performance of the prediction model trained in step S3.

[0083] Experimental setup

[0084] The experimental environment is as follows: the processor is a 13th Gen Intel(R) Core(TM) i5-13400F; the CPU frequency is 2.50 GHz; the memory is 16.00 GB; the GPU is an NVIDIA GeForce RTX 4060; the operating system is Windows 10 (64-bit); the programming language is Python 3.9.23; and the deep learning framework is PyTorch 2.5.1.

[0085] The SSA-DHAN model training settings are as follows: the text semantic feature extraction module uses a pre-trained BERT model, the regression prediction module uses the AdamW optimizer for parameter updates, and the loss function is the mean squared error loss function combined with an L2 regularization term. The learning rate, Dropout inactivation rate, number of hidden nodes in the fusion layer, and number of frozen layers in the deep hybrid attention network are used as hyperparameters to be optimized. These hyperparameters are globally optimized using a sparrow search algorithm based on the mean squared error of the validation set partitioned from the training samples. Each hyperparameter is iteratively updated within a preset search range until the maximum number of iterations or the fitness value converges.

[0086] Augmented data size optimization and fidelity verification analysis

[0087] To verify the quality of the augmented samples, this embodiment uses the Maximum Mean Difference (MMD) to evaluate the distributional consistency between the augmented samples and the original samples. As one implementation method, augmented datasets with augmentation ratios of 1:1, 1:2, 1:3, 1:4, and 1:5 were constructed based on 103 original training samples, and the MMD values ​​for each ratio were calculated. The results are shown in Table 2.

[0088]

[0089] As shown in Table 2, the MMD distance remains at a low level under different augmentation ratios, indicating that the generated augmented samples can maintain the overall distribution characteristics of the original assembly data well. As the augmentation ratio increases, the MMD distance generally shows a trend of first decreasing, reaching a minimum value of 0.022 at an augmentation ratio of 1:3, and then slightly rising and stabilizing, indicating that the augmented samples have the best consistency with the original samples in terms of distribution around 1:3.

[0090] Furthermore, the t-SNE method is used to reduce the dimensionality of the high-dimensional textual semantic features of the original and augmented samples at an augmentation ratio of 1:3 and visualize their scatter distribution as follows: Figure 7As shown in the figure, in one implementation, the original samples and augmented samples show significant overlap in the two-dimensional latent semantic space, and no isolated clusters deviating significantly from the original distribution appear, indicating that the BERT augmentation method based on physical constraints does not produce significant semantic drift. Finally, an augmentation ratio of 1:3 was determined to be the optimal augmentation scale, expanding 103 original training samples into 412 joint training samples, providing data support for the stable training of deep networks under small sample conditions.

[0091] Comparative Experiment and Result Analysis of SSA-DHAN Model

[0092] To verify the effectiveness of the method of this invention in predicting assembly time for small samples of special vehicles, five-fold cross-validation was used to obtain the out-of-fold prediction results, such as... Figure 8 As shown. Evaluation metrics include mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), and coefficient of determination (COP). ).

[0093] As one implementation method, the SSA-DHAN model described in this invention has a MAE of 3.34, an RMSE of 4.29, and a MAPE of 15.26% in five-fold cross-validation. A score of 0.85 indicates that the model can explain approximately 85% of the time fluctuations and complete assembly time prediction within a low error range.

[0094] To verify the superiority of the cross-modal fusion structure, Random Forest (RF), Support Vector Regression (SVR), Backpropagation Neural Network (BPNN), BERT, TextCNN, and Early Fusion were selected as comparison models. RF, SVR, and BPNN were used to evaluate the time prediction capability of structured physical features; BERT and TextCNN were used to evaluate the modeling capability of pure text semantic features; and Early Fusion adopted a fusion strategy that directly concatenates text semantic features with physical features. The experimental results for each model under the same augmented training set and five-fold cross-validation are shown in Table 3.

[0095]

[0096] Analysis of Table 3 shows that the prediction performance of the pure text semantic model BERT is significantly better than that of the traditional structured regression model. This indicates that the assembly process text contains a large amount of implicit semantic information related to work time. In contrast, relying solely on structured physical parameters is insufficient to fully represent the dynamic process logic and semantic relationships in complex assembly processes. Further comparison of multimodal fusion models reveals that although the Early_Fusion model, which employs a simple feature concatenation strategy, can utilize both textual semantics and physical features simultaneously, its prediction performance is still significantly lower than that of the SSA-DHAN model. Specifically, compared to the Early_Fusion model, the MAPE of the SSA-DHAN model decreased from 20.11% to 15.26%, and the coefficient of determination (... The accuracy improved from 0.72 to 0.85, indicating that traditional hard-joining methods are insufficient to effectively characterize the deep coupling between process semantics and physical parameters. This invention utilizes a cross-attention mechanism to dynamically guide physical features through process semantics, achieving better prediction accuracy and fitting ability compared to direct joining.

[0097] To further verify its engineering generalization ability, 20 original assembly samples that did not participate in the model training, parameter optimization, and feature alignment processes were selected as an independent test set. As one implementation method, the SSA-DHAN model of this invention achieved a MAE of 4.01, an RMSE of 4.77, and a MAPE of 15.10% on the independent test set. Furthermore, 80% of the test samples had prediction errors controlled within the 20% industrial tolerance range, indicating that the method of this invention has good prediction stability and engineering applicability in real-world complex equipment assembly scenarios.

[0098] SSA-DHAN model ablation experiment and results analysis

[0099] Furthermore, to verify the contribution of key modules, ablation experiments were conducted on the SSA-DHAN model proposed in this invention. Maintaining the consistency of the network structure and training data, an SSA-DHAN model without Aug was constructed to remove the data augmentation module, and an SSA-DHAN model without PM was constructed to remove the physical constraint masking mechanism. Augmented text was generated using a standard random MLM masking method. The experimental results are shown in Table 4.

[0100]

[0101] As shown in Table 4, the model's predictive performance significantly degraded after removing the data augmentation module, with MAPE increasing from 15.26% to 74.55%. The performance decreased to 0.31. This indicates that deep networks struggle to establish stable feature mapping relationships under extremely small sample conditions, and data augmentation plays a crucial role in mitigating model generalization degradation. Removing the physical constraint masking mechanism also resulted in a decline in model prediction performance. Specifically, MAPE increased from 15.26% to 18.40%. The value dropped to 0.78. This indicates that traditional random MLM masking methods easily disrupt the relationships between key physical entities and process constraints in the assembly process, leading to local semantic distortion in the generated text.

[0102] In summary, this invention effectively addresses the problems of sparse samples, complex process texts, heterogeneous physical features, and insufficient model generalization in the multi-variety, small-batch assembly production of special vehicles by using a deep hybrid attention network for prediction and result verification adjustment through physical constraint small sample augmentation, core physical feature screening, and sparrow search algorithm optimization. It provides reliable data support for time assessment, production scheduling, and cost accounting in the discrete assembly process of complex equipment.

[0103] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A physical constraint multimodal time prediction method for small-sample assembly scenarios, characterized in that, Includes the following steps: S1: Collect multimodal time data during the assembly process, construct a subset of core physical features of the assembly time based on the triangular fuzzy hierarchical analysis method, and encode, normalize and reconstruct the structured physical features in the subset of core physical features to obtain structured physical feature vectors; and perform semantic encoding on the assembly process text to extract semantic features of the assembly process text. S2: Based on the physical constraint correction strategy, small sample multimodal data augmentation is performed. Key process entities in the assembly process text are protected without masking. The physical numerical features generated by augmentation are corrected by double boundary constraints to obtain multimodal augmented samples that meet the assembly physical constraints. S3: Construct a deep hybrid attention network optimized by the sparrow search algorithm. Input the semantic features of the assembly process text and the structured physical feature vector into the deep hybrid attention network. Dynamically weight the structured physical features guided by the text semantics to generate cross-modal fusion features. Use the sparrow search algorithm to globally optimize the key hyperparameters of the deep hybrid attention network. Substitute the obtained optimal hyperparameter combination into the deep hybrid attention network for training. Output the assembly time prediction value through the regression prediction layer. S4: Verify and adjust the models and methods constructed in S2 and S3.

2. The method for predicting physical constraints and multimodal working times for small-sample assembly scenarios as described in claim 1, characterized in that: Step S1 specifically includes: Multimodal data of assembly time were collected from process documents such as two-dimensional drawings, assembly process cards, assembly detail sheets, and historical time records, and missing fields, abnormal records and duplicate samples were cleaned. A candidate time influencing factor system is constructed around the assembly object, assembly action, assembly resources and work environment constraints. The candidate time influencing factors include the number of components, component weight, number of fasteners, fit method, accuracy requirements, tooling equipment, work space constraints and collaborative operation requirements. A triangular fuzzy judgment matrix is ​​established using an expert group decision-making approach. The expert evaluation results are converted into triangular fuzzy numbers, and the triangular fuzzy numbers are defuzzified. The non-fuzzy weights of each candidate working time influencing factor are obtained by combining the probability degree calculation. Based on the ranking results of the non-fuzzy weights, core physical features are selected to construct a subset of core physical features for assembly time. The discrete features in the core physical feature subset of the assembly time are one-hot encoded, the continuous features are Min-Max normalized, and the encoded discrete feature vector is concatenated with the normalized continuous feature vector to obtain the structured physical feature vector. The assembly process text is input into the fine-tuned BERT model to extract text semantic features used to characterize the global semantic information of the assembly process text.

3. The method for predicting physical constraints and multimodal working times for small-sample assembly scenarios as described in claim 1, characterized in that: The assembly process text augmentation in step S2 specifically includes: An assembly process entity dictionary is constructed based on assembly process cards, standard operating procedures, parts lists, process specifications, and assembly technology manuals, and the underlying vocabulary of the pre-trained language model is expanded using the assembly process entity dictionary. The input assembly process text is traversed by word. When a word belongs to the assembly process entity dictionary, the mask probability of the word is set to 0. When a word does not belong to the assembly process entity dictionary, the mask probability is dynamically calculated based on the word frequency of the word in the training corpus, and augmented assembly process text is generated based on the mask language model.

4. The method for predicting physical constraints and multimodal working times for small-sample assembly scenarios as described in claim 1, characterized in that: The numerical augmentation correction in step S2 specifically includes: Perturb the continuous numerical features in the structured physical feature vector to generate candidate augmented numerical features; A dual-boundary constraint correction model is established based on the maximum and minimum values, physical baseline, and allowable fluctuation coefficient of the corresponding physical numerical features in the original training samples. The dual boundaries include an upper boundary for extreme value relaxation and a lower boundary for the physical baseline. When a candidate augmented numerical feature is higher than the upper boundary of extreme relaxation, the candidate augmented numerical feature is truncated to the maximum value of the corresponding physical numerical feature in the original training sample; when a candidate augmented numerical feature is lower than the lower boundary of the physical bottom line, the candidate augmented numerical feature is truncated to the larger of the minimum value of the corresponding physical numerical feature in the original training sample and the physical bottom line; when a candidate augmented numerical feature is between the upper boundary of extreme relaxation and the lower boundary of the physical bottom line, the candidate augmented numerical feature is retained.

5. The method for predicting physical constraints and multimodal working times for small-sample assembly scenarios as described in claim 1, characterized in that: Step S3 specifically includes: Text semantic features are mapped to a query matrix Q, and structured physical feature vectors are mapped to a key matrix K and a value matrix V; The attention weights between the query matrix Q and the key matrix K are calculated using a scaled dot product attention function, and the structured physical feature vector is dynamically weighted using these attention weights to obtain a semantically guided physical feature representation. The semantically guided physical feature representation is concatenated with the original text semantic features, and the original text semantic features are preserved through residual connection to obtain cross-modal fusion features; The initial learning rate, the number of hidden nodes in the fusion layer, the Dropout inactivation rate, and the number of frozen layers in the deep hybrid attention network are used as hyperparameters to be optimized, and these hyperparameters are encoded as individual positions in the sparrow search algorithm. Using the mean squared error of the validation set as the fitness function, the population position is iteratively updated according to the position update rules of discoverers, joiners and vigilers until the maximum number of iterations or the fitness value converges, thus obtaining the globally optimal hyperparameter combination. The deep hybrid attention network is trained based on the globally optimal hyperparameter combination. The cross-modal fusion features are input into the regression layer of the multilayer perceptron, and the mean squared error loss function combined with the L2 regularization term is used for training to output the assembly time prediction value.

6. The method for predicting physical constraints and multimodal working times for small-sample assembly scenarios as described in claim 1, characterized in that: Step S4 specifically includes: The multimodal augmented samples obtained in step S2 are subjected to distribution consistency verification and physical constraint consistency verification to determine whether the multimodal augmented samples meet the preset augmentation quality requirements. The prediction performance of the prediction model trained in step S3 is verified. The prediction performance verification is based on the validation set and independent test samples to calculate a preset prediction performance evaluation index. The preset prediction performance evaluation index includes at least one of mean square error, mean absolute error, root mean square error, mean absolute percentage error and coefficient of determination. When the multimodal augmented samples do not meet the preset augmentation quality requirements or the prediction model does not meet the preset error requirements, the data augmentation ratio, numerical feature perturbation coefficient, hyperparameter optimization range or model training parameters are adjusted, and the corresponding data augmentation, model training or parameter optimization process is re-executed. When the multimodal augmented sample meets the preset augmentation quality requirements and the prediction model meets the preset error requirements, the final assembly time prediction model and assembly time prediction results are output.

7. A physical constraint multimodal time prediction method for small sample assembly scenarios as described in any one of claims 1-6, characterized in that: This method can be used for time quota prediction, production scheduling assistance, or cost accounting in the production scenarios of special vehicles, complex equipment, or small-batch discrete assembly of multiple varieties.

Images