Case process large model method and system based on multi-teacher distillation
By training a lightweight student model using multi-teacher distillation technology, the problems of insufficient knowledge integration and catastrophic amnesia in the generation of chassis process documents are solved, achieving efficient and accurate generation of process documents, which is applicable to military and manufacturing industries.
Patent Information
- Application Number
- CN202511697360.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-03-03
AI Technical Summary
Existing methods for generating process documents suffer from insufficient knowledge integration, low model training efficiency, low accuracy, and catastrophic forgetting issues when dealing with complex products such as chassis, making it difficult to generate process documents that meet high industry standards.
A multi-teacher distillation technique was employed to train three independent teacher models focusing on chassis structure design, material selection and quality standards, and processing technology, respectively. Soft labels were generated through distillation temperature normalization, and a weighted loss function was constructed to train a lightweight student model. The model's process documentation generation was optimized by combining self-distillation mechanism and quantization processing.
It achieves efficient and accurate generation of process documents, improves the stability and applicability of the model, and is suitable for fields such as military and manufacturing, meeting the real-time requirements of industry.
Smart Images
Figure CN121598757A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a method and system for a large-scale chassis process model based on multi-teacher distillation. Background Technology
[0002] With the rapid development of intelligent and automated technologies, especially in industries with high precision and safety requirements such as military and manufacturing, the demand for automatic generation and optimization of process documents is increasing. In these industries, the accuracy and standardization of process documents directly affect production efficiency, product quality, and safety. Traditional process document generation relies on human experience, which has drawbacks such as low work efficiency, high error rate, and difficulty in handling complex product requirements. This is especially true when it comes to process documents for complex products such as chassis, where manual methods are particularly challenging.
[0003] Chassis manufacturing process documents typically involve multiple professional fields, such as chassis structural design, material selection and quality standards, and manufacturing processes, each with its unique requirements and standards. How to effectively integrate the expertise from these fields and automatically generate high-quality, industry-standard manufacturing process documents has become a pressing issue.
[0004] Existing methods for generating process documents primarily rely on rule-based automated systems or template-based techniques. However, these methods typically only handle structured data and are less capable of processing complex unstructured data, such as natural language text, design drawings, and historical process data. Furthermore, they struggle to make flexible adjustments when faced with changing actual process requirements. In addition, existing deep learning technologies face significant challenges in handling industry-specific expertise, particularly in the fusion and integration of multi-domain knowledge. Current technologies often neglect the importance of domain knowledge and suffer from insufficient accuracy, resulting in process documents that fail to meet high industry standards.
[0005] In traditional deep learning methods, the catastrophic forgetting problem is also prominent, especially when multiple models collaborate. Ensuring the effective transfer of knowledge between models from different domains and avoiding the loss of important information is a major bottleneck in current technology. Therefore, how to solve these problems through innovative technologies and improve the quality, efficiency, and intelligence level of process document generation has become a technical challenge that the industry urgently needs to overcome.
[0006] To address these issues, recent research has attempted to automate the generation of process documents using large-scale deep learning models, particularly in the field of chassis design. Existing research has focused on utilizing multimodal data (such as text, images, and time-series data) for training to improve the model's learning and generalization abilities. However, most current research concentrates on training single models, lacking effective multi-domain knowledge fusion and failing to effectively mitigate the catastrophic forgetting problem in deep learning models. Summary of the Invention
[0007] To address these issues, this invention provides a method and system for creating a large-scale chassis process model based on multi-teacher distillation. This addresses problems in existing automatic process document generation technologies, such as insufficient knowledge integration, low model training efficiency, low accuracy, and catastrophic forgetting. By integrating expertise from multiple fields through multi-teacher distillation technology, a lightweight student model is trained, thereby achieving efficient, accurate, and intelligent generation of chassis process documents. This improves the quality and efficiency of process document generation, making it suitable for fields such as military and manufacturing, and facilitating practical deployment.
[0008] To address the aforementioned technical problems, this invention provides a method for a large-scale chassis process model based on multi-teacher distillation, comprising: Collect multimodal data related to chassis manufacturing process, preprocess the multimodal data, and form training samples with hard labels; Train at least three independent teacher models, each focusing on the domains of chassis structure design, material selection and quality standards, and manufacturing process, respectively. Each teacher model is trained based on data from its corresponding domain and outputs logical results. The logical results output by each teacher model are normalized according to the set distillation temperature to generate soft labels. The soft labels are then weighted and fused based on the validation set performance indicators of each teacher model to obtain the target soft labels. A weighted loss function is constructed using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label. A lightweight student model is obtained by training by minimizing the weighted loss function. The soft loss represents the loss between the student model's output prediction result and the target soft label, and the hard loss represents the loss between the student model's output prediction result and the hard label. Based on the output of the student model, the teacher model is iteratively optimized through self-distillation according to a preset period or data increment conditions. The student model is quantized and accelerated using mixed-precision inference, and the optimized student model is used to generate chassis process documents that conform to industry standards.
[0009] In one embodiment of the present invention, the multimodal data includes text data, image data, and time series data; The multimodal data is preprocessed, including: performing word segmentation and noise reduction on text data, performing size standardization and data augmentation on image data, and performing uniform formatting and normalization on time-series data.
[0010] In one embodiment of the present invention, each of the teacher models is trained based on corresponding domain data and outputs logical results, including: A teacher model focused on chassis structure design, trained based on chassis design standard data and CAD drawing data; A teacher model focused on material selection and quality standards, trained based on a material property database and chassis quality standard data; The teacher model focuses on processing technology and is trained based on chassis processing flow data and processing equipment operation procedure data.
[0011] In one embodiment of the present invention, the soft tags are weighted and fused based on the validation set performance metrics of each of the teacher models to obtain the target soft tags, including: Using the accuracy of each teacher model on the validation set as a weighting coefficient, the soft labels generated by each teacher model are weighted and summed to obtain the target soft label.
[0012] In one embodiment of the present invention, the target soft label is obtained by weighting and summing the soft labels generated by each teacher model using the accuracy of each teacher model on the validation set as a weighting coefficient, including: The prediction results are abstracted into soft labels using the posterior probability of a single teacher neural network: ; in, It is a soft tag; It is the number of input data categories; It is the distillation temperature coefficient, used to control the degree of softening of soft tags generated by the neural network; It is the first in the data output by the neural network Logical unit value of class data; The weighted average of the soft labels in the teacher model is calculated using the following formula: ; in, It is a weighted average of the soft labels in the teacher model; It is a teacher model The weights; It is the number of teacher models; It's a soft tag.
[0013] In one embodiment of the present invention, a weighted loss function is constructed using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label, including: The soft loss is expressed as: ; in, It is a soft loss; It is the distillation temperature coefficient; These are soft labels output by the teacher model; These are the soft labels output by the student model; It is the average soft label output by the i-th class of data in the teacher model; It is the average soft label output by the i-th class of data in the student model; It is the predicted category probability output by the teacher model; The predicted class probability output by the student model; The teacher model outputs the prediction of class i. The student model outputs the prediction of class i. The hard loss is expressed as: ; in, It's a hard loss; These are hard labels for the data; It is the probability predicted by the model output; It is the probability that the model predicts it to be of class i; The weighted loss function is expressed as follows: ; in, It is a hyperparameter setting that determines the ratio of hard loss to soft loss, controlling the degree to which the student model learns from the teacher model.
[0014] In one embodiment of the present invention, the teacher model is iteratively optimized through self-distillation according to a preset period or data increment conditions, including: Using the current parameters of the teacher model as initial values, and the fusion dataset of new data and historical training data as training data, the teacher model is retrained using the knowledge distillation loss function to update the model parameters. The preset period is three months, and the data increment condition is that the amount of new data reaches 5%-10% of the total amount of current teacher model training data.
[0015] 11. A method for a large-scale chassis process model based on multi-teacher distillation according to claim 7, characterized in that the knowledge distillation loss function is expressed as: ; in, It is a hard-label loss, which measures the output of the teacher model relative to the true labels. The differences between them; It is the first The soft labels output by the teacher model during each training round; It is the first The soft labels output by the teacher model during each training round; It represents the proportion of hard labels in the loss function; It represents the proportion of soft labels in the loss function; Represented using the cross-entropy loss function: ; in, It is the index of the category, representing a traversal of all possible output prediction categories; It is a category in the real label. The value is either 0 or 1; The category in the output probability distribution of the teacher model is The predicted probability.
[0016] In one embodiment of the present invention, the student model is subjected to quantization processing and mixed-precision inference acceleration, including: The quantization uses a 4-bit quantization method and incorporates quantization-aware training during the quantization process; the mixed-precision inference acceleration uses a mixed-precision calculation method that replaces 32-bit floating-point numbers with 16-bit floating-point numbers.
[0017] This invention also provides a system for a large-scale chassis process model based on multi-teacher distillation, comprising: The training sample acquisition module is used to collect multimodal data related to chassis process, preprocess the multimodal data, and form training samples with hard labels. The teacher model logic result output module is used to train at least three independent teacher models. The three teacher models focus on the fields of chassis structure design, material selection and quality standards, and processing technology, respectively. Each teacher model is trained based on data in the corresponding field and outputs logic results. The target soft label generation module is used to normalize the logical results output by each teacher model according to the set distillation temperature, generate soft labels, and perform weighted fusion of the soft labels according to the validation set performance index of each teacher model to obtain the target soft label. The student model training module is used to construct a weighted loss function using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label, and to train a lightweight student model by minimizing the weighted loss function; wherein, the soft loss represents the loss between the student model's output prediction result and the target soft label, and the hard loss represents the loss between the student model's output prediction result and the hard label; The iterative optimization module is used to iteratively optimize the teacher model based on the output of the student model, according to a preset period or data increment conditions, through a self-distillation method. The inference acceleration module is used to perform quantization processing and mixed-precision inference acceleration on the student model, and to generate chassis process documents that conform to industry standards using the optimized student model.
[0018] The technical solution of the present invention has the following advantages compared with the prior art: This invention discloses a method and system for large-scale chassis process modeling based on multi-teacher distillation, and an automatic process document generation technology based on large-scale machine learning models. Specifically, it optimizes and improves the accuracy and efficiency of chassis process document generation through multi-teacher distillation. This invention is applicable to fields requiring accurate and efficient process document generation, such as the military, manufacturing, and other related industries, solving problems such as insufficient knowledge integration, low model efficiency, and catastrophic forgetting in existing process document generation technologies.
[0019] This invention efficiently integrates knowledge from multiple fields: employing multi-teacher distillation technology, it effectively integrates expertise from various areas such as chassis structure design, material selection and quality standards, and processing technology. Through this technology, the model can share knowledge across different fields, thereby generating more accurate and comprehensive process documents and improving the precision and quality of automated generation.
[0020] The model of this invention is highly efficient and lightweight: by training a lightweight student model, this invention significantly reduces the computational resource requirements of the model while maintaining high accuracy. This enables the method to run efficiently under limited hardware resources, facilitating large-scale practical applications and deployments.
[0021] This invention is resistant to catastrophic forgetting: Through a self-distillation mechanism and iterative optimization of the teacher model, it effectively avoids the problem of catastrophic forgetting. The knowledge of the teacher model can be updated and maintained over time, ensuring that important domain knowledge is not lost during long-term learning, thereby improving the stability and sustainability of the model.
[0022] This invention features fast inference speed: During model optimization, this invention effectively reduces memory usage and improves inference speed through quantization and mixed-precision inference acceleration techniques. This optimization enables the generated process documents to respond more quickly to actual production needs, meeting the real-time requirements of industry.
[0023] This invention has strong applicability: the method can be applied to high-precision process document generation scenarios such as military and manufacturing industries, promoting the intelligentization and standardization of process document generation. Attached Figure Description
[0024] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0025] Figure 1 This is a flowchart of the large-scale model method for chassis process based on multi-teacher distillation of the present invention.
[0026] Figure 2 This is a schematic diagram of the multi-teacher distillation method of the present invention.
[0027] Figure 3 This is a schematic diagram of the teacher's self-distillation of the present invention. Detailed Implementation
[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.
[0029] Example 1 Reference Figure 1 As shown in the figure, this embodiment presents a method for generating large-scale chassis process models based on multi-teacher distillation, applicable to chassis process document generation scenarios in the military or manufacturing industries. The method includes the following steps: S1. Collect multimodal data related to chassis process, preprocess the multimodal data, and form training samples with hard labels.
[0030] The multimodal data includes text data, image data, and time series data. Specifically, data preprocessing is a crucial step in training large models, directly impacting training effectiveness and prediction accuracy. Multimodal data related to chassis manufacturing processes is processed, including text, images, and other data types, with appropriate preprocessing steps applied to meet the input requirements of deep learning models. Preprocessing of the multimodal data includes: word segmentation and noise reduction for text data; size standardization and data augmentation for image data; and uniform formatting and normalization for time-series data.
[0031] Text data processing: The text data mainly comes from historical process documents, quality inspection records, and industry standard documents. To facilitate model training, the text is first segmented into words and then denoised to extract effective information and eliminate redundancy.
[0032] Word segmentation is the process of breaking down text into individual words or sub-words. This invention uses a word segmentation algorithm based on spaces and punctuation marks, suitable for both Chinese and English text.
[0033] The purpose of noise reduction is to remove irrelevant parts of text, such as stop words and irrelevant symbols. This invention uses stop word filtering and the removal of numbers and non-alphanumeric characters for noise reduction.
[0034] Assuming the input text Denoising the text .
[0035] For each word If it is a stop word, a number, or a non-alphabetic character, remove it from the text: ; in, It is a stop word list. It is a set of alphabetic characters. It is a set of numbers.
[0036] Image data processing: Image data includes chassis design drawings, CAD drawings, and quality inspection images. To address the high-dimensionality of images, size standardization and data augmentation are performed to improve the model's generalization ability and robustness. Image size standardization unifies images of different sizes to the same dimensions, facilitating batch processing and input into the neural network. This scheme uses bilinear interpolation to standardize image sizes. Data augmentation generates more training samples by performing various transformations on the images (such as rotation, translation, flipping, and cropping) to enhance the model's robustness.
[0037] Multimodal data integration: For data containing time-series or numerical information, such as sensor data and operation logs, uniform formatting and normalization processing is performed. Let the input dataset be... ,in For input data, For data labels, Given the dataset size, data prenormalization (performing uniform formatting and normalization on time-series data) can be represented as: ; in, and Decibels represent the mean and standard deviation of the data.
[0038] S2. Train at least three independent teacher models, each focusing on the domain of chassis structure design, material selection and quality standards, and processing technology. Each teacher model is trained based on data from its corresponding domain and outputs logical results.
[0039] This method designs and pre-trains three teacher models, each focusing on different domain knowledge. Specifically, these include: Teacher Model A: Chassis Structure Design Model. This model focuses on chassis structure design, and its learning content includes key features such as geometry, dimensional parameters, load-bearing capacity, and heat dissipation design. Training data primarily comes from chassis design standards, CAD drawings, and historical design cases, aiming to enable the model to accurately understand structural design specifications and extract relevant features.
[0040] Teacher Model B: Material Selection and Quality Standards Model. This model focuses on material performance and quality standards, comprehensively considering factors such as strength, corrosion resistance, and thermal conductivity. Training data includes a materials database, performance test records, and industry quality standards, ensuring that the model can recommend suitable materials and determine their applicability based on actual needs.
[0041] Teacher Model C: Machining Process Model. This model focuses on the machining process, covering key steps such as cutting, welding, and surface treatment. Training data comes from machining process documents, equipment operating procedures, and historical machining cases, aiming to enable the model to master the process parameters and key operational points in actual machining, ensuring the feasibility and accuracy of the process documents.
[0042] Each teacher model is trained in its focused domain, using deep learning techniques to extract knowledge from large amounts of specialized data, ensuring that the output of each teacher model contains the core information of that domain.
[0043] S3. Normalize the logical results output by each teacher model according to the set distillation temperature to generate soft labels. Then, perform weighted fusion of the soft labels based on the validation set performance indicators of each teacher model to obtain the target soft labels.
[0044] Specifically, the target soft label is obtained by weighting and summing the soft labels generated by each teacher model using the accuracy of each teacher model on the validation set as a weighting coefficient. This includes: Multi-teacher soft-label loss: The student model learns by acquiring soft labels from each teacher model. The predictions are abstracted into soft labels using the posterior probabilities of individual teacher neural networks. ; in, It is a soft tag; It is the number of input data categories; It is the distillation temperature coefficient, used to control the degree of softening of soft tags generated by the neural network; It is the first in the data output by the neural network Logical unit value of class data; The weighted average of the soft labels in the teacher model is calculated using the following formula: ; in, It is a weighted average of the soft labels in the teacher model; It is a teacher model The weights; It is the number of teacher models; It is a soft tag, that is .
[0045] S4. Construct a weighted loss function using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label, and train a lightweight student model by minimizing the weighted loss function; wherein, the soft loss represents the loss between the student model's output prediction result and the target soft label, and the hard loss represents the loss between the student model's output prediction result and the hard label.
[0046] In this stage, the goal of this method is to fuse the knowledge from multiple teacher models using multi-teacher distillation techniques to train a lightweight student model. (See reference...) Figure 2 As shown, the loss function under multi-teacher distillation consists of the soft label loss (soft loss) output by the teacher model and the hard label loss (hard loss) local to the student model.
[0047] The soft loss (soft label loss of multi-teacher distillation) is expressed as: ; in, It is a soft loss; It is the distillation temperature coefficient; These are soft labels output by the teacher model; These are the soft labels output by the student model; It is the average soft label output by the i-th class of data in the teacher model; It is the average soft label output by the i-th class of data in the student model; It is the predicted category probability output by the teacher model; The predicted class probability output by the student model; The teacher model outputs the prediction of class i. The student model outputs the prediction of class i. The hard loss (the local hard loss of the student model distilled by multiple teachers is the loss between the student model's predicted classification result and the data label) is expressed as: ; in, It's a hard loss; These are hard labels for the data; It is the probability predicted by the model output; It is the probability that the model predicts it to be of class i; The multi-teacher distillation loss function can be expressed as the loss between the student neural network's predictions and the data labels (hard loss) plus the loss between the student's soft labels and the teacher's soft labels (soft loss), i.e., the weighted loss function is expressed as: ; in, The hyperparameter setting determines the ratio of hard loss to soft loss, controlling the degree to which the student model learns from the teacher model. Multi-teacher distillation can be achieved by optimizing the student model parameters through backpropagation of the student model's loss function.
[0048] S5. Based on the output of the student model, the teacher model is iteratively optimized using a self-distillation method according to a preset period or data increment conditions. Specifically, the current parameters of the teacher model are used as initial values, and the fused dataset of new data and historical training data is used as training data. The teacher model is retrained using the knowledge distillation loss function to update the model parameters.
[0049] This method mitigates catastrophic forgetting and maintains long-term memory of key knowledge by periodically updating the teacher model based on student model output and adjusting teacher parameters using validation set performance. Knowledge updating of the teacher model is a crucial step in multi-teacher distillation. With increasing data volume, the teacher model may face catastrophic forgetting, where existing knowledge is overwritten by newly learned knowledge, leading to performance degradation. To avoid this problem, this method employs self-distillation, periodically updating the teacher model's parameters to strengthen its knowledge retention. Figure 3 As shown.
[0050] Self-distillation mechanism: Each teacher model periodically updates its knowledge using the soft labels output by the current student models as a reference. By calculating the difference between the old and new soft labels, the parameters of the teacher model can be dynamically adjusted to ensure that it maintains effective memorization of important domain knowledge in the long term.
[0051] Assume the first The soft labels output by the teacher model during each training round are , No. The soft label for wheels is Hard label Then the loss function of teacher self-distillation can be expressed as: ; in It is hard-label loss, i.e. Measuring the output of the teacher model relative to the true labels The differences between them; It is the first The soft labels output by the teacher model during each training round; It is the first The soft labels output by the teacher model during each training round; It represents the proportion of hard labels in the loss function; It represents the proportion of soft labels in the loss function; Here, the cross-entropy loss function is used to represent: ; in It is the index of the category, representing a traversal of all possible output prediction categories. It is a category in the real label. The value is usually 0 or 1; The category in the output probability distribution of the teacher model is The predicted probability.
[0052] Self-distillation iterative optimization conditions: The update cycle of the teacher model is usually set to three months. During each iteration, the teacher model is adjusted based on the performance on the validation set, selecting the knowledge most suitable for the current task. Alternatively, when the amount of data added reaches a threshold (covering 5%-10% of the teacher model's domain knowledge), an additional update is triggered to ensure that the model absorbs the latest domain knowledge.
[0053] S6. Quantize the student model and accelerate mixed-precision inference to reduce memory usage and improve inference efficiency. Use the optimized student model to generate chassis process documents that conform to industry standards.
[0054] The trained student model is quantized to reduce memory usage and accelerate the inference process, thereby improving the efficiency of the model in practical applications. Quantization usually adopts 4-bit quantization. The quantized model can greatly reduce the consumption of computing resources and improve the inference speed. The specific steps of quantization are: (1) Use integer representation to replace the original floating-point weights. (2) Simulate the quantization process during training through quantization-aware training (QAT) technology, so that the model can still maintain high accuracy even at low precision. (3) When using the quantized model, further optimization is needed in the forward inference stage of the model to ensure that the accuracy and effect of the model output are not affected.
[0055] In terms of inference acceleration, efficiency can be further improved through mixed-precision inference. Using mixed-precision computation, which replaces 32-bit floating-point numbers with 16-bit floating-point numbers, not only reduces computational load but also enables more efficient inference in hardware. Mixed-precision inference can significantly improve inference speed without substantially reducing model accuracy.
[0056] This method employs a multi-teacher distillation technique to integrate knowledge from multiple teacher models and train a lightweight student model. First, three teacher models focus on different domains, such as chassis structure design, material selection and quality standards, and manufacturing processes, generating their respective outputs. Then, the student model is trained and optimized by combining soft labels obtained from these teacher models with its local hard labels. By training a lightweight student model, high accuracy is maintained while significantly reducing the model's computational resource requirements. This allows the method to operate efficiently with limited hardware resources, facilitating large-scale practical applications and deployments. The self-distillation mechanism and iterative optimization of the teacher models effectively avoid catastrophic forgetting. The knowledge of the teacher models can be updated and maintained over time, ensuring that important domain knowledge is not lost during long-term learning, thereby improving the model's stability and sustainability.
[0057] Example 2 Based on the same inventive concept, this embodiment provides a system for a large-scale chassis process model based on multi-teacher distillation. The principle of solving the problem is similar to that of the aforementioned method for a large-scale chassis process model based on multi-teacher distillation, and the repetitions will not be repeated.
[0058] This embodiment provides a system for a large-scale chassis process model based on multi-teacher distillation, including: The training sample acquisition module is used to collect multimodal data related to chassis process, preprocess the multimodal data, and form training samples with hard labels. The teacher model logic result output module is used to train at least three independent teacher models. The three teacher models focus on the fields of chassis structure design, material selection and quality standards, and processing technology, respectively. Each teacher model is trained based on data in the corresponding field and outputs logic results. The target soft label generation module is used to normalize the logical results output by each teacher model according to the set distillation temperature, generate soft labels, and perform weighted fusion of the soft labels according to the validation set performance index of each teacher model to obtain the target soft label. The student model training module is used to construct a weighted loss function using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label, and to train a lightweight student model by minimizing the weighted loss function; wherein, the soft loss represents the loss between the student model's output prediction result and the target soft label, and the hard loss represents the loss between the student model's output prediction result and the hard label; The iterative optimization module is used to iteratively optimize the teacher model based on the output of the student model, according to a preset period or data increment conditions, through a self-distillation method. The inference acceleration module is used to perform quantization processing and mixed-precision inference acceleration on the student model, and to generate chassis process documents that conform to industry standards using the optimized student model.
[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0063] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for large-scale chassis process modeling based on multi-teacher distillation, characterized in that, include: Collect multimodal data related to chassis manufacturing process, preprocess the multimodal data, and form training samples with hard labels; Train at least three independent teacher models, each focusing on the domains of chassis structure design, material selection and quality standards, and manufacturing process, respectively. Each teacher model is trained based on data from its corresponding domain and outputs logical results. The logical results output by each teacher model are normalized according to the set distillation temperature to generate soft labels. The soft labels are then weighted and fused based on the validation set performance indicators of each teacher model to obtain the target soft labels. A weighted loss function is constructed using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label. A lightweight student model is obtained by training by minimizing the weighted loss function. The soft loss represents the loss between the student model's output prediction result and the target soft label, and the hard loss represents the loss between the student model's output prediction result and the hard label. Based on the output of the student model, the teacher model is iteratively optimized through self-distillation according to a preset period or data increment conditions. The student model is quantized and accelerated using mixed-precision inference, and the optimized student model is used to generate chassis process documents that conform to industry standards.
2. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 1, characterized in that, The multimodal data includes text data, image data, and time series data; The multimodal data is preprocessed, including: performing word segmentation and noise reduction on text data, performing size standardization and data augmentation on image data, and performing uniform formatting and normalization on time-series data.
3. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 1, characterized in that, Each of the aforementioned teacher models is trained based on data from the corresponding domain and outputs logical results, including: A teacher model focused on chassis structure design, trained based on chassis design standard data and CAD drawing data; A teacher model focused on material selection and quality standards, trained based on a material property database and chassis quality standard data; The teacher model focuses on processing technology and is trained based on chassis processing flow data and processing equipment operation procedure data.
4. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 1, characterized in that, The soft labels are weighted and fused based on the validation set performance metrics of each teacher model to obtain the target soft labels, including: Using the accuracy of each teacher model on the validation set as a weighting coefficient, the soft labels generated by each teacher model are weighted and summed to obtain the target soft label.
5. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 4, characterized in that, Using the accuracy of each teacher model on the validation set as a weighting coefficient, the soft labels generated by each teacher model are weighted and summed to obtain the target soft label, including: The prediction results are abstracted into soft labels using the posterior probability of a single teacher neural network: ； in, It is a soft tag; It is the number of input data categories; It is the distillation temperature coefficient, used to control the degree of softening of soft tags generated by the neural network; It is the first in the data output by the neural network Logical unit value of class data; The weighted average of the soft labels in the teacher model is calculated using the following formula: ； in, It is a weighted average of the soft labels in the teacher model; Teacher Model The weights; It is the number of teacher models; It's a soft tag.
6. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 1, characterized in that, A weighted loss function is constructed using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label, including: The soft loss is expressed as: ； in, It is a soft loss; It is the distillation temperature coefficient; These are soft labels output by the teacher model; These are the soft labels output by the student model; It is the average soft label output by the i-th class of data in the teacher model; It is the average soft label output by the i-th class of data in the student model; It is the predicted category probability output by the teacher model; The predicted class probability output by the student model; The teacher model outputs the prediction of class i. The student model outputs a prediction of the probability of class i. The hard loss is expressed as: ； in, It's a hard loss; These are hard labels for the data; It is the probability predicted by the model output; It is the probability that the model predicts it to be of class i; The weighted loss function is expressed as follows: ； in, It is a hyperparameter setting that determines the ratio of hard loss to soft loss, controlling the degree to which the student model learns from the teacher model.
7. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 1, characterized in that, The teacher model is iteratively optimized using a self-distillation method according to a preset period or data increment conditions, including: Using the current parameters of the teacher model as initial values, and the fusion dataset of new data and historical training data as training data, the teacher model is retrained using the knowledge distillation loss function to update the model parameters. The preset period is three months, and the data increment condition is that the amount of new data reaches 5%-10% of the total amount of current teacher model training data.
8. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 7, characterized in that, The knowledge distillation loss function is expressed as follows: ； in, It is a hard-label loss, which measures the output of the teacher model relative to the true labels. The differences between them; It is the first The soft labels output by the teacher model during each training round; It is the first The soft labels output by the teacher model during each training round; It represents the proportion of hard labels in the loss function; It represents the proportion of soft labels in the loss function; Represented using the cross-entropy loss function: ； in, It is the index of the category, representing a traversal of all possible output prediction categories; It is a category in the real label. The value is either 0 or 1; The category in the output probability distribution of the teacher model is The predicted probability.
9. The method for a large-scale chassis process model based on multi-teacher distillation according to claim 1, characterized in that, The student model undergoes quantization and mixed-precision inference acceleration, including: The quantization uses a 4-bit quantization method and incorporates quantization-aware training during the quantization process; the mixed-precision inference acceleration uses a mixed-precision calculation method that replaces 32-bit floating-point numbers with 16-bit floating-point numbers.
10. A system based on a large-scale chassis process model of multi-teacher distillation, characterized in that, include: The training sample acquisition module is used to collect multimodal data related to chassis process, preprocess the multimodal data, and form training samples with hard labels. The teacher model logic result output module is used to train at least three independent teacher models. The three teacher models focus on the fields of chassis structure design, material selection and quality standards, and processing technology, respectively. Each teacher model is trained based on data in the corresponding field and outputs logic results. The target soft label generation module is used to normalize the logical results output by each teacher model according to the set distillation temperature, generate soft labels, and perform weighted fusion of the soft labels according to the validation set performance index of each teacher model to obtain the target soft label. The student model training module is used to construct a weighted loss function using the soft loss corresponding to the target soft label and the hard loss corresponding to the hard label, and to train a lightweight student model by minimizing the weighted loss function; wherein, the soft loss represents the loss between the student model's output prediction result and the target soft label, and the hard loss represents the loss between the student model's output prediction result and the hard label; The iterative optimization module is used to iteratively optimize the teacher model based on the output of the student model, according to a preset period or data increment conditions, through a self-distillation method. The inference acceleration module is used to perform quantization processing and mixed-precision inference acceleration on the student model, and to generate chassis process documents that conform to industry standards using the optimized student model.