Policy generation method and system based on multi-modal large model

Abstract

The invention discloses a strategy generation method and system based on a multi-modal large model, and is applied to the whole process of automobile manufacturing. According to the method, production line visual images, process texts and sensor data are collected, cross-modal feature adaptive fusion is realized through a multi-modal large model, modal weights are adjusted according to production stages, and an initial strategy is generated in combination with manufacturing process knowledge; and outputting a target strategy adaptive to production through feedback iterative optimization. The system integrates a data acquisition module, a cross-modal fusion module, a strategy generation module and an optimization module, realizes weight scenarized adjustment based on large model dynamic learning, and accurately maps a production state. According to the method, a fixed weight mode is broken through, deep correlation reasoning is performed through multi-source data, the strategy adaptability and effectiveness are improved, intelligent decision making is supported, and automobile intelligent manufacturing is enabled.

Description

Technical Field

[0001] This invention relates to the field of intelligent automotive manufacturing, specifically to a strategy generation method and system based on a multimodal large model. Background Technology

[0002] Intelligent manufacturing in the automotive industry involves multiple processes and requires the integration of multimodal information to address production decision-making needs under complex operating conditions. However, traditional methods have systemic limitations, revealing shortcomings in adaptability across the entire chain from data perception to strategy output.

[0003] Single-modal analysis fragments the relationship between visual appearance, physical mechanism, and rule constraints. In the welding process, relying solely on visual inspection of surface defects fails to penetrate hidden anomalies. Fixed-weight fusion strategies ignore differences in process stages, diluting core contradictions under uniform weights and significantly reducing the strategy's universality. More importantly, the lack of a process knowledge graph and the open-loop decision-making mechanism make it difficult for the strategy to evolve with equipment aging and environmental fluctuations, resulting in adaptability and iteration efficiency lagging far behind the dynamic needs of the production line.

[0004] These shortcomings lead to a continuous accumulation of deviations between production strategies and actual working conditions, from missed defect detection to delayed decision-making, gradually restricting the accuracy and agility of intelligent manufacturing. Constructing a dynamically adaptable, knowledge-driven multimodal strategy generation method has become a core path to overcome the efficiency bottleneck in automobile manufacturing and an inevitable choice to solve the decision-making dilemma in multi-process scenarios.

[0005] Therefore, there is an urgent need to construct a new production strategy generation method based on a multimodal large model, which can improve the adaptability and accuracy of production strategies by combining visual, text, and sensor data to optimize production processes, provide early warnings for equipment maintenance, and trace the source of quality defects. Summary of the Invention

[0006] The purpose of this invention is to provide a strategy generation method based on a multimodal large model.

[0007] To achieve the above objectives, the present invention is implemented according to the following technical solution: This invention includes the following steps: Collect multi-source data from industrial vision, process text, and equipment sensors, including visual data, text data, and sensor data; Based on a serial strategy, differential feature extraction is performed on the multi-source data to obtain single-modal features. The visual data is used to obtain visual modal features through cross-layer fusion and dynamic scale attention mechanism. The text data is used to enhance semantics and temporal representation through domain terminology library to obtain text modal features. The sensor data is used to obtain sensor modal features through Transformer and Gaussian process regression algorithms. The single-modal features include visual modal features, text modal features and sensor modal features. The visual modal features and sensor modal features are used to establish a spatiotemporal mapping through a dynamic anchor matrix to generate the first cross-modal feature. The visual modal features and text modal features are used to generate the second cross-modal feature through cross-modal similarity with knowledge constraints, combined with temporal modeling and network mapping mechanisms. The sensor modal features and text modal features are used to generate the third cross-modal feature based on the attention mechanism and feature fusion principle. Based on the first cross-modal features, the second cross-modal features, the third cross-modal features, and the single-modal features, a calibration feature is obtained by fusing a three-layer linkage dynamic weight algorithm and combining intra-modal self-attention and cross-modal cross-attention to learn associated weights. The modal weights are dynamically adjusted according to multi-objective evaluation to adapt to scene changes, thereby generating a multi-modal scene adaptive optimizer. Knowledge-enhanced features are generated through knowledge cross-attention and the multimodal scene adaptive optimizer. Based on the knowledge enhancement features, a multimodal large model for intelligent vehicle manufacturing is constructed. The multimodal large model for intelligent vehicle manufacturing includes a Transformer encoder, a policy generation unit, a knowledge attention layer, a probability calculation unit, a decision evaluator, and a reward learning mechanism feedback. The reward learning mechanism feedback is obtained through conditional probability combined with time-varying risk-reward assessment.

[0008] Furthermore, the method for feature extraction from the multimodal data based on a concatenated strategy includes: The low-level features are extracted using ResNet-50. The low-level features are then fed into EfficientNet to extract high-level features for semantic judgment. Regularization terms are added to output visual modality features. Key region features are enhanced through cross-layer fusion and dynamic scale attention, as shown in the formula: , in, As a low-level feature, Characterized by high-level features To dynamically adjust the cross-layer weights at the i-th scale according to the production stage; This represents the contribution of adjusting the i-th scale; D is the defect rate sensitivity coefficient; D is the real-time defect rate. t represents the remaining welding time; t represents the welding time already completed. The spatial gradient of welding parameters along the welding torch movement direction z; The reference resistance for process stability; The current influence coefficient; For real-time welding current; These are the gradient weights for the process parameters; This represents the ideal feature vector in the process knowledge base. Given the actual visual feature vectors, dynamic scale attention is calculated using a self-attention mechanism: , Where Q, K, and V are the query matrix, key matrix, and value matrix of the feature map, respectively. It is a normalized parameter. Key region features after attention mechanism enhancement; This is a function that normalizes each row of the similarity matrix; This is the scaling factor; Features of V that are randomly discarded. The amplitude coefficient for Dropout; Based on the automotive manufacturing terminology database, the BERT model is fine-tuned to extract semantic embeddings specific to the automotive domain. Basic semantic features are generated by fine-tuning the BERT model based on the automotive manufacturing terminology database. The basic semantic features are used as input and simultaneously passed to TextCNN to extract static features and LSTM to capture time-dependent dynamic features. The weights of TextCNN and LSTM are adjusted through a gating function. The text modal features are output by combining BiLSTM and Transformer with weighted fusion. Combining TextCNN and LSTM to generate temporal-semantic cross features The formula is: , , Where T is a text sequence. For fine-tuning parameters in industrial terminology, It is a BERT model finely tuned using industry terminology; It is a domain-enhanced semantic embedding vector of a text sequence; It is the sigmoid function; The semantic modality features extracted by TextCNN; To obtain temporal semantic modal features after extraction using LSTM; For gated bias. , The coefficients are fusion coefficients; BiLSTM is a bidirectional long short-term memory network; and Transformer is a self-attention mechanism. Text modal features; The sliding window is dynamically divided according to the equipment cycle. The timing patterns and anomalies are captured by Transformer and Gaussian process regression (GPR), and the sensor modal characteristics are output.

[0009] Furthermore, a method for cross-modal differential feature enhancement of the modal features includes: Based on production condition parameters and process constraints, spatiotemporal alignment of visual modal features and sensor modal features is achieved through a time-series dynamic anchor matrix and scene adaptive factors. A cross-modal attention mechanism is used to strengthen the correlation of key regions, fusing visual modal features and sensor modal features to output the first cross-modal feature, as shown in the formula: , in, It is a spatiotemporal dynamic anchor point correlation matrix, generated by the coordinate mapping and feature correlation of the camera and sensor; Visual modal features; Sensor modal characteristics; Features for cross-temporal and spatial fusion of vision and sensors; This refers to the scene adaptive adjustment factor; It is a cross-modal topological feature; , For visual feature query matrix and sensor feature key matrix, is the scene adaptive normalization coefficient; T is the time step of the production stage. Visual modal features and text modal features are combined with knowledge-constrained cross-modal similarity, along with temporal modeling and network mapping mechanisms, to generate a second cross-modal feature: , , Where sim is a scalar measure of the correlation between visual features and text semantics. Enhancing semantic embedding vectors for the domain of text sequences; The feature fluctuation adjustment coefficient is dynamically adjusted based on the variance of visual and text features; It is a knowledge graph relation embedding function that encodes process relationships into vectors; It is a mapping function between visual features and textual semantics. It is a cross-modal feature vector of visual and textual data; It is a bidirectional long short-term memory network that captures temporal dependencies. , , It is a multilayer perceptron. To normalize attention weights; Based on the attention mechanism and feature fusion principle, a cross-modal fusion of sensor modal features and text modal features is constructed to output a third cross-modal feature: , , in, For cross-modal features of sensors and text; These are the correlation weight coefficients; Temperature compliance rate; This is the maximum temperature value; This represents the weighting modulation coefficient associated with KPIs; This is the alarm threshold; The variance of temperature; This is the text modality weight matrix; The sensor mode matrix; This is the cross-fusion weight matrix.

[0010] Furthermore, the three-layer linked dynamic weight algorithm includes: The first layer is the intramodal variance weight layer, which obtains the degree of fluctuation of the feature vector; the second layer is the cross-modal association weight layer, which updates the visual modal features; the third layer is the scene adaptation weight layer, which dynamically adjusts the modal weights. By calculating the variance and covariance of each modal feature, the degree of feature fluctuation is quantified, and the modal basis weights are determined. The formula is as follows: , , in, This refers to the basic weights of the i-th mode; Deviation between process parameters and standard values; Let i be the fundamental feature vector of the i-th mode; This represents the cross-modal characteristics of the i-th mode with other modes; KPI represents production outcome indicators. Indicates the weight of process indicators; Indicates the weight of production indicators; Textual semantic features; Indicates the adjustment factor across modes; The term represents a Gaussian penalty; the i-th modality includes: visual modality v, text modality t, and sensor modality s, and the cross-modal features include a first cross-modal feature, a second cross-modal feature, and a third cross-modal feature; Cross-modal association employs intra-modal self-attention to mine modal details, establishes inter-modal dependencies through cross-modal cross-attention, and calculates dynamic association weights through a query key-value mechanism. Based on the association weights, features are updated, multi-modal features are weighted and fused, and modal features are reconstructed by combining dynamic normalization and activation functions. The association weights between the i-th mode and other modes: , Feature update for the i-th mode: , in, This represents the association weight between the i-th mode and the j-th mode; , This represents mapping features to a projection matrix of the same dimension; Represents the L2 norm; This represents summation over all other modes; This represents the updated i-th modal feature; Represents the regularization coefficient; Based on a multi-objective optimization framework, the degree of deviation is quantified by weighted error, and weight smoothing penalty, parameter and cross-modal feature regularization are introduced to perform scene-adaptive weight optimization: , , AIM stands for Multimodal Scenario Adaptive Optimizer; This represents the predicted process deviation associated with the i-th mode; Indicates the target value for process deviation; Represents the scene-adaptive weights for the i-th mode; Regularization representing cross-modal features; The reference value representing the process deviation of the i-th mode; This represents the predicted KPI value associated with the i-th mode; Indicates the KPI target value; Represents the weight regularization coefficient; These are the regularization coefficients for the model parameters; These are the cross-modal characteristic regularization coefficients; These are the learnable parameters of the model.

[0011] Furthermore, methods for generating knowledge-enhanced features through knowledge cross-attention and multi-modal scene adaptive optimizers include: By fusing scene weights with updated modal features, combining intra-modal self-attention and cross-modal cross-attention, and introducing a knowledge graph (KG) specific to automotive manufacturing, process-related knowledge is encoded into knowledge embedding vectors to generate knowledge-enhanced features. The formula is as follows: , The formula for scene enhancement is: , The formula for modal correlation is: , The formula for the cross-attention component is: , in, Represents the modal features after scene enhancement; Represents a learnable matrix; Represents a Gaussian error linear unit; Represents the gate function; This is a global parameter representing the contribution weight of the knowledge module; This represents the minimum value; KG represents the embedding vector of the knowledge graph; This represents the normalization term of the cross product; , This is the knowledge projection matrix.

[0012] Furthermore, according to claim 1, the strategy generation method based on a multimodal large model is characterized in that, the method for obtaining the optimal strategy through probabilistic decision evaluation of the knowledge enhancement features includes: Based on the The scale is used to generate the k-th production strategy, and the formula is: , in, This represents the k-th production strategy; Indicates the first Layer-scale weights; For the first The weight matrix of the scale and the k-th strategy; Indicates the first Scale, bias of the k-th strategy; Indicates the first Variance of scale; This indicates knowledge-enhancing features; Based on knowledge graphs and augmented multimodal vectors, conditional probabilities are obtained using the following formula: , , , in, Representation strategy The conditional probability; F is the multimodal enhancement feature; The knowledge graph embedding vector for the k-th strategy; Let the target mean vector be the k-th strategy; Decisions are obtained through a time-decay Gaussian weighted average and a risk-squared weighted average mechanism, as shown in the formula: , in, This represents the optimal production strategy; express The conditional probability of the strategy; T represents the length of the time period; The variance representing time decay; Indicates the number of risk types; Representation strategy The degree of violation that leads to the first type of risk.

[0013] Furthermore, the method for constructing a reward mechanism that links time-varying risk and return includes: Collaborative modeling of returns and risks, implementing a dynamic reward function: , in, The time-varying risk coefficient; Indicates the level of risk at time t; This represents the change in production indicators at time t; Develop risk-sensitive rewards: , in, This represents the maximum historical risk before time t; Control factors indicating the severity of punishment; The gradient represents the policy. Strategy optimization goal: , in, The learnable parameters of the policy generator; Indicates the time discount factor; Let represent the learning reward at step t; M represents the number of candidate policies.

[0014] Secondly, a policy generation system based on a multimodal large model includes: Multimodal data acquisition module: used to collect multi-source data in the intelligent manufacturing process of automobiles. The multi-source data includes visual data acquired by industrial vision equipment, text data of process documents and production logs, and real-time operation data collected by sensors of production equipment. A cross-modal feature extraction module is used to extract differentiated features from the multi-source data based on a concatenated strategy to obtain single-modal features, which include visual modal features, text modal features, and sensor modal features. The visual modal features and sensor modal features are fused across modally to generate a first cross-modal feature, the visual modal features and text modal features are fused across modally to generate a second cross-modal feature, and the sensor modal features and text modal features are fused across modally to generate a third cross-modal feature. Weight fusion module: used to obtain calibration features from the first cross-modal feature, the second cross-modal feature, the third cross-modal feature and the single-modal feature; based on the calibration features, dynamically adjust the modal weights to adapt to scene changes according to multi-objective evaluation, and generate a multi-modal scene adaptive optimizer; Production strategy generation module: used to generate knowledge-enhanced features through knowledge cross-attention and the multi-modal scene adaptive optimizer; and to obtain the optimal production strategy by evaluating the knowledge-enhanced features through probabilistic decision-making. Closed-loop feedback optimization module: This module is used to construct a reward mechanism that correlates time-varying risk and return based on risk, gradient stability, and long-term return constraints. It then performs feedback optimization on the optimal production strategy according to the reward mechanism and iteratively updates the production strategy.

[0015] The beneficial effects of this invention are: This invention is a strategy generation method and system based on a multimodal large model. Compared with the prior art, this invention has the following technical advantages: This invention improves the accuracy of strategy generation for large multimodal models through preprocessing, multimodal data acquisition and cross-modal feature extraction, weight fusion, production strategy generation, and closed-loop feedback optimization. This significantly saves resources and increases work efficiency. By integrating an automotive manufacturing knowledge base, the invention uses multimodal features to perceive the current situation and professional knowledge base constraints to ensure the strategy meets actual industrial production needs. It constructs scenario-adaptive cross-modal collaborative features and, through three-layer weight fusion, allows input features to naturally carry automotive manufacturing process constraints and dynamic production scenario requirements. The generated strategy is geared towards full-process production collaboration, and through scenario-adaptive dynamic adjustment, the strategy is optimized in a closed loop according to the production line status, better meeting the continuous, collaborative, and dynamic production line requirements of intelligent automotive manufacturing. It possesses a certain degree of universality. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a strategy generation method based on a multimodal large model according to the present invention. Detailed Implementation

[0017] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0018] The present invention provides a strategy generation method and system based on a multimodal large model, comprising the following steps: like Figure 1 As shown, this embodiment includes the following steps: Collect multi-source data from industrial vision, process text, and equipment sensors, including visual data, text data, and sensor data; The system collects industrial vision weld seam images, welding process documents, and equipment sensor time-series data to form multi-source data. The vision data consists of 5000 frames of images with a resolution of 1920×1080 and a frame rate of 25fps. The text data consists of 1000 XML format process documents, each containing 10 instructions. The sensor data consists of 100Hz time-series signals of welding current and temperature, recording 500 time-series data points for 1000 five-second welding cycles. Based on a sequential strategy, differential feature extraction is performed on the multi-source data to obtain single-modal features; the single-modal features include visual modal features, text modal features, and sensor modal features; A concatenated strategy was employed to extract single-modal features. Visual data was processed using a concatenated ResNet-50 and EfficientNet structure, with a weight of 0.6 for the bottom-level features and 0.4 for the high-level semantic features. Cross-layer fusion was then performed to generate 256-dimensional visual modal features, with a statistical variance of 0.85. Text data was processed using a terminology database for automotive manufacturing welding, with a fine-tuned BERT model and embedded with specific semantics. The statistical variance of text modal features was 0.42. Sensor data was divided into sliding windows with 200 points and a step size of 100. After processing with 4 layers of Transformer and Gaussian process regression, 256-dimensional sensor modal features were generated, with a statistical variance of 0.63. The visual modal features and sensor modal features are used to establish a spatiotemporal mapping through a dynamic anchor matrix to generate the first cross-modal feature. The visual modal features and text modal features are used to generate the second cross-modal feature through cross-modal similarity with knowledge constraints, combined with temporal modeling and network mapping mechanisms. The sensor modal features and text modal features are used to generate the third cross-modal feature based on the attention mechanism and feature fusion principle. Visual and sensor modal features are spatiotemporally mapped using a dynamic anchor matrix with a value of 0.7, and then processed by a 3-layer MLP to generate a 512-dimensional first cross-modal feature with a statistical variance of 0.72. Visual and text modal features are generated based on the 0.85 cosine similarity between visual features of weld depth anomalies and textual semantics of welding parameters exceeding limits, combined with LSTM temporal modeling and MLP to generate a 512-dimensional second cross-modal feature. Sensor and text modal features are fused through an attention mechanism with an attention coefficient of 0.6 to generate a 512-dimensional third cross-modal feature with a statistical variance of 0.68. Calibration features are obtained based on the first cross-modal features, the second cross-modal features, the third cross-modal features, and the single-modal features; based on the calibration features, the modal weights are dynamically adjusted to adapt to scene changes according to multi-objective evaluation, and a multi-modal scene adaptive optimizer is generated. In actual evaluation, the balance coefficient is 0.5, and the weights of process deviation and KPI covariance are... , The values ​​are 0.6 and 0.4 respectively, with a Gaussian penalty variance of 0.04; the basic weight of the visual modality is 0.513, and the basic weights of the text and sensor modalities are derived in the same logical manner to form a closed loop; Knowledge-enhanced features are generated through knowledge cross-attention and the multimodal scene adaptive optimizer. In the actual evaluation, the intramodal variance weighting layer calculated the variances of visual, text, and sensor features to be 0.85, 0.42, and 0.63, respectively, with the base weights for visual, text, and sensor features being 0.45, 0.22, and 0.33, respectively; the association weight was 0.7, and the priority configuration of process quality targets was temperature stability 0.8, spatter 0.6, and weld strength 0.7, mapped to scene weights [0.35, 0.33, 0.32]; the scene-based multimodal weights were [0.18, 0.055, 0.10]. The knowledge enhancement features are evaluated using probabilistic decision-making to obtain the optimal strategy. A reward mechanism that correlates time-varying risk and return is constructed based on risk, gradient stability, and long-term return constraints. The optimal strategy is then optimized based on feedback from the reward mechanism.

[0019] In the actual evaluation, the conditional probability combined with the time-varying risk-return assessment λ=0.3; the learnable parameter scale of the strategy generation unit is 10M, the time discount factor is 0.9, and the number of candidate strategies is set to 10; after optimization of the optimal strategy, the spatter defect rate decreased from 12.7% to 3.2%, the weld strength reached 92MPa, the standard deviation decreased by 58% compared with the previous standard deviation, the temperature standard deviation was reduced to 3.8℃, a decrease of 52.5%, and it quickly adapted to the welding of new high-strength steel, achieving a double breakthrough in quality and scenario generalization ability.

[0020] In this embodiment, the method for feature extraction from the multimodal data based on a serial strategy includes: The low-level features are extracted using ResNet-50. The low-level features are then fed into EfficientNet to extract high-level features for semantic judgment. Regularization terms are added to output visual modality features. Key region features are enhanced through cross-layer fusion and dynamic scale attention, as shown in the formula: , in, As a low-level feature, Characterized by high-level features To dynamically adjust the cross-layer weights at the i-th scale according to the production stage; This represents the contribution of adjusting the i-th scale; D is the defect rate sensitivity coefficient; D is the real-time defect rate. t represents the remaining welding time; t represents the welding time already completed. The spatial gradient of welding parameters along the welding torch movement direction z; The reference resistance for process stability; The current influence coefficient; For real-time welding current; These are the gradient weights for the process parameters; This represents the ideal feature vector in the process knowledge base. Given the actual visual feature vectors, dynamic scale attention is calculated using a self-attention mechanism: , Where Q, K, and V are the query matrix, key matrix, and value matrix of the feature map, respectively. It is a normalized parameter. Key region features after attention mechanism enhancement; This is a function that normalizes each row of the similarity matrix; This is the scaling factor; Features of V that are randomly discarded. The amplitude coefficient for Dropout; Based on the automotive manufacturing terminology database, the BERT model is fine-tuned to extract semantic embeddings specific to the automotive domain. Basic semantic features are generated by fine-tuning the BERT model based on the automotive manufacturing terminology database. The basic semantic features are used as input and simultaneously passed to TextCNN to extract static features and LSTM to capture time-dependent dynamic features. The weights of TextCNN and LSTM are adjusted through a gating function. The text modal features are output by combining BiLSTM and Transformer with weighted fusion. Combining TextCNN and LSTM to generate temporal-semantic cross features The formula is: , , Where T is a text sequence. For fine-tuning parameters in industrial terminology, It is a BERT model finely tuned using industry terminology; It is a domain-enhanced semantic embedding vector of a text sequence; It is the sigmoid function; The semantic modality features extracted by TextCNN; To obtain temporal semantic modal features after extraction using LSTM; For gated bias. , The coefficients are fusion coefficients; BiLSTM is a bidirectional long short-term memory network; and Transformer is a self-attention mechanism. Text modal features; The sliding window is dynamically divided according to the equipment cycle. Temporal patterns and anomalies are captured by Transformer and Gaussian process regression (GPR), and sensor modal characteristics are output. .

[0021] In this embodiment, the method for cross-modal differential feature enhancement of the modal features includes: Based on production condition parameters and process constraints, spatiotemporal alignment of visual modal features and sensor modal features is achieved through a time-series dynamic anchor matrix and scene adaptive factors. A cross-modal attention mechanism is used to strengthen the correlation of key regions, fusing visual modal features and sensor modal features to output the first cross-modal feature, as shown in the formula: , in, It is a spatiotemporal dynamic anchor point correlation matrix, generated by the coordinate mapping and feature correlation of the camera and sensor; Visual modal features; Sensor modal characteristics; Features for cross-temporal and spatial fusion of vision and sensors; This refers to the scene adaptive adjustment factor; It is a cross-modal topological feature; , For visual feature query matrix and sensor feature key matrix, is the scene adaptive normalization coefficient; T is the time step of the production stage. Visual modal features and text modal features are combined with knowledge-constrained cross-modal similarity, along with temporal modeling and network mapping mechanisms, to generate a second cross-modal feature: , , Where sim is a scalar measure of the correlation between visual features and text semantics. Enhancing semantic embedding vectors for the domain of text sequences; The feature fluctuation adjustment coefficient is dynamically adjusted based on the variance of visual and text features; It is a knowledge graph relation embedding function that encodes process relationships into vectors; It is a mapping function between visual features and textual semantics. It is a cross-modal feature vector of visual and textual data; It is a bidirectional long short-term memory network that captures temporal dependencies. , , It is a multilayer perceptron. To normalize attention weights; Based on the attention mechanism and feature fusion principle, a cross-modal fusion of sensor modal features and text modal features is constructed to output a third cross-modal feature: , , in, For cross-modal features of sensors and text; These are the correlation weight coefficients; Temperature compliance rate; This is the maximum temperature value; This represents the weighting modulation coefficient associated with KPIs; This is the alarm threshold; The variance of temperature; This is the text modality weight matrix; The sensor mode matrix; This is the cross-fusion weight matrix.

[0022] In this embodiment, the three-layer linkage dynamic weight algorithm is adopted, including: The first layer is the intramodal variance weight layer, which obtains the degree of fluctuation of the feature vector; the second layer is the cross-modal association weight layer, which updates the visual modal features; the third layer is the scene adaptation weight layer, which dynamically adjusts the modal weights. By calculating the variance and covariance of each modal feature, the degree of feature fluctuation is quantified, and the modal basis weights are determined. The formula is as follows: , , in, This refers to the basic weights of the i-th mode; Deviation between process parameters and standard values; Let i be the fundamental feature vector of the i-th mode; This represents the cross-modal characteristics of the i-th mode with other modes; KPI represents production outcome indicators. Indicates the weight of process indicators; Indicates the weight of production indicators; Textual semantic features; Indicates the adjustment factor across modes; The term represents a Gaussian penalty; the i-th modality includes: visual modality v, text modality t, and sensor modality s, and the cross-modal features include a first cross-modal feature, a second cross-modal feature, and a third cross-modal feature; Cross-modal association employs intra-modal self-attention to mine modal details, establishes inter-modal dependencies through cross-modal cross-attention, and calculates dynamic association weights through a query key-value mechanism. Based on the association weights, features are updated, multi-modal features are weighted and fused, and modal features are reconstructed by combining dynamic normalization and activation functions. The association weights between the i-th mode and other modes: , Feature update for the i-th mode: , in, This represents the association weight between the i-th mode and the j-th mode; , This represents mapping features to a projection matrix of the same dimension; Represents the L2 norm; This represents summation over all other modes; This represents the updated i-th modal feature; Represents the regularization coefficient; Based on a multi-objective optimization framework, the degree of deviation is quantified by weighted error, and weight smoothing penalty, parameter and cross-modal feature regularization are introduced to perform scene-adaptive weight optimization: , , AIM stands for Multimodal Scenario Adaptive Optimizer; This represents the predicted process deviation associated with the i-th mode; Indicates the target value for process deviation; Represents the scene-adaptive weights for the i-th mode; Regularization representing cross-modal features; The reference value representing the process deviation of the i-th mode; This represents the predicted KPI value associated with the i-th mode; Indicates the KPI target value; Represents the weight regularization coefficient; These are the regularization coefficients for the model parameters; These are the cross-modal characteristic regularization coefficients; These are the learnable parameters of the model.

[0023] In this embodiment, the method for generating knowledge-enhanced features through knowledge cross-attention and a multi-modal scene adaptive optimizer includes: By fusing scene weights with updated modal features, combining intra-modal self-attention and cross-modal cross-attention, and introducing a knowledge graph (KG) specific to automotive manufacturing, process-related knowledge is encoded into knowledge embedding vectors to generate knowledge-enhanced features. The formula is as follows: , , , The formula for the knowledge cross-attention component is as follows: , in, Represents the modal features after scene enhancement; Represents a learnable matrix; Represents a Gaussian error linear unit; Represents the gate function; This is a global parameter representing the contribution weight of the knowledge module; This represents the minimum value; KG represents the embedding vector of the knowledge graph; This represents the normalization term of the cross product; , This is the knowledge projection matrix.

[0024] In this embodiment, the method for obtaining the optimal strategy through probabilistic decision evaluation of the knowledge-enhanced features includes: Based on the The scale is used to generate the k-th production strategy, and the formula is: , in, This represents the k-th production strategy; Indicates the first Layer-scale weights; For the first The weight matrix of the scale and the k-th strategy; Indicates the first Scale, bias of the k-th strategy; Indicates the first Variance of scale; This indicates knowledge-enhancing features; Based on knowledge graphs and augmented multimodal vectors, conditional probabilities are obtained using the following formula: , , , in, Representation strategy The conditional probability; F is the multimodal enhancement feature; The knowledge graph embedding vector for the k-th strategy; Let the target mean vector be the k-th strategy; Decisions are obtained through a time-decay Gaussian weighted average and a risk-squared weighted average mechanism, as shown in the formula: , in, This represents the optimal production strategy; express The conditional probability of the strategy; T represents the length of the time period; The variance representing time decay; Indicates the number of risk types; Representation strategy The degree of violation that leads to the first type of risk.

[0025] In this embodiment, the method for constructing a reward mechanism that links time-varying risk and return includes: Collaborative modeling of returns and risks, implementing a dynamic reward function: , in, The time-varying risk coefficient; Indicates the level of risk at time t; This represents the change in production indicators at time t; Develop risk-sensitive rewards: , in, This represents the maximum historical risk before time t; Control factors indicating the severity of punishment; The gradient represents the policy. Strategy optimization goal: , in, The learnable parameters of the policy generator; Indicates the time discount factor; Let represent the learning reward at step t; M represents the number of candidate policies.

[0026] Secondly, a policy generation system based on a multimodal large model includes: Multimodal data acquisition module: used to collect multi-source data in the intelligent manufacturing process of automobiles. The multi-source data includes visual data acquired by industrial vision equipment, text data of process documents and production logs, and real-time operation data collected by sensors of production equipment. A cross-modal feature extraction module is used to extract differentiated features from the multi-source data based on a concatenated strategy to obtain single-modal features, which include visual modal features, text modal features, and sensor modal features. The visual modal features and sensor modal features are fused across modally to generate a first cross-modal feature, the visual modal features and text modal features are fused across modally to generate a second cross-modal feature, and the sensor modal features and text modal features are fused across modally to generate a third cross-modal feature. Weight fusion module: used to obtain calibration features from the first cross-modal feature, the second cross-modal feature, the third cross-modal feature and the single-modal feature; based on the calibration features, dynamically adjust the modal weights to adapt to scene changes according to multi-objective evaluation, and generate a multi-modal scene adaptive optimizer; Production strategy generation module: used to generate knowledge-enhanced features through knowledge cross-attention and the multi-modal scene adaptive optimizer; and to obtain the optimal production strategy by evaluating the knowledge-enhanced features through probabilistic decision-making. Closed-loop feedback optimization module: This module is used to construct a reward mechanism that correlates time-varying risk and return based on risk, gradient stability, and long-term return constraints. It then performs feedback optimization on the optimal production strategy according to the reward mechanism and iteratively updates the production strategy.

[0027] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A strategy generation method based on a multimodal large model, characterized in that, Includes the following steps: Collect multi-source data from industrial vision, process text, and equipment sensors; Based on a sequential strategy, differential feature extraction is performed on the multi-source data to obtain single-modal features; the single-modal features include visual modal features, text modal features, and sensor modal features; The visual modal features and sensor modal features are used to establish a spatiotemporal mapping through a dynamic anchor matrix to generate the first cross-modal feature. The visual modal features and text modal features are used to generate the second cross-modal feature through cross-modal similarity with knowledge constraints, combined with temporal modeling and network mapping mechanisms. The sensor modal features and text modal features are used to generate the third cross-modal feature based on attention mechanisms and feature fusion principles. Calibration features are obtained based on the first cross-modal feature, the second cross-modal feature, the third cross-modal feature, and the single-modal feature; Based on the calibration features, the modal weights are dynamically adjusted to adapt to scene changes according to the multi-objective evaluation, and a multi-modal scene adaptive optimizer is generated. Knowledge-enhanced features are generated through knowledge cross-attention and the multimodal scene adaptive optimizer. The knowledge enhancement features are evaluated using probabilistic decision-making to obtain the optimal strategy. A reward mechanism that correlates time-varying risk and return is constructed based on risk, gradient stability, and long-term return constraints. The optimal strategy is then optimized based on feedback from the reward mechanism.

2. The strategy generation method based on a multimodal large model according to claim 1, characterized in that, A method for feature extraction from the multi-source data based on a serial strategy includes: The low-level features are extracted using ResNet-50 and then fed into EfficientNet as input to extract high-level features for semantic judgment. Regularization terms are added to output visual modality features. Key region features are enhanced through cross-layer fusion and dynamic scale attention, as shown in the formula: ， in, As a low-level feature, Characterized by high-level features To dynamically adjust the cross-layer weights at the i-th scale according to the production stage; This represents the contribution of adjusting the i-th scale; D is the defect rate sensitivity coefficient; D is the real-time defect rate. t represents the remaining welding time; t represents the welding time already completed. The spatial gradient of welding parameters along the welding torch movement direction z; The reference resistance for process stability; The current influence coefficient; For real-time welding current; These are the gradient weights for the process parameters; This represents the ideal feature vector in the process knowledge base. Given the actual visual feature vectors, dynamic scale attention is calculated using a self-attention mechanism: ， Where Q, K, and V are the query matrix, key matrix, and value matrix of the feature map, respectively. It is a normalized parameter. Key region features after attention mechanism enhancement; This is a function that normalizes each row of the similarity matrix; This is the scaling factor; Features of V that are randomly discarded. The amplitude coefficient for Dropout; Based on the automotive manufacturing terminology database, the BERT model is fine-tuned to extract semantic embeddings specific to the automotive domain. Basic semantic features are generated by fine-tuning the BERT model based on the automotive manufacturing terminology database. The basic semantic features are used as input and simultaneously passed to TextCNN to extract static features and LSTM to capture time-dependent dynamic features. The weights of TextCNN and LSTM are adjusted through a gating function. The text modal features are output by combining BiLSTM and Transformer with weighted fusion. Combining TextCNN and LSTM to generate temporal-semantic cross features The formula is: ， ， Where T is a text sequence. For fine-tuning parameters in industrial terminology, It is a BERT model finely tuned using industry terminology; It is a domain-enhanced semantic embedding vector of a text sequence; It is the sigmoid function; The semantic modality features extracted by TextCNN; To obtain temporal semantic modal features after extraction using LSTM; For gated bias. , The coefficients are fusion coefficients; BiLSTM is a bidirectional long short-term memory network; and Transformer is a self-attention mechanism. Text modal features; The sliding window is dynamically divided according to the equipment cycle. The timing patterns and anomalies are captured by Transformer and Gaussian process regression (GPR), and the sensor modal characteristics are output.

3. The strategy generation method based on a multimodal large model according to claim 1, characterized in that, A method for cross-modal differential feature enhancement of the modal features includes: Based on production condition parameters and process constraints, spatiotemporal alignment of visual modal features and sensor modal features is achieved through a time-series dynamic anchor matrix and scene adaptive factors. A cross-modal attention mechanism is used to strengthen the correlation of key regions, fusing visual modal features and sensor modal features to output the first cross-modal feature, as shown in the formula: ， in, It is a spatiotemporal dynamic anchor point correlation matrix, generated by the coordinate mapping and feature correlation of the camera and sensor; Visual modal features; Sensor modal characteristics; Features for cross-temporal and spatial fusion of vision and sensors; This refers to the scene adaptive adjustment factor; It is a cross-modal topological feature; , For visual feature query matrix and sensor feature key matrix, is the scene adaptive normalization coefficient; T is the time step of the production stage. Visual modal features and text modal features are combined with knowledge-constrained cross-modal similarity, along with temporal modeling and network mapping mechanisms, to generate a second cross-modal feature: ， ， Where sim is a scalar measure of the correlation between visual features and text semantics. Enhancing semantic embedding vectors for the domain of text sequences; The feature fluctuation adjustment coefficient is dynamically adjusted based on the variance of visual and text features; It is a knowledge graph relation embedding function that encodes process relationships into vectors; It is a mapping function between visual features and textual semantics. It is a cross-modal feature vector of visual and textual data; It is a bidirectional long short-term memory network that captures temporal dependencies. , , It is a multilayer perceptron. To normalize attention weights; Based on the attention mechanism and feature fusion principle, a cross-modal fusion of sensor modal features and text modal features is constructed to output a third cross-modal feature: ， ， in, For cross-modal features of sensors and text; These are the correlation weight coefficients; Temperature compliance rate; This is the maximum temperature value; This represents the weight modulation coefficient associated with KPIs; This is the alarm threshold; The variance of temperature; This is the text modality weight matrix; The sensor mode matrix; This is the cross-fusion weight matrix.

4. The strategy generation method based on a multimodal large model according to claim 1, characterized in that, The three-layer linkage dynamic weight algorithm includes: The first layer is the intramodal variance weight layer, which obtains the degree of fluctuation of the feature vector; the second layer is the cross-modal association weight layer, which updates the visual modal features; the third layer is the scene adaptation weight layer, which dynamically adjusts the modal weights. By calculating the variance and covariance of each modal feature, the degree of feature fluctuation is quantified, and the modal basis weights are determined. The formula is as follows: ， ， in, This refers to the basic weights of the i-th mode; Deviation between process parameters and standard values; Let i be the fundamental feature vector of the i-th mode; This represents the cross-modal characteristics of the i-th mode with other modes; KPI represents production outcome indicators. Indicates the weight of process indicators; Indicates the weight of production indicators; Textual semantic features; Indicates the adjustment factor across modes; The term represents a Gaussian penalty; the i-th modality includes: visual modality v, text modality t, and sensor modality s, and the cross-modal features include a first cross-modal feature, a second cross-modal feature, and a third cross-modal feature; Cross-modal association employs intra-modal self-attention to mine modal details, establishes inter-modal dependencies through cross-modal cross-attention, and calculates dynamic association weights through a query key-value mechanism. Based on the association weights, features are updated, multi-modal features are weighted and fused, and modal features are reconstructed by combining dynamic normalization and activation functions. The association weights between the i-th mode and other modes: ， Feature update for the i-th mode: ， in, This represents the association weight between the i-th mode and the j-th mode; , This represents mapping features to a projection matrix of the same dimension; Represents the L2 norm; This represents summation over all other modes; This represents the updated i-th modal feature; Represents the regularization coefficient; It is a numerical stability factor; Based on a multi-objective optimization framework, the degree of deviation is quantified by weighted error, and weight smoothing penalty, parameter and cross-modal feature regularization are introduced to perform scene-adaptive weight optimization: ， ， AIM stands for Multimodal Scenario Adaptive Optimizer; This represents the predicted process deviation associated with the i-th mode; Indicates the target value for process deviation; Represents the scene-adaptive weights for the i-th mode; Regularization representing cross-modal features; The reference value representing the process deviation of the i-th mode; This represents the predicted KPI value associated with the i-th mode; Indicates the KPI target value; Represents the weight regularization coefficient; These are the regularization coefficients for the model parameters; These are the cross-modal characteristic regularization coefficients; These are the learnable parameters of the model.

5. The strategy generation method based on a multimodal large model according to claim 1, characterized in that, Methods for generating knowledge-enhanced features through knowledge cross-attention and multimodal scene adaptive optimizers include: By fusing scene weights with updated modal features, combining intra-modal self-attention and cross-modal cross-attention, and introducing a knowledge graph (KG) specific to automotive manufacturing, process-related knowledge is encoded into knowledge embedding vectors to generate knowledge-enhanced features. The formula is as follows: ， ， ， The formula for the knowledge cross-attention component is as follows: ， in, Represents the modal features after scene enhancement; Represents a learnable matrix; Represents a Gaussian error linear unit; Represents the gate function; This is a global parameter representing the contribution weight of the knowledge module; is the minimum value, the numerical stability factor; KG represents the embedding vector of the knowledge graph; Represents the normalization term of the cross product; , This is the knowledge projection matrix.

6. The strategy generation method based on a multimodal large model according to claim 1, characterized in that, The method for obtaining the optimal strategy through probabilistic decision evaluation of the knowledge-enhanced features includes: Based on the The scale is used to generate the k-th production strategy, and the formula is: ， in, This represents the k-th production strategy; Indicates the first Layer-scale weights; For the first The weight matrix of the scale and the k-th strategy; Indicates the first Scale, bias of the k-th strategy; Indicates the first Variance of scale; This indicates knowledge-enhancing features; Based on knowledge graphs and augmented multimodal vectors, conditional probabilities are obtained using the following formula: ， ， ， in, Representation strategy The conditional probability; F is the multimodal enhancement feature; The knowledge graph embedding vector for the k-th strategy; Let the target mean vector be the k-th strategy; Decision-making is achieved through a time-decay Gaussian weighted average and a risk-squared weighted average mechanism, as shown in the formula: ， in, This represents the optimal production strategy; express The conditional probability of the strategy; T represents the length of the time period; The variance representing time decay; Indicates the number of risk types; Representation strategy The degree of violation that leads to the first type of risk.

7. The strategy generation method based on a multimodal large model according to claim 1, characterized in that, The method for constructing a reward mechanism that links time-varying risk and return includes: Collaborative modeling of returns and risks, implementing a dynamic reward function: ， in, The time-varying risk coefficient; Indicates the level of risk at time t; This represents the change in production indicators at time t; Develop risk-sensitive rewards: ， in, This represents the maximum historical risk before time t; Control factors indicating the severity of punishment; The gradient represents the policy. Strategy optimization goal: ， in, Represents the learnable parameters of the policy generator; Indicates the time discount factor; Let represent the learning reward at step t; M represents the number of candidate policies.

8. A policy generation system based on a multimodal large model, used to execute the method described in any one of claims 1-7, characterized in that, include: Multimodal data acquisition module: used to collect multi-source data in the intelligent manufacturing process of automobiles. The multi-source data includes visual data acquired by industrial vision equipment, text data of process documents and production logs, and real-time operation data collected by sensors of production equipment. A cross-modal feature extraction module is used to extract differentiated features from the multi-source data based on a concatenated strategy to obtain single-modal features, which include visual modal features, text modal features, and sensor modal features. The visual modal features and sensor modal features are fused across modally to generate a first cross-modal feature, the visual modal features and text modal features are fused across modally to generate a second cross-modal feature, and the sensor modal features and text modal features are fused across modally to generate a third cross-modal feature. Weight fusion module: used to obtain calibration features from the first cross-modal feature, the second cross-modal feature, the third cross-modal feature, and the single-modal feature; Based on the calibration features, the modal weights are dynamically adjusted to adapt to scene changes according to the multi-objective evaluation, and a multi-modal scene adaptive optimizer is generated. Production strategy generation module: used to generate knowledge-enhanced features through knowledge cross-attention and the multi-modal scene adaptive optimizer; The optimal production strategy is obtained by evaluating the knowledge-enhanced features using probabilistic decision-making. Closed-loop feedback optimization module: This module is used to construct a reward mechanism that correlates time-varying risk and return based on risk, gradient stability, and long-term return constraints. It then performs feedback optimization on the optimal production strategy according to the reward mechanism and iteratively updates the production strategy.

Images