A strategy generation method and system based on a multi-modal large model

Abstract

The application discloses a strategy generation method and system based on a multi-modal large model, which is applied to the whole process of automobile manufacturing. The method collects production line visual images, process texts and sensor data, realizes cross-modal feature adaptive fusion through a multi-modal large model, adjusts the modal weight according to the production stage, combines the manufacturing process knowledge to generate an initial strategy, iteratively optimizes through feedback, and outputs a target strategy adapted to production. The system integrates data collection, cross-modal fusion, strategy generation and optimization modules, relies on dynamic learning of the large model to realize scene-based adjustment of the weight, and accurately maps the production state. The application breaks through the fixed weight mode, uses deep correlation reasoning of multi-source data to improve the adaptability and effectiveness of the strategy, supports intelligent decision-making, and empowers automobile intelligent manufacturing.

Description

Technical Field

[0001] This invention relates to the field of intelligent automotive manufacturing, specifically to a strategy generation method and system based on a multimodal large model. Background Technology

[0002] Intelligent manufacturing in the automotive industry involves multiple processes and requires the integration of multimodal information to address production decision-making needs under complex operating conditions. However, traditional methods have systemic limitations, revealing shortcomings in adaptability across the entire chain from data perception to strategy output.

[0003] Single-modal analysis fragments the relationship between visual appearance, physical mechanism, and rule constraints. In the welding process, relying solely on visual inspection of surface defects fails to penetrate hidden anomalies. Fixed-weight fusion strategies ignore differences in process stages, diluting core contradictions under uniform weights and significantly reducing the strategy's universality. More importantly, the lack of a process knowledge graph and the open-loop decision-making mechanism make it difficult for the strategy to evolve with equipment aging and environmental fluctuations, resulting in adaptability and iteration efficiency lagging far behind the dynamic needs of the production line.

[0004] These shortcomings lead to a continuous accumulation of deviations between production strategies and actual working conditions, from missed defect detection to delayed decision-making, gradually restricting the accuracy and agility of intelligent manufacturing. Constructing a dynamically adaptable, knowledge-driven multimodal strategy generation method has become a core path to overcome the efficiency bottleneck in automobile manufacturing and an inevitable choice to solve the decision-making dilemma in multi-process scenarios.

[0005] Therefore, there is an urgent need to construct a new production strategy generation method based on a multimodal large model, which can improve the adaptability and accuracy of production strategies by combining visual, text, and sensor data to optimize production processes, provide early warnings for equipment maintenance, and trace the source of quality defects. Summary of the Invention

[0006] The purpose of this invention is to provide a strategy generation method based on a multimodal large model.

[0007] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0008] This invention includes the following steps:

[0009] Collect multi-source data from industrial vision, process text, and equipment sensors, including visual data, text data, and sensor data;

[0010] Based on a serial strategy, differential feature extraction is performed on the multi-source data to obtain single-modal features. The visual data is used to obtain visual modal features through cross-layer fusion and dynamic scale attention mechanism. The text data is used to enhance semantics and temporal representation through domain terminology library to obtain text modal features. The sensor data is used to obtain sensor modal features through Transformer and Gaussian process regression algorithms. The single-modal features include visual modal features, text modal features and sensor modal features.

[0011] The visual modal features and sensor modal features are used to establish a spatiotemporal mapping through a dynamic anchor matrix to generate the first cross-modal feature. The visual modal features and text modal features are used to generate the second cross-modal feature through cross-modal similarity with knowledge constraints, combined with temporal modeling and network mapping mechanisms. The sensor modal features and text modal features are used to generate the third cross-modal feature based on the attention mechanism and feature fusion principle.

[0012] Based on the first cross-modal features, the second cross-modal features, the third cross-modal features, and the single-modal features, a calibration feature is obtained by fusing a three-layer linkage dynamic weight algorithm and combining intra-modal self-attention and cross-modal cross-attention to learn associated weights. The modal weights are dynamically adjusted according to multi-objective evaluation to adapt to scene changes, thereby generating a multi-modal scene adaptive optimizer.

[0013] Knowledge-enhanced features are generated through knowledge cross-attention and the multimodal scene adaptive optimizer.

[0014] Based on the knowledge enhancement features, a multimodal large model for intelligent vehicle manufacturing is constructed. The multimodal large model for intelligent vehicle manufacturing includes a Transformer encoder, a policy generation unit, a knowledge attention layer, a probability calculation unit, a decision evaluator, and a reward learning mechanism feedback. The reward learning mechanism feedback is obtained through conditional probability combined with time-varying risk-reward assessment.

[0015] Furthermore, the method for feature extraction from the multimodal data based on a concatenated strategy includes:

[0016] The low-level features are extracted using ResNet-50. The low-level features are then fed into EfficientNet to extract high-level features for semantic judgment. Regularization terms are added to output visual modality features.

[0017] Key region features are enhanced through cross-layer fusion and dynamic scale attention, as shown in the formula:

[0018] ,

[0019] in, As a low-level feature, Characterized by high-level features To dynamically adjust the cross-layer weights at the i-th scale according to the production stage; This represents the contribution of adjusting the i-th scale; D is the defect rate sensitivity coefficient; D is the real-time defect rate. t represents the remaining welding time; t represents the welding time already completed. The spatial gradient of welding parameters along the welding torch movement direction z; The reference resistance for process stability; The current influence coefficient; For real-time welding current; These are the gradient weights for the process parameters; This represents the ideal feature vector in the process knowledge base. Given the actual visual feature vectors, dynamic scale attention is calculated using a self-attention mechanism:

[0020] ,

[0021] Where Q, K, and V are the query matrix, key matrix, and value matrix of the feature map, respectively. It is a normalized parameter. Key region features after attention mechanism enhancement; This is a function that normalizes each row of the similarity matrix; This is the scaling factor; Features of V that are randomly discarded. The amplitude coefficient for Dropout;

[0022] Based on the automotive manufacturing terminology database, the BERT model is fine-tuned to extract semantic embeddings specific to the automotive domain. Basic semantic features are generated by fine-tuning the BERT model based on the automotive manufacturing terminology database. The basic semantic features are used as input and simultaneously passed to TextCNN to extract static features and LSTM to capture time-dependent dynamic features. The weights of TextCNN and LSTM are adjusted through a gating function. The text modal features are output by combining BiLSTM and Transformer with weighted fusion.

[0023] Combining TextCNN and LSTM to generate temporal-semantic cross features The formula is:

[0024] ,

[0025] ,

[0026] Where T is a text sequence. For fine-tuning parameters in industrial terminology, It is a BERT model finely tuned using industry terminology; It is a domain-enhanced semantic embedding vector of a text sequence; It is the sigmoid function; The semantic modality features extracted by TextCNN; To obtain temporal semantic modal features after extraction using LSTM; For gated bias. , The coefficients are fusion coefficients; BiLSTM is a bidirectional long short-term memory network; and Transformer is a self-attention mechanism. Text modal features;

[0027] The sliding window is dynamically divided according to the equipment cycle. The timing patterns and anomalies are captured by Transformer and Gaussian process regression (GPR), and the sensor modal characteristics are output.

[0028] Furthermore, a method for cross-modal differential feature enhancement of the modal features includes:

[0029] Based on production condition parameters and process constraints, spatiotemporal alignment of visual modal features and sensor modal features is achieved through a time-series dynamic anchor matrix and scene adaptive factors. A cross-modal attention mechanism is used to strengthen the correlation of key regions, fusing visual modal features and sensor modal features to output the first cross-modal feature, as shown in the formula:

[0030] ,

[0031] in, It is a spatiotemporal dynamic anchor point correlation matrix, generated by the coordinate mapping and feature correlation of the camera and sensor; Visual modal features; Sensor modal characteristics; Features for cross-temporal and spatial fusion of vision and sensors; This refers to the scene adaptive adjustment factor; It is a cross-modal topological feature; , For visual feature query matrix and sensor feature key matrix, is the scene adaptive normalization coefficient; T is the time step of the production stage.

[0032] Visual modal features and text modal features are combined with knowledge-constrained cross-modal similarity, along with temporal modeling and network mapping mechanisms, to generate a second cross-modal feature:

[0033] ,

[0034] ,

[0035] Where sim is a scalar measure of the correlation between visual features and text semantics. Enhancing semantic embedding vectors for the domain of text sequences; The feature fluctuation adjustment coefficient is dynamically adjusted based on the variance of visual and text features; It is a knowledge graph relation embedding function that encodes process relationships into vectors; It is a mapping function between visual features and textual semantics. It is a cross-modal feature vector of visual and textual data; It is a bidirectional long short-term memory network that captures temporal dependencies. , , It is a multilayer perceptron. To normalize attention weights;

[0036] Based on the attention mechanism and feature fusion principle, a cross-modal fusion of sensor modal features and text modal features is constructed to output a third cross-modal feature:

[0037] ,

[0038] ,

[0039] in, For cross-modal features of sensors and text; These are the correlation weight coefficients; Temperature compliance rate; This is the maximum temperature value; This represents the weighting modulation coefficient associated with KPIs; This is the alarm threshold; The variance of temperature; This is the text modality weight matrix; The sensor mode matrix; This is the cross-fusion weight matrix.

[0040] Furthermore, the three-layer linked dynamic weight algorithm includes:

[0041] The first layer is the intramodal variance weight layer, which obtains the degree of fluctuation of the feature vector; the second layer is the cross-modal association weight layer, which updates the visual modal features; the third layer is the scene adaptation weight layer, which dynamically adjusts the modal weights.

[0042] By calculating the variance and covariance of each modal feature, the degree of feature fluctuation is quantified, and the modal basis weights are determined. The formula is as follows:

[0043] ,

[0044] ,

[0045] in, This refers to the basic weights of the i-th mode; Deviation between process parameters and standard values; Let i be the fundamental feature vector of the i-th mode; This represents the cross-modal characteristics of the i-th mode with other modes; KPI represents production outcome indicators. Indicates the weight of process indicators; Indicates the weight of production indicators; Indicates the adjustment factor across modes; The term represents a Gaussian penalty; the i-th modality includes: visual modality v, text modality t, and sensor modality s, and the cross-modal features include a first cross-modal feature, a second cross-modal feature, and a third cross-modal feature;

[0046] Cross-modal association employs intra-modal self-attention to mine modal details, establishes inter-modal dependencies through cross-modal cross-attention, and calculates dynamic association weights through a query key-value mechanism. Based on the association weights, features are updated, multi-modal features are weighted and fused, and modal features are reconstructed by combining dynamic normalization and activation functions.

[0047] The association weights between the i-th mode and other modes:

[0048] ,

[0049] Feature update for the i-th mode:

[0050] ,

[0051] in, This represents the association weight between the i-th mode and the j-th mode; , This represents mapping features to a projection matrix of the same dimension; Represents the L2 norm; This represents summation over all other modes; This represents the updated i-th modal feature; Represents the regularization coefficient;

[0052] Based on a multi-objective optimization framework, the degree of deviation is quantified by weighted error, and weight smoothing penalty, parameter and cross-modal feature regularization are introduced to perform scene-adaptive weight optimization:

[0053] ,

[0054] ,

[0055] AIM stands for Multimodal Scenario Adaptive Optimizer; This represents the predicted process deviation associated with the i-th mode; Indicates the target value for process deviation; Represents the scene-adaptive weights for the i-th mode; Regularization representing cross-modal features; The reference value representing the process deviation of the i-th mode; This represents the predicted KPI value associated with the i-th mode; Indicates the KPI target value; Represents the weight regularization coefficient; These are the regularization coefficients for the model parameters; These are the cross-modal characteristic regularization coefficients; These are the learnable parameters of the model.

[0056] Furthermore, methods for generating knowledge-enhanced features through knowledge cross-attention and multi-modal scene adaptive optimizers include:

[0057] By fusing scene weights with updated modal features, combining intra-modal self-attention and cross-modal cross-attention, and introducing a knowledge graph (KG) specific to automotive manufacturing, process-related knowledge is encoded into knowledge embedding vectors to generate knowledge-enhanced features. The formula is as follows:

[0058] ,

[0059] The formula for scene enhancement is:

[0060] ,

[0061] The formula for modal correlation is:

[0062] ,

[0063] The formula for the cross-attention component is:

[0064] ,

[0065] in, Represents the modal features after scene enhancement; Represents a learnable matrix; Represents a Gaussian error linear unit; Represents the gate function; This is a global parameter representing the contribution weight of the knowledge module; This represents the minimum value; KG represents the embedding vector of the knowledge graph; This represents the normalization term of the cross product; , This is the knowledge projection matrix.

[0066] Furthermore, the method for obtaining the optimal strategy through probabilistic decision evaluation of the knowledge-enhanced features includes:

[0067] Based on the The scale is used to generate the k-th production strategy, and the formula is:

[0068] ,

[0069] in, This represents the k-th production strategy; Indicates the first Layer-scale weights; For the first The weight matrix of the scale and the k-th strategy; Indicates the first Scale, bias of the k-th strategy; Indicates the first Variance of scale; This indicates knowledge-enhancing features;

[0070] Based on knowledge graphs and augmented multimodal vectors, conditional probabilities are obtained using the following formula:

[0071] ,

[0072] ,

[0073] ,

[0074] in, Representation Strategy The conditional probability; F is the multimodal enhancement feature; The knowledge graph embedding vector for the k-th strategy; Let the target mean vector be the k-th strategy;

[0075] Decision-making is achieved through a time-decay Gaussian weighted average and a risk-squared weighted average mechanism, as shown in the formula:

[0076] ,

[0077] in, This represents the optimal production strategy; express The conditional probability of the strategy; T represents the length of the time period; The variance representing time decay; Indicates the number of risk types; Representation Strategy The degree of violation that leads to the first type of risk.

[0078] Furthermore, the method for constructing a reward mechanism that links time-varying risk and return includes:

[0079] Collaborative modeling of returns and risks, implementing a dynamic reward function:

[0080] ,

[0081] in, The time-varying risk coefficient; Indicates the level of risk at time t; This represents the change in production indicators at time t;

[0082] Develop risk-sensitive rewards:

[0083] ,

[0084] in, This represents the maximum historical risk before time t; Control factors indicating the severity of punishment; The gradient represents the policy.

[0085] Strategy optimization goal:

[0086] ,

[0087] in, The learnable parameters of the policy generator; Indicates the time discount factor; Let represent the learning reward at step t; M represents the number of candidate policies.

[0088] Secondly, a policy generation system based on a multimodal large model includes:

[0089] Multimodal data acquisition module: used to collect multi-source data in the intelligent manufacturing process of automobiles. The multi-source data includes visual data acquired by industrial vision equipment, text data of process documents and production logs, and real-time operation data collected by sensors of production equipment.

[0090] A cross-modal feature extraction module is used to extract differentiated features from the multi-source data based on a concatenated strategy to obtain single-modal features, which include visual modal features, text modal features, and sensor modal features. The visual modal features and sensor modal features are fused across modally to generate a first cross-modal feature, the visual modal features and text modal features are fused across modally to generate a second cross-modal feature, and the sensor modal features and text modal features are fused across modally to generate a third cross-modal feature.

[0091] Weight fusion module: used to obtain calibration features from the first cross-modal feature, the second cross-modal feature, the third cross-modal feature and the single-modal feature; based on the calibration features, dynamically adjust the modal weights to adapt to scene changes according to multi-objective evaluation, and generate a multi-modal scene adaptive optimizer;

[0092] Production strategy generation module: used to generate knowledge-enhanced features through knowledge cross-attention and the multi-modal scene adaptive optimizer; and to obtain the optimal production strategy by evaluating the knowledge-enhanced features through probabilistic decision-making.

[0093] Closed-loop feedback optimization module: This module is used to construct a reward mechanism that correlates time-varying risk and return based on risk, gradient stability, and long-term return constraints. It then performs feedback optimization on the optimal production strategy according to the reward mechanism and iteratively updates the production strategy.

[0094] The beneficial effects of this invention are:

[0095] This invention is a strategy generation method and system based on a multimodal large model. Compared with the prior art, this invention has the following technical advantages:

[0096] This invention improves the accuracy of strategy generation for large multimodal models through preprocessing, multimodal data acquisition and cross-modal feature extraction, weight fusion, production strategy generation, and closed-loop feedback optimization. This significantly saves resources and increases work efficiency. By integrating an automotive manufacturing knowledge base, the invention uses multimodal features to perceive the current situation and professional knowledge base constraints to ensure the strategy meets actual industrial production needs. It constructs scenario-adaptive cross-modal collaborative features and, through three-layer weight fusion, allows input features to naturally carry automotive manufacturing process constraints and dynamic production scenario requirements. The generated strategy is geared towards full-process production collaboration, and through scenario-adaptive dynamic adjustment, the strategy is optimized in a closed loop according to the production line status, better meeting the continuous, collaborative, and dynamic production line requirements of intelligent automotive manufacturing. It possesses a certain degree of universality. Attached Figure Description

[0097] Figure 1 This is a flowchart illustrating the steps of a strategy generation method based on a multimodal large model according to the present invention. Detailed Implementation

[0098] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0099] The present invention provides a strategy generation method and system based on a multimodal large model, comprising the following steps:

[0100] like Figure 1 As shown, this embodiment includes the following steps:

[0101] Collect multi-source data from industrial vision, process text, and equipment sensors, including visual data, text data, and sensor data;

[0102] The system collects industrial vision weld seam images, welding process documents, and equipment sensor time-series data to form multi-source data. The vision data consists of 5000 frames of images with a resolution of 1920×1080 and a frame rate of 25fps. The text data consists of 1000 XML format process documents, each containing 10 instructions. The sensor data consists of 100Hz time-series signals of welding current and temperature, recording 500 time-series data points for 1000 five-second welding cycles.

[0103] Based on a sequential strategy, differential feature extraction is performed on the multi-source data to obtain single-modal features; the single-modal features include visual modal features, text modal features, and sensor modal features;

[0104] A concatenated strategy was employed to extract single-modal features. Visual data was processed using a concatenated ResNet-50 and EfficientNet structure, with a weight of 0.6 for the bottom-level features and 0.4 for the high-level semantic features. Cross-layer fusion was then performed to generate 256-dimensional visual modal features, with a statistical variance of 0.85. Text data was processed using a terminology database for automotive manufacturing welding, with a fine-tuned BERT model and embedded with specific semantics. The statistical variance of text modal features was 0.42. Sensor data was divided into sliding windows with 200 points and a step size of 100. After processing with 4 layers of Transformer and Gaussian process regression, 256-dimensional sensor modal features were generated, with a statistical variance of 0.63.

[0105] The visual modal features and sensor modal features are used to establish a spatiotemporal mapping through a dynamic anchor matrix to generate the first cross-modal feature. The visual modal features and text modal features are used to generate the second cross-modal feature through cross-modal similarity with knowledge constraints, combined with temporal modeling and network mapping mechanisms. The sensor modal features and text modal features are used to generate the third cross-modal feature based on the attention mechanism and feature fusion principle.

[0106] Visual and sensor modal features are spatiotemporally mapped using a dynamic anchor matrix with a value of 0.7, and then processed by a 3-layer MLP to generate a 512-dimensional first cross-modal feature with a statistical variance of 0.72. Visual and text modal features are generated based on the 0.85 cosine similarity between visual features of weld depth anomalies and textual semantics of welding parameters exceeding limits, combined with LSTM temporal modeling and MLP to generate a 512-dimensional second cross-modal feature. Sensor and text modal features are fused through an attention mechanism with an attention coefficient of 0.6 to generate a 512-dimensional third cross-modal feature with a statistical variance of 0.68.

[0107] Calibration features are obtained based on the first cross-modal features, the second cross-modal features, the third cross-modal features, and the single-modal features; based on the calibration features, the modal weights are dynamically adjusted to adapt to scene changes according to multi-objective evaluation, and a multi-modal scene adaptive optimizer is generated.

[0108] In actual evaluation, the balance coefficient is 0.5, and the weights of process deviation and KPI covariance are... , The values ​​are 0.6 and 0.4 respectively, with a Gaussian penalty variance of 0.04; the basic weight of the visual modality is 0.513, and the basic weights of the text and sensor modalities are derived in the same logical manner to form a closed loop;

[0109] Knowledge-enhanced features are generated through knowledge cross-attention and the multimodal scene adaptive optimizer.

[0110] In the actual evaluation, the intramodal variance weighting layer calculated the variances of visual, text, and sensor features to be 0.85, 0.42, and 0.63, respectively, with the base weights for visual, text, and sensor features being 0.45, 0.22, and 0.33, respectively; the association weight was 0.7, and the priority configuration of process quality targets was temperature stability 0.8, spatter 0.6, and weld strength 0.7, mapped to scene weights [0.35, 0.33, 0.32]; the scene-based multimodal weights were [0.18, 0.055, 0.10].

[0111] The knowledge enhancement features are evaluated using probabilistic decision-making to obtain the optimal strategy. A reward mechanism that correlates time-varying risk and return is constructed based on risk, gradient stability, and long-term return constraints. The optimal strategy is then optimized based on feedback from the reward mechanism.

[0112] In the actual evaluation, the conditional probability combined with the time-varying risk-return assessment λ=0.3; the learnable parameter scale of the strategy generation unit is 10M, the time discount factor is 0.9, and the number of candidate strategies is set to 10; after optimization of the optimal strategy, the spatter defect rate decreased from 12.7% to 3.2%, the weld strength reached 92MPa, the standard deviation decreased by 58% compared with the previous standard deviation, the temperature standard deviation was reduced to 3.8℃, a decrease of 52.5%, and it quickly adapted to the welding of new high-strength steel, achieving a double breakthrough in quality and scenario generalization ability.

[0113] In this embodiment, the method for feature extraction from the multimodal data based on a serial strategy includes:

[0114] The low-level features are extracted using ResNet-50. The low-level features are then fed into EfficientNet to extract high-level features for semantic judgment. Regularization terms are added to output visual modality features.

[0115] Key region features are enhanced through cross-layer fusion and dynamic scale attention, as shown in the formula:

[0116] ,

[0117] in, As a low-level feature, Characterized by high-level features To dynamically adjust the cross-layer weights at the i-th scale according to the production stage; This represents the contribution of adjusting the i-th scale; D is the defect rate sensitivity coefficient; D is the real-time defect rate. t represents the remaining welding time; t represents the welding time already completed. The spatial gradient of welding parameters along the welding torch movement direction z; The reference resistance for process stability; The current influence coefficient; For real-time welding current; These are the gradient weights for the process parameters; This represents the ideal feature vector in the process knowledge base. Given the actual visual feature vectors, dynamic scale attention is calculated using a self-attention mechanism:

[0118] ,

[0119] Where Q, K, and V are the query matrix, key matrix, and value matrix of the feature map, respectively. It is a normalized parameter. Key region features after attention mechanism enhancement; This is a function that normalizes each row of the similarity matrix; This is the scaling factor; Features of V that are randomly discarded. The amplitude coefficient for Dropout;

[0120] Based on the automotive manufacturing terminology database, the BERT model is fine-tuned to extract semantic embeddings specific to the automotive domain. Basic semantic features are generated by fine-tuning the BERT model based on the automotive manufacturing terminology database. The basic semantic features are used as input and simultaneously passed to TextCNN to extract static features and LSTM to capture time-dependent dynamic features. The weights of TextCNN and LSTM are adjusted through a gating function. The text modal features are output by combining BiLSTM and Transformer with weighted fusion.

[0121] Combining TextCNN and LSTM to generate temporal-semantic cross features The formula is:

[0122] ,

[0123] ,

[0124] Where T is a text sequence. For fine-tuning parameters in industrial terminology, It is a BERT model finely tuned using industry terminology; It is a domain-enhanced semantic embedding vector of a text sequence; It is the sigmoid function; The semantic modality features extracted by TextCNN; To obtain temporal semantic modal features after extraction using LSTM; For gated bias. , The coefficients are fusion coefficients; BiLSTM is a bidirectional long short-term memory network; and Transformer is a self-attention mechanism. Text modal features;

[0125] The sliding window is dynamically divided according to the equipment cycle. Temporal patterns and anomalies are captured by Transformer and Gaussian process regression (GPR), and sensor modal characteristics are output. .

[0126] In this embodiment, the method for cross-modal differential feature enhancement of the modal features includes:

[0127] Based on production condition parameters and process constraints, spatiotemporal alignment of visual modal features and sensor modal features is achieved through a time-series dynamic anchor matrix and scene adaptive factors. A cross-modal attention mechanism is used to strengthen the correlation of key regions, fusing visual modal features and sensor modal features to output the first cross-modal feature, as shown in the formula:

[0128] ,

[0129] in, It is a spatiotemporal dynamic anchor point correlation matrix, generated by the coordinate mapping and feature correlation of the camera and sensor; Visual modal features; Sensor modal characteristics; Features for cross-temporal and spatial fusion of vision and sensors; This refers to the scene adaptive adjustment factor; It is a cross-modal topological feature; , For visual feature query matrix and sensor feature key matrix, is the scene adaptive normalization coefficient; T is the time step of the production stage.

[0130] Visual modal features and text modal features are combined with knowledge-constrained cross-modal similarity, along with temporal modeling and network mapping mechanisms, to generate a second cross-modal feature:

[0131] ,

[0132] ,

[0133] Where sim is a scalar measure of the correlation between visual features and text semantics. Enhancing semantic embedding vectors for the domain of text sequences; The feature fluctuation adjustment coefficient is dynamically adjusted based on the variance of visual and text features; It is a knowledge graph relation embedding function that encodes process relationships into vectors; It is a mapping function between visual features and textual semantics. It is a cross-modal feature vector of visual and textual data; It is a bidirectional long short-term memory network that captures temporal dependencies. , , It is a multilayer perceptron. To normalize attention weights;

[0134] Based on the attention mechanism and feature fusion principle, a cross-modal fusion of sensor modal features and text modal features is constructed to output a third cross-modal feature:

[0135] ,

[0136] ,

[0137] in, For cross-modal features of sensors and text; These are the correlation weight coefficients; Temperature compliance rate; This is the maximum temperature value; This represents the weighting modulation coefficient associated with KPIs; This is the alarm threshold; The variance of temperature; This is the text modality weight matrix; The sensor mode matrix; This is the cross-fusion weight matrix.

[0138] In this embodiment, the three-layer linkage dynamic weight algorithm is adopted, including:

[0139] The first layer is the intramodal variance weight layer, which obtains the degree of fluctuation of the feature vector; the second layer is the cross-modal association weight layer, which updates the visual modal features; the third layer is the scene adaptation weight layer, which dynamically adjusts the modal weights.

[0140] By calculating the variance and covariance of each modal feature, the degree of feature fluctuation is quantified, and the modal basis weights are determined. The formula is as follows:

[0141] ,

[0142] ,

[0143] in, This refers to the basic weights of the i-th mode; Deviation between process parameters and standard values; Let i be the fundamental feature vector of the i-th mode; This represents the cross-modal characteristics of the i-th mode with other modes; KPI represents production outcome indicators. Indicates the weight of process indicators; Indicates the weight of production indicators; Indicates the adjustment factor across modes; The term represents a Gaussian penalty; the i-th modality includes: visual modality v, text modality t, and sensor modality s, and the cross-modal features include a first cross-modal feature, a second cross-modal feature, and a third cross-modal feature;

[0144] Cross-modal association employs intra-modal self-attention to mine modal details, establishes inter-modal dependencies through cross-modal cross-attention, and calculates dynamic association weights through a query key-value mechanism. Based on the association weights, features are updated, multi-modal features are weighted and fused, and modal features are reconstructed by combining dynamic normalization and activation functions.

[0145] The association weights between the i-th mode and other modes:

[0146] ,

[0147] Feature update for the i-th mode:

[0148] ,

[0149] in, This represents the association weight between the i-th mode and the j-th mode; , This represents mapping features to a projection matrix of the same dimension; Represents the L2 norm; This represents summation over all other modes; This represents the updated i-th modal feature; Represents the regularization coefficient;

[0150] Based on a multi-objective optimization framework, the degree of deviation is quantified by weighted error, and weight smoothing penalty, parameter and cross-modal feature regularization are introduced to perform scene-adaptive weight optimization:

[0151] ,

[0152] ,

[0153] AIM stands for Multimodal Scenario Adaptive Optimizer; This represents the predicted process deviation associated with the i-th mode; Indicates the target value for process deviation; Represents the scene-adaptive weights for the i-th mode; Regularization representing cross-modal features; The reference value representing the process deviation of the i-th mode; This represents the predicted KPI value associated with the i-th mode; Indicates the KPI target value; Represents the weight regularization coefficient; These are the regularization coefficients for the model parameters; These are the cross-modal characteristic regularization coefficients; These are the learnable parameters of the model.

[0154] In this embodiment, the method for generating knowledge-enhanced features through knowledge cross-attention and a multi-modal scene adaptive optimizer includes:

[0155] By fusing scene weights with updated modal features, combining intra-modal self-attention and cross-modal cross-attention, and introducing a knowledge graph (KG) specific to automotive manufacturing, process-related knowledge is encoded into knowledge embedding vectors to generate knowledge-enhanced features. The formula is as follows:

[0156] ,

[0157] ,

[0158] ,

[0159] The formula for the knowledge cross-attention component is as follows:

[0160] ,

[0161] in, Represents the modal features after scene enhancement; Represents a learnable matrix; Represents a Gaussian error linear unit; Represents the gate function; This is a global parameter representing the contribution weight of the knowledge module; This represents the minimum value; KG represents the embedding vector of the knowledge graph; This represents the normalization term of the cross product; , This is the knowledge projection matrix.

[0162] In this embodiment, the method for obtaining the optimal strategy through probabilistic decision evaluation of the knowledge-enhanced features includes:

[0163] Based on the The scale is used to generate the k-th production strategy, and the formula is:

[0164] ,

[0165] in, This represents the k-th production strategy; Indicates the first Layer-scale weights; For the first The weight matrix of the scale and the k-th strategy; Indicates the first Scale, bias of the k-th strategy; Indicates the first Variance of scale; This indicates knowledge-enhancing features;

[0166] Based on knowledge graphs and augmented multimodal vectors, conditional probabilities are obtained using the following formula:

[0167] ,

[0168] ,

[0169] ,

[0170] in, Representation Strategy The conditional probability; F is the multimodal enhancement feature; The knowledge graph embedding vector for the k-th strategy; Let the target mean vector be the k-th strategy;

[0171] Decision-making is achieved through a time-decay Gaussian weighted average and a risk-squared weighted average mechanism, as shown in the formula:

[0172] ,

[0173] in, This represents the optimal production strategy; express The conditional probability of the strategy; T represents the length of the time period; The variance representing time decay; Indicates the number of risk types; Representation Strategy The degree of violation that leads to the first type of risk.

[0174] In this embodiment, the method for constructing a reward mechanism that links time-varying risk and return includes:

[0175] Collaborative modeling of returns and risks, implementing a dynamic reward function:

[0176] ,

[0177] in, The time-varying risk coefficient; Indicates the level of risk at time t; This represents the change in production indicators at time t;

[0178] Develop risk-sensitive rewards:

[0179] ,

[0180] in, This represents the maximum historical risk before time t; Control factors indicating the severity of punishment; The gradient represents the policy.

[0181] Strategy optimization goal:

[0182] ,

[0183] in, The learnable parameters of the policy generator; Indicates the time discount factor; Let represent the learning reward at step t; M represents the number of candidate policies.

[0184] Secondly, a policy generation system based on a multimodal large model includes:

[0185] Multimodal data acquisition module: used to collect multi-source data in the intelligent manufacturing process of automobiles. The multi-source data includes visual data acquired by industrial vision equipment, text data of process documents and production logs, and real-time operation data collected by sensors of production equipment.

[0186] A cross-modal feature extraction module is used to extract differentiated features from the multi-source data based on a concatenated strategy to obtain single-modal features, which include visual modal features, text modal features, and sensor modal features. The visual modal features and sensor modal features are fused across modally to generate a first cross-modal feature, the visual modal features and text modal features are fused across modally to generate a second cross-modal feature, and the sensor modal features and text modal features are fused across modally to generate a third cross-modal feature.

[0187] Weight fusion module: used to obtain calibration features from the first cross-modal feature, the second cross-modal feature, the third cross-modal feature and the single-modal feature; based on the calibration features, dynamically adjust the modal weights to adapt to scene changes according to multi-objective evaluation, and generate a multi-modal scene adaptive optimizer;

[0188] Production strategy generation module: used to generate knowledge-enhanced features through knowledge cross-attention and the multi-modal scene adaptive optimizer; and to obtain the optimal production strategy by evaluating the knowledge-enhanced features through probabilistic decision-making.

[0189] Closed-loop feedback optimization module: This module is used to construct a reward mechanism that correlates time-varying risk and return based on risk, gradient stability, and long-term return constraints. It then performs feedback optimization on the optimal production strategy according to the reward mechanism and iteratively updates the production strategy.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal large model-based strategy generation method, characterized in that, The method comprises the following steps: Collecting industrial vision, process text and device sensor to obtain multi-source data; Based on the series strategy, the multi-source data is differentiated and the single modal feature is extracted to obtain the single modal feature; the single modal feature includes visual modal feature, text modal feature and sensor modal feature; The visual modal feature and the sensor modal feature are mapped in space and time through a dynamic anchor point matrix to generate a first cross-modal feature; the visual modal feature and the text modal feature are combined with time sequence modeling and network mapping mechanism to generate a second cross-modal feature through cross-modal similarity with knowledge constraint; the sensor modal feature and the text modal feature are combined according to the attention mechanism and the feature fusion principle to generate a third cross-modal feature; Based on the first cross-modal feature, the second cross-modal feature, the third cross-modal feature and the single modal feature, a calibration feature is obtained; Based on the calibration feature, a three-layer linkage dynamic adjustment modal weight is adopted to adapt to the scene change, and a multi-modal scene self-adaptive optimizer is generated; the first layer is a modal internal variance weight layer, which obtains the fluctuation degree of the feature vector; The second layer is a cross-modal correlation weight layer, which updates the visual modal feature; the third layer is a scene self-adaptive weight layer, which dynamically adjusts the modal weight; A knowledge enhanced feature is generated through knowledge cross-attention and the multi-modal scene self-adaptive optimizer; The knowledge enhanced feature is evaluated by a probability decision to obtain an optimal strategy, a reward mechanism related to time-varying risk and return is constructed based on risk and gradient stability and long-term return constraint, and the optimal strategy is optimized according to the reward mechanism.

2. The strategy generation method based on a multi-modal large model according to claim 1, characterized in that, The method for extracting features from multi-source data based on a series strategy comprises: ResNet-50 is used to complete bottom feature extraction, and the bottom feature is provided as input to EfficientNet to extract semantic judgment high-level feature; a regularization term is added, and a visual modal feature is output; Cross-layer fusion and dynamic scale attention are used to enhance key region features, and the formula is: ， wherein, is the bottom layer feature, is the high layer feature, is the cross-layer weight at the i-th scale dynamically adjusted according to the production stage; represents the contribution of the i-th scale adjustment; is the defect rate sensitive coefficient; D is the real-time defect rate; is the remaining welding time; t is the welded time; is the spatial gradient of the welding parameter along the welding torch movement direction z; is the process stability benchmark resistance; is the current influence coefficient; is the real-time welding current; is the process parameter gradient weight; is the ideal feature vector in the process knowledge base, is the actual visual feature vector, and the dynamic scale attention is calculated through a self-attention mechanism: ， Wherein Q, K, V are the query matrix, key matrix and value matrix of the feature map, is a normalization parameter, refers to the key region feature after attention mechanism enhancement; is a function for normalizing each row of the similarity matrix; is a scaling coefficient; is a random discarded V feature, is the amplitude coefficient of Dropout. A BERT model is fine-tuned based on an automobile manufacturing terminology library to extract automobile field exclusive semantic embedding; a basic semantic feature is generated by BERT based on the automobile manufacturing terminology library fine-tuning, and the basic semantic feature is input to TextCNN to extract static features and LSTM to capture time sequence dependent dynamic features; the weights of TextCNN and LSTM are adjusted through a gating function, and the weighted fusion of BiLSTM and Transformer is output as a text modal feature; Combining textcnn with lstm to generate time-semantic cross features The formula is: ， ， wherein T is a text sequence, is an industrial term fine-tuning parameter, is a BERT model fine-tuned by an industrial term; is a domain enhanced semantic embedding vector of the text sequence; is a sigmoid function; is a semantic modal feature extracted by TextCNN; is a time-series semantic modal feature extracted by LSTM; is a gate bias, , is a fusion coefficient, BiLSTM is a bidirectional long short-term memory network, and Transformer is a self-attention mechanism; is a text modal feature; Sliding windows are dynamically divided according to device cycles, and time sequence rules and abnormalities are captured through Transformer and Gaussian process regression GPR to output a sensor modal feature.

3. The strategy generation method based on a multi-modal large model according to claim 1, characterized in that, The method for obtaining the first cross-modal feature, the second cross-modal feature and the third cross-modal feature comprises: Based on production condition parameters and process constraint rules, time sequence dynamic anchor point matrix and scene adaptive factor are used to realize space-time alignment of visual modal feature and sensor modal feature; cross-modal attention mechanism is used to strengthen key region correlation; visual modal feature and sensor modal feature are fused to output a first cross-modal feature, and the formula is: ， wherein, is a spatio-temporal dynamic anchor association matrix generated by the coordinate mapping and feature association of the camera and sensor; is a visual modality feature; is a sensor modality feature; is a visual and sensor cross-spatio-temporal fusion feature; refers to a scene adaptive adjustment factor; is a cross-modality topological feature; , is a visual feature query matrix and a sensor feature key matrix, is a scene adaptive normalization coefficient; T is a time step of the production stage; The visual modality feature and the text modality feature generate a second cross-modality feature through a cross-modality similarity with knowledge constraints, combined with time sequence modeling and network mapping mechanism: ， ， wherein sim is a relevance scalar of visual features and text semantics, is a domain enhanced semantic embedding vector for the text sequence; is a feature fluctuation adjustment coefficient, dynamically adjusted based on the variance of visual and text features; is a knowledge graph relationship embedding function, encoding the process correlation relationship into a vector; is a process relationship mapping function of visual features and text semantics, is a visual and text cross-modal fusion feature vector; is a bidirectional long short-term memory network, capturing temporal dependencies, , , is a multi-layer perceptron, is a normalized attention weight; A cross-modality fusion of the sensor modality feature and the text modality feature is constructed based on an attention mechanism and a feature fusion principle, and a third cross-modality feature is output: ， ， wherein, is a cross-modal feature of the sensor and the text; is an associated weight coefficient; is a temperature pass rate; is a maximum temperature value; represents a weight modulation coefficient associated with the KPI; is an alarm threshold; is a variance of the temperature; is a text modality weight matrix; is a sensor modality matrix; is a cross-fusion weight matrix.

4. The strategy generation method based on a multi-modal large model according to claim 1, characterized in that, The three-layer dynamic weight algorithm comprises: The first layer is an intra-modality variance weight layer, which obtains the fluctuation degree of the feature vector; the second layer is a cross-modality correlation weight layer, which updates the visual modality feature; and the third layer is a scene adaptive weight layer, which dynamically adjusts the modality weight; The fluctuation degree of the feature is quantified by calculating the variance and covariance of each modality feature, and the modality basic weight is determined, and the formula is: ， ， wherein, is a base weight of the i-th modality; is a deviation of a process parameter from a standard value; is a base eigenvector of the i-th modality; is a cross-modality feature of the i-th modality with other modalities; KPI represents a production result indicator; is a weight of a process indicator; is a weight of a production indicator; is an adjustment coefficient of a cross-modality; is a Gaussian penalty term; the i-th modality includes: a visual modality v, a text modality t, a sensor modality s, and the cross-modality feature includes a first cross-modality feature, a second cross-modality feature, and a third cross-modality feature; The cross-modality correlation uses intra-modality self-attention to mine modality details, and cross-modality cross-attention to establish the dependence between modalities, and calculates the dynamic correlation weight through the query key mechanism; based on the correlation weight, the features are updated, the multi-modality features are fused, and the modality features are reconstructed through dynamic normalization and activation function; The correlation weight of the i-th modality with other modalities is: ， The feature update of the i-th modality is: ， wherein, represents the association weight of the i-th modality and the j-th modality; , represents a projection matrix that maps the features to the same dimension; represents the L2 norm; represents summing over all other modalities; represents the updated i-th modality feature; represents a regularization coefficient; is a numerical stability factor; Based on the construction of a multi-objective optimization framework, the deviation degree is quantified by weighted error, and weight smoothing penalty, parameter and cross-modality feature regularization are introduced for scene adaptive weight optimization: ， ， wherein, AIM denotes a multi-modal scenario self-adaptive optimizer; denotes the process bias prediction associated with the i-th modality; denotes the target value of the process bias; denotes the scenario self-adaptive weight of the i-th modality; denotes the regularization of cross-modality features; denotes the baseline value of the process bias of the i-th modality; denotes the KPI prediction value associated with the i-th modality; denotes the KPI target value; denotes the weight regularization coefficient; is a model parameter regularization coefficient; is a cross-modality feature regularization coefficient; is a model learnable parameter. 5.The multi-modal large model based strategy generation method according to claim 1, wherein, The method for generating knowledge enhanced features through knowledge cross-attention and multi-modal scene self-adaptive optimizer comprises: The scene weight and the updated modality feature are fused, the intra-modality self-attention and the cross-modality cross-attention are combined, the automobile manufacturing exclusive knowledge graph KG is introduced, the process correlation knowledge is encoded into knowledge embedding vectors, and the knowledge enhanced features are generated, and the formula is: ， ， ， The formula of the knowledge cross-attention part is: ， wherein, denotes the modality features after scene enhancement; denotes the learnable matrix; denotes the Gaussian error linear unit; denotes the gating function; is a global parameter, denoting the contribution weight of the knowledge module; is a minimum value, a numerical stability factor; KG denotes the embedding vector of the knowledge graph; denotes the normalization term of the cross-point product; , is a knowledge projection matrix. 6.The multi-modal large model based strategy generation method according to claim 1, wherein, The method for obtaining the optimal strategy through probability decision evaluation of the knowledge enhanced features comprises: Based on the first The kth production strategy is generated based on the scale, and the formula is: ， wherein, represents the kth production strategy; represents the kth weight of the layer scale; is the kth weight matrix of the scale, the kth strategy; represents the kth bias of the scale, the kth strategy; represents the kth variance of the scale; represents the knowledge-enhanced feature; Based on the knowledge graph and the enhanced multi-modal feature vector, the conditional probability is obtained, and the formula is: ， ， ， wherein, representing a policy the conditional probability; F is a multimodal enhanced feature; is a knowledge graph embedding vector of the kth policy; is a target mean vector of the kth policy; The decision is obtained through the Gaussian weighted sum of time decay and the square weighted mechanism of risk, and the formula is: ， wherein, represents the optimal production strategy; represents the conditional probability of the strategy; T represents the length of the time period; represents the variance of the time decay; represents the number of risk types; represents the strategy causing the degree of violation of the first risk type. 7.The multi-modal large model based strategy generation method according to claim 1, wherein, The method for constructing the reward mechanism associated with time-varying risk and return comprises: The return and risk are cooperatively modeled, and a dynamic reward function is implemented: ， wherein, is a time-varying risk coefficient; represents the risk level at time t; represents the change in production indicator at time t; A risk-sensitive reward is constructed: ， wherein, represents the historical maximum risk before time t; represents a control factor of the punishment degree; represents the gradient of the strategy; The strategy optimization target is: ， wherein, denote learnable parameters of the policy generator; denotes a time discount factor; denotes the learning reward at the t-th step; M denotes the number of candidate policies.

8. A multi-modal large model based strategy generation system for performing the method of any one of claims 1-7. It comprises: A multi-modality data acquisition module is used to acquire multi-source data in the automobile intelligent manufacturing process, and the multi-source data comprises visual data acquired by industrial visual equipment, text data of process documents and production logs, and real-time running data collected by production equipment sensors; A cross-modality feature extraction module is used to extract single modality features by differentiating features based on a serial strategy, and the single modality features comprise visual modality features, text modality features, and sensor modality features; the visual modality features and the sensor modality features are cross-modality fused to generate first cross-modality features, the visual modality features and the text modality features are cross-modality fused to generate second cross-modality features, and the sensor modality features and the text modality features are cross-modality fused to generate third cross-modality features; A weight fusion module is used to obtain calibrated features from the first cross-modality features, the second cross-modality features, the third cross-modality features, and the single modality features; Based on the calibration feature, a multi-modal scene self-adaptive optimizer is generated by dynamically adjusting the modal weight according to the multi-target evaluation and adapting to the scene changes; A production strategy generation module is configured to generate knowledge-enhanced features by knowledge cross-attention and the multi-modal scene self-adaptive optimizer; The knowledge-enhanced features are evaluated by a probability decision to obtain an optimal production strategy; A closed-loop feedback optimization module is configured to construct a reward mechanism associated with time-varying risk and return based on risk and gradient stability and long-term return constraints, and to perform feedback optimization on the optimal production strategy according to the reward mechanism to iteratively update the production strategy.

Images