Quality safety optimization method and system based on multi-modal vertical large model technology

Abstract

The present invention discloses a quality and safety optimization method and system based on multimodal vertical large model technology, which relates to the technical field of quality and safety optimization system, including: collecting quality and safety related data from multimodal data sources and preprocessing; fusing data through features and decision levels; using fused data to build a vertical large model of improved Transformer architecture, with quality feature extraction and safety rule embedding layers, trained with adaptive learning rate and gradient clipping; inputting new data into the large model to analyze the quality and safety status, including defect prediction, risk assessment and problem warning; based on the analysis results, generating optimization suggestions through rule reasoning and reinforcement learning. The present invention can comprehensively collect and preprocess multimodal quality and safety data. The vertical large model is highly targeted and accurately analyzes quality defects, safety risks and potential problems. The optimization suggestion module combines rules and reinforcement learning to generate suggestions with expected effects and priorities, thereby improving the quality and safety level and resource utilization efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of quality safety optimization systems, and in particular to a quality safety optimization method and system based on multi-modal vertical large model technology. Background Art

[0002] In modern industrial production and the operation of complex systems, quality and safety issues are always the top priority. With the continuous development of technology and the expansion of production scale, ensuring quality and safety faces unprecedented challenges.

[0003] From the production perspective, industries such as manufacturing involve a large number of complex process flows, various types of production equipment, and numerous raw materials. Traditional quality control methods mainly rely on manual inspection and post-analysis. For example, on the production line, quality inspectors conduct random inspections of products through visual observation and simple tools. This method is not only inefficient, but also prone to missed inspections and misjudgments. For safety issues, most of them are prevented based on established safety regulations and standards through regular equipment inspections and personnel training, but these measures often lack real-time and pertinence, and cannot effectively respond to safety hazards that may arise at any time in a complex and changing production environment.

[0004] In terms of data utilization, there is a huge amount of multimodal data in current enterprises and production systems. These data include continuous numerical data from production equipment sensors, text data in quality inspection reports, image data captured by surveillance cameras, and text information in operation record documents. However, traditional data analysis methods are difficult to effectively integrate and utilize these multimodal data. Data of different modalities have different structures and semantics, and previous technologies cannot fully explore the associations and potential values ​​between them, resulting in the failure to fully play the role of data resources in quality and safety optimization.

[0005] At the same time, with the expansion of production scale and the increase of system complexity, the consequences of quality problems and safety accidents are becoming more serious. A small quality defect may lead to product recall, damage to corporate reputation and huge economic losses, while safety accidents are more likely to endanger the safety of people's lives and the environment. Therefore, a more intelligent and comprehensive quality and safety optimization method and system is urgently needed to meet these challenges.

[0006] At the technological development level, although big data and artificial intelligence technologies have made significant progress, existing general models have limitations when processing multimodal data in specific fields (such as quality and safety). General models are often not specifically designed for key quality and safety features, and cannot accurately extract information related to quality and safety from complex multimodal data. They cannot effectively predict and evaluate quality defects and safety risks, and it is even more difficult to generate targeted optimization suggestions to prevent problems from occurring. In summary, existing technologies have many shortcomings in quality and safety optimization, and it is difficult to meet the high requirements of modern industrial production and complex system operation for quality and safety assurance, and innovative solutions are urgently needed. Summary of the invention

[0007] The present invention proposes a quality safety optimization method and system based on multimodal vertical large model technology to solve the problems mentioned in the above-mentioned prior art.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: a quality safety optimization method based on multi-modal vertical large model technology, comprising the following steps:

[0009] S1. Data collection step: collecting data related to quality and safety from multiple data sources of different modes, wherein the multiple modes include but are not limited to text data, image data, audio data, and sensor data, wherein the data sources include production equipment sensors, quality inspection reports, surveillance cameras, and operation record documents, wherein the text data is preprocessed using natural language processing technology, including lexical analysis, syntactic analysis, and entity recognition; the image data is subjected to feature extraction and normalization processing; the audio data is subjected to noise reduction and spectrum analysis; and the sensor data is subjected to data calibration and outlier processing;

[0010] S2. Multimodal data fusion step: The pre-processed different modal data are fused based on the feature-level fusion and decision-level fusion methods. The feature-level fusion constructs a multimodal feature vector and uses a deep learning model to extract joint feature representations. The decision-level fusion makes a comprehensive decision based on the independent decision results of different modal data through the Bayesian reasoning method to generate a fused quality and safety data representation. In the feature-level fusion, the fusion weight of the multimodal feature vector is calculated. The formula is: ,in Indicates Modal features With reference feature The similarity measure of To control the parameters of weight distribution, is the number of modes, represents a natural constant, The way to obtain is: ,in represents the dot product of vectors, and They represent the magnitude of the vector respectively;

[0011] S3, vertical large model construction step: Use the fused quality and safety data to represent the vertical large model. The architecture of the vertical large model is based on the Transformer architecture improvement, including specific layers and parameter adjustments for the quality and safety field, where the specific layers are used for reinforcement learning of key features of quality and safety. The adaptive learning rate and gradient clipping technology are used in the model training process. In the vertical large model training, the adaptive learning rate The update formula uses the AdamW algorithm: ,in is the initial learning rate, , are algorithm parameters, and The t in represents the number of training steps. Indicates that as the number of training steps increases, the algorithm parameters The specific value at step t, Indicates that as the number of training steps increases, the algorithm parameters The specific value at step t, , is the modified gradient first-order moment estimate and second-order moment estimate, is a small constant to prevent division by zero, and the gradient clipping formula is: ,in is the gradient vector, is the clipping threshold;

[0012] S4, quality and safety analysis step: input the new multimodal data to be analyzed into the constructed vertical large model, and the model outputs the analysis results related to quality and safety, including quality defect prediction, safety risk assessment, and potential problem warning. The analysis results are based on a predefined quality and safety indicator system, which includes dimensions from raw material quality to production process safety and finished product quality. The defect probability is calculated in the quality defect prediction. The formula is: ,in is the weight of the sth decision tree in the ensemble model, is the number of decision trees, is the defect probability predicted by the s-th decision tree. The s-th decision tree is trained using multimodal data samples with quality defect annotations. During the training process, the decision tree learns and optimizes according to the characteristics of the data and the quality defect labels. When predicting the defect probability of new data, the new data is input into the trained decision tree. The decision tree will classify and judge the new data according to the rules and patterns it has learned. If the decision tree classifies the new data into a defect category, a defect probability value of 0.7 is set; if it is classified as a non-defect category, a lower defect probability value of 0.3 is set. This probability value is the defect probability predicted by the s-th decision tree; the risk value of safety risk assessment The calculation formula is: ,in is the probability of an accident happening, is the degree of accident impact, and the two are determined according to the risk assessment matrix; the observation probability formula for calculating the hidden Markov model in potential problem warning is: ,in is the state transition probability, is in state Observed below The probability of is the number of states;

[0013] S5. Optimization suggestion generation step: Based on the quality and safety analysis results, targeted quality and safety optimization suggestions are generated through a rule-based reasoning engine and machine learning algorithm. The optimization suggestions include process parameter adjustment, equipment maintenance plan, and personnel training plan. The optimization suggestions are accompanied by expected effect evaluation and implementation priority. In the generation of optimization suggestions by reinforcement learning, the value function The update formula is: ,in is the learning rate, It’s an instant reward. is the discount factor, is the current state, is the next state; the learning rate is within a predetermined range of 0.001 to 0.1, with a step size of 0.005 for grid search, and the model is trained using different learning rates, and the final learning rate is selected based on the accuracy of the model on the validation set and the performance of the loss function value; the instant reward is obtained by quantifying indicators related to quality and safety, and determining the instant reward based on changes in the indicators, including setting a fixed reward value for every 1% increase in product qualification rate and a fixed reward value for every reduction in the number of safety accidents; the discount factor is between 0 and 1, and is determined based on the time span of the problem and the importance attached to future rewards.

[0014] Furthermore, in the data collection step:

[0015] For the preprocessing of text data, a pre-trained language model is used for word embedding representation. Lexical analysis is implemented based on finite state automata. Syntactic analysis uses a dependency syntax analysis algorithm. Entity recognition uses a conditional random field model. The recognized entities include quality standard entities, safety specification entities, production equipment entities, and raw material entities.

[0016] Convolutional neural networks are used to extract features from image data. The extracted features include object shape, texture, and color distribution. Normalization processing maps image pixel values ​​to a specific interval so that image data from different sources have the same scale.

[0017] Wavelet transform is used to reduce the noise of audio data, and spectrum analysis obtains the frequency components and amplitude information of audio through fast Fourier transform;

[0018] The calibration of sensor data is carried out according to the sensor's calibration curve and calibration equation, and the outlier processing adopts the 3σ principle based on statistical methods or the anomaly detection algorithm based on machine learning.

[0019] Furthermore, in the multimodal data fusion step:

[0020] The deep learning model of feature-level fusion is a hybrid architecture of convolutional neural network and long short-term memory network. The input is the feature vector of different modal data, and the output is the fused joint feature vector. The contrastive learning loss function is used in the training process to enhance the correlation between the features of different modal data. The formula is: ,in , is a positive sample feature pair, N represents the total number of samples in the sample set, and e represents a natural constant. is the similarity function, and the calculation formula is: ,in represents the dot product of vectors, and They represent the magnitude of the vector respectively;

[0021] The steps for obtaining the positive sample feature pair are as follows:

[0022] First, the data is preprocessed, and a convolutional neural network is used to extract high-level feature representations of the image, and the sensor data is calibrated and feature extracted;

[0023] Then, compare all samples in the dataset and check their quality and safety level labels;

[0024] When a sample with an excellent quality safety level is found, the different modal features of the sample, including image features and sensor features, are combined as a positive sample feature pair;

[0025] In the Bayesian reasoning method of decision-level fusion, a conditional probability model is first established for each modal data, the prior probability is determined based on historical data and expert knowledge, and then the posterior probability is calculated using the Bayesian formula, and the final fusion decision is obtained by combining the posterior probabilities of each modality. The Bayesian formula is: ,in is the posterior probability, is the likelihood probability, is the prior probability, It is the probability of evidence.

[0026] Furthermore, in the vertical large model construction step:

[0027] The specific layers for the quality and safety field include the quality feature extraction layer and the safety rule embedding layer. The quality feature extraction layer focuses on quality-related features through the attention mechanism, and the safety rule embedding layer embeds safety specifications and standards into the model parameters in the form of vectors.

[0028] The adaptive learning rate uses the AdamW optimization algorithm to dynamically adjust the learning rate according to the gradient information during model training. The gradient clipping technology limits the norm of the gradient to a preset range to prevent gradient explosion from causing unstable model training.

[0029] Furthermore, in the quality and safety analysis step:

[0030] Quality defect prediction is based on the quality dimension indicators in the quality safety indicator system. The possible quality defect types and probabilities are predicted through a classification model. The classification model is an integrated model based on support vector machine and random forest.

[0031] Safety risk assessment is based on safety dimension indicators and uses a risk assessment matrix to calculate risk levels. The risk assessment matrix is ​​constructed based on historical accident data and safety standards;

[0032] Potential problem warning is achieved through time series analysis of quality and safety indicators, using hidden Markov models to identify abnormal trends in indicators and issue warning signals in advance.

[0033] Furthermore, in the optimization suggestion generating step:

[0034] The rule-based reasoning engine contains a quality and safety rule base, in which the rules are expressed in an "if-then" form, and generates preliminary optimization suggestions based on the quality and safety analysis results and matching rules;

[0035] The machine learning algorithm adopts a reinforcement learning algorithm, takes the quality and safety status as the environment and the optimization suggestions as the actions, and learns the optimal optimization suggestion generation strategy through a reward mechanism. The reward mechanism is determined based on the improvement of quality and safety indicators after the implementation of the optimization suggestions.

[0036] Furthermore, a quality safety optimization system based on multi-modal vertical large model technology includes:

[0037] A data acquisition module is used to collect data related to quality and safety from multiple data sources of different modes, and pre-process different types of data. The multiple modes include text data, image data, audio data, and sensor data. The data sources include production equipment sensors, quality inspection reports, surveillance cameras, and operation record documents.

[0038] The multimodal data fusion module is used to fuse the pre-processed data of different modalities based on the feature-level fusion and decision-level fusion methods to generate fused quality and safety data representation;

[0039] A vertical large model building module, used to build a vertical large model using the fused quality and safety data representation, wherein the architecture of the vertical large model is improved based on the Transformer architecture, including specific layers and parameter adjustments for the quality and safety field;

[0040] The quality and safety analysis module is used to input new multimodal data to be analyzed into the constructed vertical large model and output analysis results related to quality and safety, including quality defect prediction, safety risk assessment, and potential problem warning;

[0041] The optimization suggestion generation module is used to generate targeted quality and safety optimization suggestions based on the quality and safety analysis results through a rule-based reasoning engine and machine learning algorithm. The optimization suggestions include process parameter adjustments, equipment maintenance plans, and personnel training plans, and the optimization suggestions are accompanied by expected effect evaluation and implementation priorities.

[0042] Furthermore, the data acquisition module includes:

[0043] A text processing unit, used to pre-process text data using natural language processing technology, including lexical analysis, syntactic analysis, and entity recognition;

[0044] An image processing unit, used for performing feature extraction and normalization processing on image data;

[0045] An audio processing unit for performing noise reduction and spectrum analysis on audio data;

[0046] The sensor data processing unit is used to perform data calibration and outlier processing on the sensor data.

[0047] Furthermore, the multimodal data fusion module includes:

[0048] Feature-level fusion unit, whose deep learning model is a hybrid architecture of convolutional neural network and long short-term memory network, is used to extract joint feature representation by constructing multimodal feature vectors using deep learning models;

[0049] The decision-level fusion unit is used to make comprehensive decisions based on the independent decision results of different modal data through the Bayesian reasoning method.

[0050] Furthermore, the vertical large model construction module includes:

[0051] Specific layer construction unit, used to construct specific layers for the quality and safety field, including quality feature extraction layer and safety rule embedding layer;

[0052] The training optimization unit is used to use adaptive learning rate and gradient clipping techniques during model training to prevent gradient explosion and speed up convergence.

[0053] The quality and safety analysis module includes:

[0054] The quality defect prediction unit is used to predict the type and probability of possible quality defects based on the quality dimension indicators in the quality safety indicator system through a classification model. The classification model is an integrated model based on support vector machine and random forest;

[0055] A security risk assessment unit is used to calculate the risk level using a risk assessment matrix based on security dimension indicators;

[0056] The potential problem early warning unit is used to analyze the time series of quality and safety indicators, use the hidden Markov model to identify abnormal change trends of indicators, and issue early warning signals.

[0057] The optimization suggestion generation module includes:

[0058] A rule reasoning unit, used to generate preliminary optimization suggestions through a rule-based reasoning engine, the reasoning engine including a quality and safety rule base;

[0059] The machine learning optimization unit is used to learn the optimal optimization suggestion generation strategy using a reinforcement learning algorithm and determine the optimization suggestions based on the quality and safety status and reward mechanism.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention collects data of multiple modalities, including text, images, audio, and sensor data, and uses advanced natural language processing, feature extraction, noise reduction calibration and other pre-processing technologies to comprehensively mine information related to quality and safety. For example, the lexical and syntactic analysis and entity recognition of text data, and the shape and texture feature extraction of image data can accurately capture the connotation of the data, avoid the one-sidedness of traditional single-modality analysis, and lay the foundation for subsequent precise analysis.

[0062] The present invention adopts a unique multimodal data fusion method. The feature-level fusion uses a formula to calculate the fusion weight, and the decision-level fusion is based on Bayesian reasoning. It fully considers the relationship between different modal data and their respective decision results, so that the fused data representation can better reflect the true status of quality and safety. Compared with traditional simple fusion methods, the ability of fused data to characterize quality and safety conditions is greatly improved.

[0063] This invention builds a vertical large model based on the improved Transformer architecture, adopts adaptive learning rate formula and gradient clipping technology such as AdamW algorithm, and specific layers designed for the field of quality and safety, including quality feature extraction layer and safety rule embedding layer, so that the model can accurately focus on key features of quality and safety. Compared with general models, it is more accurate and reliable in quality defect prediction, safety risk assessment and potential problem warning, and hidden Markov model related formulas, and can effectively deal with quality and safety issues in complex production environments.

[0064] The present invention generates optimization suggestions through a rule-based reasoning engine and machine learning algorithm, especially the update mechanism of the value function in reinforcement learning, so that the optimization suggestions can be dynamically adjusted according to the results of quality and safety analysis. The accompanying expected effect evaluation and implementation priority can help enterprises to reasonably allocate resources and give priority to key issues. This is an advantage that traditional quality and safety optimization methods do not have. It can significantly reduce the losses caused by quality problems and safety accidents and improve the overall quality and safety level. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic block diagram of a quality safety optimization system based on multi-modal vertical large model technology proposed in the present invention;

[0066] Figure 2 This is a schematic block diagram of a quality safety optimization method based on multimodal vertical large model technology proposed in the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise" and "counterclockwise" indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0069] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, and it can be the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0070] Reference Figure 1-2 :A quality safety optimization method based on multi-modal vertical large model technology, comprising the following steps:

[0071] S1. Data collection step: collecting data related to quality and safety from multiple data sources of different modes, wherein the multiple modes include but are not limited to text data, image data, audio data, and sensor data, wherein the data sources include production equipment sensors, quality inspection reports, surveillance cameras, and operation record documents, wherein the text data is preprocessed using natural language processing technology, including lexical analysis, syntactic analysis, and entity recognition; the image data is subjected to feature extraction and normalization processing; the audio data is subjected to noise reduction and spectrum analysis; and the sensor data is subjected to data calibration and outlier processing;

[0072] S2. Multimodal data fusion step: The pre-processed different modal data are fused based on the feature-level fusion and decision-level fusion methods. The feature-level fusion constructs a multimodal feature vector and uses a deep learning model to extract joint feature representations. The decision-level fusion makes a comprehensive decision based on the independent decision results of different modal data through the Bayesian reasoning method to generate a fused quality and safety data representation. In the feature-level fusion, the fusion weight of the multimodal feature vector is calculated. The formula is: ,in Indicates Modal features With reference feature The similarity measure of To control the parameters of weight distribution, is the number of modes, represents a natural constant, The way to obtain is: ,in represents the dot product of vectors, and They represent the magnitude of the vector respectively;

[0073] S3, vertical large model construction step: Use the fused quality and safety data to represent the vertical large model. The architecture of the vertical large model is based on the Transformer architecture improvement, including specific layers and parameter adjustments for the quality and safety field, where the specific layers are used for reinforcement learning of key features of quality and safety. The adaptive learning rate and gradient clipping technology are used in the model training process. In the vertical large model training, the adaptive learning rate The update formula uses the AdamW algorithm: ,in is the initial learning rate, , are algorithm parameters, and The t in represents the number of training steps. Indicates that as the number of training steps increases, the algorithm parameters The specific value at step t, Indicates that as the number of training steps increases, the algorithm parameters The specific value at step t, , is the modified gradient first-order moment estimate and second-order moment estimate, is a small constant to prevent division by zero, and the gradient clipping formula is: ,in is the gradient vector, is the clipping threshold;

[0074] S4, quality and safety analysis step: input the new multimodal data to be analyzed into the constructed vertical large model, and the model outputs the analysis results related to quality and safety, including quality defect prediction, safety risk assessment, and potential problem warning. The analysis results are based on a predefined quality and safety indicator system, which includes dimensions from raw material quality to production process safety and finished product quality. The defect probability is calculated in the quality defect prediction. The formula is: ,in is the weight of the sth decision tree in the ensemble model, is the number of decision trees, is the defect probability predicted by the s-th decision tree. The s-th decision tree is trained using multimodal data samples with quality defect annotations. During the training process, the decision tree learns and optimizes according to the characteristics of the data and the quality defect labels. When predicting the defect probability of new data, the new data is input into the trained decision tree. The decision tree will classify and judge the new data according to the rules and patterns it has learned. If the decision tree classifies the new data into a defect category, a defect probability value of 0.7 is set; if it is classified as a non-defect category, a lower defect probability value of 0.3 is set. This probability value is the defect probability predicted by the s-th decision tree; the risk value of safety risk assessment The calculation formula is: ,in is the probability of an accident happening, is the degree of accident impact, and the two are determined according to the risk assessment matrix; the observation probability formula for calculating the hidden Markov model in potential problem warning is: ,in is the state transition probability, is in state Observed below The probability of is the number of states;

[0075] S5. Optimization suggestion generation step: Based on the quality and safety analysis results, targeted quality and safety optimization suggestions are generated through a rule-based reasoning engine and machine learning algorithm. The optimization suggestions include process parameter adjustment, equipment maintenance plan, and personnel training plan. The optimization suggestions are accompanied by expected effect evaluation and implementation priority. In the generation of optimization suggestions by reinforcement learning, the value function The update formula is: ,in is the learning rate, It’s an instant reward. is the discount factor, is the current state, is the next state; the learning rate is within a predetermined range of 0.001 to 0.1, with a step size of 0.005 for grid search, and the model is trained using different learning rates, and the final learning rate is selected based on the accuracy of the model on the validation set and the performance of the loss function value; the instant reward is obtained by quantifying indicators related to quality and safety, and determining the instant reward based on changes in the indicators, including setting a fixed reward value for every 1% increase in product qualification rate and a fixed reward value for every reduction in the number of safety accidents; the discount factor is between 0 and 1, and is determined based on the time span of the problem and the importance attached to future rewards.

[0076] In the present invention, in the data collection step:

[0077] For the preprocessing of text data, a pre-trained language model is used for word embedding representation. Lexical analysis is implemented based on finite state automata. Syntactic analysis uses a dependency syntax analysis algorithm. Entity recognition uses a conditional random field model. The recognized entities include quality standard entities, safety specification entities, production equipment entities, and raw material entities.

[0078] Convolutional neural networks are used to extract features from image data. The extracted features include object shape, texture, and color distribution. Normalization processing maps image pixel values ​​to a specific interval so that image data from different sources have the same scale.

[0079] Wavelet transform is used to reduce the noise of audio data, and spectrum analysis obtains the frequency components and amplitude information of audio through fast Fourier transform;

[0080] The calibration of sensor data is carried out according to the sensor's calibration curve and calibration equation, and the outlier processing adopts the 3σ principle based on statistical methods or the anomaly detection algorithm based on machine learning.

[0081] In the present invention, in the multimodal data fusion step:

[0082] The deep learning model of feature-level fusion is a hybrid architecture of convolutional neural network and long short-term memory network. The input is the feature vector of different modal data, and the output is the fused joint feature vector. The contrastive learning loss function is used in the training process to enhance the correlation between the features of different modal data. The formula is: ,in , is a positive sample feature pair, is the feature of the sample set including positive samples and negative samples, is the temperature parameter, N represents the total number of samples in the sample set, and e represents the natural constant. is the similarity function, and the calculation formula is: ,in represents the dot product of vectors, and They represent the magnitude of the vector respectively;

[0083] The steps for obtaining the positive sample feature pair are as follows:

[0084] First, the data is preprocessed, and a convolutional neural network is used to extract high-level feature representations of the image, and the sensor data is calibrated and feature extracted;

[0085] Then, compare all samples in the dataset and check their quality and safety level labels;

[0086] When a sample with an excellent quality safety level is found, the different modal features of the sample, including image features and sensor features, are combined as a positive sample feature pair;

[0087] In the Bayesian reasoning method of decision-level fusion, a conditional probability model is first established for each modal data, the prior probability is determined based on historical data and expert knowledge, and then the posterior probability is calculated using the Bayesian formula, and the final fusion decision is obtained by combining the posterior probabilities of each modality. The Bayesian formula is: ,in is the posterior probability, is the likelihood probability, is the prior probability, It is the probability of evidence.

[0088] In the present invention, in the vertical large model construction step:

[0089] The specific layer for the field of quality and safety plays a vital role in improving the professionalism and pertinence of the entire model. Among them, the quality feature extraction layer cleverly uses the attention mechanism, which is like a precise filter. When processing large amounts of data, it can assign different weights according to the degree to which the data is related to quality, thereby focusing attention on those key features that really affect quality. For example, when analyzing product production data, it can highlight parameters that are closely related to product quality standards, such as dimensional accuracy data of parts and components, composition ratio data of raw materials, etc., and appropriately weaken those secondary information, thereby improving the model's ability to capture and analyze quality features.

[0090] The safety rule embedding layer innovatively handles complex safety regulations and standards. It cleverly embeds these rules and standards into the model parameters in the form of vectors. In this way, the model can inherently follow safety regulations during operation, rather than relying solely on later safety checks. For example, safety parameters in chemical production, such as safety thresholds of temperature and pressure, and regulations for handling hazardous substances, are all integrated into the model in the form of vectors, so that the model naturally takes safety factors into account when analyzing and predicting the production process, greatly enhancing the model's ability to control safety issues.

[0091] The adaptive learning rate uses the AdamW optimization algorithm, which has shown excellent performance in the model training process. It can dynamically adjust the learning rate according to the gradient information continuously generated during the model training process. In the early stage of training, when the model parameters are far from the optimal value, it can set a larger learning rate so that the model can quickly adjust the parameters in the optimal direction and accelerate the convergence speed. As the training progresses, when the model gradually approaches the optimal solution, the learning rate will automatically decrease to avoid crossing the optimal solution due to excessive learning rate, causing the model to oscillate back and forth near the optimal value. At the same time, the gradient clipping technology is like a safety guard, which strictly limits the norm of the gradient to a preset range. In the process of model training, especially when dealing with complex quality and safety data, the problem of gradient explosion may occur, which will cause the model training to become extremely unstable or even unable to converge. The gradient clipping technology effectively prevents this from happening and ensures the stability and reliability of model training.

[0092] In the present invention, in the quality and safety analysis step:

[0093] Quality defect prediction is based on the quality dimension indicators in a complete quality safety indicator system. This indicator system covers quality-related parameters from raw material quality to each link in the production process. In the prediction process, a powerful classification model is used, which is an integrated model based on support vector machine and random forest. Support vector machine has excellent classification ability, especially when processing high-dimensional data and small sample data. It can distinguish different categories of data by finding the optimal hyperplane. Random forest is an integrated learning method based on multiple decision trees. It improves the generalization ability and stability of the model by integrating the results of multiple decision trees. The combination of these two methods makes the classification model more accurate and reliable in predicting the type and probability of quality defects. For example, when predicting the quality defects of solder joints in the production of electronic equipment, the model can comprehensively consider multiple quality dimension indicators such as the shape, size, welding temperature, and welding time of the solder joints, and accurately determine the possible types of defects such as cold solder joints and leaking solder joints and their probability of occurrence.

[0094] Safety risk assessment is based on safety dimension indicators, which comprehensively cover various safety-related factors in the production process. The risk level is calculated by using a carefully constructed risk assessment matrix, which is based on a large amount of historical accident data and strict safety standards. Historical accident data provides a rich practical basis for risk assessment, and safety standards ensure the standardization and scientific nature of the assessment. For example, in the field of construction, the risk assessment matrix will consider safety dimension indicators such as high altitude working height, personnel density at the construction site, and the operating status of construction equipment. Through the quantitative analysis of these indicators, combined with the risk assessment matrix, the safety risk level in the construction process can be accurately assessed, such as low risk, medium risk or high risk, providing strong decision-making support for safety management.

[0095] The early warning of potential problems is achieved through the time series analysis of quality and safety indicators. The hidden Markov model is used, which has a unique advantage in processing time series data with implicit states. It can identify abnormal change trends of quality and safety indicators in time series. In the production process, quality and safety indicators usually show certain rules and trends, and when potential problems occur, these indicators will change subtly. The hidden Markov model can keenly capture these abnormal changes through dynamic analysis of indicator data. For example, in the process of mechanical processing, if the wear of the tool gradually increases, then the quality and safety indicators related to the tool, such as processing accuracy, processing surface roughness, etc., will show a specific change pattern in the time series. The hidden Markov model can detect the changes in this pattern in time and send out early warning signals in advance, so that timely measures can be taken to avoid quality problems or safety accidents.

[0096] In the present invention, in the optimization suggestion generation step:

[0097] The rule-based reasoning engine is an important part of the entire optimization suggestion generation process. It contains a carefully designed quality and safety rule base. The rules in this rule base are expressed in a clear "if-then" form. For example, "If the temperature of a certain production equipment exceeds the set threshold continuously, it is recommended to check and maintain the cooling system of the equipment." In actual operation, according to the results output by the quality and safety analysis module, the system automatically matches the corresponding rules in the rule base. These rules are summarized based on long-term production practice experience, industry standards and expert knowledge. In this way, the system can quickly generate preliminary optimization suggestions and provide a basic direction for subsequent more in-depth optimization.

[0098] The machine learning algorithm uses a reinforcement learning algorithm, which regards the complex quality and safety status as a dynamically changing environment, and the optimization suggestions as actions taken in this environment. Through a cleverly designed reward mechanism, the system can learn the optimal optimization suggestion generation strategy. This reward mechanism is determined based on the improvement of quality and safety indicators after the implementation of the optimization suggestions. For example, if the quality defect rate is significantly reduced or the safety risk level is effectively improved after the implementation of a certain optimization suggestion, the system will give a corresponding positive reward, otherwise a negative reward. By constantly trying different optimization suggestions in this quality and safety environment and learning based on reward feedback, the system can gradually find the optimal suggestion generation strategy for different quality and safety conditions, thereby generating more realistic and targeted optimization suggestions.

[0099] In the present invention, a quality safety optimization system based on multi-modal vertical large model technology includes:

[0100] The data acquisition module is the information entry of the entire system. It undertakes the important task of collecting quality and safety related data from multiple data sources of different modes, and performs corresponding preprocessing on different types of data. The multimodal data sources mentioned here cover a wide variety of data types, including but not limited to text data, image data, audio data, sensor data, etc. These data sources are widely distributed in various links of the production process. For example, the production equipment sensors can obtain the operating parameters of the equipment in real time. These parameters are crucial to judging whether the equipment is operating normally and the impact on product quality; the quality inspection report records the detailed inspection results of product quality in text form, including key information such as whether the quality is qualified and the type of quality problems; the image data captured by the monitoring camera can intuitively reflect the actual situation of the production site, such as whether the workers' operations are standardized and whether there are abnormal objects around the equipment; the operation record document records the process and parameter settings of the production operation in detail, providing an important basis for analyzing quality and safety issues.

[0101] For different types of data, each subunit in the data acquisition module will perform professional preprocessing. Among them, the text processing unit will use advanced natural language processing technology to preprocess the text data. In this process, lexical analysis will decompose the text into words and identify their parts of speech, such as accurately distinguishing the noun "quality indicators" and the verb "detection" in the quality inspection report. Syntactic analysis further analyzes the grammatical relationship between words and constructs the grammatical structure of the sentence, which helps to understand the semantics of the text. Entity recognition can accurately identify entities related to quality and safety in the text, such as quality standard entities (such as qualified standard parameters of products), safety specification entities (such as specific regulations in safe operation procedures), production equipment entities (such as the name and model of equipment), raw material entities (such as the type and specifications of raw materials), etc. Through these preprocessing steps, text data is converted into a form that computers can better understand and process, providing clear and accurate information for subsequent analysis.

[0102] The image processing unit is responsible for processing the image data. It first performs feature extraction, using advanced algorithms to extract valuable features from the image, such as the shape features of the object (important for identifying whether the appearance of the production equipment is normal and whether the shape of the product meets the specifications), texture features (can be used to detect quality problems on the surface of the product, such as scratches, cracks, etc.), color distribution features (play a role in the quality inspection of some products with strict color requirements), etc. Then, through normalization processing, the image pixel values ​​are mapped to a specific interval, so that image data from different sources, different resolutions and lighting conditions have the same scale, which is convenient for subsequent fusion and analysis. In this way, whether the image data comes from different models of surveillance cameras or different shooting environments, it can be processed under a unified standard, which improves the consistency and comparability of the data.

[0103] The audio processing unit processes audio data. In a production environment, audio data may contain a wealth of information. For example, the sound characteristics of the equipment when it is running can reflect whether the equipment is faulty. The audio processing unit first uses advanced noise reduction methods such as wavelet transform to remove background noise and interference signals in the audio, making the useful audio information clearer. Then, through spectrum analysis, the frequency components and amplitude information of the audio are obtained using technologies such as fast Fourier transform. For example, by analyzing the spectrum of the sound of the machine running, it can be determined whether the machine has problems such as loose parts and wear. This information is of great value for quality and safety analysis.

[0104] The sensor data processing unit is responsible for processing data from various sensors. Since sensors may deviate during long-term use, the data must first be calibrated according to the sensor's calibration curve and calibration equation to ensure data accuracy. At the same time, outliers may appear in the sensor data, which may be caused by sensor failure or other emergencies. To address this problem, the 3σ principle based on statistical methods or the anomaly detection algorithm based on machine learning is used to handle outliers, ensure the quality of sensor data, and provide reliable data support for subsequent quality and safety analysis.

[0105] The multimodal data fusion module is a bridge connecting different modal data. It is responsible for fusing the pre-processed different modal data based on the feature-level fusion and decision-level fusion methods to generate fused quality and safety data representation. This fusion method gives full play to the advantages of different modal data, excavates the inherent relationship between them, and provides more comprehensive and valuable data for subsequent quality and safety analysis and optimization.

[0106] Among them, the deep learning model of the feature-level fusion unit is a hybrid architecture of convolutional neural networks and long short-term memory networks. This hybrid architecture combines the advantages of convolutional neural networks in processing spatially structured data such as images and the expertise of long short-term memory networks in processing sequence data. In the fusion process, by constructing multimodal feature vectors, the data features of different modalities are represented in vector form, and the joint feature representation is extracted using this deep learning model. For example, combining the feature vector of image data with the time series feature vector of sensor data enables the model to learn the relationship between the state of the object in the image and the change in the physical quantity reflected by the sensor, thereby mining deeper quality and safety information.

[0107] The decision-level fusion unit makes a comprehensive decision based on the independent decision results of different modal data through the Bayesian reasoning method. Each modal data has its own characteristics and decision-making capabilities. For example, text data may make a preliminary judgment on quality issues through natural language processing, and image data may judge the status of equipment or products based on the appearance characteristics of the object. The decision-level fusion unit establishes a conditional probability model for each modal data. These models are constructed based on a priori probabilities determined by a large amount of historical data and expert knowledge. Then the posterior probability is calculated using the Bayesian formula, and the final fusion decision is obtained by combining the posterior probabilities of each modality. This fusion method can make full use of the independent judgment ability of each modal data, and at the same time organically combine them through Bayesian reasoning, thereby improving the accuracy and reliability of decision-making.

[0108] The vertical large model building module is one of the core parts of the entire system. It is responsible for building a vertical large model using the fused quality and safety data representation. The architecture of this vertical large model is based on the improved Transformer architecture, and specific layers and parameters are adjusted for the quality and safety field to better meet the needs of quality and safety analysis and optimization.

[0109] The specific layer building unit is specifically used to build specific layers for the quality and safety field, including the quality feature extraction layer and the safety rule embedding layer. The quality feature extraction layer focuses on quality-related features through the sophisticated use of the attention mechanism. As mentioned earlier, it can highlight information closely related to quality in complex data and provide strong support for quality analysis. The safety rule embedding layer embeds safety specifications and standards into model parameters in the form of vectors, making safety requirements an inherent attribute of the model and ensuring that the model always follows safety principles during analysis and prediction.

[0110] The training optimization unit plays a key role in the model training process. It uses adaptive learning rate and gradient clipping technology to prevent gradient explosion and accelerate convergence. The adaptive learning rate dynamically adjusts the learning rate according to the gradient information during the model training process through the AdamW optimization algorithm. At different stages of training, the learning rate can automatically adapt to the optimization needs of the model to ensure the efficiency of training. The gradient clipping technology limits the norm of the gradient to a preset range to avoid model instability caused by excessive gradients during training. Especially when processing complex quality and safety data, this technology effectively ensures the smooth progress of model training and improves the training quality and stability of the model.

[0111] The quality and safety analysis module is a key module in the system for in-depth analysis of quality and safety conditions. Its main function is to input new multimodal data to be analyzed into the constructed vertical large model, and output analysis results related to quality and safety, including quality defect prediction, safety risk assessment, potential problem warning and other important contents, providing a basis for the subsequent generation of optimization suggestions.

[0112] The quality defect prediction unit works based on the quality dimension indicators in the quality safety indicator system. The classification model it uses is an integrated model based on support vector machines and random forests. This powerful model combination can accurately predict the types and probabilities of possible quality defects. In practical applications, it will comprehensively consider many quality dimension indicators, such as the quality parameters of raw materials, production process parameters, and the physical and chemical properties of products. Through the analysis and model calculation of these indicators, it outputs the prediction results of quality problems such as whether the product has surface defects, internal structural defects, and the corresponding probabilities, providing powerful guidance for quality control.

[0113] The safety risk assessment unit calculates the risk level based on the safety dimension indicators using a carefully constructed risk assessment matrix. This risk assessment matrix is ​​constructed based on a large amount of historical accident data and strict safety standards, covering safety factors in all aspects of the production process. During the assessment process, the safety risk assessment unit will conduct quantitative analysis of safety dimension indicators such as equipment operating status, personnel operating specifications, environmental factors, etc., and determine the risk level in combination with the risk assessment matrix, for example, to determine whether a certain production link is at low risk, medium risk or high risk, so that the company can take corresponding safety measures in a timely manner.

[0114] The potential problem early warning unit uses the hidden Markov model to identify abnormal change trends of indicators through time series analysis of quality and safety indicators. In the production process, the changes in quality and safety indicators usually have certain regularities. The potential problem early warning unit can keenly capture abnormal fluctuations in indicators in the time series through long-term monitoring and analysis of these indicators and the advantages of the hidden Markov model. For example, when certain key parameters of production equipment show continuous small changes, which may indicate that the equipment is about to fail or product quality problems are about to occur, the potential problem early warning unit can detect this trend in time and issue early warning signals in advance, which can buy valuable time for enterprises to take preventive measures and avoid quality accidents and safety accidents.

[0115] The optimization suggestion generation module is an important module that provides enterprises with specific optimization measures based on the results of quality and safety analysis. It generates targeted quality and safety optimization suggestions through a rule-based reasoning engine and machine learning algorithm. These optimization suggestions include process parameter adjustments, equipment maintenance plans, personnel training programs, etc., and each optimization suggestion is accompanied by an expected effect evaluation and implementation priority, providing comprehensive support for the company's quality and safety management decisions.

[0116] The rule reasoning unit generates preliminary optimization suggestions through a rule-based reasoning engine. The quality and safety rule base in the reasoning engine contains a large number of rules expressed in the form of "if-then", which are summarized based on long-term production practice experience, industry standards and expert knowledge. When the quality and safety analysis results are input into the rule reasoning unit, it will automatically match the corresponding rules in the rule base. For example, if the quality analysis results show that a certain quality indicator of the product does not meet the standard, the rule reasoning unit may adjust the production process parameters or inspect and maintain the production equipment according to the relevant rules in the rule base. These preliminary optimization suggestions provide direction and basis for subsequent in-depth optimization.

[0117] The machine learning optimization unit uses a reinforcement learning algorithm to learn the optimal optimization suggestion generation strategy. It regards the complex quality and safety status as a dynamically changing environment, and the optimization suggestions as actions taken in this environment. Through a carefully designed reward mechanism, the reward value is determined according to the improvement of quality and safety indicators after the implementation of the optimization suggestions. For example, if the quality defect rate is significantly reduced or the safety risk level is effectively improved after the implementation of a certain optimization suggestion, then the machine learning optimization unit will give a corresponding positive reward, otherwise it will give a negative reward. By constantly trying different optimization suggestions in this quality and safety environment and learning based on reward feedback, the machine learning optimization unit can gradually find the optimal suggestion generation strategy for different quality and safety conditions, thereby generating more realistic and targeted optimization suggestions, helping enterprises to more effectively improve their quality and safety levels.

[0118] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can replace or change the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A quality safety optimization method based on multi-modal vertical large model technology, characterized in that: The following steps are involved: S1, data collection step: collecting data related to quality and safety from multiple data sources of different modes, wherein the multiple modes include text data, image data, audio data, and sensor data, and the data sources include production equipment sensors, quality inspection reports, surveillance cameras, and operation record documents, wherein the text data is preprocessed using natural language processing technology, including lexical analysis, syntactic analysis, and entity recognition; the image data is subjected to feature extraction and normalization processing; the audio data is subjected to noise reduction and spectrum analysis; and the sensor data is subjected to data calibration and outlier processing; S2. Multimodal data fusion step: The pre-processed different modal data are fused based on the feature-level fusion and decision-level fusion methods. The feature-level fusion constructs a multimodal feature vector and uses a deep learning model to extract joint feature representations. The decision-level fusion makes a comprehensive decision based on the independent decision results of different modal data through the Bayesian reasoning method to generate a fused quality and safety data representation. In the feature-level fusion, the fusion weight of the multimodal feature vector is calculated. The formula is: ,in Indicates Modal features With reference feature The similarity measure of To control the parameters of weight distribution, is the number of modes, represents a natural constant, The way to obtain is: ,in represents the dot product of vectors, and They represent the magnitude of the vector respectively; S3, vertical large model construction step: Use the fused quality and safety data to represent the vertical large model. The architecture of the vertical large model is based on the Transformer architecture improvement, including specific layers and parameter adjustments for the quality and safety field, where the specific layers are used for reinforcement learning of key features of quality and safety. The adaptive learning rate and gradient clipping technology are used in the model training process. In the vertical large model training, the adaptive learning rate The update formula uses the AdamW algorithm: ,in is the initial learning rate, , are algorithm parameters, and The t in represents the number of training steps. Indicates that as the number of training steps increases, the algorithm parameters The specific value at step t, Indicates that as the number of training steps increases, the algorithm parameters The specific value at step t, , is the modified gradient first-order moment estimate and second-order moment estimate, is a small constant to prevent division by zero, and the gradient clipping formula is: ,in is the gradient vector, is the clipping threshold; S4, quality and safety analysis step: input the new multimodal data to be analyzed into the constructed vertical large model, and the model outputs the analysis results related to quality and safety, including quality defect prediction, safety risk assessment, and potential problem warning. The analysis results are based on a predefined quality and safety indicator system, which includes dimensions from raw material quality to production process safety and finished product quality. The defect probability is calculated in the quality defect prediction. The formula is: ,in is the weight of the sth decision tree in the ensemble model, is the number of decision trees, is the defect probability predicted by the s-th decision tree. The s-th decision tree is trained using multimodal data samples with quality defect annotations. During the training process, the decision tree learns and optimizes according to the characteristics of the data and the quality defect labels. When predicting the defect probability of new data, the new data is input into the trained decision tree. The decision tree will classify and judge the new data according to the rules and patterns it has learned. If the decision tree classifies the new data into a defect category, a defect probability value of 0.7 is set; if it is classified as a non-defect category, a lower defect probability value of 0.3 is set. This probability value is the defect probability predicted by the s-th decision tree; the risk value of safety risk assessment The calculation formula is: ,in is the probability of an accident happening, is the degree of accident impact, and the two are determined according to the risk assessment matrix; the observation probability formula for calculating the hidden Markov model in potential problem warning is: ,in is the state transition probability, is in state Observed below The probability of is the number of states; S5. Optimization suggestion generation step: Based on the quality and safety analysis results, targeted quality and safety optimization suggestions are generated through a rule-based reasoning engine and machine learning algorithm. The optimization suggestions include process parameter adjustment, equipment maintenance plan, and personnel training plan. The optimization suggestions are accompanied by expected effect evaluation and implementation priority. In the generation of optimization suggestions by reinforcement learning, the value function The update formula is: ,in is the learning rate, It’s an instant reward. is the discount factor, is the current state, is the next state; the learning rate is within a predetermined range of 0.001 to 0.1, with a step size of 0.005 for grid search, and the model is trained using different learning rates, and the final learning rate is selected based on the accuracy of the model on the validation set and the performance of the loss function value; the instant reward is obtained by quantifying indicators related to quality and safety, and determining the instant reward based on changes in the indicators, including setting a fixed reward value for every 1% increase in product qualification rate and a fixed reward value for every reduction in the number of safety accidents; the discount factor is between 0 and 1, and is determined based on the time span of the problem and the importance attached to future rewards.

2. The quality safety optimization method based on multi-modal vertical large model technology according to claim 1 is characterized in that: In the data collection step: For the preprocessing of text data, a pre-trained language model is used for word embedding representation. Lexical analysis is implemented based on finite state automata. Syntactic analysis uses a dependency syntax analysis algorithm. Entity recognition uses a conditional random field model. The recognized entities include quality standard entities, safety specification entities, production equipment entities, and raw material entities. Convolutional neural networks are used to extract features from image data. The extracted features include object shape, texture, and color distribution. Normalization processing maps image pixel values ​​to a specific interval so that image data from different sources have the same scale. Wavelet transform is used to reduce the noise of audio data, and spectrum analysis obtains the frequency components and amplitude information of audio through fast Fourier transform; The calibration of sensor data is carried out according to the sensor's calibration curve and calibration equation, and the outlier processing adopts the 3σ principle based on statistical methods or the anomaly detection algorithm based on machine learning.

3. The quality safety optimization method based on multi-modal vertical large model technology according to claim 1 is characterized in that: In the multimodal data fusion step: The deep learning model of feature-level fusion is a hybrid architecture of convolutional neural network and long short-term memory network. The input is the feature vector of different modal data, and the output is the fused joint feature vector. The contrastive learning loss function is used in the training process to enhance the correlation between the features of different modal data. The formula is: ,in , is a positive sample feature pair, is the feature of the sample set including positive samples and negative samples, is the temperature parameter, N represents the total number of samples in the sample set, and e represents the natural constant. is the similarity function, and the calculation formula is: ,in represents the dot product of vectors, and They represent the magnitude of the vector respectively; The steps for obtaining the positive sample feature pair are as follows: First, the data is preprocessed, and a convolutional neural network is used to extract high-level feature representations of the image, and the sensor data is calibrated and feature extracted; Then, compare all samples in the dataset and check their quality and safety level labels; When a sample with an excellent quality safety level is found, the different modal features of the sample, including image features and sensor features, are combined as a positive sample feature pair; In the Bayesian reasoning method of decision-level fusion, a conditional probability model is first established for each modal data, the prior probability is determined based on historical data and expert knowledge, and then the posterior probability is calculated using the Bayesian formula. The final fusion decision is obtained by combining the posterior probabilities of each modality. The Bayesian formula is: ,in is the posterior probability, is the likelihood probability, is the prior probability, It is the probability of evidence.

4. The quality safety optimization method based on multi-modal vertical large model technology according to claim 1 is characterized in that: In the vertical large model construction step: The specific layers for the quality and safety field include the quality feature extraction layer and the safety rule embedding layer. The quality feature extraction layer focuses on quality-related features through the attention mechanism, and the safety rule embedding layer embeds safety specifications and standards into the model parameters in the form of vectors. The adaptive learning rate uses the AdamW optimization algorithm to dynamically adjust the learning rate according to the gradient information during model training. The gradient clipping technology limits the norm of the gradient to a preset range to prevent gradient explosion from causing unstable model training.

5. The quality safety optimization method based on multi-modal vertical large model technology according to claim 1 is characterized in that: In the quality and safety analysis step: Quality defect prediction is based on the quality dimension indicators in the quality safety indicator system. The possible quality defect types and probabilities are predicted through a classification model. The classification model is an integrated model based on support vector machine and random forest. Safety risk assessment is based on safety dimension indicators and uses a risk assessment matrix to calculate risk levels. The risk assessment matrix is ​​constructed based on historical accident data and safety standards; Potential problem warning is achieved through time series analysis of quality and safety indicators, using hidden Markov models to identify abnormal trends in indicators and issue warning signals in advance.

6. The quality safety optimization method based on multi-modal vertical large model technology according to claim 1 is characterized in that: In the optimization suggestion generation step: The rule-based reasoning engine contains a quality and safety rule base, in which the rules are expressed in the form of if and then, and preliminary optimization suggestions are generated based on the matching rules according to the quality and safety analysis results; The machine learning algorithm adopts a reinforcement learning algorithm, takes the quality and safety status as the environment and the optimization suggestions as the actions, and learns the optimal optimization suggestion generation strategy through a reward mechanism. The reward mechanism is determined based on the improvement of quality and safety indicators after the implementation of the optimization suggestions.

7. A quality safety optimization system based on multimodal vertical large model technology, used to implement the quality safety optimization method based on multimodal vertical large model technology according to any one of claims 1 to 6, characterized in that: include: The data acquisition module is used to collect data related to quality and safety from multiple data sources of different modes and pre-process different types of data. The multi-modalities include text data, image data, audio data, and sensor data. The data sources include production equipment sensors, quality inspection reports, surveillance cameras, and operation record documents. The multimodal data fusion module is used to fuse the pre-processed data of different modalities based on the feature-level fusion and decision-level fusion methods to generate fused quality and safety data representation; A vertical large model building module, used to build a vertical large model using the fused quality and safety data representation, wherein the architecture of the vertical large model is improved based on the Transformer architecture, including specific layers and parameter adjustments for the quality and safety field; The quality and safety analysis module is used to input new multimodal data to be analyzed into the constructed vertical large model and output analysis results related to quality and safety, including quality defect prediction, safety risk assessment, and potential problem warning; The optimization suggestion generation module is used to generate targeted quality and safety optimization suggestions based on the quality and safety analysis results through a rule-based reasoning engine and machine learning algorithm. The optimization suggestions include process parameter adjustments, equipment maintenance plans, and personnel training plans, and the optimization suggestions are accompanied by expected effect evaluation and implementation priorities.

8. The quality safety optimization system based on multi-modal vertical large model technology according to claim 7 is characterized in that: The data acquisition module comprises: A text processing unit, used to pre-process text data using natural language processing technology, including lexical analysis, syntactic analysis, and entity recognition; An image processing unit, used for performing feature extraction and normalization processing on image data; An audio processing unit for performing noise reduction and spectrum analysis on audio data; The sensor data processing unit is used to perform data calibration and outlier processing on the sensor data.

9. The quality safety optimization system based on multi-modal vertical large model technology according to claim 7 is characterized in that: The multimodal data fusion module includes: Feature-level fusion unit, whose deep learning model is a hybrid architecture of convolutional neural network and long short-term memory network, is used to extract joint feature representation by constructing multimodal feature vectors using deep learning models; The decision-level fusion unit is used to make comprehensive decisions based on the independent decision results of different modal data through the Bayesian reasoning method.

10. The quality safety optimization system based on multi-modal vertical large model technology according to claim 7 is characterized in that: The vertical large model building module includes: Specific layer construction unit, used to construct specific layers for the quality and safety field, including quality feature extraction layer and safety rule embedding layer; The training optimization unit is used to use adaptive learning rate and gradient clipping techniques during model training to prevent gradient explosion and speed up convergence; The quality and safety analysis module includes: The quality defect prediction unit is used to predict the type and probability of possible quality defects based on the quality dimension indicators in the quality safety indicator system through a classification model. The classification model is an integrated model based on support vector machine and random forest; A security risk assessment unit is used to calculate the risk level using a risk assessment matrix based on security dimension indicators; Potential problem early warning unit, which is used to analyze the time series of quality and safety indicators, identify abnormal change trends of indicators using hidden Markov models, and issue early warning signals; The optimization suggestion generation module includes: A rule reasoning unit, used to generate preliminary optimization suggestions through a rule-based reasoning engine, the reasoning engine including a quality and safety rule base; The machine learning optimization unit is used to learn the optimal optimization suggestion generation strategy using a reinforcement learning algorithm and determine the optimization suggestions based on the quality and safety status and reward mechanism.

Images