A method for optimizing manufacturing product production based on multimodal large models

Abstract

The present invention relates to a method for optimizing the production of manufacturing products based on a multimodal large model. The method steps include: judging whether the current product production line state is abnormal according to the deep fusion analysis results; if the current product production line state is normal, archiving and backing up the multiple modal original data and the deep fusion analysis results; if the current product production line state is abnormal, analyzing the modal data of the shared semantic subspace through the multimodal large model to obtain production fault information; classifying the production fault information to obtain the fault type corresponding to the production fault information. The correlation and complementarity between multimodal data can be fully utilized to obtain comprehensive and accurate analysis results, and the multimodal large model's ability to understand and analyze multimodal data can be fully utilized, thereby providing data support for optimizing production line resource scheduling.

Description

Technical Field

[0001] The invention relates to a manufacturing product production optimization method based on a multi-modal large model, and belongs to the field of manufacturing automation production. Background Art

[0002] Manufacturing is the main body of the national economy, the core of the real economy, and the key to revitalizing the nation. In the traditional manufacturing industry production line, every process of product production is a key link that affects production efficiency. How to optimize and upgrade each assembly line link is the core step to improve product production efficiency. With the development of Internet of Things technology, through a series of sensors and monitoring equipment, enterprises can record the data generated by each production link, but how to process and analyze these data has become a new technical difficulty. The large amount of monitoring data generated by the production line every day brings huge work pressure to data analysis technicians. Therefore, it is urgent to develop a manufacturing product production optimization method based on a multimodal large model, combining the data understanding and data analysis capabilities of the multimodal large model to assist enterprise data analysis technicians in analyzing the efficiency bottlenecks of product production lines, and then optimize product production steps. Summary of the invention

[0003] The present invention provides a method for optimizing the production of manufacturing products based on a multimodal large model, aiming to solve at least one of the technical problems existing in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides a method for optimizing manufacturing product production based on a multimodal large model, the method comprising the following steps:

[0005] Collecting multiple modal raw data associated with the product production process from the IoT devices; the multiple modal raw data is unprocessed multiple modal raw data obtained from at least one of the IoT devices in the product production line;

[0006] Processing each type of the modal raw data by a preprocessing module respectively, establishing association relationship information between different modal data by timestamp, and obtaining preprocessed modal data; the association relationship information is used to characterize the internal connection or mutual relationship between different modal data;

[0007] Inputting the pre-processed modal data into the pre-trained multi-modal large model to perform deep fusion analysis to obtain deep fusion analysis results;

[0008] Determine whether the current product production line status is abnormal based on the deep fusion analysis result; if the current product production line status is normal, archive and back up the multiple modal original data and the deep fusion analysis result; if the current product production line status is abnormal, analyze the modal data of the shared semantic subspace through the multimodal large model to obtain production fault information;

[0009] Classify the production fault information to obtain the fault type corresponding to the production fault information;

[0010] Determine whether to execute the intelligent optimization decision-making process according to the fault type, query and perform secondary fusion analysis on the deep fusion analysis results through the multimodal large model, analyze whether there are abnormal indicators in the shared semantic subspace, and generate maintenance suggestions for the abnormal indicators.

[0011] Optionally, processing the modal raw data by a preprocessing module includes:

[0012] Performing missing value processing on each type of the original data of the modality, and performing outlier processing on each type of the original data of the modality;

[0013] The step of processing missing values ​​for each type of the modal original data comprises: deleting missing values ​​from the modal original data, or filling missing values ​​with fixed values, or calculating the average value of the modal original data to fill missing values, or calculating the median of the modal original data to fill missing values;

[0014] The step of performing outlier processing on the modal raw data includes: identifying outliers and deleting the outliers.

[0015] Optionally, the deep fusion analysis includes the following steps:

[0016] Based on the joint structure of the multimodal large model, encoding the single modal data to obtain the single modal semantics; the single modal data is the data in the preprocessed modal data;

[0017] Projecting the unimodal semantic representation into a shared semantic subspace, and projecting the preprocessed modal data into the shared semantic subspace;

[0018] Based on the logistic regression binary classification algorithm of the multimodal large model, the single modality data is used as a feature vector, the weighted sum of each element in the feature vector is taken, and the activation function is input to obtain the probability value of the current product production line status being abnormal.

[0019] Optionally, the product production line includes a plurality of production equipment, and classifying the production fault information to obtain a fault type corresponding to the production fault information includes:

[0020] Using the multi-modal large model to classify different production fault information to obtain different fault categories;

[0021] After classifying the production fault information and obtaining the fault type corresponding to the production fault information, the method further includes:

[0022] Search and summarize the fault repair manual to determine the faulty production equipment;

[0023] According to the positioning coordinate data corresponding to the faulty production equipment, the location of the faulty production equipment is determined, and an alarm message is generated. Optionally, before collecting multiple modal raw data associated with the product production process from the Internet of Things device, it also includes:

[0024] Inputting training data into the multimodal large model to be trained for pre-training; the training data is the training modality data pre-processed by the pre-processing module;

[0025] Constructing a fine-tuning data set, wherein the fine-tuning data set includes preset product production line status indicator annotations and text processing annotations of production equipment status data;

[0026] Inputting the fine-tuning data set into the multimodal large model to be trained to obtain prediction results of product production line status indicator annotation and text processing prediction results of production equipment status data;

[0027] The multimodal large model is trained according to the prediction results of the product production line status indicator annotation and the text processing prediction results of the production equipment status data until convergence to obtain the multimodal large model.

[0028] Optionally, the multimodal large model to be trained is obtained according to the prediction result of the product production line status indicator labeling and the text processing prediction result of the production equipment status data until convergence, including:

[0029] When the multimodal large model is trained according to the prediction results of the product production line status indicator labeling and the text processing prediction results of the production equipment status data, the feedback information is iteratively revised according to the product manufacturing management personnel's annotations of the product production line status indicator and the text processing annotations of the production equipment status data;

[0030] The determined and revised product production line status indicator annotations and text processing annotations of production equipment status data are input into the multimodal large model, the output results are compared with the prediction results of the product manufacturing management personnel, and training is carried out until the product manufacturing management personnel are satisfied with the indicators, so as to obtain the multimodal large model.

[0031] In a second aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the processor implements the above method when executing a computer program stored in the memory.

[0032] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above-mentioned method when executed by a processor.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the processor implements the above method when executing a computer program stored in the memory.

[0034] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor.

[0035] The beneficial effects of the present invention are as follows.

[0036] Through the above-mentioned manufacturing product production optimization method based on multimodal big model, after preprocessing a variety of different modal data, various types of data are fused and analyzed in real time through the multimodal big model. The correlation and complementarity between multimodal data can be fully utilized to obtain comprehensive and accurate analysis results, and the multimodal big model's ability to understand and analyze multimodal data can be fully utilized, thereby providing data support for optimizing production line resource scheduling, making intelligent decisions in the event of equipment failure to shorten equipment failure repair time, and improving production line production efficiency.

[0037] In addition, additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is an overall flow chart of a manufacturing product production optimization method based on a multimodal large model according to an embodiment of the present invention.

[0039] Figure 2 It is an architecture diagram of a large model for manufacturing production optimization based on multimodality according to an embodiment of the present invention.

[0040] Figure 3 It is a big model modeling flow chart based on multimodal manufacturing product production optimization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0042] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0043] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, will not impose limitations on the scope of the present invention.

[0044] Reference Figures 1 to 3 In some embodiments, the present invention discloses a method for optimizing manufacturing product production based on a multimodal large model, the method comprising the following steps:

[0045] Step S1, collecting multiple modal raw data associated with a product production process from an IoT device; the multiple modal raw data is unprocessed multiple modal raw data obtained from at least one IoT device in a product production line;

[0046] Step S2, processing each type of the modal raw data by a preprocessing module, establishing association relationship information between different modal data by timestamp, and obtaining preprocessed modal data; the association relationship information is used to characterize the internal connection or mutual relationship between different modal data;

[0047] Step S3: input the pre-processed modal data into the pre-trained multi-modal large model, perform deep fusion analysis, and obtain a deep fusion analysis result; the deep fusion analysis result is the multi-modal data after deep fusion analysis;

[0048] Step S4: judging whether the current product production line status is abnormal according to the deep fusion analysis result; if the current product production line status is normal, archiving and backing up the multiple modal original data and the deep fusion analysis result; if the current product production line status is abnormal, analyzing the modal data of the shared semantic subspace through the multimodal large model to obtain production fault information;

[0049] Step S5: classify the production fault information to obtain the fault type corresponding to the production fault information;

[0050] Step S6: determine whether to execute the intelligent optimization decision-making process according to the fault type, query and perform secondary fusion analysis on the deep fusion analysis results through the multimodal large model, analyze whether there are abnormal indicators in the shared semantic subspace, and generate maintenance suggestions for the abnormal indicators.

[0051] Deploy multiple IoT devices in the product production line, and collect multiple modal data associated with the product production process from the IoT devices. The IoT devices include sensors, noise recorders, and high-definition cameras associated with the product production process, and the modal data include production equipment status data, environmental parameter data, and raw material quality data.

[0052] Through the above-mentioned manufacturing product production optimization method based on multimodal big model, after preprocessing a variety of different modal data, various types of data are fused and analyzed in real time through the multimodal big model. The correlation and complementarity between multimodal data can be fully utilized to obtain comprehensive and accurate analysis results, and the multimodal big model's ability to understand and analyze multimodal data can be fully utilized, thereby providing data support for optimizing production line resource scheduling, making intelligent decisions in the event of equipment failure to shorten equipment failure repair time, and improving production line production efficiency.

[0053] In one embodiment, processing the modal raw data by a preprocessing module includes:

[0054] Performing missing value processing on each type of the original data of the modality, and performing outlier processing on each type of the original data of the modality;

[0055] The step of processing missing values ​​for each type of the modal original data comprises: deleting missing values ​​from the modal original data, or filling missing values ​​with fixed values, or calculating the average value of the modal original data to fill missing values, or calculating the median of the modal original data to fill missing values;

[0056] The step of performing outlier processing on the modal raw data includes: identifying outliers and deleting the outliers.

[0057] In the actual data collection process, there may be a variety of reasons that lead to data loss and vacancies. How to deal with these missing data depends on the distribution characteristics of the variables and their importance in the prediction. Modal data outliers are a common situation in the data collection process. The modal data outliers include abnormal data and outliers. First, the data is abnormal due to abnormal machine operation; second, there are problems in some links in the collection process, and the data itself is out of the normal data range. This kind of data is also called an outlier.

[0058] The processing of missing values ​​in modal data includes deleting the modal data, filling with fixed values, calculating the mean and median statistics, or filling with models, such as predicting missing values ​​through regression models.

[0059] The outlier processing of the modal data includes identifying outliers and deleting the outliers. Common detection methods include simple statistical analysis, such as finding outliers based on box plots, or excluding and discarding small clusters far away from other clusters based on clustering algorithms.

[0060] After performing the above operations on the collected modal data, the quality and accuracy of the data are basically guaranteed. In addition, the correspondence and correlation between different modal data need to be established. For the correspondence between data, data features such as timestamps can often be used to select data of different modalities collected at the same time. For the correlation between data, meaningful data in different modalities can be extracted through feature engineering according to actual application scenarios, and then the correlation between modal data can be constructed.

[0061] In one embodiment, the deep fusion analysis includes the following steps:

[0062] Based on the joint structure of the multimodal large model, single modal data is encoded to obtain single modal semantics; the single modal data is data in the preprocessed modal data;

[0063] Projecting the unimodal semantic representation into a shared semantic subspace, and projecting the preprocessed modal data into the shared semantic subspace;

[0064] Based on the logistic regression binary classification algorithm of the multimodal large model, the single modality data is used as a feature vector, the weighted sum of each element in the feature vector is taken, and the activation function is input to obtain the probability value of the current product production line status being abnormal.

[0065] The product production line includes a plurality of production equipments, and the production fault information is classified to obtain the fault type corresponding to the production fault information, including:

[0066] Using the multi-modal large model to classify different production fault information to obtain different fault categories;

[0067] After classifying the production fault information and obtaining the fault type corresponding to the production fault information, the method further includes:

[0068] Search and summarize the fault repair manual to determine the faulty production equipment;

[0069] According to the positioning coordinate data corresponding to the faulty production equipment, the position of the faulty production equipment is determined and alarm information is generated.

[0070] The pre-processed modal data is input into the multimodal large model for deep fusion analysis to determine whether the current product production line status is abnormal; if the current product production line status is normal, the collected original multi-modal data and the multi-modal data after fusion analysis are archived and backed up; if the current product production line status is abnormal, the modal data of the current shared semantic subspace is analyzed by the multi-modal large model, and production fault information is obtained through multi-classification algorithms such as random forest and K nearest neighbor algorithm, and different fault information is classified.

[0071] The product production line includes multiple production equipment, and the positioning coordinate data of the production equipment is marked. After the fault information is classified, the multimodal large model is used to search and summarize according to different fault categories in combination with the fault repair manual, and the production equipment that caused the fault is output. Combined with the positioning coordinate data marked on the production equipment, the position that caused the production fault is located, and an alarm message is generated, and the alarm message is notified to the technician.

[0072] Specifically, the deep fusion analysis includes the following steps: the multimodal large model encodes the single modal data through the joint structure, projects the single modal semantic representation into the shared semantic subspace, and projects the multimodal data into the shared semantic subspace. In the multimodal large model, the single modal data and the multimodal data are used as feature vectors through the logistic regression binary classification algorithm, and the weighted sum of each element in the feature vector is input into the activation function to obtain the probability value of the abnormal state of the current product production line. The binary classification algorithm in the decision tree is a tree established based on strategy selection. Each non-leaf node represents a test on a feature attribute, each branch represents the output of this feature attribute in a certain value range, and each leaf node stores a category. The decision tree makes a decision starting from the root node, tests the corresponding feature attributes in the item to be classified in turn, and selects the output branch according to the test value until it reaches the leaf node, and the category stored in the leaf node is used as the decision result. The main idea of ​​the support vector machine binary classification algorithm is to establish an optimal decision hyperplane so that the distance between the two types of samples closest to the plane on both sides of the plane is maximized, thereby providing good generalization ability for classification problems. In actual application scenarios, one or more algorithms can be selected according to the specific situation to perform integrated learning on the multimodal large model in order to improve the accuracy and robustness of anomaly detection of the large model. At the same time, based on the characteristics of industrial production, the classification performance of the multimodal large model can be optimized in combination with domain knowledge and experience to further improve the performance and accuracy of anomaly detection. This joint structural fusion analysis method can effectively narrow the heterogeneity differences between modalities while maintaining the integrity of the specific semantics of each modality. When performing deep fusion analysis on preprocessed data, this method can make full use of the complementarity between different modalities, reduce information loss, and improve the large model's ability to identify production status anomalies.

[0073] In one embodiment, before collecting multiple modality raw data associated with the product production process from the IoT device, the method further includes:

[0074] Inputting training data into the multimodal large model to be trained for pre-training; the training data is the training modality data pre-processed by the pre-processing module;

[0075] Constructing a fine-tuning data set, wherein the fine-tuning data set includes preset product production line status indicator annotations and text processing annotations of production equipment status data;

[0076] Inputting the fine-tuning data set into the multimodal large model to be trained to obtain prediction results of product production line status indicator annotation and text processing prediction results of production equipment status data;

[0077] The multimodal large model is trained according to the prediction results of the product production line status indicator annotation and the text processing prediction results of the production equipment status data until convergence to obtain the multimodal large model.

[0078] The multimodal large model to be trained is obtained according to the prediction result of the product production line status indicator labeling and the text processing prediction result of the production equipment status data until convergence, including:

[0079] When the multimodal large model is trained according to the prediction results of the product production line status indicator labeling and the text processing prediction results of the production equipment status data, the feedback information is iteratively revised according to the product manufacturing management personnel's annotations of the product production line status indicator and the text processing annotations of the production equipment status data;

[0080] The determined and revised product production line status indicator annotations and text processing annotations of production equipment status data are input into the multimodal large model, the output results are compared with the prediction results of the product manufacturing management personnel, and training is carried out until the product manufacturing management personnel are satisfied with the indicators, so as to obtain the multimodal large model.

[0081] Training a large multimodal model consists of the following steps:

[0082] Inputting the preprocessed partial modal data into the multimodal large model for pre-training;

[0083] A fine-tuning dataset is constructed, wherein the fine-tuning dataset includes preset product production line status indicator annotations and text processing annotations of production equipment status data.

[0084] The product production line status index and the production equipment status data are input into the multimodal large model to obtain the prediction results of the product production line status index labeling and the text processing prediction results of the production equipment status data.

[0085] The multimodal large model is trained by fine-tuning instructions based on the prediction results of the product production line status indicator annotation and the text processing prediction results of the production equipment status data.

[0086] During fine-tuning training, iterative revisions are made based on feedback from product manufacturing managers on the text processing annotations of product production line status indicators and production equipment status data.

[0087] The revised product production line status indicator annotations and text processing annotations of production equipment status data are input into the multimodal large model, and the output results are compared with the prediction results of product manufacturing managers. The training is carried out until the product manufacturing managers are satisfied with the indicators.

[0088] The above-mentioned fine-tuning dataset training enables the multimodal large model to accurately perform operations to determine whether the product production line is predicted to be normal or abnormal.

[0089] Determine whether to execute the intelligent optimization decision-making process based on the fault type. The intelligent optimization decision-making process includes querying the archived backup modal data and performing secondary fusion analysis through the multimodal large model on a daily or weekly basis, analyzing whether there are high indicators in the shared semantic subspace, and generating maintenance suggestions for high indicators. Manual review is performed to check whether there are potential faults in the equipment and prepare prevention plans in advance; the model can also be used to analyze the comprehensive energy consumption and generate intelligent production scheduling suggestions or plans to reduce costs and increase efficiency; the model can also be used to analyze the gap between the production plan and the actual production capacity, and the multimodal data generated by the production line during the production cycle can be integrated and analyzed to find the bottleneck of the product line, thereby optimizing the equipment addition or resource scheduling of the production line and improving production efficiency.

[0090] The method also discloses a computer device, comprising a memory and a processor, wherein the processor implements the above method when executing a computer program stored in the memory.

[0091] The present method also discloses a computer-readable storage medium, on which program instructions are stored, and the program instructions are implemented as described above when executed by a processor. It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-temporary computer-readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose, the program can run on a programmed dedicated integrated circuit.

[0092] In addition, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0093] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0094] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0095] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A method for optimizing manufacturing product production based on a multimodal large model, characterized in that: The method comprises the following steps: Collecting multiple modal raw data associated with the product production process from the IoT devices; the multiple modal raw data is unprocessed multiple modal raw data obtained from at least one of the IoT devices in the product production line; Processing each type of the modal raw data by a preprocessing module respectively, establishing association relationship information between different modal data by timestamp, and obtaining preprocessed modal data; the association relationship information is used to characterize the internal connection or mutual relationship between different modal data; Inputting the pre-processed modal data into the pre-trained multi-modal large model to perform deep fusion analysis to obtain deep fusion analysis results; The deep fusion analysis includes the following steps: Based on the joint structure of the multimodal large model, encoding the single modal data to obtain the single modal semantics; the single modal data is the data in the preprocessed modal data; Projecting the unimodal semantic representation into a shared semantic subspace, and projecting the preprocessed modal data into the shared semantic subspace; Based on the logistic regression binary classification algorithm of the multimodal large model, the single modality data is used as a feature vector, each element in the feature vector is weighted and summed, and input into an activation function to obtain a probability value of the current product production line state being abnormal, and the probability value is used as a deep fusion analysis result; The method further comprises: Determine whether the current product production line status is abnormal based on the deep fusion analysis result; if the current product production line status is normal, archive and back up the multiple modal original data and the deep fusion analysis result; if the current product production line status is abnormal, analyze the modal data of the shared semantic subspace through the multimodal large model to obtain production fault information; Classify the production fault information to obtain the fault type corresponding to the production fault information; Determine whether to execute the intelligent optimization decision-making process according to the fault type, query and perform secondary fusion analysis on the deep fusion analysis results through the multimodal large model, analyze whether there are abnormal indicators in the shared semantic subspace, and generate maintenance suggestions for the abnormal indicators; Before collecting raw data of multiple modalities associated with the product production process from IoT devices, it also includes: Inputting training data into the multimodal large model to be trained for pre-training; the training data is the training modality data pre-processed by the pre-processing module; Constructing a fine-tuning data set, wherein the fine-tuning data set includes preset product production line status indicator annotations and text processing annotations of production equipment status data; Inputting the fine-tuning data set into the multimodal large model to be trained to obtain prediction results of product production line status indicator annotation and text processing prediction results of production equipment status data; Training the multimodal large model to be trained according to the prediction results of the product production line status indicator labeling and the text processing prediction results of the production equipment status data until the multimodal large model is obtained through convergence; The multimodal large model to be trained is trained according to the prediction results of the product production line status indicator labeling and the text processing prediction results of the production equipment status data until convergence to obtain the multimodal large model to be trained, including: When the multimodal large model is trained according to the prediction results of the product production line status indicator labeling and the text processing prediction results of the production equipment status data, the feedback information is iteratively revised according to the product manufacturing management personnel's annotations of the product production line status indicator and the text processing annotations of the production equipment status data; The determined and revised product production line status indicator annotations and text processing annotations of production equipment status data are input into the multimodal large model, the output results are compared with the prediction results of the product manufacturing management personnel, and training is carried out until the product manufacturing management personnel are satisfied with the indicators, so as to obtain the multimodal large model.

2. The method according to claim 1, characterized in that Processing the modal raw data through a preprocessing module includes: Performing missing value processing on each type of the original data of the modality, and performing outlier processing on each type of the original data of the modality; The step of processing missing values ​​for each type of the modal original data comprises: deleting missing values ​​from the modal original data, or filling missing values ​​with fixed values, or calculating the average value of the modal original data to fill missing values, or calculating the median of the modal original data to fill missing values; The step of performing outlier processing on the modal raw data includes: identifying outliers and deleting the outliers.

3. The method according to claim 1, characterized in that The product production line includes a plurality of production equipments, and the production fault information is classified to obtain the fault type corresponding to the production fault information, including: Using the multi-modal large model to classify different production fault information to obtain different fault types; After classifying the production fault information and obtaining the fault type corresponding to the production fault information, the method further includes: Search and summarize the fault repair manual to determine the faulty production equipment; According to the positioning coordinate data corresponding to the faulty production equipment, the position of the faulty production equipment is determined and alarm information is generated.

4. A computer device comprising a memory and a processor, characterized in that: The method according to any one of claims 1 to 3 is implemented when the processor executes the computer program stored in the memory. 5 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method according to claim 1 .

Images