Industrial production line optimization method and system fusing multi-modal large model and digital twin

Abstract

This invention discloses an industrial production line optimization method and system integrating a multimodal large-scale model and digital twin. The method includes: establishing a digital twin model of the production line and configuring standard operating procedures; collecting real-time operational information through multimodal sensing, and generating structured descriptive information through the multimodal large-scale model; comparing information within the digital twin model to identify operational errors and providing feedback on operational guidance information. It also includes material traceability to form a traceability data chain; coordinated production line rhythm control by dynamically adjusting material flow rate based on real-time operational load assessment; and closed-loop traceability and optimization of quality defects based on the traceability data chain to locate problematic links, analyze root causes, and implement optimization guidance. The standard operating procedure is an evolvable digital rule set, updated collaboratively by the multimodal large-scale model and the digital twin model. This invention achieves real-time guidance, adaptive control, and closed-loop optimization of manual operation links, improving production line compliance, efficiency, and product quality.

Description

Technical Field

[0001] This application relates to the field of industrial production line control methods, and in particular to industrial production line optimization methods and systems that integrate multimodal large models and digital twins. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing technologies, industrial production lines have gradually achieved automated control of production equipment, aiming to improve production efficiency and product consistency. However, in many precision assembly, complex processes, or special material handling scenarios, there are still a large number of production processes that rely on the experience and judgment of operators. The standardization and proficiency of these manual operations directly affect the quality and yield of the final product, but due to the flexibility and subjectivity of human operation, their standardization is difficult to be precisely controlled and guaranteed like that of automated equipment.

[0003] Currently, the supervision and quality assurance of manual operations mainly rely on post-event monitoring methods such as regular manual inspections and sampling tests of work-in-process or finished products. This approach has significant limitations and lags: First, inspections or sampling cannot cover the entire operation process, making it difficult to detect and correct non-standard practices in real time. Second, when defective products are discovered at the final quality inspection stage, a large number of subsequent products have often already been produced, resulting in high rework costs and making it difficult to accurately and quickly trace quality problems back to specific preceding production stages, corresponding operators, or material batches. This leads to low efficiency in identifying the root cause of problems, and similar issues may recur.

[0004] Digital twin technology can construct virtual mappings of physical production lines for monitoring and simulation, but its traditional applications lack depth in understanding unstructured and complex human-machine interactions. While multimodal large models possess powerful capabilities for understanding multi-source information such as images and videos, there is still a lack of systematic solutions for deeply integrating them with digital twins in industrial scenarios and applying them to real-time compliance assessment of manual operations, dynamic collaborative control of production lines, and intelligent traceability and closed-loop optimization of quality issues. Therefore, there is an urgent need for a technology that can integrate multimodal perception, intelligent analysis, and digital twin simulation and control capabilities to achieve real-time guidance, adaptive optimization, and continuous improvement of manual processes on production lines. Summary of the Invention

[0005] This invention aims to provide an industrial production line optimization method and system that integrates multimodal large-scale models and digital twins to address the problems in existing technologies, such as the lack of real-time and precise monitoring of manual operations, difficulty in tracing and locating quality issues, and the inability to adaptively adjust production line rhythm. By constructing a digital twin model synchronized with the physical production line and integrating the analytical capabilities of the multimodal large-scale model, this invention achieves real-time analysis and compliance comparison of operator behavior, enabling immediate identification of operational errors and providing feedback and guidance. By establishing a traceability data chain with all elements linked, it achieves precise source tracing and closed-loop optimization of quality issues. By integrating digital twin simulation and multimodal behavioral analysis, it assesses real-time operational load and dynamically adjusts production line rhythm to balance efficiency and pressure. Finally, through the deep integration of multimodal large-scale models and digital twins, it drives the self-evolution of standard operating procedures, forming a continuously optimized intelligent production line management closed loop, thereby comprehensively improving the standardization, traceability, adaptability, and overall efficiency of production line operations.

[0006] To achieve the above objectives, the industrial production line optimization method and system integrating multimodal large models and digital twins provided in this application adopts the following technical solution:

[0007] Firstly, this application discloses an industrial production line optimization method that integrates multimodal large models and digital twins, including:

[0008] Establish a digital twin model corresponding to the physical production line, and configure standard operating procedures for the target production links, including manual operations, in the digital twin model;

[0009] Real-time acquisition of operational information from the target production process;

[0010] The operation site information is input into the multimodal large model for parsing, generating structured description information that represents the current operation behavior. The structured description information is obtained by parsing the video and image data in the operation site information, including the action type, the state of the operation object, and the operation sequence.

[0011] Within the digital twin model, structured description information is compared with standard operating procedures in real time to identify operational errors. Specifically, structured description information is injected into the virtual workstation corresponding to the digital twin model in real time. The digital twin model then compares the incoming factual action descriptions with the pre-configured standard operating procedures item by item. The dimensions of the comparison include the logical sequence of actions, the location of action execution, the type of object, and the duration of key actions.

[0012] Based on operational errors, operational guidance information is generated and fed back to the target production stage.

[0013] Preferably, material traceability operations are also included:

[0014] Based on the structured description information generated by the multimodal large model, the characteristic information and material identification of the materials currently flowing into the target production process are obtained;

[0015] The structured description information, material characteristic information, material identification, and current operator identification are linked and bound to the product identification to form a traceability data chain.

[0016] Preferably, it also includes a production line rhythm coordination and control step, which includes:

[0017] Based on the material flow simulation status in the digital twin model and the analysis results of the multimodal large model on the operation site information, the real-time operation load of the target production link is evaluated.

[0018] When the real-time operating load exceeds the first threshold, a control command is generated through the digital twin model to reduce the material flow rate from the physical production line to the target production stage.

[0019] During the reduction of material flow rate, the changes in real-time operating load are monitored, and when the real-time operating load recovers to the second threshold, the material flow rate is adjusted to return to normal through a digital twin model, and the first threshold is greater than the second threshold.

[0020] Preferably, the real-time operational load of the target production stage is assessed as follows:

[0021] The first load index is calculated by the digital twin model based on the queue status in front of the virtual workstation;

[0022] The action video stream in the operation site information is analyzed by multimodal large model, and features representing the operation rhythm or state are extracted to generate a second load index.

[0023] The real-time operating load is obtained by merging the first load index and the second load index.

[0024] Preferably, it also includes a closed-loop traceability and optimization method for quality defects, including:

[0025] Obtain the defective products and defect types detected at the end of the production line;

[0026] Based on the traceability data chain, the target production process and corresponding historical operation batches associated with defective products are located in the digital twin model;

[0027] By using a multimodal large model to analyze the operation site information within historical operation batches, abnormal behavior patterns are identified, and causal simulation analysis is performed using a digital twin model to determine the root cause of the defect type.

[0028] Based on the root cause, generate and execute optimized guidance information for the relevant target production process.

[0029] Preferably, determining the root cause of the defect type specifically includes:

[0030] The multimodal large model extracts habitual action sequences that do not conform to standard operating procedures from the operation site information of historical operation batches;

[0031] The digital twin model receives a sequence of habitual actions, executes them, and outputs an assessment of their impact on product quality indicators.

[0032] Based on the impact assessment results, the correlation between habitual action sequences and defect types is quantified to determine the root cause.

[0033] Preferably, the optimized boot information includes at least one of the following:

[0034] Rule update information used to correct standard operating procedures in digital twin models;

[0035] Multimodal training materials generated from a multimodal large model, including comparisons of defect cases and standard operating procedures;

[0036] Configuration information used to adjust the process parameters of physical equipment in related production processes.

[0037] Preferably, the multimodal large model extracts habitual action sequences that do not conform to the standard operating procedure from the operation site information of historical operation batches, specifically including:

[0038] Retrieve all operation site information stored within the historical operation batches located in the location;

[0039] The multimodal large model spatiotemporally aligns and compares the video sequence of each round of operation with the ideal action sequence corresponding to the standard operation process stored in the digital twin model;

[0040] Identify recurring action patterns in the batch that systematically deviate from standard procedures.

[0041] Preferably, the standard operating procedure is represented in the digital twin model as an evolvable set of digital rules, the evolution of which includes:

[0042] The multimodal large model performs self-supervised learning on massive amounts of compliant structured description information to extract the implicit features and high-order patterns of optimal operations.

[0043] Based on implicit features and higher-order patterns, the digital twin model performs parameter tuning and logic optimization on the digital rule set to generate an optimized standard operating procedure.

[0044] The root cause and optimization guidance information output by the closed-loop tracing and optimization method for quality defects serve as feedback signals to trigger or correct the evolutionary process.

[0045] Secondly, this application discloses an industrial production line optimization system that integrates multimodal large-scale models and digital twins, applied to the industrial production line optimization system integrating multimodal large-scale models and digital twins as described in the first aspect, including:

[0046] The digital twin engine is used to build a digital twin model corresponding to the physical production line and to configure standard operating procedures for target production links, including manual operations, in the digital twin model.

[0047] Multimodal sensing networks are used to collect real-time operational information from the target production process.

[0048] The multimodal analysis module is used to input operational field information into a multimodal large model for analysis, generating structured descriptive information that characterizes the current operational behavior;

[0049] The compliance assessment module is used to compare structured description information with standard operating procedures in real time within the digital twin model in order to identify operational errors.

[0050] The real-time guidance module is used to generate operation guidance information based on operation errors and feed it back to the target production stage.

[0051] Compared with existing technologies, this invention provides an industrial production line optimization method and system that integrates multimodal large models and digital twins, and has the following beneficial effects:

[0052] 1. Real-time collected operational information is transformed into a structured description and dynamically compared with a preset standard process in a digital twin model. Once an operational error is identified, visual guidance information is immediately generated and fed back. This changes the traditional lagging supervision model that relies on manual spot checks, transforming operational standardization from "post-event error correction" to "process error prevention". It significantly improves the operational accuracy and product consistency of single-point processes, reducing the risk of quality fluctuations from the source.

[0053] 2. When defective products are detected at the end point, the system can quickly locate the specific production process, batch, and related materials based on the traceability data chain. Utilizing collaborative analysis of multimodal large-scale models and digital twin simulations, it deeply explores the causal relationships between abnormal behavior patterns and defects, thereby accurately identifying the root cause. Based on this, the system can automatically generate and execute targeted optimization measures, forming a complete quality control closed loop of "problem discovery - root cause tracing - improvement implementation," greatly improving problem response speed and resolution efficiency, and effectively preventing the recurrence of similar problems.

[0054] 3. The system can comprehensively analyze material flow simulation status and personnel operation behavior to intelligently assess real-time operational load. When the load is too high, it automatically adjusts the material flow rate to create a buffer period for operators, thereby alleviating personnel pressure and avoiding production bottlenecks while ensuring operational quality. Standard operating procedures are represented as an evolvable set of digital rules. Based on the learning results of a multimodal large model on massive amounts of compliant operational data and feedback signals from quality closed-loop optimization, it can continuously and autonomously optimize its rules and parameters, enabling production line control strategies to have the ability to continuously improve themselves, ultimately driving the overall production system towards a more efficient and intelligent direction. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the steps of the industrial production line optimization method that integrates multimodal large models and digital twins according to an embodiment of this application. Figure 2 This is a flowchart of the production line rhythm coordinated control steps in the industrial production line optimization method that integrates multimodal large model and digital twin in the embodiments of this application; Figure 3 This is a flowchart illustrating the steps of the closed-loop tracing and optimization method for quality defects in the industrial production line optimization method that integrates multimodal large models and digital twins, as described in this application embodiment. Detailed Implementation

[0056] The following is in conjunction with the appendix Figure 1-3 This application will be described in further detail.

[0057] This application discloses an industrial production line optimization method and system that integrates multimodal large models and digital twins.

[0058] Firstly, referring to Figure 1 This application discloses an industrial production line optimization method that integrates multimodal large models and digital twins, including:

[0059] S1. Establish a digital twin model corresponding to the physical production line, and configure standard operating procedures for the target production links, including manual operations, in the digital twin model;

[0060] This involves using the 3D design drawings, equipment parameters, and sensor network topology of the physical production line to construct a digital twin model that corresponds to the geometric, physical, and logical relationships of the physical production line using a digital twin engine, such as Unity 3D.

[0061] The digital twin model includes a three-dimensional model of automated equipment such as conveyor belts and robotic arms in the physical production line and their motion logic. For target production links such as manual assembly of precision parts and visual inspection stations, virtual workbenches and virtual operator models are established.

[0062] Furthermore, the standard operating procedures (SOPs) for the target production processes are digitized and structured.

[0063] For example, in the "circuit board insertion" stage, a series of action steps such as material picking, orientation identification, insertion, and pressing confirmation, as well as the standard posture, standard timing of each action and key quality checkpoints, are transformed into machine-readable rules and parameters, which are then configured into the digital twin model instance corresponding to the target production stage to form the standard operating procedure for that target production stage.

[0064] S2. Real-time acquisition of operational information from the target production process;

[0065] Among them, the multimodal large model analyzes the video and image data in the operation site information to output structured description information containing action type, operation object status and operation sequence, and synchronously drives the status update of virtual operators and virtual materials in the digital twin model.

[0066] Among them, a multimodal sensing unit consisting of high-definition industrial cameras, depth sensors and microphones is deployed at the target production station of the physical production line. When the operator starts working, the multimodal sensing unit starts synchronously and continuously collects on-site information of the operation, including RGB video stream, depth point cloud data and environmental audio.

[0067] For example, a high-definition industrial camera captures visual images of operator hand movements, tool usage, and materials being handled at 30 frames per second.

[0068] At the same time, the depth sensor provides spatial coordinate information of the action, which is transmitted in real time to the edge computing node or central server via industrial Ethernet.

[0069] S3. Input the operation site information into the multimodal large model for parsing and generate structured description information representing the current operation behavior;

[0070] The received multimodal raw data will be input into a pre-trained multimodal large model.

[0071] Similarly, in a specific embodiment, for example, the RGB video stream is analyzed frame by frame to identify a series of actions such as the operator's hand moving to the material box, the thumb and forefinger picking up the type A capacitor, and moving the type A capacitor to position B above the PCB board. The spatial trajectory of the above series of actions is determined by combining depth information. After processing, the multimodal large model outputs the structured description information of the current operation cycle.

[0072] S4. Within the digital twin model, the structured description information is compared with the standard operating procedure in real time to identify operational errors.

[0073] The structured description information is injected into the virtual workstation corresponding to the digital twin model in real time, and the digital twin model compares the incoming factual action description with the pre-configured standard operating procedures item by item.

[0074] The dimensions for comparison include the logical sequence of actions, the location of action execution, the object type, and the duration of key actions.

[0075] For example, the standard operating procedure specifies that the Type A capacitor must be inserted vertically, but the angle of the Type A capacitor deviates from the vertical direction by more than 5° in the real-time structured description information; or, the standard operating procedure specifies that the Type A capacitor must remain stable for 1 second after positioning, but the holding time in the real-time structured description information is less than 0.8 seconds. Such deviations will be identified and marked as operating errors, and the type of operating error, the event that occurred, and the degree of deviation will be recorded.

[0076] S5. Based on operational errors, generate operational guidance information and feed it back to the target production stage.

[0077] Specifically, when an operational error is detected, the system will match the digital twin model with its corresponding prompt library or dynamically generate specific operational guidance information. For example, for "insertion angle deviation," a highlighted prompt box will be generated with the message "Please adjust the capacitor to a vertical position," and this message will be overlaid on the AR glasses display or workstation LCD screen in front of the operator, while an arrow animation will indicate the correct angle. For "insufficient holding time," a countdown visual prompt will be displayed on the screen, and the operational guidance information will be fed back to the workstation in the target production process, aiming to correct errors as they are performed.

[0078] Furthermore, this also includes material traceability operations:

[0079] S01. Based on the structured description information generated by the multimodal large model, obtain the feature information and material identifier of the material currently flowing into the target production process;

[0080] When an operator begins processing a specific product unit, the multimodal large model, while generating structured description information for the current operation cycle, triggers parallel material information acquisition threads, including:

[0081] Material identification acquisition: RFID readers or fixed barcode scanners located at the material inlet of the workstation automatically read the material identification carried by the currently operated object, that is, the material or semi-finished product flowing into the target production process. The material identification can be a unique product serial number, batch number or work order number.

[0082] The product unit mentioned above refers to the product that needs to be divided into several target production stages when the product is assembled using assembly line operations during the product manufacturing process.

[0083] S02. Material Feature Information Acquisition: In sync with acquiring on-site operation information, the multimodal large model calls upon a dedicated visual inspection unit for materials. Using a high-definition industrial camera, the appearance, key dimensions, or preset feature points of the materials are quickly photographed and measured before or during operation.

[0084] The structured description information, material characteristic information, material identification, and current operator identification are linked and bound to the product identification to form a traceability data chain.

[0085] When the processing of the current material is completed in the target production stage and the product is about to be transferred to the next stage, the system executes the core data association operation:

[0086] Data encapsulation: The multimodal large model collects all relevant data packets at this moment, including:

[0087] Operation data packet: a structured description of the current operation cycle generated by step S3, which implicitly includes the quality judgment result of the operation action;

[0088] Material data package: includes material identifiers and material characteristic information obtained in step S1;

[0089] Identity data packet: The identifier of the current operator obtained through the workstation login system or wearable device identification;

[0090] Output Identifier: Generate or inherit a unique product identifier for the product produced in this stage, for example, add a stage code to the original material serial number.

[0091] Furthermore, refer to Figure 2 It also includes production line rhythm coordination and control steps, which include:

[0092] S10. Based on the material flow simulation status in the digital twin model and the analysis results of the multimodal large model on the operation site information, evaluate the real-time operation load of the target production link.

[0093] This includes:

[0094] State awareness of the digital twin model: The digital twin engine runs continuously, and its built-in material flow simulation logic calculates in real time the length of the virtual material queue waiting to be processed in front of the virtual workstation, the average waiting time, etc.

[0095] For example, the simulation showed that there were 5 workpieces backed up in the virtual buffer zone in front of the "manual welding station" and the average dwell time exceeded 30 seconds;

[0096] Behavioral analysis of the multimodal large model: The multimodal analysis module continuously processes operational information from the workstation. The multimodal large model further analyzes the operator's motion video stream, extracting features that characterize the rhythm or state of the operation, such as whether the frequency of the motion is abnormally high, whether there are unnatural pauses or repetitions between motions, and whether the operating posture shows signs of fatigue (such as frequent shoulder rubbing).

[0097] Fusion Assessment: The collaborative control module receives the above two inputs and performs fusion analysis using a preset algorithm model. For example, when the digital twin displays queue backlog and the multimodal large model analysis shows that the operator's movements become rapid and the accuracy rate decreases, the system comprehensively judges that the real-time operational load of the target production link has entered a "high load" state.

[0098] S11. When the real-time operating load exceeds the first threshold, control instructions are generated through the digital twin model to reduce the material flow rate from the physical production line to the target production stage.

[0099] If the estimated real-time operational load exceeds a preset first threshold, the system immediately triggers a control action:

[0100] The first threshold is illustrated by an example: queue backlog length > 4 and action error rate > 5%.

[0101] Instruction Generation: The collaborative control module sends a request to the digital twin engine. Within its virtual control logic, the digital twin engine simulates and generates a control instruction for the upstream material delivery point or the conveyor belt of the previous workstation. For example, the core content of this instruction is: requesting a reduction in the material delivery or conveying rate to the current "manual welding workstation" by a preset percentage.

[0102] Command issuance and execution: The generated control commands are converted into physical control signals and issued through the real-time communication interface between the digital twin engine and the physical production line control system.

[0103] For example, when the PLC controlling the upstream vibrating feeder receives a command, it adjusts the vibration frequency of the feeder from 50Hz to 35Hz, thereby effectively reducing the flow rate of the material to the target station.

[0104] S12. During the reduction of material flow rate, monitor the changes in real-time operating load, and when the real-time operating load recovers to the second threshold, adjust the material flow rate to return to normal through a digital twin model.

[0105] After the control measures are initiated, the system enters the dynamic monitoring and recovery phase:

[0106] Continuous monitoring: The collaborative control module and multimodal analysis module continue to operate, closely monitoring the effects of the reduced flow rate. The system observes whether the length of the virtual queue in the digital twin model begins to shorten, and whether the operator's actions in the multimodal large model analysis tend to stabilize and whether the accuracy rate recovers.

[0107] Recovery Judgment and Execution: When monitoring data indicates that the real-time operating load has dropped below the second threshold, the collaborative control module determines that the pressure has been relieved;

[0108] The second threshold is illustrated by an example: queue length < 2 and action error rate < 2%.

[0109] Recovery command: The system generates and issues another control command through the digital twin engine, instructing the upstream equipment to gradually or all at once restore the material flow rate to the normal preset level.

[0110] Furthermore, assessing the real-time operational load of the target production process includes:

[0111] The first load index is calculated by the digital twin model based on the queue status in front of the virtual workstation;

[0112] Status Acquisition: The digital twin model synchronizes the status of the physical production line in real time. For the virtual workstation corresponding to the target production link, the model continuously monitors the real-time status of its upstream virtual buffer, i.e., the "queue in front of the virtual workstation".

[0113] The real-time status data includes, but is not limited to: queue length, i.e., the number of virtual workpieces waiting to be processed, the total queue waiting time, and the distribution of time intervals for workpiece arrival;

[0114] Index Calculation: The model's built-in load calculation submodule maps the above queue status data into a scalar value, namely the first load index, according to a preset algorithm. The weighted formula is as follows:

[0115]

[0116] The weighting coefficients a, b, and a can be set according to the actual production situation, and are not limited here.

[0117] Where X is the first load index, L is the current queue length, T is the average waiting time, and a is the standard beat rate.

[0118] When the queue length is empty and the flow is smooth, the first load index approaches 0;

[0119] As queue length increases or waiting time lengthens, the first load index will rise linearly or non-linearly to reflect the digital twin load from the material flow itself.

[0120] The action video stream in the operation site information is analyzed by multimodal large model, and features representing the operation rhythm or state are extracted to generate a second load index.

[0121] This step, which runs on an edge computing unit or server, focuses on the behavioral analysis of "operators," specifically:

[0122] Video stream analysis: The multimodal analysis module directs the RGB video stream from the operational site information collected from the target production process to a dedicated behavior analysis branch of the multimodal large model. This branch model is trained to recognize micro-expressions, gesture frequencies, and body postures related to workload and stress in the video sequence;

[0123] Feature extraction: The model extracts a series of temporal features from the video stream in real time. Feature extraction includes:

[0124] Operation rhythm characteristics: such as the deviation between the average duration of a single operation cycle and the standard duration, and the shortening trend of the interval between consecutive operations.

[0125] Characteristics of movement state: such as the amplitude of hand shaking, the frequency of unnecessary small movements, and whether the neck and shoulders show fatigue-induced stiffness.

[0126] Attention characteristics: such as the frequency and duration of eye movement away from the focus of work.

[0127] Indicator generation:

[0128] Specifically, the real-time operational load assessment of the target production process is conducted in the digital space, automatically completed by the simulation kernel driven by the digital twin engine:

[0129] Furthermore, the original features are fed into a lightweight regression or classification sub-model, which outputs a comprehensive score, i.e., the second load index.

[0130] For example, a rule-based system might be configured to increase the indicator value by 0.3 when it detects "a 10% reduction in operation cycle and an increase in hand tremor"; and increase the indicator value by 0.2 when it detects "frequent eye movement". This indicator changes continuously from 0 (relaxed) to 1 (extremely stressed), quantifying the behavioral analysis load.

[0131] The real-time operating load is obtained by fusing the first load index and the second load index, including:

[0132] Data synchronization and normalization: The first load index from the digital twin model and the second load index from the multimodal large model are timestamped and normalized respectively to map them to the same numerical range.

[0133] Fusion Strategy: The final real-time operational load is calculated using a preset fusion strategy. The fusion strategy can be a weighted summation, calculated using the following formula:

[0134]

[0135] Where G is the real-time operating load and N is the second load index;

[0136] in The weights are adjustable and can be set according to the characteristics of each stage.

[0137] Furthermore, the maximum value is taken for the operational load.

[0138] Fuzzy logic or rule-based fusion, defined as follows: if the first load index is high and the second load index is medium, then the total load is a high rule;

[0139] Output decision: The calculated real-time operating load is a single, comprehensive quantitative value, and the real-time operating load value is directly compared with the preset threshold to determine whether to trigger production line rhythm regulation.

[0140] Furthermore, refer to Figure 3 It also includes closed-loop traceability and optimization methods for quality defects, including:

[0141] X1. Obtain the defective products and defect types detected at the end of the production line;

[0142] At the final quality inspection station of the physical production line, automated quality inspection units are deployed. When a product completes all production stages and arrives at this station:

[0143] X11. Automated Inspection: Performs full inspection or spot checks on key characteristics of products according to preset inspection procedures. For example, a vision system captures images of the product's appearance and compares them with standard images;

[0144] X12. Result Judgment and Marking: The detection system automatically determines whether the product is qualified.

[0145] For products determined to be defective, while recording their defect type, a unique product identifier for the final product is automatically read or generated.

[0146] Furthermore, defective product identification, defect type, and inspection timestamp are encapsulated into quality event data packets and sent to the closed-loop optimization module in real time.

[0147] X2. Based on the traceability data chain, locate the target production process and corresponding historical operation batch associated with defective products in the digital twin model;

[0148] Upon receiving the aforementioned quality event, the closed-loop optimization module immediately initiates the traceability procedure, including:

[0149] X21. Data Chain Query: Using the final product identifier of the defective product as a unique key, a query is initiated to the traceability data chain database. Utilizing the pre-established relationships within the data chain, the system can trace back in reverse, quickly retrieving the historical records of the product across all target production stages;

[0150] X22. Digital Twin Spatiotemporal Positioning: The closed-loop optimization module inputs the traced process information and corresponding timestamps into the digital twin engine. The digital twin engine replays and positions the information on its virtual timeline. In the three-dimensional scene of the digital twin model, the virtual path through which the defective product flows is highlighted, and the related production process that may have caused the current defect type is accurately located.

[0151] X3. Utilize multimodal large-scale model analysis to identify operational site information within historical operation batches, identify abnormal behavior patterns, and combine this with digital twin model for causal simulation analysis to determine the root causes of defect types, including:

[0152] X31. Multimodal historical data analysis: The closed-loop optimization module retrieves all the original data of the operation site information stored in the historical operation batch located in step X22, and inputs the video data of this batch into the multimodal large model for centralized and in-depth offline analysis to identify abnormal behavior patterns.

[0153] X32. Digital Twin Causal Simulation: Transforms suspicious abnormal behavior patterns identified by a multimodal large model into parameter inputs for a digital twin model.

[0154] In a digital twin environment, the digital twin engine drives virtual operators and virtual products to perform causal simulation analysis, that is, to strictly reproduce the entire operation process in the virtual environment according to the identified abnormal patterns.

[0155] Through physics engine calculations, simulations can quantitatively assess whether an abnormal action will cause the virtual product to produce stress distributions or contact marks similar to real defects, thereby determining the causal relationship between the behavior and the defect in the virtual world, i.e., the root cause.

[0156] X4. Based on the root cause, generate and execute optimization guidance information for the relevant target production process, including:

[0157] Adjustment information for equipment and tools will be dispatched to the equipment maintenance system as maintenance work orders;

[0158] Updates to processes and standards will be pushed to the process management system and trigger online updates to the standard operating procedures in the digital twin model.

[0159] For enhancements related to operator skills, the information will be immediately sent to the workstation display screen of the relevant production process, or incorporated into the operator's next training session.

[0160] Furthermore, identifying the root cause of the defect type specifically includes:

[0161] X301, a multimodal large model, extracts habitual action sequences that do not conform to standard operating procedures from the operation site information of historical operation batches;

[0162] The closed-loop optimization module calls the multimodal analysis module for offline analysis to uncover potential patterns and deviations in operator behavior, including:

[0163] X3011, Data Preparation: Retrieve all operation site information stored in the historical operation batch located in step X2;

[0164] X3012, Sequence Alignment and Pattern Discovery: The multimodal large model activates its sub-network specifically designed for temporal behavior analysis. The sub-network spatiotemporally aligns and compares the video sequence of each round of operations with the ideal action sequence corresponding to the standard operation process stored in the digital twin model.

[0165] X3013. Extracting Habitual Deviations: Identifying recurring action patterns in the batch that systematically deviate from the standard procedure, i.e., habitual action sequences.

[0166] For example, the standard operating procedure requires "pressing the part vertically until a click is heard," but the model analysis of the video revealed that in over 60% of the operations, the operator performed a subtle but continuous additional movement step: "first tilting slightly to the left and then pressing down." This habitual movement sequence was extracted by the model and encoded into a series of motion key point data with timestamps and spatial coordinates.

[0167] X302: The digital twin model receives habitual action sequences, drives the virtual model to execute them in the simulation environment, and outputs the impact assessment results on product quality indicators.

[0168] This step places the identified behavioral patterns in a digital twin environment for physical verification to establish a causal chain between behavior and defects:

[0169] X3021, Virtual Reproduction: The closed-loop optimization module sends the extracted habitual action sequence to the digital twin engine.

[0170] The digital twin engine creates a virtual operator agent in the corresponding virtual workstation of the digital twin model and drives it to operate the virtual product strictly according to the received deviation sequence.

[0171] X3022, Simulation Execution and Physical Calculation: The high-fidelity physics engine built into the digital twin model begins to run, simulating the deformation, stress distribution, and interaction process between the virtual component and its mating parts under the action of a non-perpendicular force of "leftward tilt-downward pressure".

[0172] The simulation process calculates various physical quantities in real time, such as peak contact stress, internal strain of components, and engagement depth of snap fasteners.

[0173] X3023. Quantitative Output of Results: After the simulation is completed, the system outputs a detailed impact assessment result based on the preset quality judgment logic.

[0174] For example, "Due to the tilting and downward pressure, the probability of the stress at the root of the clip exceeding the material fatigue limit increases to 85%", and "The simulated size of the micro-deformation on the inner wall of the left side of the component is 0.05 mm". Furthermore, the impact assessment results are directly related to specific habitual action sequences and potential product quality indicator failure modes.

[0175] X303. Based on the impact assessment results, quantify the correlation between habitual action sequences and defect types to determine the root cause.

[0176] X3031. Correlation Quantification: The closed-loop optimization module compares and analyzes the simulation impact assessment results with the actual defect types obtained in step X1. The system uses statistical or logical methods to quantify the correlation between the two.

[0177] For example, it can be calculated that the defect rate of the "left-tilt down" sequence in this historical operating batch is statistically significantly different from the defect rate under standard operation.

[0178] X3032. Based on statistical correlation and reasonable physical interpretation, the system can determine that the extracted habitual action sequence is the root cause of this defect, rather than other accidental factors.

[0179] Furthermore, optimizing the boot information includes at least one of the following:

[0180] X41. Rule update information used to correct standard operating procedures in digital twin models;

[0181] When root cause analysis indicates that the problem stems from the root cause and the impact assessment results, the system automatically drafts rules to update information.

[0182] Similarly, to better understand, let's take an example:

[0183] If the root cause is that the buckle insertion angle deviation is within 5° and is not judged as an error, then the updated information is as follows: In the standard operating procedure for buckle assembly, a compliance criterion is added: the insertion angle deviation from the vertical direction is less than 3°, and if it exceeds this, a real-time alarm is triggered.

[0184] Furthermore, the rule update information is first sent to the digital twin engine, which updates the standard operating procedure rule library for the virtual workstation online without shutting down.

[0185] Furthermore, the updated standard operating procedure rule base will take effect immediately. At the start of the next work cycle, the compliance assessment module will use the new angle threshold for real-time comparison. Any deviation exceeding 3° will be immediately identified as an operational error and trigger the real-time guidance module for on-site correction.

[0186] X42. Multimodal training materials generated from a multimodal large model, including comparisons of defect cases and standard operating procedures;

[0187] When root cause analysis points to the operator's habitual action sequence, personalized training content will be generated, including:

[0188] X421. Material Generation: The closed-loop optimization module instructs the multimodal analysis module to start the training content generation program. The large multimodal model performs the following operations:

[0189] X4211, Case Extraction: From the videos of historical operation batches, cut out typical video clips that contain confirmed habitual action sequences and ultimately lead to defective products, as defect cases;

[0190] X4212, Standard Demonstration Generation: Call the digital twin model and, based on the optimized standard operating procedure, render and generate a 3D animation or high-fidelity simulation video demonstrating the correct actions as a standard operating demonstration.

[0191] X4213. Combine defect case videos with standard demonstration videos in a split-screen or picture-in-picture format, and use the natural language generation capabilities of the large model to automatically add keyframe markers, arrow indicators, and voice narration to point out specific deviations, potential mechanical effects, and correct techniques, ultimately generating a multimodal training material.

[0192] X422. Material Execution: Through the production line management system, materials are pushed to the work terminals or mobile devices of operators linked by the traceability data chain; the operators are required to complete the learning and confirmation before the start of the next shift, and a short online quiz may be set up to verify the effect.

[0193] X43. Configuration information used to adjust the process parameters of physical equipment in related production processes.

[0194] When root cause analysis indicates a closer coupling between the defect and equipment parameter settings, the following guidance information is generated:

[0195] X431, Information Generation: The closed-loop optimization module requests the digital twin engine to perform parameter optimization simulation. In the virtual environment, the engine performs multiple rounds of simulation analysis on relevant equipment parameters around the problem area to find the optimal parameter combination that can offset operational fluctuations or reduce the defect rate;

[0196] X432. Information Execution: Configuration information is automatically sent to the corresponding physical equipment controllers via the interface between the digital twin engine and the manufacturing execution system or equipment management system. After receiving the instructions, the controllers automatically adjust the parameters and send a confirmation signal to the system after the parameter adjustments are complete. The corresponding virtual equipment parameters in the digital twin model are also updated synchronously to maintain consistency between the virtual and real systems. The system will subsequently monitor the product quality data of the adjusted workstation to verify the optimization effect.

[0197] Furthermore, standard operating procedures are represented in the digital twin model as an evolvable set of digital rules, the evolution of which includes:

[0198] The multimodal large model performs self-supervised learning on massive amounts of compliant structured description information to extract the implicit features and high-order patterns of optimal operations.

[0199] Specifically, the system continuously collects and filters structured descriptive information generated during the massive production process, including real-time data, as well as historical operational data marked by the compliance assessment module as fully compliant or producing high-quality products, forming a best practice dataset.

[0200] Furthermore, self-supervised learning: Multimodal large models initiate their self-supervised learning mode, using structured descriptive information as input or contrastive learning tasks, and train within the model to extract latent implicit features and higher-order patterns that lead to successful results from the structured descriptive information.

[0201] Based on implicit features and higher-order patterns, the parameters of the digital rule set are optimized using a digital twin model to generate an optimized standard operating procedure.

[0202] Specifically, rule set tuning: The self-evolutionary optimization module transforms the implicit features and high-order patterns learned by the multimodal large model into specific optimization suggestions, and performs parameter tuning and logic optimization on the digital rule set stored internally that controls the virtual production logic.

[0203] Furthermore, the process version generation: After tuning and optimization, the digital twin engine generates a new, optimized version of the standard operating procedure. This new version is then verified through simulation in a virtual environment and marked as a candidate version for release.

[0204] The root cause and optimization guidance information output by the closed-loop tracing and optimization method for quality defects serve as feedback signals to trigger or correct the evolutionary process.

[0205] Specifically, feedback signal reception: After the closed-loop tracing and optimization method for quality defects is executed, the root cause and optimization guidance information output by it will be sent to the self-evolutionary optimization module in real time as a feedback signal;

[0206] Furthermore, based on the feedback signal, evolution is triggered or corrected:

[0207] Triggering evolution: If the feedback signal reveals a completely new defect pattern that the rule set has not covered, the feedback signal will trigger a new, targeted learning-optimization cycle;

[0208] Corrective evolution: If the feedback signal indicates that the current rule set is still insufficient to prevent a certain type of known defect, the feedback signal will be used to correct the ongoing or next evolutionary process.

[0209] Secondly, this application discloses an industrial production line optimization system that integrates multimodal large-scale models and digital twins, applied to the industrial production line optimization method integrating multimodal large-scale models and digital twins as described in the first aspect, including:

[0210] The digital twin engine is used to build a digital twin model corresponding to the physical production line and to configure standard operating procedures for target production links, including manual operations, in the digital twin model.

[0211] The digital twin engine is the core of the system's virtual world. Based on the CAD model, PLC logic, and sensor network topology of the physical production line, it builds and runs a digital twin model synchronized with the physical production line status in real time on a server or high-performance edge computing node. It is responsible for receiving real-time data from the physical world, driving the movement and status updates of the virtual model, and performing simulation calculations in the virtual environment.

[0212] Multimodal sensing networks are used to collect real-time operational information from the target production process.

[0213] Among them, the multimodal sensing network is the physical sensing layer of the system, which consists of multimodal sensing units widely deployed in the target production links of the production line. The multimodal sensing unit integrates high-definition industrial cameras, depth sensors, microphone arrays and RFID / barcode readers, etc., and is responsible for collecting raw multimodal data from the operation site in real time. It then streams high-bandwidth, low-latency operation site information and material characteristic information to the system's data processing center through industrial Ethernet or 5G private network.

[0214] The multimodal analysis module is used to input operational field information into a multimodal large model for analysis, generating structured descriptive information that characterizes the current operational behavior;

[0215] Among them, the multimodal analysis module is the intelligent perception hub of the system. It integrates a pre-trained multimodal large model. The multimodal analysis module receives the raw data stream from the perception network and uses the visual understanding, action recognition and cross-modal fusion capabilities of the multimodal large model to perform real-time analysis of the operation site information. Its core function is to generate semantically rich structured description information.

[0216] The compliance assessment module is used to compare structured description information with standard operating procedures in real time within the digital twin model in order to identify operational errors.

[0217] The compliance assessment module is embedded in or tightly coupled to the digital twin engine. It receives structured description information from the multimodal analysis module and compares this information with the preset standard operating procedures in real time at the millisecond level in the virtual context provided by the digital twin model. By executing the preset rule logic, the compliance assessment module identifies the operational error between the current operation and the standard and outputs the error type, degree and occurrence time.

[0218] The real-time guidance module is used to generate operation guidance information based on operation errors and feed it back to the target production stage;

[0219] The real-time guidance module is the human-machine interface of the system. Based on the operation error information output by the compliance assessment module, it dynamically generates specific and visualized operation guidance information. The real-time guidance module feeds back the guidance information to the corresponding operators on the physical production line through human-machine interfaces such as industrial tablets and industrial control screens.

[0220] The traceability management module is used to associate and bind structured description information, material characteristic information, material identification, and current operator identification with the product identification of the output, and form a traceability data chain.

[0221] Among them, the traceability management module is responsible for building a full-process data chain, obtaining structured description information from the multimodal analysis module, obtaining material identification and feature information from the sensing network, and obtaining operator identification and product identification from the production management system.

[0222] The core function of the traceability management module is to execute material traceability steps, align and strongly associate these multi-source information with timestamps, form and persistently store traceability data chains in the database, and provide a complete data foundation for quality backtracking.

[0223] The collaborative control module is used for collaborative control of production line rhythm.

[0224] Among them, the collaborative control module is the decision center for the dynamic balance of the production line, and is used to execute the collaborative control steps of the production line rhythm.

[0225] On the one hand, the collaborative control module connects to the material flow simulation data of the digital twin engine;

[0226] On the other hand, by using the analysis results of operator behavior by the multimodal analysis module, the collaborative control module assesses the real-time operating load, and when the load exceeds the limit, the digital twin engine generates and issues control commands to the physical production line control system to achieve adaptive dynamic adjustment of material flow rate.

[0227] The closed-loop optimization module uses a multimodal large model to perform self-supervised learning on massive amounts of compliant structured description information, extracting implicit features and higher-order patterns of optimal operation; the digital twin model, based on implicit features and higher-order patterns, performs parameter tuning and logic optimization on the digital rule set, generating optimized standard operating procedures; the root cause and optimization guidance information output by the closed-loop tracing and optimization method for quality defects serve as feedback signals to trigger or correct the evolutionary process.

[0228] Among them, the closed-loop optimization module is the central hub for handling quality issues, used to execute closed-loop tracing and optimization steps for quality defects;

[0229] The closed-loop optimization module receives defective product information from the end-of-line quality inspection system, then quickly locates the problem link in the digital twin model based on the traceability data chain, coordinates the multimodal analysis module to deeply mine historical data, and combines the simulation analysis capabilities of the digital twin engine to determine the root cause. Subsequently, the closed-loop optimization module generates specific optimization guidance information.

[0230] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. An industrial production line optimization method integrating multimodal large models and digital twins, characterized in that, include: Establish a digital twin model corresponding to the physical production line, and configure standard operating procedures for the target production links, including manual operations, in the digital twin model; Real-time acquisition of operational information from the target production process; The operation site information is input into the multimodal large model for parsing, generating structured description information that represents the current operation behavior. The structured description information is obtained by parsing the video and image data in the operation site information, including the action type, the state of the operation object, and the operation sequence. Within the digital twin model, structured description information is compared with standard operating procedures in real time to identify operational errors. Specifically, structured description information is injected into the virtual workstation corresponding to the digital twin model in real time. The digital twin model then compares the incoming factual action descriptions with the pre-configured standard operating procedures item by item. The dimensions of the comparison include the logical sequence of actions, the location of action execution, the type of object, and the duration of key actions. Based on operational errors, operational guidance information is generated and fed back to the target production stage.

2. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 1, characterized in that, This also includes material traceability operations: Based on the structured description information generated by the multimodal large model, the characteristic information and material identification of the materials currently flowing into the target production process are obtained; The structured description information, material characteristic information, material identification, and current operator identification are linked and bound to the product identification to form a traceability data chain.

3. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 1, characterized in that, It also includes production line rhythm coordination and control steps, which include: Based on the material flow simulation status in the digital twin model and the analysis results of the multimodal large model on the operation site information, the real-time operation load of the target production link is evaluated. When the real-time operating load exceeds the first threshold, a control command is generated through the digital twin model to reduce the material flow rate from the physical production line to the target production stage. During the reduction of material flow rate, the changes in real-time operating load are monitored, and when the real-time operating load recovers to the second threshold, the material flow rate is adjusted to return to normal through a digital twin model, and the first threshold is greater than the second threshold.

4. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 3, characterized in that, Assessing the real-time operational load of the target production process includes: The first load index is calculated by the digital twin model based on the queue status in front of the virtual workstation; The action video stream in the operation site information is analyzed by multimodal large model, and features representing the operation rhythm or state are extracted to generate a second load index. The real-time operating load is obtained by merging the first load index and the second load index.

5. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 2, characterized in that, It also includes closed-loop traceability and optimization methods for quality defects, including: Obtain the defective products and defect types detected at the end of the production line; Based on the traceability data chain, the target production process and corresponding historical operation batches associated with defective products are located in the digital twin model; By using a multimodal large model to analyze the operation site information within historical operation batches, abnormal behavior patterns are identified, and causal simulation analysis is performed using a digital twin model to determine the root cause of the defect type. Based on the root cause, generate and execute optimized guidance information for the relevant target production process.

6. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 5, characterized in that, Determining the root cause of a defect type specifically includes: The multimodal large model extracts habitual action sequences that do not conform to standard operating procedures from the operation site information of historical operation batches; The digital twin model receives a sequence of habitual actions, executes them, and outputs an assessment of their impact on product quality indicators. Based on the impact assessment results, the correlation between habitual action sequences and defect types is quantified to determine the root cause.

7. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 5, characterized in that, Optimized boot information includes at least one of the following: Rule update information used to correct standard operating procedures in digital twin models; Multimodal training materials generated from a multimodal large model, which include comparisons of defect cases and standard operating procedures; Configuration information used to adjust the process parameters of physical equipment in related production processes.

8. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 7, characterized in that, The multimodal large model extracts habitual action sequences that deviate from standard operating procedures from historical operation batches of operational field information. Specifically, these include: Retrieve all operation site information stored within the historical operation batches located in the location; The multimodal large model spatiotemporally aligns and compares the video sequence of each round of operation with the ideal action sequence corresponding to the standard operation process stored in the digital twin model; Identify recurring action patterns in the batch that systematically deviate from standard procedures.

9. The industrial production line optimization method integrating multimodal large model and digital twin as described in claim 1, characterized in that, Standard operating procedures are represented in the digital twin model as an evolvable set of digital rules. The evolution of this set of rules includes: The multimodal large model performs self-supervised learning on massive compliant structured description information to extract the implicit features and high-order patterns of optimal operation. Based on implicit features and higher-order patterns, the digital twin model performs parameter tuning and logic optimization on the digital rule set to generate an optimized standard operating procedure. The root cause and optimization guidance information output by the closed-loop tracing and optimization method for quality defects serve as feedback signals to trigger or correct the evolutionary process.

10. An industrial production line optimization system integrating multimodal large-scale models and digital twins, applied to the industrial production line optimization method integrating multimodal large-scale models and digital twins as described in any one of claims 1-9, characterized in that, include: The digital twin engine is used to build a digital twin model corresponding to the physical production line and to configure standard operating procedures for target production links, including manual operations, in the digital twin model. Multimodal sensing networks are used to collect real-time operational information from the target production process. The multimodal analysis module is used to input operational field information into a multimodal large model for analysis, generating structured descriptive information that characterizes the current operational behavior; The compliance assessment module is used to compare structured description information with standard operating procedures in real time within the digital twin model in order to identify operational errors. The real-time guidance module is used to generate operation guidance information based on operation errors and feed it back to the target production stage.

Images