An assembly progress detection method and system based on few-shot vision and RAG

Abstract

The application discloses an assembly progress detection method and system based on few-shot vision and RAG. The core of the method is: through a large language model to analyze the assembly process procedure, automatically extract the process logic and part attribute to construct a lightweight knowledge graph; at the same time, extract the part diagram in the procedure, combine a small amount of real part samples to train the target detection model, and generate a part detection model. In the assembly process, the system tracks the part state in real time, when the key working condition state transition of the core component is detected, the dynamic graph RAG mechanism is triggered, the context semantics of the current process and the SOP guidance are retrieved, and they are fused with the real-time assembly picture, and input into the multimodal large language model for end-to-end process integrity and compliance reasoning. Finally, the reasoning result is mapped to the digital twin system to realize error-proofing warning and closed-loop control. The application realizes rapid changeover in high complexity assembly scene and intelligent monitoring with deep process understanding.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and human-machine collaboration, specifically to an assembly progress detection method and system based on few-sample vision and RAG. Background Technology

[0002] In the assembly and manufacturing of complex products, real-time monitoring of assembly progress is a key technical aspect for ensuring production plan execution, shortening production cycles, and reducing manufacturing costs. Existing assembly process management systems suffer from the following technical deficiencies:

[0003] 1. Poor model generalization ability and high changeover costs: Traditional detection solutions heavily rely on pre-trained closed-set object detection models. In the face of "multiple varieties and small batches" scenarios, each time a new part or process is introduced, a lot of manual labor is required to re-acquire images and annotate anchor boxes, retrain and fine-tune the model, making it extremely difficult for the visual monitoring system to keep up with the changeover pace of flexible production lines.

[0004] 2. Insufficient knowledge integration and lagging status perception: On the one hand, assembly process knowledge (such as SOP, process logic, material BOM, etc.) is mostly stored in unstructured form. The rule engine can only perform simple Boolean logic judgments and lacks the ability to understand and analyze the deep process semantics of the assembly scenario, which makes it difficult to trace the source of anomalies. On the other hand, traditional detection systems mostly rely on manual offline reporting, which results in delayed information feedback and an inability to link the status of materials and processes in real time. When the assembly process progresses to a specific node, it is only then that the required materials are found to be missing, resulting in a work stoppage due to material shortages. This causes frequent production line shutdowns and seriously affects the continuity and cycle stability of production.

[0005] To address the aforementioned issues, there is an urgent need for an assembly progress detection solution that combines high precision, real-time performance, and intelligence. This solution should be able to integrate multi-source sensing data with structured process knowledge to enable rapid production line changes and a robust assembly monitoring system with strong semantic interpretation capabilities. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as poor model generalization ability leading to high changeover costs, insufficient knowledge fusion, and lagging state perception, this invention provides an assembly progress detection method and system based on few-sample vision and dynamic graph (RAG), aiming to achieve intelligent assembly monitoring with rapid changeover and deep semantic understanding capabilities.

[0007] An assembly progress detection method based on few-sample vision and RAG includes the following steps:

[0008] S100, parse the process specification document to construct a knowledge graph, extract the part illustrations in the document, and combine a small number of real part samples to train the pre-trained target detection model with few samples to generate a part detection model.

[0009] S200: Based on the part detection model, acquire real-time sensing data of the assembly site; maintain the part state machine according to the real-time sensing data, and monitor the state transition events in the part state machine.

[0010] S300, in response to the detected state transition event, and when the time elapsed since the last input of data to the multimodal reasoning model exceeds the anti-shake cooling time, the local context information is retrieved from the knowledge graph using the current process node as the index.

[0011] S400, the local context information and the real-time perception data when the state transition event is triggered are integrated and input into the multimodal reasoning model to obtain the process progress determination result.

[0012] Furthermore, in step S100, the input assembly process specification document is parsed using a large language model, and process entities, part entities and their topological relationships are automatically extracted to form a standardized dataset containing part types, quantities and process requirements. An initial lightweight knowledge graph is constructed based on the dataset, wherein each process node is accompanied by a state attribute, and each part node is accompanied by a real-time quantity attribute and a state attribute.

[0013] Furthermore, in step S100, generating the part detection model includes: parsing the process specification document using a large language model to extract the graphic data (two-dimensional graphic representations or three-dimensional model screenshots of the part); combining a small amount of real sample data, fine-tuning the pre-trained YOLO model through data augmentation and transfer learning techniques to generate the part detection model, which can identify all part categories defined in the process specification document.

[0014] Furthermore, step S200 specifically involves: real-time visual perception and dynamic knowledge graph update based on the trained model. A part inspection model is deployed at the assembly site to acquire real-time video streams and perform material fit-out inspection. Based on the fit-out inspection results, a part state machine is established in memory. The collected images, process, and material data are preprocessed and structured, and the fit-out inspection results and process status are injected into the dynamic knowledge graph in real time.

[0015] Furthermore, in step S200, the part state machine includes a missing state, a transient state, and a stable state; monitoring state transition events includes: when the object is continuously detected for a duration that reaches a preset stable duration, thereby transitioning from a missing state or a transient state to a stable state, the state transition event is determined to be detected.

[0016] Furthermore, in step S200, after acquiring the real-time sensing data of the assembly site, the method further includes: comparing the number of objects detected in the real-time sensing data with the number of objects required for the current process node in the knowledge graph, and performing a completeness check; injecting the completeness check result into the knowledge graph and updating the attribute status of the corresponding node.

[0017] Furthermore, step S300 specifically involves: triggering dynamic graph (RAG) retrieval based on state transitions, i.e., continuously tracking the state changes of parts based on state machines and dynamic knowledge graphs; when a preset key state transition is detected in a part related to the current core process, RAG retrieval is triggered in the lightweight knowledge graph using the current process node as an index to obtain a local graph context containing process name, SOP guidance, and real-time status of associated parts.

[0018] Furthermore, a state transition event is defined as: an event in which the state of a part related to the current process transitions from a transient or missing state to a stable state; wherein, the basis for judging the transition is: the duration for which the part is continuously identified by the part detection model reaches or exceeds a preset stable duration threshold.

[0019] Furthermore, the detection of a state transition event must simultaneously satisfy the following conditions: the state of a certain component in the component state machine has transitioned from a missing state or a transient state to a stable state, and this transition is triggered by the component being continuously detected for a preset stability threshold.

[0020] Furthermore, in step S400, the local context information includes: the status of the preceding task, the list of required objects, the real-time number of objects, and the operation guidance text; the local context information is serialized into a structured data format.

[0021] Furthermore, step S400 specifically includes: guiding the multimodal reasoning model to perform an existence check based on the real-time number of objects in the local context information; referring to the operation guidance text, determining the assembly action intention based on the real-time perception data; if a unique object of the next process node appears in the real-time perception data, and the object of the current process node is not detected due to occlusion, then the current process node is determined to be completed.

[0022] Furthermore, step S400 specifically involves: structurally encoding the local map context retrieved in step S300, aligning it with the key frame image of the assembly site at the current triggering moment for feature integration and data fusion, and inputting it into the multimodal large language model; the multimodal large language model combines the SOP context with the material status and part assembly relationship in the visual image to infer and output the judgment result of whether the current process is completed and the reason for deep semantic diagnosis.

[0023] Furthermore, in S400, after obtaining the process progress determination result, the process further includes: updating the state attributes of the corresponding process node in the knowledge graph based on the process progress determination result; mapping the process progress determination result to a digital twin model to visualize the assembly progress and abnormal locations in a three-dimensional scene.

[0024] Furthermore, the process progress determination results are mapped to a digital twin model, including: displaying the assembly progress, kit status, and abnormal locations in the 3D model in real time through color coding and text annotation; when an assembly abnormality or material shortage risk is detected, generating a multimodal early warning command and pushing it to the industrial terminal.

[0025] Furthermore, the process progress determination result includes process completion determination, diagnosis reasons, and early warning information; the diagnosis reasons include at least one of insufficient number of objects, assembly actions not conforming to the operation instruction text, or abnormal object status.

[0026] Compared with existing technologies, this system, through modular and lightweight design, offers the following advantages while ensuring real-time performance and controllable deployment costs:

[0027] (1) Training samples are automatically generated by parsing the process specifications through a large language model. The pre-trained YOLO model is fine-tuned with a small number of real images. It can quickly adapt to the new production line without large-scale manual annotation, solving the problems of high production change costs and long cycles in traditional solutions.

[0028] (2) Introducing a multimodal large language model and dynamic graph RAG mechanism breaks through the limitation that the rule engine can only handle simple Boolean logic, realizes the understanding of deep process semantics in the assembly scenario, and can quickly trace and locate the root cause of the problem when an anomaly occurs by combining the SOP context.

[0029] (3) By capturing key working condition state transitions through the part state machine, the material status and process progress are linked in real time; the vision and material data are linked to generate status reports, and the twin visualization marks the abnormal location, providing accurate basis for process adjustment; the adjustment response speed is accelerated and the rework rate is reduced. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below.

[0031] Figure 1 This is a schematic diagram of the assembly progress detection system architecture according to an embodiment of the present invention;

[0032] Figure 2 This is a flowchart of an assembly progress detection method according to an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with the accompanying drawings and embodiments, further illustrates this invention. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0034] Example 1

[0035] like Figures 1 to 2 As shown, this embodiment discloses an assembly progress detection method based on few-sample vision and RAG, including the following steps:

[0036] S100: Parse the process specification document, construct a knowledge graph, and generate a part inspection model based on the illustrated data in the document.

[0037] In this step, the system first acquires the process specification document corresponding to the assembly line. This document can be an electronic document in PDF format or a scanned copy of a paper document, containing process logic, required Bill of Materials (BOM), Standard Operating Procedures (SOP), and part illustrations. The system utilizes document parsing technology to automatically extract structured information from the document, such as the topological relationship between "processes" and parts, and the required number of parts, and constructs a knowledge graph based on this. This knowledge graph is stored in the form of a graph database, where nodes represent processes or parts, and edges represent dependencies or inclusion relationships, thus transforming unstructured text specifications into a structured knowledge network that computers can understand. Simultaneously, the system also extracts part illustration data embedded in the document, such as 2D engineering drawings or 3D model screenshots. This illustration data serves as prior knowledge and is used to assist in generating a part inspection model. In this way, the system does not need to collect massive amounts of labeled data from scratch; it only needs to combine a small number of real samples to quickly generate a visual inspection model adapted to the current production line, significantly reducing the model training cost and time cycle during production changeovers.

[0038] This embodiment includes:

[0039] S101, import the "Electronic Cabin Assembly Process Manual" document into the system. The built-in large language model performs deep analysis of the document through preset prompt word templates, automatically identifying and extracting the structured information, including entities such as processes, parts, SOPs, and their relationships.

[0040] For example, in the text "Step 7. Install the calculation module" and the accompanying illustrations (such as part drawings and installation diagrams), the LLM will understand the text and images and identify and extract key information. In this embodiment, the LLM extracts process nodes and part nodes:

[0041] Process node: Process 7;

[0042] property:

[0043] Procedure name: Install the computing module;

[0044] Operating Procedure (SOP): Insert the computing module into the outer cylinder of the electronics compartment according to the positioning position, ensuring accurate installation and avoiding squeezing the installed cables;

[0045] List of required parts, etc.

[0046] Part node: Calculation module;

[0047] Relationship: (Operation 7)-[:REQUIRES {qty:1}]->(Part 'Calculation Module'); This relationship has the attribute that Operation 7 requires 1 calculation module;

[0048] Dependency relationship between processes: (process 7)-[:PRECEDED_BY]->(process 6); This means that process 7 follows process 6 immediately, that is, the completion of process 6 is a prerequisite for starting process 7.

[0049] In this way, LLM transforms unstructured process documents into structured entity-relationship data, forming a standardized dataset that includes part types, quantities, and process requirements.

[0050] S102. Based on the standardized dataset mentioned above, a lightweight knowledge graph is constructed in the Neo4j graph database. The graph covers all 25 processes and their associated parts and SOP information. Each process node is accompanied by a status attribute (pending / in_progress / completed), and each part node is accompanied by a current quantity (current_count) and a real-time status (status), laying the foundation for subsequent dynamic updates and reasoning. The graph is constructed using a concise triple structure, ensuring query efficiency while preserving the complete logical relationships between processes.

[0051] S103, matching the application scenario of this application, involves the assembly of electromechanical products with standard model parts, clear illustrations, and installation processes. Two-dimensional illustrations of all key components, such as the "computing module," "19-pin connector," and "straight-wave antenna," are automatically extracted from the manual. Simultaneously, a small number of real images of each component are collected on-site. The specification illustrations and real images are mixed, and the training set is expanded through data augmentation (random rotation, cropping, Gaussian blur, etc.). To fine-tune the pre-trained YOLO model using few samples, a weight freezing and local retraining fine-tuning scheme is adopted, which differs from the traditional full-parameter fine-tuning method. The specific steps are as follows: Load the YOLO model pre-trained on a large-scale general image dataset, retaining the weights of most network layers of the pre-trained model unchanged; freeze the weights of all network layers except the last 3-5 layers (including the classification head and regression head), preserving the feature extraction capabilities of deep networks and avoiding the risk of deep weight destruction and overfitting caused by few samples; replace the model's general classification head with a specific classification head adapted to the part categories in this embodiment, and fine-tune the initial weights of the regression head to adapt to the part size and position prediction requirements; adopt a training strategy of mini-batch gradient descent and low learning rate decay, using the enhanced sample set as input, to train the replaced classification head and the unfrozen deep networks. Through multiple iterations, the model accurately maps general visual features to the specific part-specific features of this embodiment, achieving the dual goals of general feature reuse and specific feature adaptation, and further reducing the cost of manual annotation.

[0052] S200, Based on the part detection model, acquire real-time sensing data of the assembly site; based on the real-time sensing data, maintain the object state machine and monitor the state transition events in the object state machine.

[0053] After constructing the model and atlas, the real-time monitoring phase begins. The part detection model is deployed on edge computing nodes at the assembly site, analyzing the real-time video stream captured by industrial cameras frame by frame. It outputs the category, location, and confidence level of each part in the image, forming real-time perception data. Unlike traditional methods that directly use detection results for rule-based judgment, this application introduces an object state machine mechanism. The system maintains an independent state machine in memory for each critical part. This state machine dynamically updates its internal state based on the part's appearance and disappearance in the real-time perception data. By maintaining the state machine, the system can filter out noise signals caused by brief occlusions or detection jitter, focusing only on changes in operating conditions that have practical assembly significance. Monitoring state transition events means that the system continuously tracks the internal flow logic of the state machine. When a critical change in the part's state that meets preset conditions is detected (e.g., from a long-absent "missing" state to a persistently existing "stable" state), a trigger signal is generated. This event-driven mechanism based on state transitions avoids the system performing complex logical reasoning on each frame of the image, thus significantly reducing the consumption of computing resources.

[0054] This embodiment includes:

[0055] S201 captures real-time images from an industrial camera above the workstation. An improved part detection model mounted on the edge computing node detects each frame of the image, outputting the category, bounding box coordinates, and confidence score for each detected part, and counting the number of each part in the current frame. For example, when a computing module appears in the frame, the model outlines it with high confidence and updates its current count to 1.

[0056] S202: Obtain the current work order information by initializing the knowledge graph, call the knowledge graph constructed in step S100, and query the current required process and the list of required parts for the process. In this embodiment, it is confirmed that the part required for the current process is "Computation Module: 1", that is, the current process is "Install Computation Module", and the process requirement is 1 "Computation Module". Based on the comparison between the detection result and the requirement list, perform material completeness detection. That is, call the vision model to scan the workstation. Assuming that the camera sees a part, the YOLO model recognizes it as a "Computation Module" with a quantity of 1, compare the system's view (1 "Computation Module") with the knowledge graph requirement (1 "Computation Module"). If the type and quantity match perfectly (Q... vis = 1), then the material completeness test passes, and the process can start. At the same time, the comparison result (Q) will be sent. vis = 1) Write back to the knowledge graph and update the status of the process node.

[0057] Kitting inspection uses a visual perception unit to identify the parts in real time and compares this information with the process requirements of the current operation (from a knowledge graph) to confirm whether the required materials are complete and available. If a missing part, incorrect model, or insufficient quantity is detected, the system will issue a warning and intercept the operation before it begins, ensuring the continuity and stability of the production line. The specific inspection logic is as follows: the input to kitting inspection consists of visual recognition results and a process requirement list. The visual recognition results include the number N of part types identified by the YOLO model from the real-time image. tm and the actual number Q of the i-th type of parts identified im The process requirements list, derived from the attributes of the current process node in the knowledge graph, includes the number N of part types required for the current process, obtained by parsing the process specification. tr The number of parts required for the process of type i, Q ir Matching by type and quantity, i.e., checking N. tm =N tr That is, the types of parts identified visually must completely cover all types required by the process, with no more and no less; for each type of part required by the process (i from 1 to N) tr Check if the actual quantity is greater than or equal to the required quantity, i.e., Q.im ≥Q ir When both conditions are met, the system determines that the system is homogeneous and outputs Q. vis = 1 (Materials are complete), otherwise Q vis =0 (Materials not fully stocked, triggering a material shortage warning). Q vis As a dynamic attribute, it is updated in real time to the corresponding process node in the knowledge graph.

[0058] Simultaneously, a state machine is created for each component in memory, with the initial state uniformly set to "missing," and timing information such as the first appearance time and continuous detection duration is recorded. In this embodiment, upon confirming the existence of a component, a state machine is created in the background for the "computation module" component. The state machine is a proactive deployment that defines three states a component may experience during assembly: missing, flashed, and stable_present. "Missing" is the initial state, indicating that it has not been detected; "flashed" occurs when the component first appears on the screen, indicating that it may be in the worker's hands and ready for installation; and "stable_present" is the target state, which is reached only after the component has been continuously and stably detected for a period of time, at which point it may have already been installed.

[0059] S203 preprocesses and structures the collected images, process and material data, and injects the kitting inspection results, process execution progress and real-time status of parts into the Neo4j knowledge graph in batches using Cypher statements, updates the corresponding node attributes, and realizes the synchronization between the virtual graph and the physical assembly site.

[0060] Step S300: In response to detecting the state transition event, local context information is retrieved from the knowledge graph using the current job node as an index.

[0061] S301, the state machine continuously tracks the state of the "computation module". Upon monitoring startup, the initial state is "missing"; when the part is detected for the first time, the state transitions to a "flashed" state, and the time of its first appearance is recorded. If the part is continuously detected in subsequent frames, the continuous detection time will be... Accumulation; when Exceeding the preset stability threshold At 3 seconds, it indicates that the part has been stably present at the workstation (e.g., installed in place), and the state machine updates its state to stable_present. If the part disappears during the detection for more than 1 second, the state reverts to missing, indicating that the part was just "passing by" rather than "in place".

[0062] S302, when the transition from `flashed` to `stable_present` is determined to be a "critical operating condition transition", it indicates that the assembly operation may have been completed. The system checks the time interval since the last multimodal inference; if it is greater than the debouncing cooling time... If so, the RAG retrieval process is immediately triggered, proceeding to step S303.

[0063] S303: Using the currently executing process node as an index, perform a fast query and traversal in the real-time updated lightweight knowledge graph to obtain a local graph context containing information such as current process details, SOP, and real-time status of associated parts. In this embodiment, using the current process node "Process 7" as an index, perform a graph traversal query in Neo4j to extract a local subgraph centered on this node and expanding outward by 1 to 2 hops, containing information such as the status of previous processes, a list of required parts and their real-time quantities, SOP text, and safety specifications. Serialize the search results into a structured context in JSON format, for example:

[0064] {

[0065] "current_process": "Process 7 - Install Calculation Module",

[0066] "sop": "Send the computing module into the outer tube of the electronics compartment according to the positioning location..."

[0067] "pre_process_status": "completed",

[0068] "parts": [{"name": "Computation Module", "required": 1, "current": 1, "status":"stable_present"}]

[0069] }

[0070] S400: Based on the structured process knowledge and real-time images retrieved in step S300, joint reasoning is performed based on a multimodal large language model.

[0071] S401, the system combines the aforementioned JSON-formatted local context information (including parts list, real-time quantities, SOP text, etc.) with the current keyframe image (Base64 encoded) at the trigger moment into a multimodal Prompt. The Prompt embeds the following logical constraints to guide model inference:

[0072] ① Part status priority: The existence of the part must be checked based on the current_count and status in the context;

[0073] ② Action-assisted verification: Refer to the SOP description. If the installation intention is shown on the screen (such as the parts have been put in place and the worker's hands have been removed), it can be judged as passing, not just the presence of the parts;

[0074] ③ Fault-tolerant progress mechanism: If a typical part for the next process has appeared in the image, it can help determine whether the current process is complete. This mechanism is an intelligent fault-tolerant design. If the part for the current process is not seen due to occlusion, but a typical part for the next process has appeared in the image, this mechanism can help determine whether the current process is complete, avoiding production stoppages caused by simple visual occlusion.

[0075] S402, after receiving the Prompt, the multimodal large language model extracts visual features from the image to understand objects, locations, and actions within the scene; simultaneously, it parses semantic information from the JSON context to understand process requirements and part status. It fuses data from both modalities through a cross-modal attention mechanism, ultimately outputting a structured JSON result. For example:

[0076] {

[0077] "is_current_step_complete": true,

[0078] "reason": "The screen shows that the calculation module has been placed in the hull slot and the worker has let go. Combined with the fact that the current quantity of this part in the diagram is 1, the SOP requirements are met."

[0079] "alerts": []

[0080] }

[0081] If an anomaly is detected, the alerts field will contain the specific alert type and description.

[0082] S403. After receiving the completion signal, the system automatically updates the knowledge graph, executes the Cypher statement to update the status attribute of "Process 7" in Neo4j to "completed", advances the current process pointer to the next process, and resets the state machine of the relevant parts to prepare for subsequent monitoring.

[0083] S500, digital twin visualization and closed-loop feedback.

[0084] S501 is a digital twin model built on the Unity3D engine, mapped 1:1 to the physical assembly workstations, accurately reproducing the workstation layout, the 3D shape of core parts, and the process flow logic. The twin model subscribes to the updated node states in the Neo4j knowledge graph in real time via a WebSocket interface, achieving synchronous evolution between the virtual image and the physical world.

[0085] On the S502 remote process engineer's monitoring interface, the corresponding location of the virtual model of the aircraft compartment, the "Calculation Module," changed from a semi-transparent, pending installation state to a solid, colored state, highlighted with a green box. The overall assembly progress bar advanced from (6 / 25) to (7 / 25), while the right side of the interface displayed the current process name, part status, and LLM inference text.

[0086] S503: If a cable connection error occurs in subsequent "Step S101", the multimodal large model will return a warning message. The twin model will immediately highlight the error location in the 3D scene and push the warning through multiple channels, including workstation terminal pop-ups, workshop industrial screen prompts, and voice broadcasts. Engineers can remotely issue rework instructions to complete closed-loop control.

[0087] Example 2

[0088] This embodiment discloses an assembly progress detection system based on few-sample vision and RAG, including:

[0089] The multi-source perception module includes a process specification recognition subunit, which is used to collect input assembly process specification documents; and a visual perception subunit, which is equipped with a part detection model generated by training with few samples, for real-time visual perception of the assembly site and outputs the recognition results and quantity of parts.

[0090] The knowledge graph construction and management module has a built-in large language model for parsing the process specification document, automatically extracting process entities, part entities and their relationships, and constructing an initial lightweight knowledge graph; it is also used to receive the real-time recognition results of the visual perception subunit and dynamically update the real-time quantity and status attributes of part nodes in the knowledge graph.

[0091] The status monitoring and retrieval enhancement generation module includes a state machine maintenance unit, used to maintain a state machine for each part based on the real-time visual perception results. The state machine includes at least a missing state, a transient emergent state, and a stable existence state, and monitors key state transition events for the part to transition from a transient emergent state or a missing state to a stable existence state; a retrieval triggering unit, used to generate a retrieval command when the key state transition event is detected and the time since the last retrieval trigger exceeds the anti-shake cooling time; and a graph retrieval unit, used to respond to the retrieval command, using the current process node as an index, to perform retrieval enhancement generation operations in the lightweight knowledge graph to obtain local context information including current process details, standard operating procedure text, and the real-time status of related parts.

[0092] The multimodal reasoning module is used to receive the real-time perceived image and the local context information when the state transition event is triggered, fuse the two and input them into the multimodal reasoning model, perform joint reasoning under the guidance of preset logical constraints, and output the process progress judgment result containing process completion judgment, diagnosis reasons and early warning information.

[0093] The digital twin and interaction module includes: a twin modeling unit for constructing a digital twin 3D model mapped to the physical assembly station; a visualization mapping unit for mapping the process progress determination results and the real-time status in the knowledge graph to the digital twin model for visualization display; and an early warning and closed-loop control unit for generating multimodal early warning instructions based on the early warning information in the process progress determination results and pushing them to the industrial terminal, while receiving process adjustment feedback to form closed-loop control.

[0094] The implementation and collaboration workflow for each module is as follows:

[0095] (1) Initialization Phase: Knowledge Injection and Model Generation (corresponding to the Multi-Source Sensing Module and the Map Construction and Management Module). During this phase, the process engineer imports the "Electronic Cabin Assembly Process Manual" into the system. The process procedure identification sub-unit reads this document.

[0096] The large language model in the graph construction and management module automatically parses manual text and illustrations. For example, from "Step 7. Install the calculation module", the process entity "Process 7" (with attributes containing SOP text) and the part entity "Calculation Module" are extracted, and the relationships "(Process 7)-[:REQUIRES {qty:1}]->(Calculation Module)" and "(Process 7)-[:PRECEDED_BY]->(Process 6)" are established. All processes, parts, and relationships are constructed into an initial lightweight knowledge graph in the Neo4j graph database.

[0097] The same large language model extracts illustrations of all parts, such as "computation module" and "antenna," from the manual. These illustrations are mixed with real photos of each part taken on-site. After data augmentation, they are used to fine-tune the pre-trained YOLO model with few samples, generating a dedicated part detection model, which is then loaded into the visual perception subunit.

[0098] (2) Runtime Phase: Event-Driven. Assume the assembly process is currently at "Process 7". At this time, the industrial camera continuously captures images of the workstation. The part detection model of the visual perception subunit identifies the parts in the image in real time and identifies one "computation module". This result (type, quantity, location) is sent to the graph construction and management module, which updates the current_count of the "computation module" node in the graph to 1 and marks its status as flashed using Cypher statements. The state machine maintenance unit in the status monitoring and retrieval enhancement generation module starts timing for this part.

[0099] When the "computing module" is identified in consecutive video frames and the cumulative duration exceeds a preset threshold, the state machine maintenance unit determines that its state transitions from flashed to stable_present. The retrieval trigger unit checks that the anti-shake cooling time has passed and then triggers the retrieval. The map retrieval unit starts from the "process 7" node, retrieves the status of the preceding process, the list of required parts (including real-time quantity 1), and the SOP text "Send the computing module into the outer cylinder of the electronic cabin according to the positioning position...", and serializes it into local context information in JSON format.

[0100] The multimodal inference module receives the aforementioned JSON and the workstation image (Base64 encoded) at the state transition moment. It constructs a Prompt according to preset logical constraints (such as "must verify part quantity" and "determine installation intention based on SOP") and calls the multimodal large language model for inference. The model outputs a structured result: {"is_current_step_complete": true, "reason": "The screen shows the part has been stably placed in the mounting slot, and the worker's hands have been removed, which matches the installation characteristics described in the SOP."}

[0101] The inference results are sent to the graph construction and management module, which performs an update, changing the status of the "Process 7" node to "completed". Simultaneously, the visualization mapping unit of the digital twin and interaction module subscribes to graph changes in real time. In the 3D twin model, the corresponding "Computation Module" component changes from semi-transparent to highlighted green, and the overall progress bar updates from 6 / 25 to 7 / 25. The diagnostic reasons generated by the LLM are displayed synchronously on the right side of the interface.

[0102] (3) Anomaly Handling and Closed-Loop Control. If the model detects a cable connection error (such as a mismatch in model number) in "Process 11", the output of the multimodal inference module will include a warning message. The warning and closed-loop control unit will immediately highlight the error location in the 3D model, display a pop-up message on the industrial terminal, and issue a voice warning via the workshop broadcast. The process engineer can view the details at the workstation and remotely issue the correct assembly instructions or drawings to the terminal at that workstation to guide the workers to rework, thus completing the closed-loop control from anomaly detection to correction.

[0103] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An assembly progress detection method based on few-sample vision and RAG, characterized in that, Includes the following steps: S100, parse the process specification document to construct a knowledge graph, extract the part illustrations in the document, and combine a small number of real part samples to train the pre-trained target detection model with few samples to generate a part detection model. The construction of the knowledge graph includes: extracting process entities, part entities and their topological relationships from the process specification document to form a standardized dataset; constructing a lightweight knowledge graph containing process nodes, part nodes and inter-process dependencies based on the standardized dataset; each process node is attached with a state attribute, and each part node is attached with a real-time quantity attribute and a state attribute. S200: Based on the part detection model, acquire real-time sensing data of the assembly site; maintain the part state machine according to the real-time sensing data, and monitor the state transition events in the part state machine. The part state machine includes a missing state, a transient emergent state, and a stable existence state; the state transition event is defined as: an event in which the part state related to the current process transitions from a transient emergent state or a missing state to a stable existence state. The criteria for determining the transition are: the duration for which the part is continuously identified by the part detection model reaches or exceeds a preset stable duration threshold. S300, in response to the detected state transition event, and when the time elapsed since the last input of data to the multimodal reasoning model exceeds the anti-shake cooling time, the local context information is retrieved from the knowledge graph using the current process node as the index. S400, the local context information is fused with the real-time perception data when the state transition event is triggered, and input into the multimodal reasoning model to obtain the process progress determination result; wherein, the local context information includes: the status of the previous operation, the list of required objects, the real-time number of objects, and the operation guidance text; the local context information is serialized into a structured data format; Specifically, the multimodal reasoning model is guided to perform an existence check based on the real-time number of objects in the local context information; the assembly action intention is determined based on the real-time perception data, referring to the operation guidance text; if a unique object of the next process node appears in the real-time perception data, and the object of the current process node is not detected due to occlusion, then the current process node is determined to be completed.

2. The method according to claim 1, characterized in that, In step S100, generating the part detection model includes: parsing the process specification document using a large language model to extract graphic data; combining real sample data, adjusting the pre-trained YOLO model through data augmentation and transfer learning techniques to generate the part detection model, which can identify all part categories defined in the process specification document.

3. The method according to claim 1, characterized in that, In step S200, after acquiring the real-time sensing data of the assembly site, the method further includes: comparing the number of objects detected in the real-time sensing data with the number of objects required for the current process node in the knowledge graph, and performing a completeness check; injecting the completeness check result into the knowledge graph and updating the attribute status of the corresponding node.

4. The method according to claim 1, characterized in that, In S400, after obtaining the process progress determination result, the process progress determination result is further included as follows: based on the process progress determination result, the state attributes of the corresponding process node in the knowledge graph are updated; the process progress determination result is mapped to the digital twin model, and the assembly progress and abnormal location are visualized in the three-dimensional scene.

5. The method according to claim 4, characterized in that, Mapping the process progress determination results to a digital twin model includes: displaying the assembly progress, kit status, and abnormal locations in the 3D model in real time through color coding and text annotation; and generating multimodal early warning instructions and pushing them to the industrial terminal when assembly abnormalities or material shortage risks are detected.

6. An assembly progress detection system based on few-sample vision and RAG, characterized in that, The system employing the assembly progress detection method according to claim 4 or 5 includes: The multi-source sensing module is used to collect process specification documents and is equipped with a part detection model generated by training with a small number of samples to obtain real-time sensing data of the assembly site. The graph construction and management module is used to parse the process specification document to construct a lightweight knowledge graph, and dynamically update the attribute status of nodes in the graph based on the real-time sensing data. The status monitoring and retrieval enhancement generation module is used to maintain the part state machine based on the real-time perception data, monitor key state transition events, and retrieve local context information of the current process from the knowledge graph when the event is triggered. The multimodal reasoning module is used to fuse the local context information with the real-time perception data of the triggering event, and output the process progress determination result through the multimodal reasoning model; The digital twin and interaction module is used to visualize and display the process progress determination results, and to perform early warning and closed-loop control based on the results.

Images