Mobile projection assembly process guidance method and system

Abstract

This technology belongs to the field of assembly manufacturing, and specifically relates to a manual assembly operation process guidance system for the discrete manufacturing industry. The assembly process guidance system includes a process design module, a process projection guidance module, and a state detection module. The process design module is used to generate a three-dimensional structured process data packet for the assembly; the process projection guidance module realizes the virtual-real fusion of the process guidance model and the physical object, projects the projection surface and process information at the specified position of the assembly object, and presents it to the operator in an intuitive form; the state detection module is used to inspect the assembly results. The assembly process guidance system can avoid mistakes such as confusion, forgetfulness, and transition operations, minimize non-value-added operations such as process document review and labeling, and improve product assembly efficiency and quality stability.

Description

Field of the Invention

[0001] This technology belongs to the field of assembly manufacturing, and specifically relates to a manual assembly operation process guidance system for the discrete manufacturing industry. Background Art

[0002] Assembly is a crucial step in the manufacturing process, directly impacting performance, cost, and reliability. In discrete manufacturing industries, such as aerospace, automation is low due to the complex structures, large number of parts, mechanical / electrical / thermal coupling, and poor incoming material consistency. Operators often handle multiple assembly steps. Each task requires extensive process information. Traditional assembly work manuals or Kanban systems require operators to interpret and translate this information to obtain useful information. This process is time-consuming, inefficient, and places a strain on operators' memory and cognitive abilities. This is particularly true in satellite assembly, where each station encompasses dozens of processes, each comprised of thousands of steps. The sheer volume of process information required to review this information makes it inevitable that oversights, omissions, and operational errors may occur. Furthermore, in-process inspection models require interspersed inspection requirements within assembly process documentation, requiring operators and inspectors to coordinate and inspect simultaneously, which is time-consuming and labor-intensive, significantly impacting production efficiency.

[0003] The introduction of augmented reality technology has greatly improved the level of visualization of assembly process information. By superimposing virtual process information on the real assembly scene in real time, it is presented to operators in a more intuitive form, enabling them to complete assembly tasks quickly and with high quality. The document "Chinese Invention Patent Application Publication No. CN109491497A" discloses an artificially assisted assembly application system based on augmented reality technology, including a content editing subsystem and a real-time guidance subsystem. It edits information such as lightweight models, part postures, and assembly processes into enhanced assembly process step files, renders and integrates them with assembly site videos, and guides on-site workers' assembly operations. The document "Chinese Invention Patent Application Publication No. CN112734945B" discloses an assembly guidance method, system and application based on augmented reality, including a visual module, a calculation module and a playback module, which highlights assembly information and part contour information to guide operators to complete assembly operations. The document "Chinese Invention Patent Application Publication No. CN113220121A" discloses an AR fastener assisted assembly system and method based on projection display, including a process database module, an AR intelligent guidance module, an assembly detection module and an integrated control module. The assembly guidance process is projected on the assembly site in an AR visual manner, and the assembly results are given real-time detection feedback, thereby improving assembly efficiency and reducing assembly error rate.

[0004] The above-mentioned assembly guidance systems are all suitable for fixed workstations. They use visual recognition of specific logos to perform scene positioning and model registration. They need to set targets in the three-dimensional model and the actual assembly scene in advance. The amount of preparatory work such as data preparation and on-site calibration is large, and the efficiency improvement in multi-variety mixed-line production is limited.

[0005] The present invention intends to use a three-dimensional model as a carrier to construct a visual process data package, and use a mobile projection device to superimpose the digital model and work instructions on the physical object in the work area in real time, assisting operators in correctly understanding and executing the process, avoiding errors such as confusion, forgetfulness, and transitional operations, minimizing non-value-added operations such as process document review and labeling, and improving product assembly efficiency and quality stability. Summary of the Invention

[0006] The purpose of the present invention is to propose a mobile projection assembly process guidance system and method, which performs target tracking and positioning through contour matching of three models and actual scenes, projects three-dimensional structured assembly information to the corresponding position of the assembly object, and while guiding operators to perform assembly operations, collects image information of the assembly results for automatic recording and interpretation of the technical status. It has the characteristics of fast tracking speed, high positioning accuracy, high detection accuracy, and less on-site preparation work, and is particularly suitable for assembly guidance and detection of large and complex structures such as satellites and spacecraft.

[0007] The present invention provides a mobile projection type assembly process guidance system, comprising a process design module, a process projection guidance module, and a state detection module; the process design module is used to generate a three-dimensional structured process data packet for an assembly, which reads in a three-dimensional model of an assembly with a structure tree and a product attribute table, generates a process number for a part to be assembled according to the assembly sequence defined in the structure tree, and generates process content, wherein the process content includes a process guidance model, a projection surface, annotation information, and an optimal viewpoint; the process guidance model is a model of an assembly base part corresponding to the current process before the implementation of the current process, and is used to track and locate the base part; the projection surface refers to a footprint and an indication mark of the part to be assembled on the projection plane; the annotation information is a textual description of the assembly process of the part to be assembled, which is obtained through the product attribute table. Obtain; the optimal viewpoint is the pose transformation matrix of the virtual camera relative to the part to be assembled, which can be obtained through the viewpoint generation algorithm to ensure that the projection information is clearly and accurately presented on the assembly object; the final output is a Json file and an STL file. The Json file is divided into two levels: process and step. The process level contains the process number, process guidance model name, and optimal projection viewpoint. The step level contains the projection surface name and process annotation information. The STL file is organized according to the process task and corresponds to the name in the Json file one by one; the process projection guidance module is used to project the three-dimensional structured process to the assembly site, which includes a mobile robot, a projection device, a camera 1, a computer, a human-computer interaction unit, a target tracking unit, and a projection path control unit. The mobile The robot end is fixedly connected to the projection device and camera 1. The projection device receives instructions from the human-computer interaction unit and is used to present the annotation information and projection surface on the part to be assembled. The camera 1 is used to collect the assembly site image and transmit it to the target tracking unit. The computer is electrically connected to the mobile robot, the projection device and the camera 1 to run the human-computer interaction unit, the target tracking unit and the projection path control unit. The human-computer interaction unit receives the Json file and STL file output by the process design module, and parses the process guidance model, projection surface, annotation information and optimal viewpoint corresponding to each process according to the assembly sequence defined by the structure tree, transmits the process guidance model to the target tracking unit, publishes the projection surface and annotation information to the projection device, and transmits the optimal viewpoint to the projection path control unit. The tracking unit extracts the contour information of the process guidance model and the assembly site image for matching, obtains the real-time pose matrix of camera 1, and transmits it to the projection path control unit. The projection path control unit plans the motion path according to the current pose of camera 1, and drives the robot to move camera 1 to the optimal viewpoint position; the state detection module includes camera 2 and an assembly state recognition unit. The camera 2 is fixedly connected to the end of the mobile robot together with the projection device and camera 1 in the process projection guidance module. The mutual pose conversion relationship can be obtained through calibration. The state detection module is used to inspect the assembly results; the product attribute table contains the component ID, name and process information, and the process information is the force measurement requirements, glue sealing requirements and thermal grease coating requirements.

[0008] Preferably, the assembly three-dimensional model is a CAD model.

[0009] Preferably, the indicator mark includes an R point, an installation sequence number and a position indication arrow.

[0010] Preferably, the mutual posture conversion relationship is calibrated by the calibration method described in the Chinese invention patent application with application number CN202210497995.1, and the specific steps are as follows: setting a reference surface, wherein the reference surface is set in the common field of view of the camera device and the projection device, and the Z axes of the camera device coordinate system and the projection device coordinate system are both facing the reference surface;

[0011] Projecting a first image onto the reference surface to obtain a second image, wherein the first image includes a plurality of first identification points;

[0012] photographing the second image to obtain a third image, wherein the third image has a plurality of third identification points;

[0013] Traversing and solving the Euclidean distance between the first identification point and the third identification point to obtain a matching point set, the matching point set including at least four pairs of the first identification point and the corresponding third identification point whose Euclidean distance is less than a set threshold;

[0014] Inputting the set of matching points into a mapping function to obtain a homography matrix of the first image and the third image;

[0015] Setting a calibration plate under the field of view of the camera device;

[0016] photographing the calibration plate to obtain a first calibration image;

[0017] Inputting the first calibration image and the homography matrix into an inversion function to obtain a second calibration image, wherein the coordinate system of the second calibration image is the image coordinate system of the projection device;

[0018] The first calibration image is calibrated by a single object to obtain a first extrinsic parameter matrix and a first distortion vector of the camera device; the second calibration image is calibrated by a single object to obtain a second extrinsic parameter matrix and a second distortion vector of the projection device;

[0019] The first extrinsic parameter matrix and the second extrinsic parameter matrix are input into a conversion function to obtain a rotation matrix and a translation matrix between the projection device coordinate system and the camera device coordinate system.

[0020] Preferably, the optimal viewpoint can be achieved through an algorithm or can be specified manually.

[0021] The present invention also provides a mobile projection assembly process guidance method, comprising the following steps:

[0022] 1) Read assembly information, import the assembly 3D model with a structure tree into the process design module, read the product attribute table containing the ID of the part to be assembled and the process information, and associate the 3D model of the part to be assembled with the process information;

[0023] 2) Generate process annotation information for the assembly, establish assembly process tasks, select the parts to be assembled on the assembly structure tree, and define the process annotation information of the parts to be assembled that need to be projected, including the name, code, fastener specifications and quantity, sealing requirements, and force measurement requirements;

[0024] 3) Generate a projection patch of the part to be installed. Select the part to be installed as the projection object, analyze the triangular block model of the part to be installed, select the installation position on the base component as the projection plane, project the vertices of the triangular block of the part to be installed onto the projection plane, redraw the projection image to be the projection footprint of the part to be installed, specify the R point of the part to be installed according to the model annotation, add indicator symbols such as arrows and numbers, and repeat the above steps to generate the corresponding projection patch;

[0025] 4) Set the optimal projection viewpoint for the part to be mounted, read the calibration configuration files of the projection device, camera 1, camera 2, and the robot, view the projection effect from the perspective of the projection device, and adjust the perspective for occlusion, stacking, etc. The pose matrix of the projection device relative to the part to be mounted is the optimal viewpoint;

[0026] 5) Set up the process guidance model, select the assembly base part as the process guidance model for tracking and positioning the parts to be assembled; select the assembly base part and the parts to be assembled as the inspection model for assembly status identification and interpretation;

[0027] 6) 3D process data package compilation, defining the process guidance sequence of the parts to be assembled, assembling the process annotation information, projection surface patches, and optimal viewpoint information step by step, and merging them with the process guidance model and inspection model into a structured process task data package. Set the assembly process regulations for the assembly, sort the process task data packages according to the assembly steps, and generate a 3D process data package;

[0028] 7) Parsing the three-dimensional process data packet, the human-computer interaction unit reads the three-dimensional process data packet, extracts the process guidance model of the current process, and transmits it to the target tracking unit;

[0029] 8) Initial alignment of the process guidance model: In the human-computer interaction unit, the size and position of the process guidance model are manually adjusted so that it is initially aligned with the assembly basic parts in the video captured in real time by camera 1;

[0030] 9) The assembly base part model is precisely aligned. The target tracking unit extracts the contour information of the process guidance model and the assembly base part respectively, matches the virtual and real contours, estimates the pose transformation matrix of the camera 1 relative to the assembly base part, and transmits it to the path control unit;

[0031] 10) Projection path planning and execution: The path control unit uses the current pose of camera 1 as the initial pose and the optimal projection viewpoint as the target pose to plan the robot's motion path and drive the robot to move the projection device to the target pose. During this process, the target tracking unit estimates the pose transformation matrix of camera 1 in real time and corrects the robot's motion path until the pose deviation is less than the preset value;

[0032] 11) Assembly process projection guidance: the projection device projects the process information and projection surface onto the corresponding position of the assembly base parts, and the operator completes the assembly of the parts to be assembled according to the guidance information;

[0033] 12) Online detection of assembly status: Camera 2 captures images of the assembly site, and the assembly status recognition unit extracts the contour information of the scene image and the inspection model respectively, and uses the contour matching algorithm to determine whether there are any errors or omissions in the assembly parts;

[0034] 13) Repeat steps 9 to 12 until all parts are assembled.

[0035] The guide information is used to guide the assembly of the parts to be assembled, and includes a footprint diagram, marking information and indication marks.

[0036] The process projection guidance module realizes the virtual-real fusion of the process guidance model and the physical object, projects the projection surface and process information at the specified position of the assembly object, and presents them to the operator in an intuitive form.

[0037] The advantages and beneficial effects of this method are:

[0038] (1) The projection-type assembly process guidance system proposed by the present invention breaks away from the constraints of wearable devices and supports multi-user collaborative operations in shared scenes. For users, the field of vision is complete and unrestricted, and there is no discomfort when using it for a long time, making it more suitable for assembly operation scenarios;

[0039] (2) The mobile robot is equipped with a projection device, which breaks through the limitation of the projector's field of view on the working range. It is not only suitable for the assembly guidance of large-scale products, but also can quickly switch between different workstations, with good flexibility;

[0040] (3) The introduction of the optimal viewpoint effectively solves the problem of mutual occlusion between components, and the projection surface generated based on the spatial relationship can achieve a three-dimensional projection effect, ensuring that the footprint map and text can be accurately and clearly presented on the assembly object;

[0041] (4) Compared with the traditional augmented reality assisted assembly method, this method uses a model-based monocular camera pose estimation method to achieve tracking and positioning of the assembly scene. It does not require the setting of markers, simplifies the preparation work before the assembly operation, and avoids the risk of model registration failure caused by marker detachment or position change;

[0042] (5) Through image processing, the technical status is identified to achieve the integration of operation and inspection and improve the continuity of assembly operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a step diagram of the mobile projection assembly process guidance method of the present invention.

[0044] Figure 2 This is a schematic diagram of the mobile projection-based assembly process guidance system of the present invention. In the figure, 1 is a mobile robot, 2 is a projection device, 3 is camera 1, 4 is camera 2, 5 is a computer, 6 is component 1 to be assembled, 7 is an assembly base component, 8 is component 2 to be assembled, 9 is component 3 to be assembled, 10 is component 4 to be assembled, 11 is component 5 to be assembled, and 12 is component 6 to be assembled.

[0045] Figure 3 1 is a diagram of the footprint of the component to be installed, 2 is a process description, and 3 is an indication mark of the installation sequence of the fastener.

[0046] Figure 4 It is an implementation effect diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] Example 1

[0049] The present invention provides a mobile projection assembly process guidance system, which designs a visual process data package based on a three-dimensional product model and process files, and projects it to the corresponding position of the assembly object after on-site analysis, guiding the assembly operation in a virtual-real fusion manner; the assembly process guidance system includes a process design module, a process projection guidance module, and a status detection module.

[0050] The process design module is used to generate a three-dimensional structured process data package. It reads in a three-dimensional model with a structure tree and a product attribute table. Designers define assembly processes based on the product structure tree. Each process corresponds to a part to be assembled. The process number is the assembly order. The process content includes a process guidance model, projection surface, annotation information, and the best viewpoint. The product attribute table contains information such as part ID, process requirements, and process instructions, such as tightening torque, whether it is sealed, whether thermal grease is applied, etc. The process guidance model is the assembly base part model corresponding to the current process before the implementation, which is used to track and locate the assembly object; the projection surface refers to the footprint map, R point, arrow and other indicator marks of the part to be assembled on the projection plane; the annotation information is the text description projected on the assembly object, which is obtained through the product attribute table; the best viewpoint is the posture transformation matrix of the virtual camera relative to the assembly base, which can be obtained through the viewpoint generation algorithm or manually specified to ensure that the projection information is clearly and accurately presented on the assembly object. The process design module outputs Json files and STL files. The Json file is divided into two levels: process and step. The process level contains the process number, process guidance model name, and optimal projection viewpoint, and the step level contains the projection surface name and process annotation information; the STL file is organized according to process tasks and corresponds one-to-one to the name in the Json file.

[0051] The process projection guidance module includes a mobile robot, a projection device, a camera 1, a computer, a human-computer interaction unit, a target tracking unit, and a projection path control unit. The end of the mobile robot is fixedly connected to the projection device, camera 1, and camera 2, and the mutual posture conversion relationship can be obtained through calibration. For details, see the document "Chinese Invention Patent Application No. CN202210497995.1"; the projection device is a high-brightness, high-resolution industrial projector, which receives instructions from the human-computer interaction unit and presents process information and projection surfaces on the assembly base; camera 1 is used to collect images of the assembly site and transmit them to the target tracking unit; the computer is electrically connected to the mobile robot, the projection device, camera 1, and camera 2 to run the human-computer interaction unit, the target tracking unit, and the projection path control unit; the human-computer interaction unit receives the process information and projection surface. The design module outputs a three-dimensional structured process data packet, which parses the process guidance model, projection surface, process information, and optimal projection point corresponding to each process according to the process flow, transmits the process guidance model to the target tracking unit, publishes the projection surface and process information to the projection device, and transmits the optimal projection viewpoint to the projection path control unit; the target tracking unit extracts the contour information of the process guidance model and the assembly site image for matching, obtains the real-time pose matrix of camera 1, and transmits it to the projection path control unit; the projection path control unit plans the motion path according to the current pose of camera 1, and drives the robot to move camera 1 to the optimal projection viewpoint position.

[0052] Furthermore, the target tracking unit adopts a method based on edge contour matching to achieve target tracking, which specifically includes four steps: S1, reading in the process guidance model, adjusting the initial posture of the model so that it is roughly aligned with the image captured by camera 1, projecting the three-dimensional model to obtain the model edge contour, and sampling to obtain 2D contour points; S2, extracting local color features from the real image frame based on the 2D contour points projected by the model; S3, establishing a Gauss-Newton optimization function, solving the posture change between the previous and next two frames, and updating the current posture; S4, if tracking is successful, updating the local color features and performing iterative tracking.

[0053] The state detection module includes a camera 2 and an assembly state recognition unit, which is used to verify the assembly results. Furthermore, the assembly state recognition unit uses the positioning results of the target tracking unit to render the assembly model in the OpenGL environment to obtain a two-dimensional virtual image of the CAD model. The edge contour gradient features of the assembly model are extracted as a recognition template, and similarity matching is performed on the real image captured by camera 2. If the similarity value exceeds the set threshold, it means that the assembly is in place. If it is lower than the set threshold, it means that the assembly is missing or the direction is wrong. The similarity calculation method is as follows:

[0054]

[0055] Where ε represents the similarity between the template image and the real image, ori(O,r) represents the gradient direction of position r in the template image O, ori(I,t) represents the gradient direction of point t in the real image I, p is a list of r, T = (O,p) represents the template image, Represents the neighborhood area with c+r as the center and τ as the radius.

[0056] Example 2

[0057] The present invention also provides a mobile projection type assembly process guidance method, the method step flow chart is as follows Figure 1 As shown, taking the installation of the device on the deck as an example, the steps of this method are as follows:

[0058] Step 1: Import the deck assembly model with a structure tree into the computer of the process design module, read the product attribute table containing process information such as component ID and assembly requirements, and confirm the association between the 3D model of the component and the process information;

[0059] Step 2: Create an assembly process task, select the parts to be assembled on the assembly structure tree, and define the process annotation information that needs to be projected ( Figure 2 (in), including the name and code of the parts to be assembled, the specifications and quantity of the fasteners, the requirements for adhesive sealing, the requirements for force measurement, etc.;

[0060] Step 3: Select the part to be installed as the projection object ( Figure 26, 8, 9, 10, 11, 12), analyze the triangular block model of the part to be installed, and select the installation position on the base part as the projection plane ( Figure 2 7), project the vertices of the triangular face of the component to be assembled onto the projection plane, and redraw the projection image to be the projection footprint of the component to be assembled ( Figure 3 2); specify the R point of the part to be assembled according to the model annotation ( Figure 3 1), add the numerical identification of the fastener installation sequence ( Figure 3 In step 4), repeat the above steps to generate the projection patch corresponding to the mark.

[0061] Step 4: Read the projection device ( Figure 2 2), Camera 1( Figure 2 3), Camera 2 ( Figure 2 4) and robots ( Figure 2 In the calibration configuration file in 1), the projection effect is viewed from the perspective of the projection device, and the perspective is adjusted for occlusion, stacking, etc. The pose matrix of the projection device relative to the assembly base is the optimal projection viewpoint.

[0062] Step 5: Select the assembly base as the process guide model ( Figure 2 7), used for tracking and positioning of assembly objects; select the model of the part to be assembled as the inspection model ( Figure 2 6), used for assembly status identification and interpretation.

[0063] Step 6: Define the process guidance sequence for the parts to be assembled, assemble the process annotation information, projection surface patches, and optimal viewpoint information step by step, and merge them with the process guidance model and inspection model into a structured process task data package; set the product assembly process procedures, sort the process task data packages according to the assembly steps, and generate a three-dimensional process data package.

[0064] Step 7: The human-computer interaction unit reads the three-dimensional process data packet, extracts the process guidance model of the current process, and transmits it to the target tracking unit.

[0065] Step 8: In the human-computer interaction unit, manually adjust the size and posture of the process guidance model so that it is preliminarily aligned with the assembly basic parts in the video captured in real time by camera 1.

[0066] Step 9: The target tracking unit extracts the contour information of the process guidance model and the assembly base part respectively, matches the virtual and real contours, estimates the pose conversion matrix of the camera 1 relative to the assembly base part, and transmits it to the path control unit.

[0067] Step 10: The path control unit uses the current posture of camera 1 as the initial posture and the optimal projection viewpoint as the target posture, plans the robot's motion path, and drives the robot to move the projection device to the target posture; during this process, the target tracking unit estimates the posture transformation matrix of camera 1 in real time and corrects the robot's motion path until the posture deviation is less than the preset value.

[0068] Step 11: The projection device projects the process information and the projection surface to the corresponding position of the assembly base component, and the operator completes the assembly of the component to be assembled according to the guidance information.

[0069] Step 12: Camera 2 captures the assembly scene image, and the assembly state recognition unit extracts the contour information of the scene image and the inspection model respectively, and determines whether there are any wrong or missing assembly parts through the contour matching algorithm.

[0070] Step 13: Repeat steps 9 to 12 until all parts are assembled.

[0071] The implementation effect of the present invention is as follows Figure 4 shown.

Claims

1. A mobile projection assembly process guidance system, characterized in that: include: The process design module is used to generate a three-dimensional structured process data package for the assembly. The process design module reads in the three-dimensional model of the assembly with a structure tree and the product attribute table, generates a process number for the part to be assembled according to the assembly sequence defined in the structure tree, and generates the process content. The process content includes a process guidance model, a projection surface, annotation information, and an optimal viewpoint. The process guidance model is the assembly base part model corresponding to the current process before the implementation of the current process, which is used to track and locate the base part. The projection surface refers to the footprint map and indicator mark of the part to be assembled on the projection plane. The annotation information is a text description of the assembly process of the part to be assembled, which is obtained through the product attribute table. The optimal viewpoint is the posture transformation matrix of the virtual camera relative to the base part, which can be obtained through the viewpoint generation algorithm to ensure that the projection information is clearly and accurately presented on the assembly object. The final output is a Json file and an STL file. The Json file is divided into two levels: process and step. The process level contains the process number, process guidance model name, and optimal projection viewpoint. The step level contains the projection surface name and process annotation information. The STL file is organized according to the process task and corresponds one-to-one with the name in the Json file. The process projection guidance module is used to project the three-dimensional structured process to the assembly site. The process projection guidance module includes a mobile robot, a projection device, a camera 1, a computer, a human-computer interaction unit, a target tracking unit, and a projection path control unit. The end of the mobile robot is fixedly connected to the projection device and the camera 1. The projection device receives instructions from the human-computer interaction unit and is used to present the annotation information and the projection surface on the basic component. The camera 1 is used to collect the assembly site image and transmit it to the target tracking unit. The computer is electrically connected to the mobile robot, the projection device and the camera 1 to run the human-computer interaction unit, the target tracking unit, and the projection path control unit. The human-computer interaction unit receives instructions from the process The Json file and STL file output by the process design module are parsed according to the assembly sequence defined in the structure tree, and the process guidance model, projection surface, annotation information, and optimal viewpoint corresponding to each process are transmitted to the target tracking unit. The projection surface and annotation information are published to the projection device, and the optimal viewpoint is transmitted to the projection path control unit. The target tracking unit extracts the contour information of the process guidance model and the assembly site image for matching, obtains the real-time pose matrix of camera 1, and transmits it to the projection path control unit. The projection path control unit plans the motion path according to the current pose of camera 1 and drives the robot to move camera 1 to the optimal viewpoint position. A state detection module, comprising a camera 2 and an assembly state recognition unit. The camera 2, the projection device in the process projection guidance module, and the camera 1 are fixedly connected to the end of the mobile robot. The mutual posture conversion relationship can be obtained through calibration. The assembly state recognition unit is used to verify the assembly results. The product attribute table contains component ID, name and process information. The process information includes force measurement requirements, glue sealing requirements and thermal grease coating requirements.

2. The mobile projection assembly process guidance system according to claim 1, characterized in that: The assembly three-dimensional model is a CAD model.

3. The mobile projection assembly process guidance system according to claim 1, characterized in that: The indicator mark includes an R point, an installation sequence number and a position indication arrow.

4. The mobile projection assembly process guidance system according to claim 1, characterized in that: The optimal viewpoint can be achieved through an algorithm or specified manually.

5. A mobile projection assembly process guidance method comprising the following steps: 1) Read assembly information, import the assembly 3D model with a structure tree into the process design module, read in the product attribute table containing the ID of the part to be assembled and the process information, and associate the 3D model of the part to be assembled with the process information; 2) Generate process annotation information for the assembly, establish assembly process tasks, select the parts to be assembled on the assembly structure tree, and define the process annotation information of the parts to be assembled that need to be projected, including the name, code, fastener specifications and quantity, sealing requirements, and force measurement requirements; 3) Generate a projection patch of the part to be installed. Select the part to be installed as the projection object, analyze the triangular block model of the part to be installed, select the installation position on the base component as the projection plane, project the vertices of the triangular block of the part to be installed onto the projection plane, and redraw the projection image to obtain the projection footprint of the part to be installed. Specify the R point of the part to be installed according to the model annotation, add position indication arrows and installation sequence digital indication symbols, and repeat the above steps to generate the corresponding projection patch; 4) Set the optimal projection viewpoint for the part to be mounted, read the calibration configuration files of the projection device, camera 1, camera 2, and the robot, view the projection effect from the perspective of the projection device, and adjust the perspective for occlusion and stacking. The pose matrix of the projection device relative to the part to be mounted is the optimal viewpoint; 5) Set up the process guidance model and select the assembly base as the process guidance model for tracking and positioning the assembly site; select the assembly base and the assembly to be assembled as the inspection model for assembly status identification and interpretation; 6) 3D process data package compilation, defining the process guidance sequence of the parts to be assembled, assembling the process annotation information, projection surface patches, and optimal viewpoint information step by step, and merging them with the process guidance model and inspection model into a structured process task data package. Set the assembly process regulations for the assembly, sort the process task data packages according to the assembly steps, and generate a 3D process data package; 7) Parsing the 3D process data packet, the human-computer interaction unit reads the 3D process data packet, extracts the process guidance model of the current process, and transmits it to the target tracking unit; 8) Initial alignment of the process guidance model: In the human-computer interaction unit, the size and posture of the process guidance model are manually adjusted to align it with the assembly base part in the real-time video captured by camera 1; 9) The assembly base part model is precisely aligned. The target tracking unit extracts the contour information of the process guidance model and the assembly base part, matches the virtual and real contours, estimates the pose transformation matrix of camera 1 relative to the assembly base part, and transmits it to the path control unit. 10) Projection path planning and execution: The path control unit uses the current pose of camera 1 as the initial pose and the optimal projection viewpoint as the target pose to plan the robot's motion path and drive the robot to move the projection device to the target pose. During this process, the target tracking unit estimates the pose transformation matrix of camera 1 in real time and corrects the robot's motion path until the pose deviation is less than the preset value. 11) Assembly process projection guidance: the projection device projects the process information and projection surface to the corresponding position of the assembly base part, and the operator completes the assembly of the parts to be assembled according to the guidance information; 12) Online detection of assembly status: Camera 2 captures images of the assembly site. The assembly status recognition unit extracts the contour information of the scene image and the inspection model respectively, and uses the contour matching algorithm to determine whether there are any errors or omissions in the assembly parts. 13) Repeat steps 9 to 12 until all parts are assembled; The guide information is used to guide the assembly of the parts to be assembled, and includes a footprint diagram, marking information and indication marks.

Images