An ai vision positioning method and system for robot automatic assembly

Abstract

The application relates to the technical field of robots, and discloses an AI vision positioning method and system for robot automatic assembly, which comprises a composite perception system, an active physical excitation module and a central processing unit; the composite perception system is integrated with a polarization camera, a thermal imager and a structured light projector; and the central processing unit is internally arranged with a physical causal decoupling network and a health state manifold model. The method comprises the following steps: in an offline stage, a physical characteristic enhanced point cloud of a standard part is collected through a detection, excitation and response protocol, so as to train the physical causal decoupling network and construct a health state manifold representing the state of qualified materials; in an online stage, the state of a material to be assembled is diagnosed, and an expected process path connecting the current state and the target state is generated on the manifold. The application solves the positioning and control problems in complex scenes such as high reflectivity and transparency, and realizes high-precision in-situ closed-loop assembly.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to an AI visual positioning method and system for automated robot assembly. Background Technology

[0002] Robotics automation technology plays a crucial role in modern precision manufacturing, especially in high-precision assembly tasks. Currently, robotic assembly systems commonly employ machine vision for initial positioning, supplemented by force / torque sensors for force control during the contact phase; this combination has become the standard technical approach in the industry.

[0003] However, when faced with increasingly complex industrial scenarios, this technical approach, which relies on traditional optical imaging, exposes its inherent limitations. For highly reflective metals or transparent / semi-transparent materials, severe specular reflection or light transmission occurs on their surfaces, which seriously interferes with the effective extraction of image information. This causes traditional geometric feature matching and 3D reconstruction algorithms to fail, making it impossible to obtain accurate pose information of the material.

[0004] While force / torque sensors can provide feedback during the contact phase, their effect is often delayed. They can only react when the contact force or torque has exceeded the safety threshold, making it difficult to anticipate and avoid the risk of jamming or scratching caused by minor deviations during assembly. More importantly, existing vision systems and force control systems are usually separate or loosely coupled, lacking a unified framework to perceive and understand the deep physical state changes of the assembly interface, such as changes in microscopic material properties caused by contact stress or thermal effects. This makes it impossible to achieve truly online, closed-loop, adaptive control at the physical level.

[0005] Therefore, this invention proposes an AI visual positioning method and system for automated robot assembly to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an AI vision positioning method and system for automated robot assembly, which solves the perception limitations of traditional vision when dealing with complex materials with high reflectivity, transparency, and other physical properties, and enables online closed-loop control of the physical state during the precision assembly process of robots.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an AI visual positioning method for automated robot assembly, the method comprising the following steps:

[0008] S1. Collect the dynamic physical response of the material to be assembled under active physical excitation, and generate a physical property enhanced point cloud containing three-dimensional spatial coordinates, linear polarization degree, polarization angle and surface temperature information;

[0009] S2. Input the enhanced physical property point cloud into the physical causal decoupling network to extract physical feature vectors that characterize the intrinsic physical properties of the material;

[0010] S3. Project the physical feature vector onto the pre-constructed healthy state manifold to calculate the initial state point of the material;

[0011] S4. Generate the desired process path connecting the initial state point and the target state point on the healthy state manifold;

[0012] S5. During the assembly process, the deviation vector between the actual process trajectory of the material and the expected process path is calculated in real time, and a composite physical correction command is generated based on the deviation vector to adjust the assembly action.

[0013] Preferably, in step S1, the step of collecting the dynamic physical response of the material to be assembled under active physical excitation and generating a physical property-enhanced point cloud containing three-dimensional spatial coordinates, degree of linear polarization, polarization angle, and surface temperature information includes:

[0014] The polarization camera, thermal imager, and structured light projector in the composite sensing system are pre-calibrated coaxially;

[0015] The composite sensing system simultaneously captures polarization images and thermal images of the material to be assembled under active physical excitation.

[0016] Based on the polarization image, the Stokes vector is calculated, and the linear polarization degree and polarization angle of each pixel are solved according to the Stokes vector.

[0017] The structured light projector projects a structured light pattern onto the surface of the material to be assembled, and the polarization camera captures the structured light pattern. The three-dimensional spatial coordinates of the material are then reconstructed using the triangulation principle.

[0018] For each coordinate point in the three-dimensional spatial coordinates, associate a physical property vector containing the degree of linear polarization, polarization angle, and surface temperature information extracted from the thermal image.

[0019] Preferably, the polarization camera is a focal plane polarization camera; the calculation of the Stokes vector is based on the polarization intensity images of the four orthogonal directions captured by the polarization camera in the composite sensing system in a single exposure.

[0020] Preferably, in step S2, the physical causal decoupling network includes multiple parallel sub-networks constructed based on prior physical knowledge; the sub-networks include at least:

[0021] The specular reflection decoupling subnet takes geometric and polarization information of each point and its local neighborhood in the physical property enhancement point cloud as input and outputs an estimated value of the surface normal direction of each point in the physical property enhancement point cloud.

[0022] The transparent material penetration analysis subnet takes polarization and temperature gradient information from the physical property enhancement point cloud as input, weights the polarization and temperature gradient information through an attention mechanism, and outputs the probability value of each point in the physical property enhancement point cloud belonging to the transparent surface.

[0023] Preferably, in step S3, the step of projecting the physical feature vector onto a pre-constructed healthy state manifold to calculate the initial state point of the material includes:

[0024] Collect enhanced point clouds of physical properties from multiple standard parts;

[0025] The enhanced point cloud of the physical properties of multiple standard parts is input into a physical causal decoupling network to extract the corresponding set of physical feature vectors.

[0026] A neighborhood graph is constructed based on the Euclidean distance between vectors in the set of physical feature vectors.

[0027] Calculate the shortest path length between any two vectors on the neighborhood graph, and use it as the geodesic distance between the two vectors to form a geodesic distance matrix.

[0028] Applying a multidimensional scaling algorithm to the geodesic distance matrix yields a point set in a low-dimensional space, which is the healthy manifold.

[0029] Preferably, in step S4, the step of generating the desired process path connecting the initial state point and the target state point on the healthy state manifold includes:

[0030] On the neighborhood graph corresponding to the healthy state manifold, with the initial state point and the target state point as the starting and target nodes, the shortest path algorithm is applied to calculate a sequence composed of discrete state points, which is the desired process path.

[0031] Each discrete state point in the desired process path is input into a pre-trained inverse mapping model, which maps the discrete state points into a nominal configuration vector containing robot pose, active physical excitation parameters, and force control parameters, thereby generating a series of nominal command sequences.

[0032] Preferably, in step S5, the composite physical correction instruction includes a combination of at least one or more of the following instructions:

[0033] Robot joint space fine-tuning commands used to fine-tune the position of robot joints;

[0034] Active excitation parameter adjustment command for adjusting the excitation parameters of the active physical excitation module;

[0035] Force control adjustment commands are used to adjust the force or torque applied by the robot's end effector.

[0036] Preferably, step S5, which involves calculating the deviation vector between the actual process trajectory of the material and the desired process path in real time during the robot's assembly operation, includes:

[0037] During the assembly process performed by the robot, the physical property enhancement point cloud of the material is captured in real time through the physical interaction interface.

[0038] The real-time generated physical property enhanced point cloud is input into the physical causal decoupling network, and the output of the physical causal decoupling network is the actual process trajectory.

[0039] Preferably, step S5, which involves generating a composite physical correction instruction based on the deviation vector to adjust the assembly action, includes:

[0040] The composite physical correction command is superimposed with a nominal command sequence pre-generated based on the desired process path to form the final control command applied to the robot and the active physical excitation module.

[0041] The present invention also provides an AI vision positioning system for automated robot assembly, the system comprising:

[0042] The active physical excitation module is used to apply active physical excitation to the materials to be assembled;

[0043] A composite sensing system is used to collect the dynamic physical response of the material to be assembled under active physical excitation, so as to generate a physical property enhanced point cloud containing three-dimensional spatial coordinates, linear polarization degree, polarization angle and surface temperature information;

[0044] The intelligent analysis module is used to receive the physical property enhancement point cloud and extract physical feature vectors representing the intrinsic physical properties of the material through a physical causal decoupling network; and to project the physical feature vectors onto a pre-constructed healthy state manifold to calculate the initial state point of the material.

[0045] The decision control module is used to generate a desired process path connecting the initial state point and the target state point on the health state manifold; and during the robot's assembly action, to calculate in real time the deviation vector between the actual process trajectory of the material and the desired process path, and to generate a composite physical correction command based on the deviation vector to adjust the assembly action.

[0046] This invention provides an AI-based visual positioning method and system for automated robot assembly. It offers the following advantages:

[0047] 1. This invention enhances point clouds by collecting physical property data containing linear polarization degree and polarization angle information, and processes this information using a physical causal decoupling network. This effectively separates optical interference from highly reflective or transparent material surfaces, thereby extracting stable material features. Compared to traditional methods that rely solely on geometric or color information, this invention improves the positioning accuracy and environmental adaptability for optically complex materials.

[0048] 2. This invention projects the physical feature vectors of the materials to be assembled onto a pre-constructed healthy state manifold, achieving not only geometric positioning but also a quantitative assessment of the materials' intrinsic physical state. This assembly feasibility diagnosis mechanism based on manifold distance can effectively identify materials with potential defects before assembly begins, thereby avoiding assembly failures due to material quality issues and improving the overall yield of the production process.

[0049] 3. This invention generates the desired process path on a healthy manifold and treats the entire assembly process as a controlled physical state evolution process. By monitoring the deviation between the actual process trajectory and the desired process path in real time, it can proactively detect potential physical anomalies during assembly, achieving dynamic process control of assembly quality, rather than the post-event detection of traditional methods.

[0050] 4. When the assembly process deviates from the expected process path, the composite physical correction command generated by this invention can coordinately adjust the robot's geometric position, the energy output of the active physical excitation module, and the force of the end effector. This multi-dimensional, integrated closed-loop feedback control method enables the system to cope with complex physical interactions, actively eliminate potential assembly obstacles, and enhance the system's robustness and assembly success rate under non-ideal conditions. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0052] Figure 2 This is a flowchart of the method of the present invention;

[0053] Figure 3This is a schematic diagram of the physical property enhancement point cloud data structure of the present invention;

[0054] Figure 4 This is a schematic diagram of the in-situ self-consistency verification and control closed loop of the present invention.

[0055] Among them, 10 is the robot; 20 is the composite sensing system; 21 is the polarization camera; 22 is the thermal imager; 23 is the structured light projector; 30 is the active physical excitation module; 40 is the central processing unit; 41 is the intelligent analysis module; 42 is the decision control module; 50 is the robot controller; 60 is the material to be assembled; 70 is the physical property enhanced point cloud; 80 is the health state manifold; 81 is the desired process path; 82 is the actual process trajectory; 83 is the deviation vector; and 84 is the composite physical correction instruction. Detailed Implementation

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] See attached document Figure 1 , Figure 1 This is a schematic diagram of the overall system architecture according to an embodiment of the present invention. The present invention provides an AI vision positioning system for automated robot assembly, which may include: a robot 10, a composite perception system 20, an active physical excitation module 30, a central processing unit 40, and a robot controller 50.

[0058] Robot 10 is a multi-degree-of-freedom industrial robot, serving as the actuator for assembly actions. The composite sensing system 20 and the active physical excitation module 30 are mounted to the end effector interface of robot 10 via a flange. The composite sensing system 20 integrates a polarization camera 21, a thermal imager 22, and a structured light projector 23; the optical axes of these three devices are coaxially calibrated to ensure alignment of their field of view centers. The active physical excitation module 30 is a controllable-power infrared laser. The central processing unit 40 is an industrial computer, which houses an intelligent analysis module 41 and a decision control module 42.

[0059] In terms of hardware connectivity, the composite sensing system 20 and the active physical excitation module 30 communicate and exchange commands with the central processing unit 40 via a gigabit industrial Ethernet interface. The central processing unit 40 is connected to the robot controller 50 via an industrial fieldbus protocol (such as Ether-CAT or PROFINET) to send motion control commands.

[0060] To ensure that all parts of the system operate under a unified spatial reference, coordinate system calibration and unification must be completed during system initialization. First, using a hand-eye calibration method, the homogeneous transformation matrix of the coordinate system of the composite sensing system 20 relative to the coordinate system of the end effector flange of the robot 10 is calculated. Second, the internal calibration of the composite sensing system 20 is completed, determining the internal parameters of the polarization camera 21, thermal imager 22, and structured light projector 23, as well as their relative pose transformation relationships. Finally, the beam axis of the active physical excitation module 30 is calibrated relative to the coordinate system of the composite sensing system 20. Through the above calibration, the data collected by all sensors and the action point of the excitation module can be transformed to the base coordinate system of the robot 10, achieving spatial coordinate unification.

[0061] In a specific calibration process, robot 10, carrying a composite sensing system 20, observes a fixed calibration board (e.g., a checkerboard calibration board) from multiple different poses. At each pose, the composite sensing system 20 captures an image of the calibration board and calculates its pose in the sensor coordinate system. Simultaneously, robot controller 50 records the pose of the current end effector flange coordinate system relative to the base coordinate system. The hand-eye transformation matrix can be accurately calculated by solving a matrix equation of the form AX = XB.

[0062] See attached document Figure 2 , Figure 2 This is a flowchart of a method according to an embodiment of the present invention. The AI ​​visual positioning method for automated robot assembly provided by the present invention is divided into an offline stage and an online stage in its overall process.

[0063] The core task of the offline phase is to learn and construct a mathematical model that characterizes the intrinsic physical properties of qualified assembly materials. This phase takes a batch of standard parts as input, collects their physical property enhanced point cloud 70 through a probe, excitation, and response protocol, then trains a physical causal decoupling network, and finally uses the set of feature vectors extracted by the network to construct a healthy state manifold 80.

[0064] The core task of the online phase is to use the offline-built model to diagnose, plan, and implement in-situ closed-loop control of the materials 60 to be assembled on-site. This phase first diagnoses the condition of the materials, and then plans the desired process path 81 based on the health state manifold 80. During the assembly process, the system continuously monitors the actual process trajectory 82 of the materials, calculates its deviation vector 83 from the desired process path 81, and generates composite physical correction instructions 84 to adjust the assembly actions in real time until the task is completed.

[0065] The offline phase of this invention aims to construct a baseline model, namely the healthy state manifold 80, representing the physical state of qualified materials for subsequent online assembly tasks. The first step of this phase is to collect and generate a data structure containing multi-dimensional physical information, defined in this invention as a Physically-Augmented-Point-Cloud (PAPC) 70. Subsequently, this phase also includes training a physically-causal decoupling network and using PAPC to learn and construct the healthy state manifold 80.

[0066] See attached document Figure 3 , Figure 3 This is a schematic diagram of a physical property enhancement point cloud data structure according to an embodiment of the present invention. The acquisition and generation of the physical property enhancement point cloud 70 is performed on a batch of pre-selected standard parts that meet quality requirements. For each standard part, the system executes a set of detection, excitation, and response protocols. First, in an unexcited state, the composite sensing system 20 acquires static data of the material once. Subsequently, the active physical excitation module 30 (controllable infrared laser) excites a specific area on the surface of the material according to preset parameters. The excitation mode can be single-point pulse, line scan, or area scan. The excitation parameters are precisely controlled; for example, the laser power is set to 500mW, the pulse width is 100ms, and it acts on a circular area with a diameter of 1mm.

[0067] Simultaneously with the initiation of active physical excitation, an external synchronization trigger signal is sent to the composite sensing system 20 and the active physical excitation module 30. The polarization camera 21 and thermal imager 22 of the composite sensing system 20 are triggered to continuously capture the dynamic physical response process of the material surface at a high frame rate (e.g., 100 fps). The polarization camera 21 is a focal plane array (DoFP) camera, with each 2x2 pixel macrocell on its sensor covering a micro-polarizer array in four directions: 0°, 45°, 90°, and 135°. Therefore, polarization intensity images (I0, I...) in four orthogonal directions can be simultaneously captured in a single exposure. 45 ,I 90 ,I 135 ).

[0068] Based on these four polarization intensity images, the system calculates the first three components S0, S1, and S2 of the normalized Stokes vector:

[0069] S0=I0+I 90 ;

[0070] S1=I0-I 90 ;

[0071] S2=I 45 -I 135 ;

[0072] In the formula, S0 represents the total light intensity, and S1 and S2 represent the linear polarization components.

[0073] Subsequently, the linear polarization degree L and polarization angle A of each pixel are calculated using the following formula:

[0074]

[0075] In the formula, atan2(y,x) is a bivariate arctangent function. Simultaneously, the structured light projector 23 projects a structured light pattern onto the material surface, which is then captured by the polarization camera 21 (using its total light intensity image S0). The three-dimensional spatial coordinates (X,Y,Z) of the material surface are reconstructed using the triangulation principle. The thermal imager 22 simultaneously records the temperature field distribution T generated on the material surface due to thermal excitation. Finally, through the calibrated relative pose relationships between the sensors, each three-dimensional spatial coordinate point is precisely registered and fused with its corresponding linear polarization degree L, polarization angle A, and surface temperature T, forming a physical property enhanced point cloud 70 containing multi-dimensional information. Its data structure is P. i =(X i ,Y i Z i ,L i A i ,T i ).

[0076] Next, a Physical Causal Decoupling Network (PCD-Net) is trained using the collected Physical Properties Augmented Point Cloud 70 dataset. The network takes the material's PAPC as input and outputs a fixed-dimensional high-dimensional physical feature vector. The PCD-Net architecture consists of a backbone network (such as PointNet++) for extracting local and global geometric features of the point cloud, and multiple parallel sub-networks connected after the backbone network for decoupling specific physical properties.

[0077] One subnet, designed for specular reflection decoupling, is based on the physical laws revealed by Fresnel equations. Its input consists of the geometric and polarization information of each point in PAPC and its local neighborhood. This subnet is trained to learn the mapping between polarization information in highly linearly polarized regions (typically caused by specular reflection) and the real surface normal. Its loss function includes a penalty term based on the cosine similarity between the predicted and real normals. Another subnet, for transparent material penetration analysis, integrates the abrupt changes in polarization angle at the edges of transparent objects with the different thermal conduction responses of the object's surface and its underlying substrate under active thermal excitation. This subnet uses an attention mechanism to weight polarization and temperature gradient information to distinguish whether a point belongs to the real surface of a transparent object or the transmitted background. Its training objective is a semantic segmentation task, and it uses cross-entropy loss. The network's total loss function is a weighted sum of the loss functions of each subnet.

[0078] Finally, the health state manifold 80 is learned and constructed. The PAPC70 values ​​of all standard parts are input one by one into the trained physical causal decoupling network, resulting in a set of high-dimensional physical feature vectors [v]. j The feature vector set is then processed using the Isomap manifold learning algorithm. First, for each feature vector in the set, its Euclidean distance to all other vectors is calculated, and a K-nearest neighbor graph is constructed. Then, Dijkstra's algorithm is used on this graph to calculate the shortest path length between any two points, which serves as an approximation of the geodesic distance between the two points, resulting in a geodesic distance matrix. Finally, this geodesic distance matrix is ​​input into a multidimensional scaling (MDS) algorithm to find a low-dimensional embedding space that optimally preserves these geodesic distances. The set of points in this low-dimensional space constitutes the health state manifold 80, representing the physical state of qualified materials. This manifold and its geometric properties are stored in the central processing unit 40 for use in the online phase.

[0079] See attached document Figure 1 and attached Figure 2 In the offline phase, a Physical Causal Decoupling Network (PCD-Net) is designed and trained using the physical property enhancement point cloud dataset collected and generated in the preceding steps. This network is deployed in the intelligent analysis module 41 of the central processing unit 40, and its function is to map the input, high-dimensional and complex physical property enhancement point cloud into a fixed-dimensional physical feature vector that can characterize the intrinsic physical properties of the material.

[0080] The neural network architecture of the Physical Causal Decoupling Network comprises a backbone network and multiple parallel, decoupled subnetworks. The backbone network employs PointNet++'s hierarchical point set feature learning structure to process the input physical property-enhanced point cloud. Through multi-level sampling and grouping operations, this backbone network effectively extracts geometric features from the local neighborhood to the global morphology of the point cloud data. Simultaneously, it aggregates the linear polarization degree, polarization angle, and surface temperature information associated with each point, ultimately generating a local feature descriptor containing rich contextual information for each point in the point cloud.

[0081] Following the backbone network, the physical causal decoupling network is connected in parallel to multiple subnetworks. Each subnetwork is designed to focus on solving a specific physical perception problem by decoupling independent physical properties using different combinations of physical information. In this embodiment, it includes at least one specular reflection decoupling subnetwork and one transparent material penetration analysis subnetwork.

[0082] A specular reflection decoupling subnet is configured to handle specular highlights caused by specular reflection from materials such as metals. This subnet utilizes physical properties to enhance the geometric features of each point and its neighborhood in the point cloud, along with the degree of linear polarization L and the polarization angle A as input. According to Fresnel's equations, specularly reflected light from smooth dielectric or metallic surfaces exhibits highly linear polarization, and its polarization angle is directly related to the incident surface. By learning this physical law, this subnet establishes a mapping between the polarization information of highly linearly polarized regions and the normal to the actual surface of the material, outputting an accurate estimate of the surface normal direction.

[0083] A transparent material penetration analysis subnet is configured to identify the true geometric boundaries of transparent materials. This subnet integrates the abrupt change in polarization angle A at the edge of the transparent material due to refraction, and the difference in surface temperature T caused by the different thermal conductivity and heat capacity of the transparent material and its underlying substrate under active physical excitation. This subnet uses an attention mechanism to weight polarization and temperature gradient information to determine whether a data point originates from the surface of the transparent material or from the background perceived after penetrating the material. Its output is the probability that each point in the point cloud belongs to the transparent surface category.

[0084] The training process of the physical causal decoupling network is based on a dataset containing a large number of standard component physical property augmented point clouds with ground truth annotations. For the specular reflection decoupling subnetwork, its loss function L... normal Designed to minimize the angular deviation between the predicted surface normal and the ground truth normal, specifically calculated using negative cosine similarity. For the i-th point in the physically enhanced point cloud, based on its predicted unit normal vector n... pred,i and the unit vector of the true value normal line n gt,i Its loss function is:

[0085]

[0086] In the formula, N p Let n be the total number of points in the point cloud. pred,i Let n be the unit vector of the surface normal at the i-th point predicted by the network. gt,i Let be the unit vector of the true surface normal of the i-th labeled point in the dataset, and · denotes the dot product of the vectors.

[0087] For the transparent material penetration analysis subnet, its loss function L seg The binary cross-entropy loss function is used to calculate and measure the accuracy of the semantic segmentation results. For the i-th point, the predicted probability q based on its belonging to the transparent surface category is calculated. i And truth label y i Its loss function is:

[0088]

[0089] In the formula, y i Let y be the truth label for the i-th point. If the point belongs to a transparent surface, then y i =1, otherwise y i =0; q i Predict the probability that the i-th point belongs to a transparent surface for the network.

[0090] The total loss function L of the network total This is a weighted sum of the loss functions of each sub-network to balance the contributions of different tasks during training.

[0091] L total =w normal L normal +w seg L seg ;

[0092] In the formula, w normal and w seg These are pre-defined hyperparameter weights used to balance different loss terms. The network is trained using the backpropagation algorithm, employing the Adam optimizer to update the network parameters until the total loss function converges to a stable value. After training, the activation values ​​extracted from specific layers of the physical causal decoupling network constitute a physical feature vector that comprehensively characterizes the physical properties of the material.

[0093] See attached document Figure 1 and attached Figure 2After the physical causal decoupling network is trained, the next step in the offline phase is to learn and construct a healthy state manifold 80. This process first utilizes the trained physical causal decoupling network to convert the physical property enhancement point cloud dataset of all standard parts into a set of high-dimensional physical feature vectors. Specifically, the physical property enhancement point cloud of each standard part is input to the input of the network, and its output is extracted from the global pooling layer after the network backbone, thereby generating a D-dimensional physical feature vector v for each standard part. After processing all M standard parts, a set of feature vectors is obtained. in

[0094] Subsequently, the isomap manifold learning algorithm is used to process the feature vector set VV to discover and represent its inherent low-dimensional geometric structure. The execution of the Isomap algorithm includes the following steps:

[0095] The first step is to construct a neighborhood graph. This involves analyzing the set of feature vectors. Each vector v in j The distance to each vector is calculated using Euclidean distance, and its k nearest neighbors are found. Based on this, an undirected graph is constructed. The vertices of the graph are the set of eigenvectors. If vector v k It is v j If one of the k nearest neighbors is v, then in v j and v k There exists an edge between them, the weight of which is their Euclidean distance d. E (v j ,v k ).

[0096] The second step is to calculate the geodesic distance. This is to estimate the distance between any two eigenvectors v. j and v k The true distance along the surface of the data manifold, i.e., the geodesic distance, is used to calculate the shortest path between all pairs of vertices on the neighborhood graph G using Dijkstra's algorithm. The length of this shortest path is the geodesic distance d between the two points. G (v j ,v k This approximates the value of ). Through this step, an M×M matrix D containing the geodesic distances between all eigenvector pairs is obtained. G .

[0097] The third step is to apply multidimensional scaling (MDS) for dimensionality reduction embedding. This involves embedding the geodesic distance matrix D... G It serves as input to the classic Multidimensional Scaling (MDS) algorithm. The goal of the MDS algorithm is to find a target low-dimensional Euclidean space. The point set in (where d << D) This process aims to make the Euclidean distance between these points as close as possible to the original geodesic distance. This is achieved by minimizing the following stress function:

[0098]

[0099] In the formula, d G (v j ,v k ) is the feature vector v j and v k Geodesic distance between them; For v j and v k The new coordinates in the target low-dimensional space; ||·|| represents the Euclidean norm.

[0100] By solving the above optimization problem, a set of low-dimensional coordinate points Y is obtained, which constitutes a discretized representation of the intrinsic structure of the original high-dimensional data. This is the health state manifold 80 in this invention. The health state manifold 80 and its geometric characteristics (such as point coordinates, adjacency relationships, etc.) are completely stored in the central processing unit 40, serving as a benchmark model for material status diagnosis, assembly path planning, and process control in the online stage.

[0101] See attached document Figure 1 Appendix Figure 2 and attached Figure 3 When the system enters the online execution phase, the primary task is to perform status diagnosis and assembly feasibility prediction on the material 60 to be assembled located in the work area. The robot 10 first moves to the preset observation position, aligning its end-effector-mounted composite sensing system 20 with the material 60 to be assembled.

[0102] The system then executes a rapid detection, excitation, and response protocol on the material 60 to be assembled, with the process consistent with the protocol used when collecting standard part data in the offline stage. The composite sensing system 20 collects the dynamic physical response of the material and generates an enhanced point cloud of the material's physical properties in real time. This enhanced point cloud is immediately transmitted to the intelligent analysis module 41 in the central processing unit 40. The intelligent analysis module 41 calls the pre-trained physical causal decoupling network to perform forward propagation calculations on the input enhanced point cloud of physical properties to extract a physical feature vector characterizing the material's intrinsic physical properties.

[0103] The physical feature vector is a fixed-dimensional (D-dimensional) vector composed of multiple concatenated sub-feature vectors. Each sub-feature vector is output by a different functional part of the physical causal decoupling network and corresponds to a specific physical or geometric property. In this embodiment, the physical feature vector v testThe composition is as follows:

[0104] Global Geometry Descriptor (v geom ): Extracted from the global feature aggregation layer of the backbone network (PointNet++ structure) of the physical causal decoupling network, this sub-vector encodes the overall three-dimensional shape, size and topology of the material.

[0105] Surface normal distribution histogram (v normal This is derived from the output of the specular reflection decoupling subnet. This subnet estimates the surface normals for a large number of points in the point cloud. The system quantizes all estimated normal directions and statistically summarizes them into a normalized orientation histogram, which is v. normal It describes the macroscopic curvature and orientation distribution of the material surface.

[0106] Transparent property descriptor (v trans ): Calculated from the output of the transparent material penetration analysis subnet. This subnet outputs the probability that each point in the point cloud belongs to a transparent surface. The descriptor v trans It consists of two values: the proportion of the total number of points identified as transparent to the total number of points in the material point cloud, and the average confidence probability of these points identified as transparent.

[0107] By concatenating the above sub-feature vectors, i.e. v test =[v geom ||v normal ||v trans This forms the final, comprehensive D-dimensional physical feature vector.

[0108] To quantify the degree of consistency between the state of the material to be assembled 60 and the qualified standard parts, the system calculates its physical characteristic vector v. test The projection distance to the healthy state manifold 80 constructed offline. In this embodiment, the projection distance is defined as the minimum Euclidean distance between the feature vector and the feature vectors of all standard components constituting the healthy state manifold 80. This is based on the set of feature vectors with the same structure generated from the standard components, obtained in the offline stage. Calculate the projected distance d proj The formula is as follows:

[0109]

[0110] In the formula, v test This represents the D-dimensional physical feature vector of the material to be assembled. For the offline stage, extracted from all standard parts, and related to v test A set of D-dimensional physical eigenvectors with the same composition, which discretely defines the healthy state manifold 80; v j For set Any eigenvector in d; E (v test ,v j ) represents vector v test With vector v j The Euclidean distance between them.

[0111] Calculate the projected distance d proj Then, the decision control module 42 compares it with a pre-set health status threshold τ. health Compare the threshold τ. health In the offline phase, based on the feature vector set of standard parts It is determined by statistical analysis of the internal distribution; for example, it can be set as a set. A specific percentage of the maximum geodesic distance between all vectors.

[0112] If d proj ≤τ health If the physical state of the material to be assembled 60 is sufficiently close to that of the standard part, and the existing physical deviations are within an acceptable range, the system determines that the material is qualified and sets v. test The nearest projection point on the healthy state manifold 80 is taken as the initial state point of the material, and subsequent assembly path planning steps are continued.

[0113] If d proj >τ health If the physical state of the material to be assembled 60 differs significantly from that of the standard part, it indicates potential problems such as material defects, residual processing stress, or geometric deformation. The system determines the material to be unqualified, immediately halts the assembly process for that material, sends an alarm signal to the central control system, and simultaneously records the material's physical characteristic vector v. test and projection distance d proj For further analysis.

[0114] See attached document Figure 1 , Figure 2 and Figure 4 After the material 60 to be assembled is determined to be qualified, the decision control module 42 then enters the generation step of the desired process path 81. This path defines a physically optimal evolution trajectory from the current state of the material to the ideal assembly completion state.

[0115] The input to this process is two points on the healthy manifold 80: the initial state point y of the material. initial and the predefined target state point y target The initial state point y initial This refers to the physical characteristic vector v of the material to be assembled in the previous diagnostic step. testThe projection point on the healthy state manifold 80. The target state point y. target The point cloud is obtained by collecting the physical characteristics of a standard part that is already in an ideal assembly state during the offline stage, extracting its feature vector through a physical causal decoupling network, and then mapping it onto the healthy state manifold 80 through a manifold learning algorithm. This point is stored in the central processing unit 40.

[0116] The expected process path 81 is calculated to connect the initial state point y. initial and target state point y target The geodesic between them. This calculation is performed on the neighborhood graph G constructed in the offline phase. Specifically, the system uses y initial and y target Using the starting and target nodes as the starting and target nodes, run Dijkstra's algorithm on graph G to find the shortest path connecting the two nodes. This shortest path consists of a series of discrete points on the manifold, which is the desired process path P. exp :

[0117] P exp =(y1,y2,...,y K );

[0118] In the formula, y k Let y1 be the k-th state point on the path. initial ,y K =y target K is the number of off-path steps.

[0119] Next, we need to define the desired process path P in the high-dimensional physical feature space. exp This is converted into a sequence of nominal instructions in physical space that can be executed by robot 10 and active physical arousal module 30. This conversion is achieved through a pre-trained inverse mapping model f. inv accomplish.

[0120] In this embodiment, the inverse mapping model f inv This is a multilayer perceptron (MLP) neural network. The number of nodes in the input layer is equal to the dimension d of the healthy state manifold 80, and the number of nodes in the output layer is equal to the configuration vector C. j The network uses paired samples (y) collected and stored in the offline phase. j C j The model is trained using paired samples (yi, yi) as training data, employing a backpropagation algorithm. The goal is to minimize the mean squared error between the predicted and actual configuration vectors. The model is trained offline, using paired samples (yi, ...). j C j ), where y jIt is a point on the healthy manifold 80, and C j It is the configuration vector corresponding to the state of the robot system when the state is acquired.

[0121] Configuration Vector C j It contains all the control parameters required to execute this physical state, and its structure is as follows:

[0122] C j =[P j,robot ||C j,excite ||C j,force ];

[0123] In the formula, P j,robot C represents the pose vector of the robot's end effector in the base coordinate system. j,excite This is the parameter vector for the active physical excitation module 30, containing excitation power and duration; C j,force This is the parameter vector of the force control module in the robot controller 50, which includes the target contact force and torque.

[0124] By passing the desired process path P exp Each state point y on k The inputs are sequentially fed into the inverse mapping model f. inv From this, a corresponding nominal configuration vector sequence can be obtained:

[0125] C k,nom =f inv (y k ), for k=1,...,K;

[0126] In the formula, C k,nom For the state point y k The nominal configuration vector.

[0127] This nominally configured vector sequence {C 1,nom C 2,nom ,...,C K,nom This constitutes the nominal instruction sequence. The decision control module 42 sends this sequence to the robot controller 50 and the active physical arousal module 30. The robot controller 50 uses inverse kinematics to convert the pose command P in the sequence into... k,robot Convert the commands into motion instructions for each joint of the robot and set the corresponding force control parameters C. k,force This drives robot 10 to begin performing assembly actions.

[0128] See attached document Figure 1 , Figure 2 and Figure 4After robot 10 begins executing the assembly actions driven by the nominal instruction sequence, the system immediately enters a high-frequency, in-situ self-consistency verification and closed-loop control loop. This loop is the core execution link of the invention, ensuring that the assembly process always closely follows the preset physical state evolution path.

[0129] During the movement of robot 10, composite sensing system 20 continuously focuses on the physical interaction interface between materials, such as the contact area between a pin and a hole. Within a fixed control cycle (e.g., every 20 milliseconds), the system performs a complete in-situ monitoring operation. This operation includes: active physical excitation module 30 applying physical excitation according to the current command; composite sensing system 20 capturing the dynamic response of the interaction interface and generating an enhanced point cloud of physical properties at that instant. This enhanced point cloud of physical properties is immediately input into a physical causal decoupling network to calculate a low-dimensional feature point y representing the current actual physical state of the material. actual (t). Over time, this series of continuously calculated actual state points constitutes an actual process trajectory 82, P, on the healthy state manifold 80. actual =(y actual (t1),y actual (t2),...).

[0130] In each control cycle t, the decision control module 42 synchronously calculates the deviation vector 83 between the actual process trajectory 82 and the desired process path 81. First, the system, based on the nominal time or completion rate of the current assembly process, calculates the deviation vector 83 from the pre-generated desired process path P. exp Obtain the target state point y at the current moment. exp (t). Subsequently, in the low-dimensional Euclidean space where the healthy manifold 80 resides, the deviation vector e(t) is calculated:

[0131] e(t) = y exp (t)-y arctan (t);

[0132] In the formula, e(t) is the deviation vector at the current time; y exp (t) represents the state point that the expected process path should reach at the current time; y arctan (t) represents the actual state point of the material calculated through in-situ monitoring.

[0133] After obtaining the deviation vector e(t), a feedback controller in the decision control module 42 converts it into a composite physical correction command 84, denoted as ΔC(t). In this embodiment, the feedback controller is a multi-input multi-output (MIMO) proportional-integral-derivative (PID) controller. Its discrete-time control law is as follows:

[0134]

[0135] In the formula, K p ,K i ,K d These are the proportional, integral, and differential gain matrices, which are determined during system calibration; the gain matrix K p ,K i ,K d The initial values ​​can be obtained through system identification methods. Specifically, during the offline phase, a series of known small disturbance control inputs are applied to the system, and the responses generated on the healthy state manifold are recorded. Based on this, a dynamic response model of the system is established. Subsequently, engineering methods such as Ziegler-Nichols or optimal control theories such as LQR (linear quadratic regulator) can be used to tune the PID parameters to obtain stable and fast response characteristics. In practical applications, these gain matrices can be full matrices with non-zero off-diagonal elements to handle coupling effects between different physical dimensions; Δt is the control cycle duration. The composite physical correction command ΔC(t) has the same structure as the nominal configuration vector, i.e., ΔC(t) = [ΔP robot (t)||ΔC excite (t)||ΔC force [(t)], which correspond to the adjustment amounts of robot pose, active excitation parameters and force control parameters, respectively.

[0136] Finally, at the end of each control cycle, the system executes the superposition and execution of instructions. The decision control module 42 combines the calculated composite physical correction instruction ΔC(t) with the nominal configuration vector C at the current moment obtained from the nominal instruction sequence. nom (t) are superimposed to generate the final control instruction C applied to the hardware. final (t):

[0137] C final (t)=C nom (t)+ΔC(t);

[0138] This final control command C final (t) is sent to the robot controller 50 and the active physics excitation module 30. The robot controller 50 adjusts the pose correction amount ΔP in the instruction. robot (t) and force control correction ΔC force (t) is used to fine-tune the robot's motion trajectory and end effector force. Simultaneously, the active physical excitation module 30 adjusts the excitation parameter correction amount ΔC. excite (t) Adjust its output. In this way, the system forms a tight closed loop, converting deviations in the physical state into cooperative corrective actions in the physical space in real time, thereby guiding the actual assembly process to always converge to the desired physical state evolution path.

[0139] To further illustrate the technical solution of the present invention, a specific embodiment will be described below.

[0140] Scenario Description: The task of this embodiment is to precisely insert a highly polished stainless steel pin into a slightly chamfered hole in an aluminum base. The pin diameter is 10mm, the hole diameter is 10.05mm, and the clearance is 0.05mm. Due to the specular reflective properties of the pin surface, traditional vision systems struggle to accurately identify its three-dimensional edges and precise orientation. Furthermore, the minute clearance necessitates precise control of the contact force and alignment during insertion.

[0141] Offline Phase: First, prepare 50 sets of standard pins and aluminum bases that meet the highest quality standards. Robot 10, in force control mode, inserts each standard pin at an extremely slow speed with precise force feedback, completing an ideal insertion process. During this process, the composite sensing system 20 continuously records the enhanced physical feature point cloud at a frequency of 50Hz, from pin separation from the orifice to initial contact and then to complete insertion. The collected thousands of enhanced physical feature point cloud data are input into a pre-trained physical causal decoupling network to generate a set of high-dimensional physical feature vectors. Finally, this vector set is processed using the Isomap algorithm to construct a healthy state manifold 80 specifically for this insertion task, representing the continuous evolution from the initial state to the completed state. A specific point in this manifold, namely the point obtained by mapping the physical feature vector of the fully inserted pin state, is defined as the target state point y. target .

[0142] Online stage diagnostics: On an automated production line, a pin to be assembled is delivered to a workstation. Robot 10 moves to the observation position and performs physical property enhancement point cloud acquisition and feature extraction on the pin to obtain its physical feature vector v. test Suppose that the pin, due to uneven cooling during heat treatment, generates internal residual stress invisible to the human eye. This stress alters the material's microcrystalline structure, resulting in a slight difference in its surface thermal conductivity response and coefficient of thermal expansion compared to a standard part when excited by a thermal pulse from the active physical excitation module 30. This difference is captured by the thermal imager 22 and reflected in the temperature component T of the physical property enhancement point cloud. Therefore, the calculated physical feature vector v test The projected distance d between the healthy manifold 80 and the manifold 80 proj Exceeded the preset threshold τ health The decision control module 42 determines that the pin is a defective product, and the robot 10 puts it into the scrap bin and waits for the next material.

[0143] Online phase control: Another pin to be assembled has passed the diagnostic and is deemed qualified. Decision control module 42 determines the qualification based on its initial state point y.initial and target state point y target The desired process path 81 is generated on the healthy state manifold 80 and converted into a nominal instruction sequence. Robot 10 begins the insertion action. When the pin tip is about to contact the chamfer of the hole, the system enters a high-frequency in-situ self-consistency verification loop. At time t, the pin makes physical contact with the hole, and the contact force causes stress concentration in a localized area on the pin surface. According to the photoelastic effect, this stress changes the polarization reflection characteristics of the metal surface, causing specific changes in the linear polarization degree L and polarization angle A in that region. The composite sensing system 20 captures this change and generates the actual state point y. actual (t) Therefore, it deviates from the corresponding point y on the expected process path. exp (t), generating a non-zero deviation vector e(t). The feedback controller immediately converts this deviation vector e(t) into a composite physical correction command ΔC(t). In this command, the pose correction amount ΔP robot (t) includes a tiny attitude adjustment (e.g., a 0.01-degree rotation around the X-axis), with a force control correction ΔC. force (t) reduces the target contact force in the Z-axis direction by 0.5 Newtons. Final control command C final (t) This correction amount is superimposed, which drives the robot to adjust its posture and force in real time, so that the tip of the pin can slide smoothly into the hole along the chamfer normal direction, thereby making the actual process trajectory 82 quickly return to the desired process path 81.

[0144] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An AI visual positioning method for automated robot assembly, characterized in that, The method includes the following steps: S1. Collect the dynamic physical response of the material to be assembled under the active physical excitation applied by the active physical excitation module, and generate a physical property enhancement point cloud containing three-dimensional spatial coordinates, linear polarization degree, polarization angle and surface temperature information. S2. Input the physical property enhancement point cloud into a physical causal decoupling network to extract physical feature vectors characterizing the intrinsic physical properties of the material; wherein, the physical causal decoupling network includes multiple parallel sub-networks; the sub-networks include at least: a specular reflection decoupling sub-network, which takes as input the geometric and polarization information of each point and its local neighborhood in the physical property enhancement point cloud, and outputs an estimated value of the surface normal direction of each point in the physical property enhancement point cloud; and a transparent material penetration analysis sub-network, which takes as input the polarization information and temperature gradient information in the physical property enhancement point cloud, weights the polarization information and temperature gradient information through an attention mechanism, and outputs the probability value of each point in the physical property enhancement point cloud belonging to a transparent surface; S3. Project the physical feature vectors onto a pre-constructed healthy state manifold to calculate the initial state points of the material; wherein, physical property enhancement point clouds of multiple standard parts are collected; the physical property enhancement point clouds of the multiple standard parts are input into a physical causal decoupling network to extract the corresponding physical feature vector set; a neighborhood graph is constructed based on the Euclidean distance between the vectors in the physical feature vector set; the shortest path length between any two vectors is calculated on the neighborhood graph and used as the geodesic distance between the two vectors, thereby forming a geodesic distance matrix; a multidimensional scaling algorithm is applied to the geodesic distance matrix to obtain a point set in a low-dimensional space, which is the healthy state manifold; S4. Generate the desired process path connecting the initial state point and the target state point on the healthy state manifold; S5. During the assembly process, the deviation vector between the actual process trajectory of the material and the expected process path is calculated in real time, and a composite physical correction command is generated based on the deviation vector to adjust the assembly action.

2. The AI ​​visual positioning method for automated robot assembly according to claim 1, characterized in that, Step S1, which involves collecting the dynamic physical response of the material to be assembled under active physical excitation and generating a physical property-enhanced point cloud containing three-dimensional spatial coordinates, linear polarization degree, polarization angle, and surface temperature information, includes the following steps: The polarization camera, thermal imager, and structured light projector in the composite sensing system are pre-calibrated coaxially; The composite sensing system simultaneously captures polarization images and thermal images of the material to be assembled under active physical excitation. Based on the polarization image, the Stokes vector is calculated, and the degree of linear polarization and polarization angle of each pixel are solved according to the Stokes vector. The structured light projector projects a structured light pattern onto the surface of the material to be assembled, and the polarization camera captures the structured light pattern. The three-dimensional spatial coordinates of the material are then reconstructed using the triangulation principle. For each coordinate point in the three-dimensional spatial coordinates, associate a physical property vector containing the degree of linear polarization, polarization angle, and surface temperature information extracted from the thermal image.

3. The AI ​​visual positioning method for automated robot assembly according to claim 2, characterized in that, The polarization camera is a focal plane polarization camera; the calculation of the Stokes vector is based on the polarization intensity images of four orthogonal directions captured by the polarization camera in the composite sensing system in a single exposure.

4. The AI ​​visual positioning method for automated robot assembly according to claim 1, characterized in that, In step S4, the step of generating the desired process path connecting the initial state point and the target state point on the healthy state manifold includes: On the neighborhood graph corresponding to the healthy state manifold, with the initial state point and the target state point as the starting and target nodes, the shortest path algorithm is applied to calculate a sequence composed of discrete state points, which is the desired process path. Each discrete state point in the desired process path is input into a pre-trained inverse mapping model, which maps the discrete state points into a nominal configuration vector containing robot pose, active physical excitation parameters, and force control parameters, thereby generating a series of nominal command sequences.

5. The AI ​​visual positioning method for automated robot assembly according to claim 1, characterized in that, In step S5, the composite physical correction instruction includes a combination of at least one or more of the following instructions: Robot joint space fine-tuning commands used to fine-tune the position of robot joints; Active excitation parameter adjustment command for adjusting the excitation parameters of the active physical excitation module; Force control adjustment commands are used to adjust the force or torque applied by the robot's end effector.

6. The AI ​​visual positioning method for automated robot assembly according to claim 1, characterized in that, Step S5, which involves calculating the deviation vector between the actual process trajectory of the material and the desired process path in real time during the robot's assembly operation, includes: During the assembly process performed by the robot, the physical property enhancement point cloud of the material is captured in real time through the physical interaction interface. The real-time generated physical property enhanced point cloud is input into the physical causal decoupling network, and the output of the physical causal decoupling network is the actual process trajectory.

7. The AI ​​visual positioning method for automated robot assembly according to claim 1, characterized in that, Step S5, which involves generating a composite physical correction command based on the deviation vector to adjust the assembly action, includes: The composite physical correction command is superimposed with a nominal command sequence pre-generated based on the desired process path to form the final control command applied to the robot and the active physical excitation module.

8. An AI vision positioning system for automated robot assembly, applied to the method described in any one of claims 1-7, characterized in that, The system includes: The active physical excitation module is used to apply active physical excitation to the materials to be assembled; A composite sensing system is used to collect the dynamic physical response of the material to be assembled under active physical excitation, so as to generate a physical property enhanced point cloud containing three-dimensional spatial coordinates, linear polarization degree, polarization angle and surface temperature information; The intelligent analysis module is used to receive the physical property enhancement point cloud and extract physical feature vectors representing the intrinsic physical properties of the material through a physical causal decoupling network; and to project the physical feature vectors onto a pre-constructed healthy state manifold to calculate the initial state point of the material. The decision control module is used to generate a desired process path connecting the initial state point and the target state point on the health state manifold; and during the robot's assembly action, to calculate in real time the deviation vector between the actual process trajectory of the material and the desired process path, and to generate a composite physical correction command based on the deviation vector to adjust the assembly action.

Images