Vision guidance method and system for intelligent assisted assembly

Abstract

The present application relates to the technical field of intelligent equipment, and provides a visual guidance method and system for intelligent auxiliary assembly, comprising the following steps: collecting original image sequences, depth coordinate data and three-axis torque data of a current assembly object, generating spatial configuration feature information, using the three-axis torque data to perform stress state evaluation analysis on the spatial configuration feature information, obtaining a strategy candidate set and a plurality of assembly state transition path sets, using the strategy candidate set to perform strategy matching on all the assembly state transition path sets, and obtaining an assembly guidance strategy. The assembly guidance strategy with high adaptability is generated through the strategy candidate set and the assembly state transition path set, solving the deficiencies of the prior art in terms of coping with assembly step changes, error deviations and real-time regulation, and improving the problem that the prior art lacks adaptive dynamic guidance strategies when encountering changes in assembly steps, assembly object assembly deviations or errors, and is unable to adjust assembly schemes and guidance modes in real time.

Description

Technical Field

[0001] This application relates to the technical field of smart devices, and in particular to a visual guidance method and system for intelligent assisted assembly. Background Technology

[0002] With the continuous development of intelligent manufacturing, assembly operations in industrial production are becoming increasingly complex and precise. Traditional manual assembly methods are gradually failing to meet the requirements of efficient, precise, and flexible production. Therefore, intelligent assembly technology has emerged and is widely used in high-precision manufacturing fields such as electronics, automobiles, and aerospace. Intelligent assembly systems, by combining cutting-edge technologies such as robotics, sensors, and vision technology, have gradually achieved real-time monitoring, analysis, and optimization of the assembly process.

[0003] Among the relevant technical methods, industrial cameras and depth sensors installed at the workstation are used to acquire image and spatial position information of the assembly objects. Image recognition and vision algorithms are then used to monitor and analyze the assembly process in real time. This allows for real-time determination of the position, orientation, and status of the assembled components, and provides guidance information to the operator through display devices, such as prompts for the next operation, component positions, and installation directions. This effectively improves assembly efficiency and reduces human error.

[0004] Regarding the above-mentioned technical solutions, although existing technologies can achieve basic monitoring and guidance of the assembly process through visual guidance, reducing operational errors and improving efficiency, they lack adaptive dynamic guidance strategies when encountering changes in assembly steps or assembly deviations or errors in the assembly objects, and cannot adjust the assembly scheme and guidance method in real time. Summary of the Invention

[0005] To address the problem that existing technologies lack adaptive dynamic guidance strategies and cannot adjust assembly schemes and guidance methods in real time when encountering changes in assembly steps or assembly deviations or errors in the assembly objects, this application provides a visual guidance method and system for intelligent assisted assembly.

[0006] This invention provides a visual guidance method for intelligent assisted assembly, comprising: acquiring the original image sequence, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station to construct a set of key assembly areas; generating spatial configuration feature information based on the set of key assembly areas; using the triaxial torque data to perform stress state assessment and analysis on the spatial configuration feature information to obtain stability index information and a dynamic change map of assembly strain; extracting features from the stability index information to obtain a stability feature vector; using the stability feature vector to perform contact pattern recognition on the dynamic change map of assembly strain to obtain a contact area state sequence; inputting the contact area state sequence into a preset assembly anomaly identification model to obtain an assembly anomaly risk assessment result; performing a hierarchical analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates; acquiring dynamic evolution data of the spatial configuration feature information based on the risk level information; classifying the dynamic evolution data to obtain a set of assembly state transition paths; and using the set of strategy candidates to perform strategy matching on all the sets of assembly state transition paths to obtain an assembly guidance strategy.

[0007] As a preferred embodiment, the step of acquiring the original image sequence, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station to construct a set of key assembly areas includes: simultaneously acquiring the original image sequence, depth coordinate data, and triaxial torque data of the force applied to the contact surface of the current assembly object using an industrial camera, depth sensing module, and force sensor installed at the assembly station; calibrating and registering the original image sequence and the depth coordinate data; combining the calibrated and registered original image sequence and depth coordinate data with the station reference coordinate system to generate a three-dimensional visual model and real-time pose information of the assembly component; fusing the three-dimensional visual model of the assembly component with the real-time pose information to obtain a posture mapping field; performing time series analysis on the triaxial torque data to obtain instantaneous stress fluctuation characteristics; jointly calculating the posture mapping field and the instantaneous stress fluctuation characteristics to generate assembly action influence domain information; and using the assembly action influence domain information to dynamically label the three-dimensional visual model to obtain a set of key assembly areas.

[0008] As a preferred embodiment, the steps of generating spatial configuration feature information based on the set of key assembly regions, and using the triaxial torque data to perform stress state evaluation and analysis on the spatial configuration feature information to obtain stability index information and assembly strain dynamic change spectrum include: constructing an assembly configuration contour surface based on the set of key assembly regions; performing topological modeling of the assembly configuration contour surface in combination with three-dimensional pose information to obtain spatial configuration feature information; wherein, the three-dimensional pose information refers to the position coordinate information and orientation attitude information of the current assembly object in the workstation coordinate system; performing multi-region loading simulation analysis on the spatial configuration feature information using the triaxial torque data to obtain the stress uniformity distribution map and contact stiffness change characteristics of each configuration region, and then performing stress state evaluation and analysis on the spatial configuration feature information to obtain stability index information and assembly strain dynamic change spectrum. The force uniformity distribution map is cross-matched with the contact stiffness variation characteristics to obtain high-risk contact sub-regions and low-stability feature point sets. The real-time contact pressure distribution of the high-risk contact sub-regions in the current assembly process is matched with the force patterns of the same contact areas in the historical task database to calculate the deviation score of each sub-region. Stability index information is constructed based on the deviation score. The three-dimensional displacement time series of the low-stability feature point set is extracted throughout the assembly process. The strain rate change trend and contact area change trend of the three-dimensional displacement time series are calculated. The strain rate change trend and contact area change trend are used to construct contact area change information. Based on the contact area change information and the stability index information, an assembly strain dynamic change map is constructed.

[0009] As a preferred embodiment, the step of using the triaxial moment data to perform multi-region loading simulation analysis on the spatial configuration feature information to obtain the stress uniformity distribution map and contact stiffness variation characteristics of each configuration region, and cross-matching the stress uniformity distribution map with the contact stiffness variation characteristics to obtain high-risk contact sub-regions and low-stability feature point sets includes: dividing each key assembly area in the spatial configuration feature information according to the triaxial moment data to obtain multiple local regions, and performing multi-physics coupling simulation on each local region to obtain loading simulation results; calculating the stress distribution of each local region based on the loading simulation results to generate a stress uniformity distribution map, extracting features from the stress uniformity distribution map to identify the contact stiffness variation characteristics of each assembly region to obtain a contact stiffness variation spectrum; and cross-matching the stress uniformity distribution map with the contact stiffness variation characteristics to analyze the correlation between stress and stiffness variation in local regions and identify existing high-risk contact sub-regions and low-stability feature point sets.

[0010] As a preferred embodiment, the step of extracting features from the stability index information to obtain a stability feature vector, and using the stability feature vector to perform contact pattern recognition on the assembly strain dynamic change spectrum to obtain a contact area state sequence includes: performing principal component analysis and dynamic time warping on the stability index information to extract steady-state feature distribution and change rate, and constructing a stability feature vector using the steady-state feature distribution and change rate; performing interval segmentation and dynamic clustering on the assembly strain dynamic change spectrum to obtain stress response patterns, jointly embedding the stability feature vector and the stress response patterns into a unified feature space to obtain a fused stability feature vector; identifying key contact events based on the fused stability feature vector, constructing a time-series contact spectrum using the key contact events, extracting the contact state transition paths of the time-series contact spectrum, and performing structural encoding processing on the contact state transition paths to obtain a complete contact area state sequence.

[0011] As a preferred embodiment, the step of inputting the contact area state sequence into a preset assembly anomaly identification model to obtain an assembly anomaly risk assessment result, and performing a hierarchical analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates includes: encoding and vectorizing the contact area state sequence according to time windows and state nodes to generate a dynamic contact behavior sequence; inputting the dynamic contact behavior sequence into the preset assembly anomaly identification model to output assembly stability deviation and anomaly pattern labels; generating an assembly anomaly risk assessment result based on the assembly stability deviation and the anomaly pattern labels; performing a similarity comparison of the assembly anomaly risk assessment result using a preset assembly database to obtain risk level information; and matching the risk level information with a preset strategy template library to obtain a set of strategy candidates.

[0012] As a preferred embodiment, the steps of obtaining dynamic evolution data of spatial configuration feature information based on the risk level information, classifying the dynamic evolution data to obtain several assembly state transition path sets, and using the strategy candidate set to perform strategy matching on all the assembly state transition path sets to obtain an assembly guidance strategy include: selecting key spatial configuration regions corresponding to the spatial configuration feature information based on the risk level information, extracting the spatial feature evolution trajectory of the key spatial configuration regions over time to obtain dynamic evolution data, performing multi-dimensional feature mapping and path structure analysis on the dynamic evolution data to obtain analysis results, dividing the analysis results into several assembly state transition units; clustering all the assembly state transition units into an assembly state transition path set according to temporal order and evolution trend, matching and scoring each strategy in the strategy candidate set with each path in the assembly state transition path set to construct a strategy path adaptation matrix; performing intervention effectiveness simulation prediction and response time evaluation on each strategy-path combination in the strategy path adaptation matrix to obtain test evaluation results, and selecting the optimal strategy path combination based on the test evaluation results to generate an assembly guidance strategy.

[0013] This application also provides a visual guidance system for intelligent assisted assembly, comprising: a data acquisition module for acquiring original image sequences, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station to construct a set of key assembly areas; an analysis module for generating spatial configuration feature information based on the set of key assembly areas, and using the triaxial torque data to perform stress state assessment and analysis on the spatial configuration feature information to obtain stability index information and a dynamic change map of assembly strain; an extraction module for extracting features from the stability index information to obtain a stability feature vector, and using the stability feature vector to perform contact pattern recognition on the dynamic change map of assembly strain to obtain a contact area state sequence; an input module for inputting the contact area state sequence into a preset assembly anomaly identification model to obtain an assembly anomaly risk assessment result, and performing a graded analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates; and a matching module for obtaining dynamic evolution data of the spatial configuration feature information based on the risk level information, classifying the dynamic evolution data to obtain a set of assembly state transition paths, and using the set of strategy candidates to perform strategy matching on all the sets of assembly state transition paths to obtain an assembly guidance strategy.

[0014] Compared with existing technologies, this application has the following advantages: high flexibility and precise guidance. By combining multimodal perception with original image sequences, depth coordinate data, and three-axis torque data, a set of key assembly areas is constructed. Stability index information and dynamic change maps of assembly strain are generated through spatial configuration feature information and multi-physics coupling simulation analysis. Then, feature extraction and pattern recognition are performed on the stability index information, and a contact area state sequence is generated based on the contact pattern to dynamically monitor abnormal states generated during the assembly process. A deep learning-based assembly anomaly identification model is used to perform hierarchical analysis of assembly anomalies, and a highly adaptable assembly guidance strategy is generated by combining dynamic evolution data and a strategy candidate set. This effectively solves the shortcomings of existing technologies in dealing with changes in assembly steps, error deviations, and real-time system control, greatly improving assembly efficiency and the adaptability and intelligence level of the assembly process. It also addresses the problem that existing technologies lack adaptive dynamic guidance strategies and cannot adjust assembly schemes and guidance methods in real time when encountering changes in assembly steps or assembly deviations or errors in the assembly objects. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0017] Figure 1 This is a flowchart illustrating the visual guidance method for intelligent assisted assembly provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic block diagram of the structure of the intelligent assisted assembly visual guidance system provided in the embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures:

[0020] 10. Visual guidance system for intelligent assisted assembly; 11. Acquisition module; 12. Analysis module; 13. Extraction module; 14. Input module; 15. Matching module. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1:

[0027] like Figure 1 As shown, this application provides a visual guidance method for intelligent assisted assembly, including steps S100 to S500.

[0028] Step S100: Collect the original image sequence, depth coordinate data and three-axis torque data of the current assembly object at the assembly station to construct a set of key assembly areas.

[0029] In this step, an industrial camera installed at the assembly station acquires the original image sequence of the current assembly object, a depth sensing module acquires the depth coordinate data of the assembly object, and a force sensor collects the triaxial torque data applied to the contact surface of the current assembly object. By calibrating and registering the original image sequence and depth coordinate data, the calibrated image data and depth coordinate data are combined with the station's reference coordinate system to generate a 3D visual model and real-time pose information of the assembly component. Then, the 3D visual model of the assembly component and the real-time pose information are jointly calculated to obtain the pose mapping field. Based on this, by performing time-series analysis and dynamic region annotation on the pose mapping field and triaxial torque data, the set of key assembly areas that meet the requirements is finally identified. Specifically, during the calibration and registration of image data and depth coordinate data, a camera calibration technique based on a checkerboard calibration method is used to perform coordinate mapping and geometric alignment of the sampling spaces of the industrial camera and the depth sensor. The generation of the pose mapping field uses a multi-resolution pyramid model to perform layered calculations on the 3D visual model, obtaining the pose state of each spatial point based on the gradient distribution of depth coordinates and pose information. In time series analysis, the instantaneous stress fluctuation characteristics of triaxial torque data are extracted by frequency domain characteristics. Combined with the dynamic region labeling algorithm of assembly components, regions with obvious stress peaks or non-uniform distribution are identified as key assembly regions.

[0030] For example, in practical operation, when an assembly component is detected to have a tilted edge as indicated by an image sequence captured by an industrial camera, the depth sensing module records the object's depth position information in space. Simultaneously, the force sensor detects the three-dimensional torque change of this assembly component at the contact point (e.g., an instantaneous increase in torque to [M]). x =2.0,M y =1.8,M z =2.3]N·m), the joint attitude mapping field can quickly identify this as an important contact critical area in the assembly process and mark it as part of the set of critical assembly areas.

[0031] Step S200: Generate spatial configuration feature information based on the set of key assembly areas, and use triaxial torque data to evaluate and analyze the stress state of the spatial configuration feature information to obtain stability index information and assembly strain dynamic change spectrum.

[0032] In this step, an assembly configuration contour surface is constructed by gathering key assembly areas. This is combined with real-time pose information for topological modeling to generate spatial configuration feature information. A loading analysis algorithm is then used to simulate each key area. The simulation results are processed using triaxial torque data to complete a multi-field fusion analysis of the stress state, obtaining the stress uniformity distribution map and contact stiffness variation characteristics of each key local area. Furthermore, by analyzing the non-uniformity of feature distribution between overall regions and combining multi-point strain analysis, a dynamic variation map of assembly strain reflecting the strain evolution relationship within the region is generated. Specifically, in topological modeling, a three-dimensional Delaunay triangulation algorithm is used to mesh the assembly configuration contour surface. Boundary conditions conforming to the continuum mechanics model are set during simulation analysis, and stress distribution simulation calculations are performed using finite element software (such as ANSYS). Triaxial torque data is used as load input in the multi-field fusion analysis. By weighting the time series and spatial distribution of the torque, information on local pressure concentration points is obtained, and the mechanical state distribution and deformation characteristics of the overall configuration are extracted.

[0033] For example, in a certain assembly station, when a sudden increase in pressure is detected at the connection point of a critical assembly area (the triaxial torque is [M]), x =-5.5,M y =3.6,M z =4.7] N·m), the finite element simulation shows that the stress concentration area in the corresponding region gradually expands. Combined with the simulation results, the contact stiffness change spectrum can be used to reflect the trend of local instability of the assembly component.

[0034] Step S300: Extract features from stability index information to obtain stability feature vectors. Use the stability feature vectors to perform contact pattern recognition on the dynamic change spectrum of assembly strain to obtain the contact area state sequence.

[0035] In this step, the stability index information is subjected to time-domain and frequency-domain feature analysis. The steady-state feature distribution and rate of change are extracted by principal component analysis (PCA) algorithm to construct a stability feature vector. This feature vector is combined with the dynamic change map of assembly strain, and the stress response mode in different time windows during the assembly process is calibrated by dynamic clustering algorithm to complete the pattern recognition of the contact area of ​​the assembly component, and finally generate a time-series contact area state sequence.

[0036] Specifically, in the main feature extraction stage, the PCA algorithm is used to reduce the dimensionality of high-dimensional data. During this process, all input variables (such as data points of stress distribution and stiffness characteristic curves) are normalized, and principal components with an explained cumulative variance contribution rate of over 95% are extracted. The dynamic clustering algorithm uses the K-means++ method, which classifies and analyzes stress fluctuation patterns by setting the number of cluster centers.

[0037] For example, in a certain assembly step, the dynamic variation spectrum of assembly strain shows that the stiffness change slope of the key contact point is as high as 0.9. Combined with the normal assembly process stiffness slope threshold of <-0.5, the current state is identified as "slip transition" through dynamic clustering algorithm.

[0038] Step S400: Input the contact area state sequence into the preset assembly anomaly identification model to obtain the assembly anomaly risk assessment result. Perform a graded analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates.

[0039] In this step, the contact area state sequence is used as input features, and an anomaly pattern is matched by a deep learning-based anomaly recognition model combined with the historical database of the assembly process. The identified anomaly risks are classified and analyzed according to loss probability and severity, and the corresponding strategy candidate set is matched from the built-in strategy template library for the generation of subsequent assembly guidance strategies.

[0040] Specifically, the anomaly identification model employs a convolutional neural network (CNN) model, a technology already in use. The time window length for the input state sequence is 10 frames. After training, the model obtains anomaly labels, such as "assembly component position deviation" or "stress concentration occurrence," by calculating the classification results output by the Softmax layer. The hierarchical analysis is based on the probability of anomaly occurrence and the magnitude of its impact, classifying them into levels from 1 to 5.

[0041] For example, after a certain state sequence is input, the CNN model classifies it as "stress mutation anomaly". Historical database matching shows that the probability of this state occurring in the past position deviation is 70%, so it is classified as risk level 4, and outputs a set of policy candidates such as "pose adjustment".

[0042] Step S500: Obtain dynamic evolution data of spatial configuration feature information based on risk level information, classify the dynamic evolution data to obtain several assembly state transition path sets, and use the strategy candidate set to perform strategy matching on all assembly state transition path sets to obtain assembly guidance strategy.

[0043] In this step, based on the obtained risk level information, the dynamic evolution trajectory of spatial configuration features over time is extracted. This dynamic evolution data is then classified using a time series classification algorithm to form different sets of state transition paths. Simultaneously, combined with a policy candidate set, the suitability of each policy and path is ranked based on matching scores, ultimately generating the optimal assembly guidance policy. Specifically, the classification algorithm uses a current time series classification method based on Dynamic Time Warping (DTW) to perform cluster analysis on feature points in the evolution data, with each state transition path corresponding to an assembly step feature. During policy matching, scores are ranked based on historical success rate and expected response time.

[0044] For example, a dynamic evolution trajectory reflects a gradual increase in the deviation angle of the assembly component's position, which belongs to state transition path type A. Combining the strategy candidate set with strategies that are applicable to "type A paths" and include "guide the operator to reposition", an assembly guidance strategy is finally generated to prompt the operator to make real-time adjustments.

[0045] In this embodiment, the original image sequence, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station are collected, and a set of key assembly areas is constructed by combining these data. Then, spatial configuration feature information is generated based on the set of key assembly areas, and the stress state of the spatial configuration feature information is evaluated and analyzed using the triaxial torque data to obtain stability index information and a dynamic change map of assembly strain. Next, features are extracted from the stability index information to obtain stability feature vectors, and contact pattern recognition is performed on the dynamic change map of assembly strain using the stability feature vectors to obtain a contact area state sequence. The contact area state sequence is input into a preset assembly anomaly identification model to obtain assembly anomaly risk assessment results, and the assembly anomaly risk assessment results are graded to obtain risk level information and a set of strategy candidates. Finally, dynamic evolution data of spatial configuration feature information is obtained based on the risk level information, and the dynamic evolution data is classified to obtain several sets of assembly state transition paths. The strategy candidate set is used to perform strategy matching on all sets of assembly state transition paths to generate an assembly guidance strategy. To address the limitations of existing technologies in real-time adjustment of assembly schemes and guidance methods, this method dynamically constructs a set of key assembly areas by real-time acquisition of raw image sequences, depth coordinate data, and triaxial torque data of the assembly object. This generates spatial configuration feature information and comprehensively assesses the assembly stress state, obtaining stability index information and dynamic strain change maps. Furthermore, by utilizing feature extraction and contact pattern recognition technologies, it monitors and assesses abnormal risks during the assembly process in real time. Through hierarchical analysis, it generates risk level information and a set of strategy candidates. Based on changes in the assembly environment, it achieves dynamic evolution data classification and assembly state transition path matching, ultimately generating an assembly guidance strategy that optimizes assembly efficiency and quality. This ensures accuracy and flexibility in the assembly process and enhances the adaptability of complex assembly tasks.

[0046] Example 2:

[0047] Step S100 involves collecting the original image sequence, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station to construct a set of key assembly areas. This step specifically includes:

[0048] The system utilizes an industrial camera, depth sensing module, and force sensor installed at the assembly station to simultaneously acquire the original image sequence, depth coordinate data, and triaxial torque data of the force applied to the contact surface of the object being assembled.

[0049] By synchronously configuring the industrial camera, depth sensing module, and force sensor, a unified timestamp is used to establish the data correspondence between the sensors. The industrial camera employs a high-speed imaging mode, continuously acquiring raw image sequences at a rate of 60 frames per second, and capturing surface texture information of the assembly object through an adapted optical lens. The depth sensing module uses structured light or LiDAR technology to acquire the depth coordinate data of the assembly object. This process generates 3D point cloud data of the assembly object by actively emitting a light beam and recording the time of reflection information. The force sensor records triaxial torque data in real time through the contact surface, storing the torque values ​​as a time series. All data is transmitted to the control system in real time, and these data are initially calibrated using the reference coordinate system of the assembly station to ensure spatial alignment and data consistency.

[0050] Specifically, during the industrial camera image acquisition process, the existing "Zhang Calibration Method" is used to calibrate the camera's intrinsic and extrinsic parameters, enabling the camera data to be mapped more accurately to three-dimensional space. The depth sensing module configuration employs the "ICP" (Iterative Closest Point) algorithm to iteratively optimize the point cloud data to eliminate noise and errors. Triaxial torque data is acquired from a force sensor mounted on the end of the contact tool, and its triaxial force values ​​([M... x M y M z Spatial mapping is achieved through matrix transformation from the body coordinate system to the workstation reference coordinate system. The data synchronization of all sensors is verified by the system through a synchronization mechanism with a unique timestamp, ensuring the temporal continuity and spatial correlation of the data.

[0051] For example, in assembly operations, an industrial camera captures a sequence of surface images of the assembled object (e.g., the edge contours and texture distribution of parts), and a depth module generates corresponding point cloud data to construct spatial depth information (e.g., the spatial surface shape of the part). Simultaneously, a force sensor records triaxial torque data applied at the contact point (e.g., torque values ​​[M...). x =3.2,M y =-1.5,M z =2.8] N·m). Through timestamp synchronization, the control system jointly encodes image, depth, and torque data and verifies the spatial consistency between the data. This information together provides high-precision raw data for the subsequent generation of key assembly area sets.

[0052] The original image sequence and depth coordinate data are calibrated and registered. The calibrated and registered original image sequence and depth coordinate data are combined with the workstation reference coordinate system to generate a 3D visual model and real-time pose information of the assembly component. The 3D visual model of the assembly component and the real-time pose information are fused to obtain the pose mapping field.

[0053] A calibration and registration technique is employed to jointly calibrate image data from the industrial camera and point cloud data from the depth sensing module, ensuring spatial alignment and data accuracy. During the calibration phase, an orthogonal checkerboard pattern is used as the calibration template to calibrate the geometric relationship between the camera and the depth sensor, and a reference coordinate system for the workstation is constructed using benchmark points. The calibrated image sequence and depth coordinate data are input into the 3D vision model generation module, where a joint filtering algorithm is used to fuse the image texture and depth data. Real-time pose information is estimated by combining feature points from consecutive frames of the assembly component with optical flow technology, updating the position and orientation of the assembly component in real time. Finally, a pose mapping field construction algorithm is used to merge the 3D vision model with the real-time pose information to obtain the overall spatial configuration of the assembly station and the dynamic pose information of the assembled objects.

[0054] Specifically, during the calibration process, the camera's intrinsic and extrinsic parameters are calibrated using the "DLT" (Direct Linear Transformation) algorithm, and the depth coordinate points are optimized using the "RANSAC" (Random Sample Consensus) algorithm to remove outliers. When generating the 3D visual model, edge texture information from the image data is superimposed with the point cloud surface data from the depth perception module using a multi-scale data fusion method to ultimately construct the spatial model. The pose mapping field is generated based on the gradient information of the depth data, combined with the dynamic changes of pose information over time, and utilizes existing distributed field computation algorithms to generate the mapping field.

[0055] For example, during assembly, when an industrial camera captures an image of the edge contour of the assembly object, combined with the high-precision point cloud from a depth sensing module, the texture information of the image edges is combined with the depth surface of the part to generate a 3D visual model. Through optical flow displacement estimation of feature points in three consecutive frames, the part position is calculated as [x=1.2, y=0.5, z=-0.3] (unit: m), with attitude angles of [α=30∘, β=-12∘, γ=45∘]. The attitude mapping field further reflects the spatial distribution around the assembly object.

[0056] Time series analysis of triaxial torque data is performed to obtain instantaneous stress fluctuation characteristics. The attitude mapping field and instantaneous stress fluctuation characteristics are jointly calculated to generate assembly action influence domain information. The assembly action influence domain information is used to dynamically annotate the three-dimensional visual model to obtain a set of key assembly areas.

[0057] Frequency and time domain analyses were performed on the triaxial torque data. Instantaneous fluctuation features were extracted using Fast Fourier Transform (FFT), and the mechanical concentration trends of key points were identified. The peak torque data were correlated with key contact areas in the attitude mapping field. Finally, a joint influence domain calculation algorithm was used to generate dynamic region annotations, thereby identifying the set of key assembly regions.

[0058] Specifically, in the Fast Fourier Transform, sudden changes in triaxial torque are extracted by analyzing the amplitude distribution of frequency components, and the mechanical imbalance region near the contact point is identified as a key area for dynamic annotation. During the joint influence domain calculation, the torque fluctuation characteristics are combined with the time-series data of the mapped field using a spatially distributed concentration function to annotate areas of concentrated force. The region partitioning logic is based on cluster analysis and fluctuation gradient calculation, defining the dynamic region as a set of key assembly regions.

[0059] For example, in actual operation, when the torque at the contact point changes (e.g., the instantaneous value is [M]), x =4.5,M y =2.1,M z =-1.8]N·m) caused the force gradient change at key points in the attitude mapping field to be greater than 0.75. Through joint analysis, the set of key assembly areas was described, including the high-risk area on the contact surface of the parts, which extends to the edge side by 10%.

[0060] In step S200, the steps of generating spatial configuration feature information based on the set of key assembly areas, and using triaxial moment data to evaluate and analyze the stress state of the spatial configuration feature information to obtain stability index information and assembly strain dynamic change spectrum specifically include:

[0061] Based on the set of key assembly regions, an assembly configuration contour surface is constructed. The assembly configuration contour surface is combined with three-dimensional pose information to perform topological modeling and obtain spatial configuration feature information. Among them, three-dimensional pose information refers to the position coordinate information and orientation attitude information of the current assembly object in the workstation coordinate system, including translation vectors and rotation matrices in three-dimensional space.

[0062] Spatial point data from each region in the key assembly area set are combined, and a surface reconstruction algorithm is used to generate the assembly configuration contour surface. Combining the 3D pose information of the assembly object in the workstation reference coordinate system, the position coordinate information and orientation attitude information (translation vector and rotation matrix) are mapped onto the point set of the generated contour surface. During topology modeling, a discrete mesh representation method is used to form the topological structure of the spatial configuration features, and the configuration attributes of each region are labeled. Through these operations, the complete geometric features and dynamic attitude information of each key region in the assembly space are obtained.

[0063] Specifically, surface construction utilizes the existing "Poisson Surface Reconstruction" algorithm, aiming to transform sparse point cloud data into a coherent and smooth assembly contour surface through global optimization. During topology modeling, a triangulation-based Delaunay algorithm is employed to generate mesh segmentation, connecting neighboring point sets in the point cloud into a continuous triangular mesh structure and labeling its spatial information. Three-dimensional pose information is derived from the coordinate translation vector [t]. x ,t y ,t z The rotation matrix R is a 3×3 orthogonal matrix used to describe the orientation of the assembled objects, and is expressed by the formula:

[0064] ;

[0065] Each element of R describes the projection of the reference coordinate axis vector into the target coordinate system. Combining this information, the contour surface and the 3D pose are fused.

[0066] For example, for a certain disk part, its outer contour is reconstructed from the point cloud of the key assembly area set, resulting in an assembly configuration contour surface with a surface accuracy of 0.2mm; assuming the current contour center coordinates are [100.5, 200.3, 50.1]mm, and the orientation is determined by the rotation matrix.

[0067] ;

[0068] Complete the topological combination of surface and pose to generate spatial configuration feature information.

[0069] Multi-region loading simulation analysis was performed using triaxial torque data to analyze the spatial configuration characteristics, resulting in the force uniformity distribution map and contact stiffness variation characteristics of each configuration region. The force uniformity distribution map and contact stiffness variation characteristics were cross-matched to obtain the high-risk contact sub-region and low-stability feature point set.

[0070] Based on the distribution characteristics of each key assembly region in the spatial configuration feature information, multi-region loading simulations were performed. During the simulation, triaxial torque data was used as the mechanical condition input to construct a multiphysics coupled simulation model to evaluate the stress distribution, contact stiffness, and force uniformity of each local region. The generated force uniformity distribution map and contact stiffness variation characteristics reflect the force concentration area and stiffness weakening area during the assembly process, respectively. By cross-matching the two types of data, high-risk contact sub-regions and low-stability feature point sets with overlapping force and stiffness variations were further screened.

[0071] Specifically, during simulation analysis, finite element analysis software (such as ANSYS) is used to apply loading conditions to the discretized assembly configuration mesh. The stress uniformity distribution diagram is calculated from the stress distribution gradient of the simulated mesh elements, and the contact stiffness characteristics are derived from the force change rate per unit displacement. For cross-matching, a matching algorithm based on correlation analysis is used to evaluate the significance of the overlapping areas in the spatial distribution of the two components.

[0072] For example, in a certain assembly, when a load is applied to a configuration area, the stress uniformity distribution map shows that the stress concentration area is located at point (30mm, 60mm); the contact stiffness variation map shows that the point where the stiffness drops significantly coincides with the above area. Thus, it is determined that this area is a high-risk contact sub-region, and at the same time, the key area of ​​the low stability feature point set is located.

[0073] The real-time contact pressure distribution of high-risk contact sub-regions in the current assembly process is matched with the force patterns of the same contact areas in the historical task database. The deviation score of each sub-region is calculated, and stability index information including spatial location, force stability score and contact duration is constructed based on the deviation score.

[0074] The system records the contact pressure distribution in high-risk contact sub-regions in real time and compares this data with standard contact patterns in the historical task database to obtain a deviation score for the current assembly process. The deviation score comprehensively considers the differences in maximum, mean, and volatility of contact pressure, and combines this with the assembly time information of the sub-regions to generate stability index information.

[0075] Specifically, in the matching calculation process, an algorithm based on cosine similarity is used to quantify the shape differences of the contact pressure distribution. Specifically, the pressure distribution is represented as a feature vector P, and the cosine similarity is:

[0076] ;

[0077] Among them, P c P is the characteristic vector of the real-time contact pressure distribution. h The feature vector of the historical contact pattern, This represents the L2 norm. The deviation score is defined as 1-S.

[0078] For example, in a certain sub-area during the assembly process, the deviation score of the real-time distribution characteristic of the contact pressure is 0.25, and the contact time is 5s. Therefore, the stability index is constructed as follows: position (50mm, 75mm), deviation score 0.25, and contact duration 5s.

[0079] The three-dimensional displacement time series of the low-stability feature point set is extracted during the entire assembly process. The strain rate change trend and contact area change trend of the three-dimensional displacement time series are calculated. The contact area change information is constructed by applying the strain rate change trend and contact area change trend.

[0080] Continuous sampling of 3D displacement data in the region containing low-stability feature points is performed to establish a displacement time series based on time. The strain rate and contact area variation trends are then calculated based on the sampled data. First, the displacement changes of feature points are mapped to strain rates, and the overall strain trend is analyzed using the relative offsets between neighboring points in the point cloud. Simultaneously, the contact area variation rate between feature points and the contact surface is calculated by combining the force concentration points in the region, reflecting the changes in mechanical distribution. Finally, the strain rate and contact area variation trends are combined, and contact area variation information is generated through unified trend encoding.

[0081] Specifically, the three-dimensional displacement time series is generated according to the following formula: feature point (x i ,y i ,z i The position change at any time t is calculated as displacement ΔL. t The formula is:

[0082] ;

[0083] Where, x i ,y i ,z i These are the three-dimensional coordinates of the feature point at time t, and the displacement change is used to analyze the strain rate of change at the contact surface. The strain rate change trend is obtained by solving the strain tensor point by point and calculating the deformation intensity change. The contact area change is calculated by dynamically adjusting the area of ​​the force-affected region, and finally, both are unified, normalized, encoded, and stored.

[0084] For example, within a specific time period, the average displacement change of feature points within a certain set of low-stability feature points is ΔL. t =0.15mm, the strain rate shows a gradual linear increase (initial strain rate is 0.03 / s, ending at 0.08 / s), and the contact area change rate is from 12mm. 2 Reduced to 5mm 2 Based on the above changes, unified trend information is encoded to generate contact area change information, and the sub-region is marked as a potentially unstable region.

[0085] A dynamic variation map of assembly strain is constructed by combining information on changes in the contact area with information on stability indicators.

[0086] By integrating contact area variation information and stability index information, key parameters (such as strain rate variation, contact area dynamics, position coordinates, stability score, etc.) are mapped onto a spatial configuration model to construct an assembly strain dynamic variation map. This map updates the dynamic state of the assembly area in real time using three-dimensional simulation technology, revealing stress concentration trends, displacement distribution, and stability evaluation results within the area.

[0087] Specifically, the generation of the assembly strain dynamic change map is based on a spatial configuration model. Contact area change information is overlaid as a dynamic change layer onto the stability index, forming a distributed change composite map. Data fusion follows a hierarchical mapping principle, where horizontal mapping marks contact area change information to arbitrary spatial sub-regions, and vertical mapping dynamically supplements regional features through stability scores. The map can be automatically adjusted in real time as the assembly data changes and is ultimately presented through graphical software.

[0088] For example, by simulating the stress distribution characteristics of an assembly object, the distribution of low-stability feature points is combined with the stability score to ultimately generate a dynamic strain change map of the assembly. The map clearly shows that the areas with high strain rate changes are concentrated in x=50-75mm and y=20-30mm, with the contact area gradually decreasing to the critical value. At the same time, the stability score warning points are located at x=75mm and y=25mm, prompting the operator to pay close attention to these areas.

[0089] The process of using triaxial moment data to perform multi-region loading simulation analysis on spatial configuration feature information to obtain the force uniformity distribution map and contact stiffness variation characteristics of each configuration region, and then cross-matching the force uniformity distribution map and contact stiffness variation characteristics to obtain high-risk contact sub-regions and low-stability feature point sets, includes:

[0090] Based on the triaxial torque data, each key assembly area in the spatial configuration feature information is divided into multiple local regions. Multiphysics coupling simulation is then performed on each local region to obtain the loading simulation results.

[0091] Based on the force characteristics of each key assembly region in the triaxial torque data, the spatial configuration feature information is divided into multiple local regions. The division of each key assembly region is based on the characteristics of force distribution and geometric topology, and is discretized using a mesh model (e.g., finite element mesh generation). Subsequently, for each local region, a multiphysics coupled simulation model of elasticity and thermodynamics is established. The triaxial torque data is used as input data for loading simulation to calculate the stress distribution, deformation behavior, and temperature field changes in the assembly region after loading, thereby generating loading simulation results.

[0092] Specifically, the mesh generation employs the existing "Adaptive Mesh Refinement" (AMR) algorithm to refine regions with drastic stress changes (such as high stress gradient regions) to improve simulation accuracy. During the simulation, the multiphysics coupling model uses finite element analysis (FEA) to define the mechanical boundary conditions: the location and magnitude of triaxial moments Mx, My, and Mz. Elastic modulus and Poisson's ratio are applied to material properties, while the thermodynamic model considers the thermal effects of volumetric deformation. The final output shows the loading simulation results for each region, including stress field, displacement field, and thermal field changes.

[0093] For example, when dividing the critical assembly area of ​​a rectangular plate-shaped component in an assembly station, the triaxial moment data shows that the points of application are located at (20,30,0) and (40,60,0), respectively. These areas are meshed into 1000 refined mesh points, and loading simulation analysis is applied. The simulation results show that the initial stress is distributed between 200 MPa and 300 MPa in the high-stress region, while the maximum displacement fluctuation reaches 0.12 mm.

[0094] The stress distribution of each local area is calculated based on the loading simulation results to generate a stress uniformity distribution map, which reflects the mechanical equilibrium state of each assembly area and analyzes the stability of the stress in each area. Feature extraction is performed on the stress uniformity distribution map to identify the contact stiffness variation characteristics of each assembly area and obtain a contact stiffness variation spectrum, which reflects the dynamic change of stiffness in each area during the assembly process.

[0095] Based on the stress field data from the loading simulation results, the stress distribution in each local region is calculated to generate a stress uniformity distribution map reflecting the stress characteristics of the assembly area. The stress uniformity and stress variation gradient of each region are analyzed to determine the state of mechanical equilibrium. Subsequently, based on the relationship between local displacement changes and loads, the corresponding contact stiffness variation characteristics are calculated, generating a contact stiffness variation map, thus comprehensively reflecting the dynamic stiffness variation characteristics of each region during the assembly process.

[0096] Specifically, the stress uniformity distribution map is generated using the maximum principal stress value of the simulated mesh elements, calculated using the formula σ=F / A, where σ is the stress per unit area, F is the applied force, and A is the area of ​​the applied region. Regions with high stress gradients in the map are marked as non-uniform mechanical regions for subsequent analysis. The contact stiffness variation characteristics are calculated using the applied force ΔF and the corresponding unit displacement change Δδ, with the rate of change of stiffness k being k=ΔF / Δδ. A three-dimensional curve is used to mark the trajectory of contact stiffness change over time, generating a contact stiffness variation map.

[0097] For example, simulation analysis of a certain assembly component shows that the stress concentration region in the stress uniformity distribution diagram is located at coordinate point (25, 50, 0), corresponding to a maximum stress value of 250 MPa; the contact stiffness variation diagram shows that the stiffness (k) in this region decreases from the initial value of 3500 N / mm to 2800 N / mm. These data indicate that stiffness attenuation during the assembly process has a tendency to lead to instability.

[0098] By cross-matching the stress uniformity distribution map with the contact stiffness variation characteristics, the correlation between stress and stiffness variation in local areas is analyzed, and high-risk contact sub-regions and low-stability feature point sets are identified.

[0099] A cross-matching algorithm was used to correlate the stress uniformity distribution map with the contact stiffness variation characteristics to evaluate the spatial distribution consistency and dynamic trend between the two sets of data. By calculating the correlation index, the intersection points of the stress unevenness and the region of significant stiffness attenuation were marked, and contact sub-regions with high-risk characteristics and key point sets (low-stability feature point sets) exhibiting low stability characteristics were further screened.

[0100] Specifically, cross-matching uses the Pearson Correlation Coefficient (PCC) calculation formula:

[0101] ;

[0102] Where Cov(X,Y) is the covariance of the force uniformity distribution data X and the contact stiffness variation characteristic data Y, and σ X ,σ Y Let X and Y be the standard deviations, respectively. Based on the correlation coefficient ρ, regions with a threshold ρ ≥ 0.85 are marked as high-risk contact sub-regions, and points outside the threshold range are labeled as low-stability feature point sets.

[0103] For example, in a certain assembly task, cross-matching of the stress uniformity distribution map and the contact stiffness variation characteristics showed that the region (30,45,0) had a correlation of 0.92 and was marked as a high-risk contact sub-region; while the point (20,35,0) exhibited low stability due to a correlation of 0.65. Further analysis showed that the stiffness decay rate of this feature point exceeded 25% within 10 seconds, indicating the presence of mechanical anomalies.

[0104] In step S300, the steps of extracting features from the stability index information to obtain a stability feature vector, and using the stability feature vector to perform contact pattern recognition on the dynamic change spectrum of assembly strain to obtain the contact area state sequence, specifically include:

[0105] Principal component analysis and dynamic time warping are performed on the stability index information to extract the steady-state feature distribution and rate of change, and the stability feature vector is constructed using the steady-state feature distribution and rate of change.

[0106] Using stability index information as input variables, principal component analysis (PCA) is first performed on the data. Dimensionality reduction maps high-dimensional feature data (such as strain rate variation, dynamic characteristics of contact stiffness, and positional stability scores) to a lower-dimensional space, thereby extracting the steady-state feature distribution. Simultaneously, the dynamic time warping (DTW) algorithm is used to analyze the time series of stability features, calculating their rate of change and correcting for temporal inconsistencies. Based on the extraction results, a stability feature vector containing both the steady-state distribution and the rate of change is constructed.

[0107] Specifically, the PCA process includes calculating the covariance matrix of the stability index information dataset, sorting the eigenvectors and eigenvalues ​​of the covariance matrix, and selecting the principal components with the highest contribution rates as the feature representations after dimensionality reduction. Dynamic time warping is used to calculate the optimal matching path between any two unequal time series, as shown in the following formula:

[0108] ;

[0109] Where D(i,j) represents the total consumption distance before the current point, and d(i,j) is the Euclidean distance between the two data points. Finally, the principal components and the rate of change over time are combined into a stability feature vector.

[0110] For example, for a certain assembly, the three main characteristics of stability index information are strain rate change, contact stiffness change, and stability score. After principal component analysis, the two principal components with the largest contribution rates are PC1=80% and PC2=15%, respectively. Dynamic time warping analysis shows that after correcting the time inconsistency between the strain rate change feature points and stiffness change, the final stability feature vector is [0.5, 0.2].

[0111] The dynamic variation spectrum of assembly strain is segmented into intervals and dynamically clustered. The stress response mode within different time windows is calibrated to obtain the stress response mode. The stability feature vector and the stress response mode are jointly embedded into a unified feature space to obtain the fused stability feature vector.

[0112] The dynamic strain variation spectrum of the assembly is divided into several small windows along the time dimension. Feature extraction is performed on the image data of each time interval, and the data are grouped using a dynamic clustering algorithm to identify the corresponding stress response mode. Subsequently, the feature vectors extracted from the stability index information and the stress response modes are fused in a unified feature space to generate a fused stability feature vector that includes both geometric and mechanical properties.

[0113] Specifically, the time window is adaptively segmented based on a fixed time interval (e.g., every 1 second) or a significant inflection point in stress change; the stress response features of each window are extracted by calculating feature vectors such as local stress peaks and gradient changes. A dynamic clustering algorithm, K-means++, is used to perform clustering by minimizing the Euclidean distance between the feature points of each window and the cluster center. The stability feature vector and the stress response pattern feature vector are concatenated to establish a unified feature vector representation in high-dimensional space.

[0114] For example, during the assembly process, the dynamic change spectrum is divided into 5 time windows within the 0-5 second interval, and the feature points of each window form a stress response pattern; for example, the stress peak value in window 1 is 50 MPa, the stress gradient is 0.8, and it is labeled as pattern A. The stability feature vector [0.8,0.5] and pattern A are combined to obtain the fused feature vector [0.8,0.5,A].

[0115] Key contact events are identified based on the fusion stability feature vector, including state changes such as contact establishment, slip transition, and contact disappearance. A time-series contact map is constructed using these key contact events, and the contact state transition paths are extracted from the time-series contact map. The contact state transition paths are then structurally encoded to obtain a complete contact area state sequence.

[0116] By leveraging the dynamic changes in fused stability feature vectors, key contact events (such as contact establishment, slip transition, and contact disappearance) occurring during assembly are identified, and the temporal and spatial locations of these events are analyzed. Based on these events, a temporal contact map is constructed to represent the evolution trajectory of contact states during assembly. Simultaneously, contact state transition paths are extracted from the map and represented as a temporal state sequence through path structure encoding.

[0117] Specifically, nodes in the temporal contact map are represented by key contact events, and the connections between nodes characterize the temporal and spatial relationships of the state transition paths. For example, a contact establishment event is defined as the transition from initial no contact to stress > 5 MPa and contact area > 10 mm². 2 The transition is characterized by a sudden drop in stiffness (>30%), and a rapid reduction in contact area. The transition path in the graph is represented by a state transition matrix, where each element contains an event number and a time interval. The path structure is encoded by traversing the entire transition path using a depth-first search, generating a temporal sequence.

[0118] For example, the contact events in a certain assembly process are identified as follows: contact is established at 0.2 seconds, slip transition is detected at 3.6 seconds, and contact disappears at 6.0 seconds. Based on this, a temporal contact map is constructed, and the transition path in the map is represented as follows:

[0119] Path = {(Create, 0.2s), (Slide, 3.6s), (Disappear, 6.0s)}.

[0120] The generated contact area state sequence after encoding is: establishment—sliding—disappearance.

[0121] In step S400, the steps of inputting the contact area state sequence into a preset assembly anomaly identification model to obtain the assembly anomaly risk assessment result, and performing a graded analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates include:

[0122] The contact area state sequence is encoded and vectorized according to time window and state node to generate dynamic contact behavior sequence. The dynamic contact behavior sequence is input into the preset assembly anomaly identification model, and the assembly stability deviation and anomaly mode label are output.

[0123] Based on the temporal and spatial variation characteristics of the contact area state sequence, the sequence data is divided into multiple time windows, and the state nodes of each window are encoded using time nodes and processed into feature vectors. These encoded dynamic contact behavior sequences are then used as input to a pre-defined assembly anomaly identification model for real-time calculation and analysis. This model identifies stability deviations and anomaly patterns during the assembly process, and outputs the results as anomaly pattern labels.

[0124] Specifically, the time window is dynamically divided based on the state change time points in the contact area state sequence (such as contact transfer time, force state abrupt change time). The state node information of each window is converted through encoding rules, for example:

[0125] ;

[0126] Among them, B i Let be the feature vector of the i-th node in the window, containing the timestamp t. i State s i (such as contact establishment, slip transition, etc.) and force state characteristics f i (Including contact area, stress, etc.). The assembly anomaly identification model uses an existing convolutional neural network (CNN) to perform anomaly analysis on dynamic contact behavior sequences through feature extraction and classification layers, and finally outputs stability deviation and anomaly labels, such as classification results like "overstress" and "force concentration".

[0127] For example, in a certain assembly process, the state sequence of the contact area is recorded as follows: the state "contact established" is generated at 0.2 seconds, the state "slip transition" occurs at 3.0 seconds, and "contact disappearance" is detected at 7.0 seconds. This is encoded as a time-series dynamic contact behavior sequence:

[0128] ;

[0129] This sequence is input into a CNN model, and the final output has a deviation of 0.35 and an anomaly pattern label "force concentration anomaly".

[0130] Assembly anomaly risk assessment results are generated based on assembly stability deviation and anomaly pattern labels, including anomaly type, severity, and trigger time period. The assembly anomaly risk assessment results are compared for similarity using a pre-set assembly database to obtain risk level information. Based on the risk level information, a pre-set strategy template library is matched to obtain a set of strategy candidates.

[0131] The deviation and anomaly labels output by the model are passed to the subsequent anomaly assessment module. Based on the degree of deviation and label type, assembly anomaly risk assessment results are generated, including information such as anomaly type, severity, and trigger time period. These results are compared with historical data in the assembly task database to calculate the probability and impact of anomaly patterns, ultimately generating risk level information (such as "low risk," "high risk," etc.). Based on the risk level information, a pre-defined strategy template library is matched to generate a set of strategy candidate sets adapted to the current assembly state.

[0132] Specifically, risk assessment and grading are based on the following rules: Let the deviation score be p, and the severity range be [1, 5]. Low risk is defined as p < 0.2, medium risk as 0.2 ≤ p < 0.5, and high risk as p ≥ 0.5. The anomaly label type is graded based on probability statistics from the historical database; for example, if the probability of the overstress type exceeds 70%, the risk level is increased. The strategy candidate set generation process uses a matching utility scoring algorithm.

[0133] ;

[0134] Where S is the utility score, e i As the current environment state, r i The content is the strategy template, sim() is the similarity function, and w i These are the weighting coefficients. Policy templates with high matching scores are added to the candidate set.

[0135] For example, in a certain assembly task, the deviation score is 0.45, the triggered anomaly type is "slip transition," and the occurrence time is from 2.5 seconds to 4.0 seconds. The risk level is confirmed as "medium risk" (65% probability of occurrence) by database comparison results. After comparison with the strategy template library, the generated strategy candidate set includes: ① adjusting torque direction, ② optimizing assembly path, and ③ dynamic position calibration.

[0136] In step S500, the dynamic evolution data of spatial configuration feature information is obtained based on risk level information. The dynamic evolution data is classified to obtain several sets of assembly state transition paths. The strategy candidate set is used to perform strategy matching on all sets of assembly state transition paths to obtain the assembly guidance strategy. The steps specifically include:

[0137] Based on the risk level information, the key spatial configuration regions corresponding to the spatial configuration feature information are selected, and the spatial feature evolution trajectory of the key spatial configuration regions over time is extracted to obtain dynamic evolution data. Multidimensional feature mapping and path structure analysis are performed on the dynamic evolution data to obtain analysis results, which are then divided into several assembly state transition units.

[0138] Based on the risk level information of the assembly task, the spatial configuration region most relevant to the current risk level is selected. Based on these key regions, their spatial feature evolution trajectories over time are extracted, with data sourced from real-time monitoring and simulation systems. On this basis, multi-dimensional feature mapping is performed to convert them into a unified feature space, capturing the changing trends of the spatial configuration during assembly. Path structure analysis is conducted on these changing data, and based on the transition relationships between various states during assembly, the results are divided into several assembly state transition units.

[0139] Specifically, the extraction of spatial feature evolution trajectories employs time series analysis and feature extraction techniques, combining features such as position, attitude, and stress to construct an evolution trajectory map. During path structure analysis, the shortest path algorithm from graph theory is used to optimize the changing paths. By constructing a state transition matrix, the continuously changing state sequence is divided into several transition units.

[0140] For example, in a certain assembly task with a risk level of "medium risk," dynamic evolution data was extracted from areas where torque changes were significant during assembly. By analyzing the pose changes in these areas (e.g., position changes of x: 5.2 mm, y: 3.5 mm), the dynamic evolution trajectory was obtained. Further path analysis showed that these changes could be divided into two assembly state transition units: the first unit was "contact establishment," and the second unit was "slip transition."

[0141] All assembly state transition units are clustered into an assembly state transition path set according to temporal order and evolution trend. Each strategy in the strategy candidate set is matched and scored with each path in the assembly state transition path set to construct a strategy path adaptation matrix.

[0142] Cluster analysis of the temporal sequence and evolution trend of all assembly state transition units is performed. Dynamic Time Warping (DTW) and other clustering algorithms are used to optimize the clustering of each state transition in the assembly process, ensuring that each state transition path reflects the actual assembly operation steps. After clustering, each policy in the policy candidate set is matched with each path to evaluate the policy's fit to the path, and a policy-path fit matrix is ​​constructed based on the scores.

[0143] Specifically, during the clustering process, the changing trend of each path is first divided into time windows, and the optimal matching path combination is found through dynamic time warping analysis. Next, similarity scoring algorithms (such as cosine similarity and Euclidean distance) are used to match each path with the policies in the policy candidate set. Finally, a policy-path adaptation matrix is ​​constructed based on the matching results, where each matrix element represents the matching degree between the policy and the path.

[0144] For example, during the assembly process, after dynamic time warping analysis, the assembly state transition units are clustered into three paths: path A (contact establishment – ​​slip transition), path B (contact disappearance – stress recovery), and path C (contact establishment – ​​torque fluctuation). For each path, by matching it with strategies in the strategy candidate set, the matching degree between path A and strategy 1 is 0.85, the matching degree between path B and strategy 2 is 0.72, and the matching degree between path C and strategy 3 is 0.90. The final strategy path adaptation matrix is ​​obtained.

[0145] For each strategy-path combination in the strategy-path adaptation matrix, the intervention effectiveness is simulated and the response time is evaluated to obtain the test evaluation results. Based on the test evaluation results, the optimal strategy-path combination is selected to generate an assembly guidance strategy that includes operation sequence, intervention content and guidance method.

[0146] An intervention effectiveness simulation system is used to simulate and predict each strategy-path combination in the strategy-path adaptation matrix, evaluating the execution effect of each strategy under the corresponding path. During the simulation, various factors, such as assembly efficiency, accuracy, and stability, are considered to assess the intervention effectiveness of each strategy. Simultaneously, response time is evaluated, taking into account the strategy implementation time and real-time reactions during the assembly process. Based on these test and evaluation results, the optimal strategy-path combination is selected, ultimately generating an assembly guidance strategy that includes detailed operation sequences, intervention content, and guidance methods.

[0147] Specifically, in the simulation prediction of intervention effectiveness, existing simulation platforms (such as MATLAB, Simulink, etc.) are used for dynamic modeling. Simulation environment parameters of the strategy path (such as torque, contact pressure, component displacement) are input, and the intervention effect is simulated through control algorithms and verified experimentally. Response time evaluation is performed by comparing time complexity analysis with experimental data to select the strategy with the shortest time and optimal effect.

[0148] For example, in a certain assembly task, path A and strategy 1 have the highest matching score. Through intervention effectiveness simulation, strategy 1 can effectively reduce the slip transition time, thereby improving assembly accuracy. After response time evaluation, the response time of this strategy is 0.3 seconds. Compared with other strategies, strategy 1 performs the best and is therefore selected as the optimal strategy. The generated assembly guidance strategy includes the following operation sequence: ① calibrate the contact point position, ② optimize the force application direction, and ③ adjust the torque value.

[0149] In this embodiment, by collecting the original image sequence, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station, a set of key assembly areas is first generated using multi-sensor fusion technology. Combined with calibration registration and data synchronization analysis, a 3D visual model, real-time pose information, and attitude mapping field of the assembly component are constructed. Instantaneous stress fluctuation characteristics are extracted based on the triaxial torque data to achieve dynamic region labeling of the assembly action. Subsequently, spatial configuration feature information is generated based on the set of key assembly areas. Through topological modeling of the 3D pose information and multiphysics coupling simulation, the uniformity of force distribution and contact stiffness variation trend in local areas are analyzed. High-risk contact sub-regions and low-stability feature point sets are identified through cross-matching. Combining real-time contact pressure distribution with historical database matching, stability index information and contact area change information are comprehensively evaluated and constructed, thereby generating a dynamic variation map of assembly strain. In the contact pattern recognition stage, steady-state feature distribution and change rate are extracted through principal component analysis and dynamic time warping of stability index information to construct a stability feature vector. Simultaneously, by combining interval segmentation and dynamic clustering of the assembly strain dynamic change spectrum, a fused stability feature vector is generated to identify key contact events during the assembly process and construct and encode the contact area state sequence. In the assembly anomaly identification and guidance strategy generation process, dynamic analysis of the contact area state sequence and the assembly anomaly identification model generates assembly anomaly risk assessment results (including deviation, risk level, and anomaly type). Using a strategy matching tool, a strategy path adaptation matrix is ​​generated based on the assembly state transition path set. Simulation evaluation is used to select the optimal strategy path combination, resulting in an assembly guidance strategy that includes operation sequences, intervention content, and guidance methods. This embodiment achieves accurate perception and adaptive adjustment of dynamic changes in complex assembly environments, effectively improving assembly efficiency and quality.

[0150] Example 3:

[0151] This application also provides a visual guidance system 10 for intelligent assisted assembly, including a data acquisition module 11, an analysis module 12, an extraction module 13, an input module 14, and a matching module 15.

[0152] like Figure 2 As shown, the acquisition module 11 is mainly used to acquire the original image sequence, depth coordinate data and three-axis torque data of the current assembly object at the assembly station in order to construct a set of key assembly areas.

[0153] The acquisition module 11 uses a high-precision industrial camera, a depth sensing module, and a force sensor to acquire and process in real time image data, depth coordinate data, and three-axis torque data of the object being assembled at the assembly station, thereby constructing a set of key assembly areas. This module ensures the integrity and real-time nature of the assembly data, providing high-quality data input for subsequent analysis.

[0154] Analysis module 12 is mainly used to generate spatial configuration feature information based on the set of key assembly areas, and to evaluate and analyze the stress state of the spatial configuration feature information using triaxial torque data to obtain stability index information and assembly strain dynamic change spectrum.

[0155] Analysis module 12 generates spatial configuration feature information based on the collected data, and performs stress state assessment and analysis on the configuration feature information using triaxial moment data to obtain stability indicators and dynamic strain change patterns during the assembly process. This module can clearly reflect the stress distribution, stress uniformity, and dynamic change trends of the assembled object, providing an important mechanical basis for identifying abnormal assembly tendencies.

[0156] Extraction module 13 is mainly used to extract features from stability index information to obtain stability feature vectors. The stability feature vectors are then used to perform contact pattern recognition on the dynamic change spectrum of assembly strain to obtain the state sequence of the contact area.

[0157] The extraction module 13 further extracts features from the stability index information generated by the analysis module 12, calculates the stability feature vector, and performs contact pattern recognition using a dynamic clustering algorithm and assembly strain dynamic change map to identify the key states of the contact area during assembly. This module can accurately identify the dynamic contact patterns of the assembled object and any contact anomalies that occur, providing input for anomaly risk assessment.

[0158] The input module 14 is mainly used to input the contact area state sequence into the preset assembly anomaly identification model to obtain the assembly anomaly risk assessment result, perform hierarchical analysis on the assembly anomaly risk assessment result, and obtain risk level information and strategy candidate set.

[0159] Input module 14 inputs the extracted contact area state sequence into a preset assembly anomaly identification model. Based on a deep learning algorithm, it identifies and classifies the anomaly risks, ultimately generating risk level information and a set of strategy candidates. This module can effectively distinguish the severity and location of different risks during the assembly process, providing a reliable basis for generating assembly guidance strategies.

[0160] The matching module 15 is mainly used to obtain dynamic evolution data of spatial configuration feature information based on risk level information, classify the dynamic evolution data to obtain several assembly state transition path sets, and use the strategy candidate set to perform strategy matching on all assembly state transition path sets to obtain the assembly guidance strategy.

[0161] The matching module 15 selects dynamic evolution data from spatial configuration feature information based on risk level information and classifies this data to generate a set of assembly state transition paths. This module intelligently matches all path sets using a strategy candidate set, and selects the optimal strategy path combination through an algorithm, ultimately generating an assembly guidance strategy that includes intervention measures and guidance content. Through this module, the assembly process can be dynamically optimized in complex assembly environments, providing rapid and effective guidance and intervention.

[0162] In this embodiment, a modular design enables real-time data acquisition, multi-dimensional mechanical analysis, anomaly and risk identification, and intelligent generation and guidance of assembly strategies, forming a complete closed-loop assembly optimization process. This system possesses significant advantages in efficiency, accuracy, and flexibility, helping to improve the automation level and assembly precision of complex assembly tasks, and addressing the shortcomings of existing technologies in responding to dynamic changes and real-time control.

[0163] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and each module described above can be referred to the corresponding process in the aforementioned Embodiment 1, and will not be repeated here.

[0164] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0165] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual guidance method for intelligent assisted assembly, characterized in that, include: Collect the original image sequence, depth coordinate data, and three-axis torque data of the current assembly object at the assembly station to construct a set of key assembly areas; Based on the set of key assembly areas, spatial configuration feature information is generated. The stress state of the spatial configuration feature information is evaluated and analyzed using the triaxial torque data to obtain stability index information and assembly strain dynamic change spectrum. The stability index information is subjected to feature extraction to obtain a stability feature vector. The stability feature vector is then used to perform contact pattern recognition on the dynamic change spectrum of the assembly strain to obtain a contact area state sequence. Specifically, principal component analysis and dynamic time warping are performed on the stability index information to extract the steady-state feature distribution and change rate, and the steady-state feature distribution and change rate are used to construct a stability feature vector. The dynamic variation spectrum of the assembly strain is segmented into intervals and dynamically clustered to obtain the stress response mode. The stability feature vector and the stress response mode are jointly embedded into a unified feature space to obtain a fused stability feature vector. Key contact events are identified based on the fusion stability feature vector, and a time-series contact map is constructed using the key contact events. The contact state transition paths of the time-series contact map are extracted, and the contact state transition paths are structurally encoded to obtain a complete contact area state sequence. The contact area state sequence is input into a preset assembly anomaly identification model to obtain assembly anomaly risk assessment results. The assembly anomaly risk assessment results are then subjected to hierarchical analysis to obtain risk level information and a set of strategy candidates. Based on the risk level information, obtain dynamic evolution data of the spatial configuration feature information, classify the dynamic evolution data to obtain several assembly state transition path sets, and use the strategy candidate set to perform strategy matching on all the assembly state transition path sets to obtain the assembly guidance strategy.

2. The visual guidance method for intelligent assisted assembly according to claim 1, characterized in that, The step of collecting the original image sequence, depth coordinate data, and triaxial torque data of the current assembly object at the assembly station to construct a set of key assembly areas includes: The system utilizes an industrial camera, depth sensing module, and force sensor installed at the assembly station to simultaneously acquire the original image sequence, depth coordinate data, and triaxial torque data of the force applied to the contact surface of the object being assembled. The original image sequence and the depth coordinate data are calibrated and registered. The calibrated and registered original image sequence and depth coordinate data are combined with the workstation reference coordinate system to generate a three-dimensional visual model and real-time pose information of the assembly component. The three-dimensional visual model of the assembly component and the real-time pose information are fused to obtain the pose mapping field. Time series analysis is performed on the triaxial torque data to obtain instantaneous stress fluctuation characteristics. The attitude mapping field and the instantaneous stress fluctuation characteristics are jointly calculated to generate assembly action influence domain information. The assembly action influence domain information is used to dynamically label the three-dimensional visual model to obtain a set of key assembly areas.

3. The visual guidance method for intelligent assisted assembly according to claim 1, characterized in that, The steps of generating spatial configuration feature information based on the set of key assembly areas, and using the triaxial moment data to perform stress state evaluation and analysis on the spatial configuration feature information to obtain stability index information and assembly strain dynamic change spectrum include: Based on the set of key assembly areas, an assembly configuration contour surface is constructed. The assembly configuration contour surface is combined with three-dimensional pose information to perform topological modeling to obtain spatial configuration feature information. The three-dimensional pose information refers to the position coordinate information and orientation attitude information of the current assembly object in the workstation coordinate system. Using the triaxial torque data, multi-region loading simulation analysis is performed on the spatial configuration feature information to obtain the force uniformity distribution map and contact stiffness variation characteristics of each configuration region. The force uniformity distribution map and the contact stiffness variation characteristics are cross-matched to obtain high-risk contact sub-regions and low-stability feature point sets. The real-time contact pressure distribution of the high-risk contact sub-region during the current assembly process is matched with the force pattern of the same contact area in the historical task database, and the deviation score of each sub-region is calculated. Stability index information is constructed based on the deviation score. Extract the three-dimensional displacement time series of the low-stability feature point set during the entire assembly process, calculate the strain rate change trend and contact area change trend of the three-dimensional displacement time series, and construct contact area change information by applying the strain rate change trend and the contact area change trend. An assembly strain dynamic change map is constructed by combining the contact area change information with the stability index information.

4. The visual guidance method for intelligent assisted assembly according to claim 3, characterized in that, The step of performing multi-region loading simulation analysis on the spatial configuration feature information using the triaxial moment data to obtain the force uniformity distribution map and contact stiffness variation characteristics of each configuration region, and cross-matching the force uniformity distribution map with the contact stiffness variation characteristics to obtain high-risk contact sub-regions and low-stability feature point sets includes: Based on the triaxial torque data, each key assembly area in the spatial configuration feature information is divided into multiple local regions, and multiphysics coupling simulation is performed on each local region to obtain loading simulation results; The stress distribution of each local area is calculated based on the loading simulation results to generate a stress uniformity distribution map. Feature extraction is performed on the stress uniformity distribution map to identify the contact stiffness variation characteristics of each assembly area and obtain a contact stiffness variation spectrum. By cross-matching the stress uniformity distribution map with the contact stiffness variation characteristics, the correlation between stress and stiffness variation in local areas is analyzed, and high-risk contact sub-regions and low-stability feature point sets are identified.

5. The visual guidance method for intelligent assisted assembly according to claim 1, characterized in that, The steps of inputting the contact area state sequence into a preset assembly anomaly identification model to obtain an assembly anomaly risk assessment result, and performing a graded analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates include: The contact area state sequence is encoded and vectorized according to time window and state node to generate dynamic contact behavior sequence. The dynamic contact behavior sequence is input into a preset assembly anomaly identification model to output assembly stability deviation and anomaly mode label. An assembly anomaly risk assessment result is generated based on the assembly stability deviation and the anomaly pattern label. The assembly anomaly risk assessment result is then compared with the pre-set assembly database to obtain risk level information. Based on the risk level information, a pre-set strategy template library is matched to obtain a set of strategy candidates.

6. The visual guidance method for intelligent assisted assembly according to claim 1, characterized in that, The steps of obtaining dynamic evolution data of spatial configuration feature information based on the risk level information, classifying the dynamic evolution data to obtain several assembly state transition path sets, and using the strategy candidate set to perform strategy matching on all the assembly state transition path sets to obtain an assembly guidance strategy include: Based on the risk level information, the key spatial configuration region corresponding to the spatial configuration feature information is selected, and the spatial feature evolution trajectory of the key spatial configuration region over time is extracted to obtain dynamic evolution data. Multidimensional feature mapping and path structure analysis are performed on the dynamic evolution data to obtain analysis results. The analysis results are divided into several assembly state transition units. All assembly state transition units are clustered into an assembly state transition path set according to temporal order and evolution trend. Each strategy in the strategy candidate set is matched and scored with each path in the assembly state transition path set to construct a strategy path adaptation matrix. For each strategy-path combination in the strategy-path adaptation matrix, the intervention effectiveness is simulated and predicted, and the response time is evaluated to obtain the test evaluation results. Based on the test evaluation results, the optimal strategy-path combination is selected to generate the assembly guidance strategy.

7. A vision guidance system for intelligent assisted assembly, characterized in that, include: The acquisition module is used to acquire the original image sequence, depth coordinate data and three-axis torque data of the current assembly object at the assembly station in order to construct a set of key assembly areas. The analysis module is used to generate spatial configuration feature information based on the set of key assembly areas, and to perform stress state evaluation and analysis on the spatial configuration feature information using the triaxial torque data to obtain stability index information and assembly strain dynamic change spectrum. The extraction module is used to extract features from the stability index information to obtain a stability feature vector, and to use the stability feature vector to perform contact pattern recognition on the dynamic change spectrum of the assembly strain to obtain a contact area state sequence; specifically, it performs principal component analysis and dynamic time warping on the stability index information to extract the steady-state feature distribution and change rate, and uses the steady-state feature distribution and change rate to construct a stability feature vector; The dynamic variation spectrum of the assembly strain is segmented into intervals and dynamically clustered to obtain the stress response mode. The stability feature vector and the stress response mode are jointly embedded into a unified feature space to obtain a fused stability feature vector. Key contact events are identified based on the fusion stability feature vector, and a time-series contact map is constructed using the key contact events. The contact state transition paths of the time-series contact map are extracted, and the contact state transition paths are structurally encoded to obtain a complete contact area state sequence. The input module is used to input the contact area state sequence into a preset assembly anomaly identification model to obtain the assembly anomaly risk assessment result, and to perform a hierarchical analysis on the assembly anomaly risk assessment result to obtain risk level information and a set of strategy candidates. The matching module is used to obtain dynamic evolution data of the spatial configuration feature information based on the risk level information, classify the dynamic evolution data to obtain several assembly state transition path sets, and use the strategy candidate set to perform strategy matching on all the assembly state transition path sets to obtain the assembly guidance strategy.

Images