Machine vision-based multi-dimensional detection control method and system for screw assembly quality

Abstract

The application provides a machine vision-based screw assembly quality multidimensional detection control method and system, relates to the technical field of image analysis and intelligent control, and through the deployment of a three-dimensional profile sensor which moves synchronously with the main shaft of an electric wrench, time-sequenced three-dimensional visual monitoring of the whole screw locking process is realized, and the three-dimensional morphological changes of the screw head and the surrounding area are reconstructed in the form of point cloud sequences, three-dimensional visual perspective of the locking process is realized, the final state of the screw can be provided, the path and attitude evolution process of the screw to achieve the final state can be recorded completely, early fault signs can be found when the torque is abnormal, diagnosis is carried out based on the abnormality, and grading control instructions are generated, the intelligent ability of autonomously coping with complex working conditions and actively maintaining the process window without human intervention is realized, process dynamic three-dimensional detection and adaptive online correction are realized, the scrap rate is reduced, and therefore the screw assembly qualified rate is improved and the assembly cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of image analysis and intelligent control technology, specifically to a multi-dimensional detection and control method and system for screw assembly quality based on machine vision. Background Technology

[0002] In the current field of automated screw assembly, ensuring the reliability of screw fastening quality is the core technological challenge. Existing methods mainly fall into two directions: one is quality inspection based on machine vision, and the other is tightening control technology based on process parameter monitoring. The first approach usually involves taking an image of the screw head with a 2D camera after screw fastening and using template matching, color analysis, or simple edge detection technology to determine whether the screw is present or misaligned. A few solutions introduce a single-point laser rangefinder to measure the height of the screw head to determine whether it is floating or sunken. The second approach relies on a smart electric screwdriver with communication capabilities to collect and monitor the torque and angle curves during the screw fastening process in real time, and to trigger an alarm when the torque value exceeds or fails to reach a preset threshold.

[0003] The existing technology has the following technical problems when used: Problem 1: Existing vision-based screw assembly quality inspection methods make binary judgments on the assembly results, which cannot perceive the dynamic behavior during the fastening process. When defects such as floating or stripping are detected, defective products have already been generated, and can only be reworked or scrapped. It cannot prevent the occurrence of defects. Moreover, 2D images lack depth information, making it difficult to accurately quantify the tilt angle and three-dimensional posture of the screw. It is impossible to perform real-time and continuous quantitative monitoring and early anomaly identification of the dynamic three-dimensional posture of the screw during the fastening process, such as the sinking trajectory and real-time tilt angle. This results in the inability to intervene during the defect occurrence process and the lack of real-time quantitative perception capabilities, which leads to an increase in the screw assembly scrap rate and a significant increase in production costs. The second problem is that while existing technologies monitor process parameters, the analysis is limited to a single dimension, only detecting the numerical relationship between torque and angle, ignoring the physical state of the interaction between the screw and the workpiece. For example, the same final torque may be caused by completely different physical processes such as normal tightening, foreign objects in the threads, or screw tilting and jamming. Torque values ​​alone cannot distinguish these essential differences, let alone pinpoint the root cause of the defect. At the same time, process parameter monitoring and visual inspection systems are independent of each other, and cannot be correlated in time and space for analysis and root cause diagnosis. They also cannot autonomously adjust strategies based on real-time status during the fastening process, lacking the control capability of multi-source information fusion for real-time diagnosis and autonomous decision-making. This leads to inaccurate screw inspection, reduced quality pass rate, and is not conducive to practical use. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a multi-dimensional detection and control method and system for screw assembly quality based on machine vision, wherein the method includes: During the screw fastening process, a time-series 3D point cloud sequence of the screw head and surrounding area is simultaneously acquired, along with real-time torque and angle data streams of the fastening tool. Dynamic visual feature sequences representing screw posture and sinking process are extracted in real time from a temporal 3D point cloud sequence. The dynamic visual feature sequence is spatiotemporally aligned and feature fused with the real-time torque and angle data stream to generate multimodal fusion features for the locking process; Based on the multimodal fusion features, real-time analysis is performed using preset association rules and diagnostic models. If a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated. According to the hierarchical control instructions, the corresponding real-time control actions are executed, including generating early warnings, adjusting the process parameters of the locking tool, executing adaptive correction motion, and stopping the locking process.

[0005] Furthermore, the synchronous acquisition of the time-series 3D point cloud sequence of the screw head and surrounding area includes: Deploy a three-dimensional profile sensor that maintains a fixed offset relationship with the spindle of the fastening tool, and ensure that the scanning axis of the three-dimensional profile sensor is synchronously tracked with the screw head; Throughout the entire process of pressing down and rotating the fastening tool, the three-dimensional contour sensor is controlled to continuously scan the screw head and its surrounding area in contact with the workpiece. The data obtained from continuous scanning is reconstructed into a time-series 3D point cloud sequence in chronological order of acquisition time. Each frame of the 3D point cloud in the time-series 3D point cloud sequence carries timestamp information.

[0006] Furthermore, the step of extracting a dynamic visual feature sequence representing the screw's posture and sinking process from the temporal 3D point cloud sequence in real time includes: Acquire each frame of the 3D point cloud in the time-series 3D point cloud sequence, calculate the real-time height of the center point of the screw head relative to the preset workpiece reference surface, and form the screw sink trajectory curve. Acquire each frame of the 3D point cloud in the time-series 3D point cloud sequence, calculate the real-time angle between the screw head plane and the preset workpiece reference plane, and form the screw tilt angle change curve. Calculate the smoothness index of the screw sinking trajectory curve and the rate of change index of the screw tilt angle change curve; The real-time height, real-time angle, smoothness index, and rate of change index are used as visual feature data and arranged in chronological order to form a dynamic visual feature sequence.

[0007] Furthermore, the step of spatiotemporally aligning and fusing the dynamic visual feature sequence with the real-time torque and angle data stream to generate multimodal fusion features for the locking process includes: Based on a unified time base, a synchronization timestamp is applied to each visual feature data in the dynamic visual feature sequence and the torque angle data acquired at each sampling point in the real-time torque angle data stream. Visual feature data with the same timestamp is paired and associated with torque angle data to form a set of multidimensional feature data, which includes torque value, rotation angle, real-time height of screw head, and real-time tilt angle of screw head. Multidimensional feature data arranged in chronological order are combined to generate multimodal fusion features for the locking process.

[0008] Furthermore, the real-time analysis based on the multimodal fusion features, using preset association rules and diagnostic models, includes: The multimodal fusion features are monitored in real time to determine whether the multidimensional feature data exceeds the preset safety process window. If any multidimensional feature data exceeds the window, a first-level anomaly identifier is generated. Identify cross-modal association patterns in multimodal fusion features, wherein the association patterns include a first association pattern, a second association pattern, and a third association pattern; If the upward slope of the torque curve extracted from the real-time torque angle data stream is lower than the first preset threshold, and the change in height of the screw sinking trajectory curve near the angle corresponding to the torque peak is lower than the fourth preset threshold, then it is judged as the first association mode. If the standard deviation of the torque curve is higher than the fifth preset threshold and the smoothness index of the screw sinking trajectory curve is lower than the second preset threshold, it is judged as the second association mode. If the rate of change of the real-time tilt angle of the screw head exceeds the third preset threshold, it is determined to be the third association mode; When the first or second association pattern is identified, it is determined that a specific abnormal condition has occurred in the locking process, and a diagnostic result containing the abnormality type code is generated, with the abnormality type code corresponding to the identified association pattern.

[0009] Furthermore, if a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated, including: The preset anomaly modes include a first association mode, a second association mode, and a third association mode; If a first-level anomaly identifier is generated, or a first association pattern is identified, then a first-class control instruction is generated.

[0010] If the second association pattern is identified, a second type of control instruction is generated; If the pattern is identified as the third association pattern, then a third type of control instruction is generated; If the multimodal fusion features of N consecutive locking cycles show that the real-time tilt angle change rate of the screw head exhibits a monotonically increasing trend, then a fourth type of control command is generated.

[0011] Furthermore, the step of executing corresponding real-time control actions according to the hierarchical control instructions includes: If a first-class control command is received, the pressing and rotating actions of the locking tool will be stopped immediately, and it will be raised to the initial waiting position before the locking process begins, while triggering an audible and visual alarm. If a third type of control instruction is received, the screw head tilt angle is corrected in real time according to the third type of control instruction while the fastening tool is under downward pressure. The change of the screw head tilt angle in real time is monitored. If the screw head tilt angle returns to within the safe process window, the correction is considered successful and the fastening process continues. Otherwise, the first type of control instruction is executed. If a second or fourth type of control command is received, a warning message will be displayed on the human-machine interface and recorded in the maintenance log.

[0012] Furthermore, the method also includes: After the screw is fastened, the final three-dimensional point cloud of the screw head and the workpiece plane is collected, the final height and final flatness of the screw head are calculated, and the panoramic data package of the screw fastening event is generated by combining the multimodal fusion features and diagnostic results generated during the fastening process. The panoramic data package is stored in the process knowledge base.

[0013] Furthermore, the method also includes: Cluster analysis is performed on the panoramic data package of M qualified locking events of the same product model stored in the process knowledge base to mine the optimal range of parameters in the multimodal fusion features; Update the baseline fastening process parameters for this product model based on the optimal range of the parameters.

[0014] A machine vision-based multi-dimensional inspection and control system for screw assembly quality, the system comprising: The multi-dimensional visual perception module simultaneously acquires a time-series 3D point cloud sequence of the screw head and surrounding area during the screw fastening process; A high-precision process sensing module is used to synchronously acquire real-time torque and angle data streams of the fastening tool; The real-time quality analysis and decision control module includes a data fusion unit, an online intelligent anomaly detection unit, and an instruction generation unit; The data fusion unit is used to extract dynamic visual feature sequences representing screw posture and sinking process from a time-series 3D point cloud sequence in real time; and to perform spatiotemporal alignment and feature fusion with real-time torque and angle data streams to generate multimodal fusion features of the fastening process. The online intelligent anomaly detection unit is used to perform real-time analysis based on the multimodal fusion features, through preset association rules and diagnostic models, to identify preset abnormal patterns. The instruction generation unit is used to generate corresponding diagnostic results and hierarchical control instructions based on the identification of preset abnormal patterns. The motion control and execution module is used to execute corresponding real-time control actions according to the hierarchical control instructions. The real-time control actions include generating early warnings, adjusting the process parameters of the locking tool, executing adaptive correction motion, and stopping the locking process. The process data management and optimization module includes a process knowledge base and an optimization analysis unit. The process knowledge base is used to store panoramic data packets with locking time. The optimization analysis unit is connected to the process knowledge base and is used to analyze historical panoramic data packets and perform process parameter optimization.

[0015] This invention provides a machine vision-based multi-dimensional detection and control method and system for screw assembly quality. It offers the following advantages: 1. This invention utilizes a 3D contour sensor that moves synchronously with the electric screwdriver spindle to achieve time-series 3D visual monitoring of the entire screw fastening process. It continuously collects and reconstructs the 3D morphological changes of the screw head and surrounding area in the form of a point cloud sequence. This allows for the real-time extraction of two key dynamic visual features: the screw's sinking trajectory curve and the screw head tilt angle change curve. Compared to existing static single-point measurements or 2D image analysis, this invention provides a 3D visual perspective of the dynamic physical process of fastening. It not only provides the final state of the screw but also fully records the path and attitude evolution process leading to that final state. Before torque anomalies occur, it can identify early fault signs such as misaligned entry holes and thread obstruction by analyzing abnormal smoothness of the sinking trajectory or abnormal increase in the tilt angle. This shifts the quality inspection point from result acceptance to process monitoring, advancing the inspection time and achieving dynamic 3D process inspection, reducing the scrap rate, thereby improving the screw assembly qualification rate and reducing assembly costs.

[0016] This invention constructs a multimodal fusion feature data layer and hierarchical intelligent diagnosis, fusing temporal three-dimensional visual features with high-synchronization-precision torque angle curves, and running preset cross-modal association rules on this fused data. This changes the single-parameter threshold judgment mode and can accurately diagnose complex root causes such as "gradual increase in tilt caused by bit wear" and "fluctuating jamming caused by foreign objects in the thread". Based on the accurate diagnosis, hierarchical control commands are generated, such as triggering millimeter-precision adaptive online correction motion to attempt to repair slight tilt, or decisively stopping in an emergency to prevent serious defects. This forms a continuous decision chain from problem occurrence to cause analysis to solution, giving the assembly system the intelligent ability to autonomously cope with complex working conditions and actively maintain the process window without human intervention, thus improving the system's robustness and process consistency. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of the multi-dimensional detection and control method for screw assembly quality based on machine vision according to the present invention. Figure 2 This is the data transmission flow of the multi-dimensional detection and control method for screw assembly quality based on machine vision according to the present invention. Figure 1 ; Figure 3 This is the data transmission flow of the multi-dimensional detection and control method for screw assembly quality based on machine vision according to the present invention. Figure 2 ; Figure 4 This is an architecture diagram of the multi-dimensional detection and control system for screw assembly quality based on machine vision, as described in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figures 1 to 3 As shown, a multi-dimensional detection and control method for screw assembly quality based on machine vision is presented. The method includes: Step S100: During the screw fastening process, a time-series three-dimensional point cloud sequence of the screw head and surrounding area is simultaneously acquired, and the real-time torque and angle data stream of the fastening tool is simultaneously acquired. Before the screw fastening process, there is also a process of guiding the screw assembly and determining the preset workpiece reference surface; First, a high-resolution 2D vision system (such as a CCD camera) deployed above the fastening station images the target workpiece, accurately identifies and locates the two-dimensional coordinates of the center of the screw mounting hole, and provides an initial alignment guide position for the end of the fastening tool. Secondly, after the fastening tool carrying the screw moves to a predetermined height above the mounting hole, the three-dimensional contour sensor, which maintains a fixed offset relationship with the spindle of the fastening tool, is activated, but the fastening tool is stationary at this time; the three-dimensional contour sensor performs a static three-dimensional scan of the local workpiece area where the screw is about to be fastened, and obtains high-precision three-dimensional point cloud data P of the area, P=P i = (x i y i , z i And i=1,…,N; where P i Represents any point in a 3D point cloud; (x i y i , z i () represents the three-dimensional coordinates of any point in the three-dimensional point cloud data; Then, the 3D point cloud data P is filtered to remove outliers and eliminate noise and irrelevant specks. A plane fitting is then performed on the filtered 3D point cloud data P, with the goal of finding a plane equation that satisfies all points P within that plane. i The sum of the squares of the perpendicular distances is minimized; Finally, solve for an M·[A, B, C, D]. T =0 eigenvalues, where M is a matrix constructed from the coordinate covariance matrix of the 3D point cloud data P. Finally, A, B, C, and D are solved to define the optimal fitting plane, whose equation is Ax + By + Cz + D = 0; where A, B, and C are the unit normal vectors n constituting the plane. bench The components of the normal vector n bench Perpendicular to the reference plane, the direction defines the frontal orientation of the reference plane and serves as the reference for calculating real-time height and real-time angle. D is a constant term in the plane equation, which, together with the normal vector, determines the plane's specific location in space. This represents the distance from the origin of the coordinate system to the reference plane. The preset workpiece reference plane obtained based on the plane equation serves as the sole spatial geometric reference for calculating the real-time height and tilt angle of the screw head during the subsequent fastening process. This extends visual guidance from a two-dimensional plane to three-dimensional space, laying an indispensable and stable spatial reference for accurate three-dimensional measurement and diagnosis of the subsequent dynamic process.

[0020] The screw fastening tool is an integrated electromechanical actuator, which mainly includes an electric screwdriver or servo motor that provides rotational power, a screwdriver bit for clamping and driving the screw, a pressing mechanism or floating device for controlling the axial feed of the screw, and, in order to achieve high-precision process monitoring, a torque sensor (usually implemented using strain gauges or magnetoelastic principles) for real-time measurement of the output torque of the rotating shaft, and an angle encoder for measuring the absolute or relative rotation angle of the spindle. Real-time torque and angle data streams refer to the sequence of torque and angle values ​​that are synchronously collected and output at a fixed sampling frequency (e.g., 1kHz) during the entire tightening process of a screw from contact with the workpiece to completion of tightening. These values ​​are arranged in chronological order and are used to quantitatively characterize the mechanical state of the tightening process, such as the upward trend, peak value, and fluctuation of torque, as well as the correspondence between angle and torque. This is a key basis for judging whether the screw has reached the preset clamping force, whether there is stripping, thread abnormalities, or bit slippage, which are traditional process abnormalities. The real-time torque and angle data stream directly acquires analog or digital signals through the sensor circuit built into the tightening tool. After analog-to-digital conversion and timestamp marking by the data acquisition card, it forms a standardized data stream synchronized with the control system clock.

[0021] Step S101: Deploy a three-dimensional contour sensor that maintains a fixed offset relationship with the spindle of the fastening tool, and keep the scanning axis of the three-dimensional contour sensor synchronously tracking the screw head; Step S102: During the entire process of pressing down and rotating the fastening tool, the three-dimensional contour sensor is controlled to continuously scan the screw head and its surrounding area in contact with the workpiece. Among them, the workpiece specifically refers to the component or product being assembled, that is, the target object that needs to be screwed in, which can be a metal plate, a plastic shell, a PCB board, or an assembly composed of multiple parts. The surrounding area specifically refers to the workpiece's upper surface region within the scanning field of view of the 3D contour sensor, extending outward from the screw head by a certain radius (e.g., 2-3 times the screw head diameter). This surrounding area includes not only the pressure-bearing surface where the screw head contacts the workpiece, but also the exposed surface of the workpiece immediately adjacent to the screw head. By monitoring the surrounding area, the morphological changes of the point cloud can be analyzed, allowing for a more stable and accurate fitting of the preset working reference surface, especially when the screw sinks, causing partial occlusion of the head. It also allows observation of whether screw fastening causes localized deformation or warping of the workpiece surface. Furthermore, it provides more comprehensive 3D morphological information to determine whether the screw is fully seated, i.e., whether the lower surface of the screw head is completely in contact with the workpiece surface. Step S103: The data obtained by continuous scanning is reconstructed into a time-series 3D point cloud sequence in the order of acquisition time. Each frame of the 3D point cloud in the time-series 3D point cloud sequence has timestamp information. The temporal 3D point cloud sequence is a frame sequence arranged in strict chronological order, recording the evolution of the 3D scene throughout the entire fastening process. Each frame of the 3D point cloud is essentially a set of 3D coordinates (X, Y, Z) of all object surfaces within the entire field of view covered by the 3D contour sensor scan line, precisely marked by a timestamp. In addition to the core timestamp information, each frame of point cloud data also contains the coordinate values ​​of tens of thousands of 3D spatial points that constitute that frame. The coordinate values ​​accurately describe the 3D shape of the top surface of the screw head, the edge contour, the partial morphology of the side of the screw head relative to the scan line, and the 3D undulation of the workpiece surface defined by the surrounding area at the corresponding moment. This discretizes and digitizes the continuous physical motion process of fastening into a set of 3D data that can be processed and analyzed frame by frame by a computer. This allows the screw's posture (height, tilt), sinking trajectory, and microscopic changes on the workpiece surface to be accurately quantified, tracked, and analyzed, providing the most original 3D spatiotemporal data source for subsequent feature extraction.

[0022] Step S200: Extract dynamic visual feature sequences representing screw posture and sinking process from the temporal 3D point cloud sequence in real time. Step S201: Obtain each frame of the 3D point cloud in the time-series 3D point cloud sequence, calculate the real-time height of the center point of the screw head relative to the preset workpiece reference surface, and form the screw sinking trajectory curve. The calculation of the real-time height and the generation of the screw sinking trajectory curve include the following processes: First, calculate the three-dimensional coordinates (X, Y, Z) of the center point of the screw head in the current frame. c ,Y c Z c Substitute the plane equation Ax+By+Cz+D=0 into the preset workpiece datum plane; Then, the directional vertical distance from the center point to the preset workpiece reference surface is calculated as the real-time height H, using the following formula: ; Where H is the real-time height of the screw head center point relative to the preset workpiece reference surface, and A, B, C, and D are plane equation coefficients; the absolute value is used to determine which side of the preset workpiece reference surface the center point is on. When it is positive, it means that the center point is on the side pointed to by the normal vector of the preset workpiece reference surface, and when it is negative, it means that it is on the other side. Finally, obtain the timestamp t of each frame in the temporally sequenced 3D point cloud sequence. i Corresponding height value H i As a set of data pairs (t) i H i ), and put all data pairs (t) i H i Plot and connect the coordinates in the coordinate system to generate the screw sinking trajectory curve.

[0023] When acquiring the center point of the screw head, for each frame of the 3D point cloud, the point set belonging to the screw head is extracted from the 3D point cloud, and then the geometric centroid of the point set in the 3D space is calculated. This geometric centroid is regarded as the center point of the screw head at that moment, representing the position of the screw head in space. Real-time height is used to quantify the depth of screw sinking along the axial direction (usually approximately the normal direction of the preset workpiece reference surface) during the fastening process. It is a core indicator for judging the fastening progress, whether it is in place, and whether the sinking movement is smooth and stable. The screw sinking trajectory curve is a two-dimensional curve depicted with time as the horizontal axis and real-time height as the vertical axis. It includes all time points from the start of fastening to the end (or the current moment) and their corresponding height value sequences, providing a visualization of the screw sinking movement process. The shape of the curve (such as slope, smoothness, and inflection points) directly reflects whether the sinking speed is uniform and whether there is any jamming or sudden movement.

[0024] The process involves acquiring each frame of a temporally sequenced 3D point cloud, fitting the plane equation of the screw head, calculating the real-time angle between the screw head plane and the preset workpiece reference plane, and generating a screw tilt angle variation curve. The calculation process for the real-time angle and the screw tilt angle variation curve includes the following steps: First, from the fitted equation A of the screw head plane in the current frame s x+B s y+C s z+D s Extract its normal vector n from =0. screw (A s B s C s ); ; Then, extract the normal vector n from the preset workpiece datum surface equation. bench (A,B,C); Calculate the real-time angle θ between these two normal vectors. This angle θ is usually an acute angle, which is the real-time tilt angle of the screw head plane. θ=arccos(|n screw ·n bench | / (|n screw |*|n bench |)); Where, |n screw ·n bench | represents the dot product of two unit normal vectors, i.e., the inner product; |n screw |*|n bench | represents the magnitude (length) of two vectors, both of which have a magnitude of 1. Therefore, the real-time included angle θ is simplified to θ = arccos(|n) in actual calculations. screw ·n bench|); Since it is already a unit vector, taking the absolute value is to ensure that the calculated real-time included angle θ is an acute angle; Finally, obtain the timestamp t of each frame in the temporally sequenced 3D point cloud sequence. i Corresponding height value H i As a set of data pairs (t) i ,θ i ), and put all data pairs (t) i ,θ i Plot and connect the coordinates to generate the screw tilt angle variation curve.

[0025] When acquiring the screw head plane, for each frame of the 3D point cloud, the set of points belonging to the screw head is extracted from the 3D point cloud. Typically, points on the top surface of the head are selected for fitting to obtain a plane equation. The fitting and calculation method of this plane equation is similar to the calculation method of the plane equation of the preset workpiece reference plane. The final plane equation is A. s x+B s y+C s z+D=0; The screw head plane represents the instantaneous attitude of the top surface of the screw head in space, and is used to calculate the tilt state of the screw.

[0026] The real-time included angle is used to quantify the degree of inclination of the screw axis relative to the normal of the workpiece reference surface. It is the same data as the real-time tilt angle mentioned below. It is a direct indicator for detecting tilted assembly defects such as misalignment of the screw in the hole, misalignment of the bit and the screw groove, and thread crossing. The screw tilt angle variation curve is a two-dimensional curve plotted with time as the horizontal axis and real-time angle as the vertical axis. It includes all time points from the start to the end of fastening (or the current moment) and their corresponding real-time tilt angle sequences, providing visualization of the stability changes of the screw's posture during fastening. A sudden increase or continuous growth trend of the real-time tilt angle is a clear signal of the occurrence of tilt defects.

[0027] Step S202: Calculate the smoothness index of the screw sinking trajectory curve and the rate of change index of the screw tilt angle change curve; The smoothness index is calculated by taking a continuous time interval of real-time height data sequence H1, H2, ..., H of the sinking trajectory. n Calculate the first-order difference sequence D of the height difference between adjacent points in the real-time height data sequence. i D i =H i+1 -H iThe standard deviation or mean absolute deviation of the first-order difference sequence is calculated. The smaller the standard deviation or deviation, the more uniform the real-time height change and the higher the smoothness index. Conversely, the larger the standard deviation or deviation, the worse the smoothness, indicating that the sinking process is violently jittery. The smoothness index is mainly used to quantify the smoothness of the screw sinking trajectory curve, that is, whether the sinking movement is stable and without jitter. It is used to identify the non-smooth sinking process caused by foreign objects in the thread, uneven material, misalignment of the hole, etc. This process may not show obvious abnormalities on the screw sinking trajectory curve, but can be warned in advance on the visual sinking trajectory. The rate of change index is calculated by taking a continuous time period of real-time tilt angle data sequence θ1, θ2, ..., θ n and their corresponding timestamps t1, t2, ..., t n Calculate the rate of change of angle between adjacent points in the real-time tilt angle data sequence, R. i R i =(θ i+1 -θ i ) / (t i+1 -t i ), i.e., instantaneous angular velocity; calculate the maximum absolute value of the rate of change sequence or the integral within a certain time window (i.e., the total change in angle). The larger the value of this index, the more drastic the change in tilt angle per unit time, and the more unstable the attitude. The rate of change index is mainly used to quantify the magnitude of the angular velocity of the screw tilt angle change curve, i.e. the degree of tilt change. It is used to capture sudden changes in screw attitude, such as the instantaneous tilt angle jump caused by the bit forcibly twisting after the screw suddenly jams, or the trend of accelerated increase in tilt angle caused by the continuous decrease in centering ability due to bit wear.

[0028] Step S203: The real-time height, real-time angle, smoothness index, and rate of change index are used as visual feature data and arranged in chronological order to form a dynamic visual feature sequence. The dynamic visual feature sequence extracts and compresses the original, high-dimensional, and redundant temporal 3D point cloud sequence into a set of low-dimensional key feature data streams with clear physical meaning and diagnostic value, arranged in chronological order. The real-time height reflects the axial position of the screw, the real-time angle reflects the tilt of the screw, the smoothness index reflects the stability of the axial movement, and the rate of change index reflects the severity of the screw tilt attitude change. Step S300: The dynamic visual feature sequence is spatiotemporally aligned and feature fused with the real-time torque and angle data stream to generate multimodal fusion features for the locking process; Step S301: Based on a unified time reference, a synchronization timestamp is applied to each visual feature data in the dynamic visual feature sequence and the torque angle data acquired at each sampling point in the real-time torque angle data stream. The unified time reference is set based on the central clock source of the entire locking and fastening control system. It usually uses a high-precision, high-stability crystal oscillator as the master clock and is deployed in the main control industrial computer or PLC of the system. The master clock is synchronized to all data acquisition units, including the controller of the 3D contour sensor and the data acquisition card of the locking tool, through hardware trigger signals (such as the IO signal at the start of the locking process) or network time protocols (such as PTP). This ensures that all data generated from the source is marked with a timestamp based on the same clock. The accuracy of the timestamp is usually in the range of milliseconds to microseconds, depending on the sensor sampling rate and the real-time requirements of the control. For example, if the visual acquisition frequency is 100Hz, the timestamp resolution is at least 10ms. The torque and angle data in the real-time torque and angle data stream refers to a set of related measurement values ​​acquired by the sensors of the clamping tool at each sampling moment. These include torque values ​​characterizing the instantaneous rotational torque output by the spindle, and angle values ​​characterizing the absolute or incremental angle of the spindle's cumulative rotation since the start of clamping or since the previous sampling point. The torque value is in Newton-meter or centi-Newton-meter, and the angle value is in degree or radian. In some more precise calculations, the rotational speed is also collected or derived simultaneously, in RPM. These data together constitute the core parameter set describing the mechanical and kinematic state of the clamping process. Step S302: Pair and associate visual feature data with the same timestamp with torque angle data to form a set of multidimensional feature data. The multidimensional feature data includes at least torque value, rotation angle, real-time height of screw head, and real-time tilt angle of screw head. The pairing and association of visual feature data and torque angle data is a precise matching process based on a unified timestamp, as follows: First, acquire the dynamic visual feature sequence and real-time torque angle data stream, both of which have been time-stamped. Each entry in the dynamic visual feature sequence contains a timestamp T. v And the corresponding visual feature vector, each entry in the real-time torque angle data stream contains a timestamp T. t And the corresponding torque and angle values; Then, a time alignment lookup is performed. Since the sampling frequencies of the dynamic visual feature sequence and the real-time torque angle data stream may be different, a timestamp T is assigned to each visual feature data. v (i) The timestamp T of each torque angle data t (j) Find the timestamp T in the real-time torque angle data stream. t (j) Closest to T v (i) Torque angle data entries; usually, the nearest neighbor matching principle is adopted, that is, to find the values ​​that satisfy |T v (i)-T t (j) | Minimum stamp T t (j) and its corresponding torque and angle values; Finally, pairing and combining are performed, combining the found visual feature vectors with the matched torque and angle values ​​to form a vector with a common timestamp (or a visual timestamp T). v (i) is the basis) a unified feature data entry, traversing all visual feature data points to complete the pairing of the whole process, forming a data sequence that is fully synchronized and fused on the time axis, that is, multidimensional feature data.

[0029] Step S303 involves combining the multidimensional feature data arranged in chronological order to generate multimodal fusion features of the fastening process. The multimodal fusion features are a multidimensional data matrix or vector sequence tightly arranged in chronological order, including timestamps, synchronously paired torque values ​​at those timestamps, rotation angles, real-time screw head height, and real-time screw head tilt angle. It also typically includes screw sinking trajectory smoothness indicators and screw tilt angle change rate indicators derived from the visual feature sequence and aligned with the time axis. The multimodal fusion features create a unified data view, combining torque signals representing mechanical force, angle signals representing motion quantity, and visual signals representing three-dimensional spatial attitude and motion smoothness in the time dimension. This allows for the analysis of cross-modal correlations and causal relationships, such as whether minute fluctuations in torque occur synchronously with height sinking pauses, or whether specific angle positions correspond to abrupt changes in tilt angles. This enables the diagnosis of complex fault modes that cannot be revealed by single-modal data.

[0030] Step S400: Based on multimodal fusion features, real-time analysis is performed using preset association rules and diagnostic models. If a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated. The preset association rules are a set of IF-THEN type logical judgment conditions summarized from process expert knowledge and historical fault data. They are used to automatically identify specific cross-modal association patterns that indicate abnormal operating conditions in the multimodal fusion feature data stream, including specific rules for triggering association patterns. The preset association rules are mainly obtained by comparing and analyzing a large amount of multimodal fusion feature data of qualified and unqualified fastening processes, and summarizing the characteristics or association features of various typical anomalies (such as stripping, tilting, and thread foreign objects) in multiple dimensions such as torque, angle, visual height, and visual tilt angle. For example, "slow torque rise and insufficient sinking" is associated with stripping risk, and "large torque fluctuation and unstable sinking" is associated with thread foreign objects.

[0031] In this embodiment, the diagnostic model is a lightweight real-time analysis module based on preset association rules. Its internal architecture mainly includes a feature input interface for receiving real-time multimodal fusion features, a rule base for storing preset association rules, an inference engine for sequentially or in parallel evaluating whether the current data triggers a rule in the rule base, and a diagnostic result generator for generating corresponding anomaly type codes and descriptions based on the triggered rules. By replacing human experience with the diagnostic model, the abnormal patterns are automated, standardized, and rapidly identified. The diagnostic model uses the latest multimodal fusion feature data in real time, and based on the inference engine, it sequentially calls each association rule in the rule base, comparing the conditions in the association rules with the corresponding indicators calculated from the real-time data. If all conditions of a rule are met, the abnormal pattern corresponding to that rule is identified, and the diagnostic result generator then outputs the diagnostic result bound to that pattern and triggers the corresponding hierarchical control logic.

[0032] Step S401: Real-time monitoring of multimodal fusion features to determine whether the multi-dimensional feature data exceeds the preset safety process window. If any multi-dimensional feature data exceeds the window, a first-level anomaly identifier is generated. The safety process window is a set of permissible fluctuation ranges or threshold intervals for each key monitoring parameter during the fastening process. It is a multi-dimensional set of parameter safety boundaries. The safety process window includes the upper and lower limits of the real-time height of the screw head, the maximum permissible value of the real-time tilt angle of the screw head, the upper and lower limits of the torque value, the range of the rotation angle, the torque rise slope, the sinking smoothness, and other derived indicators' threshold values. The setting of the safety process window is mainly based on two sources. First, product design and process specifications. For example, the nominal embedment depth of the screw determines the real-time height window, and the product structural strength requirements determine the maximum tilt angle. Second, historical qualified process data statistics. By statistically analyzing a large number of successful fastening events, the distribution range of each parameter under normal conditions is determined, and a certain confidence interval of this distribution (such as ±3σ) is taken as the dynamic process window.

[0033] The first-level anomaly identifier is a basic and rapid fault sign. When real-time monitoring detects that any dimension of the multimodal fusion features (such as the real-time instantaneous torque value or the real-time tilt angle of the screw) suddenly exceeds its preset safety process window, this identifier will be generated immediately. The first-level anomaly identifier is usually a simple Boolean value or a specific status code, which does not involve complex pattern analysis, making it easy to achieve the fastest risk interception. It is used to trigger emergency control commands that require immediate response (such as emergency stop) and is the first line of defense to ensure the safety of equipment and products.

[0034] Step S402: Identify cross-modal association patterns in the multimodal fusion features, including a first association pattern, a second association pattern, and a third association pattern; If the upward slope of the torque curve extracted from the real-time torque angle data stream is lower than the first preset threshold, and the height change of the screw sink trajectory curve is lower than the fourth preset threshold near the angle corresponding to the torque peak, it is judged as the first association mode, which represents a high risk of stripping or thread stripping. The screw rotates but fails to effectively tighten itself, resulting in a slow torque increase and the screw cannot be firmly seated. If the standard deviation of the torque curve is higher than the fifth preset threshold and the smoothness index of the screw sinking trajectory curve is lower than the second preset threshold, it is judged as the second correlation mode, which means that there are foreign objects (such as debris, colloids) or local damage to the thread in the thread. The foreign objects cause the resistance to fluctuate, which is visually manifested as the jamming of the sinking process. If the rate of change of the real-time tilt angle of the screw head exceeds the third preset threshold and is judged to be the third associated mode, it means that the screw is seriously misaligned when it enters the hole or the bit suddenly slips out of the screw slot, causing a sharp tilt. When the first or second association pattern is identified, it is determined that a specific abnormal condition has occurred in the locking process, and a diagnostic result containing the abnormality type code is generated, with the abnormality type code corresponding to the identified association pattern.

[0035] The upward slope of the torque curve refers to the rate at which the torque value increases with the increase of the screw rotation angle. It is used to characterize the efficiency of thread engagement and preload establishment, and to identify slow torque establishment caused by poor thread fit (such as stripping or thread stripping). By identifying the linear upward segment of the torque value sequence and angle value sequence in the real-time torque and angle data stream, usually from the 70%-80% range of the contact torque to the target torque, a straight line is fitted. The slope k of this straight line, in units of N·m / ° or N·m / rad, is the upward slope of the torque curve.

[0036] The angle near the peak torque is an angle range, which refers to the range from the moment before the torque value reaches its peak (maximum value) (e.g., peak angle minus Δθ1) to the moment after (peak angle plus Δθ2). Δθ1 and Δθ2 are preset small angle values ​​(e.g., 5°-15°). This position is the key stage where the screw is basically tightened and the mechanical state changes. The change in height is the decrease in the real-time height of the screw head within a specific angular range near the peak torque angle, i.e., the increase in the depth of indentation in this final stage. This is determined by setting the starting angle A within this angular range. start and ending angle A end Find the angle value closest to A from the multimodal fusion features. start and A endTwo data points were obtained, and their corresponding real-time height values ​​H were read. start and H end The difference between the two is the change in height, ΔH, where ΔH = H. start -H end ; The standard deviation of the torque curve is used to quantify the degree of fluctuation of the torque value around its mean, and is used to identify whether there are periodic jamming, slippage, or intermittent resistance changes due to foreign objects in the thread during the fastening process; by selecting a representative torque value sequence T1, T2, ..., T n Calculate the arithmetic mean μ of the torque value sequence, μ = (T1, T2, ..., T...). n ) / n; where, T i This represents the i-th torque sample value in the torque value sequence, where i = 1, 2, ..., n; n is the total number of torque value sequences involved in the calculation, i.e., the length of the torque value sequence; the standard deviation σ is calculated based on the average value. ; Among them, T i -μ represents the deviation of the i-th torque sample value from the average value μ, indicating the degree to which a single data point deviates from the average level; n is the total number of torque value sequences involved in the calculation.

[0037] The real-time tilt angle change rate of the screw head is obtained through the change rate index in step S202, that is, by calculating the first difference of the tilt angle sequence with respect to time (instantaneous angular velocity). In the preset association rule judgment, it is usually checked whether the real-time tilt angle change rate of the screw head exceeds the set threshold at a certain time point or in a short period of time. An exception type code is a predefined, unique identifier (e.g., E01, E02, W01, etc.) used to accurately locate the diagnosed exception pattern and associate specific fault phenomena such as stripped threads, excessive tilting, foreign objects in the threads with subsequent control strategies, maintenance recommendations, and data records. The preset thresholds in the correlation pattern judgment are empirical values ​​derived from extensive process experiments and statistical analysis of qualified samples of specific screws, workpieces, and fastening tools. The first preset threshold is a lower limit for the torque rise slope; below this value, the torque is considered to be built up too slowly. The specific value (e.g., 0.05 N·m / °) depends on the screw specifications and materials. The second preset threshold is a lower limit for the sinking smoothness index; below this value, the sinking process is considered to have excessive jitter. The value is based on the minimum allowable value of the smoothness of the qualified sinking trajectory. The third preset threshold is an upper limit for the tilt angle change rate; exceeding this value indicates a sudden change in tilt. The value is based on the maximum allowable instantaneous angular velocity. The fourth preset threshold is a lower limit for the height change; below this value, the screw is considered to have almost no sinking in the final tightening stage. The value is based on the minimum necessary compression of the screw head height in the final stage. The fifth preset threshold is an upper limit for the torque standard deviation; exceeding this value indicates abnormal torque fluctuation. The value is based on the statistical upper limit of the fluctuation of the qualified torque curve.

[0038] Step S403: If a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated, including: The preset exception patterns include the first association pattern, the second association pattern, and the third association pattern; If a first-level anomaly identifier is generated, or a first association pattern is identified, then a first-class control instruction is generated. If the second association pattern is identified, a second type of control instruction is generated; If the pattern is identified as the third association pattern, then a third type of control instruction is generated; If the multimodal fusion features of N consecutive locking cycles show that the real-time tilt angle change rate of the screw head exhibits a monotonically increasing trend, then a fourth type of control command is generated.

[0039] Among them, the first type of control command is the highest level of emergency response command, which includes action command, retraction command and alarm command. Action command includes immediately stopping the rotation and downward drive of the electric screwdriver. Retraction command includes controlling the locking tool to rise along the Z-axis to the preset initial waiting position. Alarm command includes triggering an audible and visual alarm of a specific frequency and mode. The second type of control command is the warning and recording command, which includes display commands and log commands. The display command sends a specific warning text, code and related key data snapshots to the human-machine interface (HMI), such as an abnormal torque curve segment. The log command marks the current locking event as a "warning" and records the multimodal fusion features and diagnostic results completely to the system maintenance log database. The third type of control command is the online adaptive correction command, which includes correction motion parameters and execution logic. The correction motion parameters include the calculated correction direction and correction angle. The correction direction is a vector direction in a two-dimensional plane, which is derived by back-calculating from the direction with the largest tilt angle. The correction angle is a preset, controlled small angle value, such as 3°. The execution logic instructs the control system to drive the end of the locking tool to perform a two-dimensional circular arc compensation movement centered on the current point, with the correction angle as the amplitude, and along the correction direction while maintaining the current downward pressure. The fourth type of control instruction is predictive maintenance warning instructions, which include trend warning instructions and maintenance suggestion instructions. Trend warning instructions send a prompt message to the HMI, indicating that there is potential wear (such as bit wear) at a specific workstation or bit that will cause the tilt angle to continue to increase. Maintenance suggestion instructions generate a record with a "predictive maintenance" label in the maintenance log, suggesting that the relevant parts be checked or replaced.

[0040] Step S500: According to the hierarchical control instructions, execute the corresponding real-time control actions, including generating early warnings, adjusting the process parameters of the locking tool, executing adaptive correction motion, and stopping the locking process.

[0041] If a first-class control command is received, the pressing and rotating actions of the locking tool will be stopped immediately, and it will be raised to the initial waiting position before the locking process begins, while triggering an audible and visual alarm. If a third type of control instruction is received, while the fastening tool is under downward pressure, the end of the fastening tool is controlled to perform an arc compensation movement with an amplitude controlled within the preset maximum correction angle, according to the correction direction and angle in the third type of control instruction. The real-time tilt angle of the screw head is monitored in real time. If the real-time tilt angle of the screw head returns to within the safe process window, the correction is judged to be successful and the fastening process continues; otherwise, the first type of control instruction is executed. If a second or fourth type of control command is received, a warning message will be displayed on the human-machine interface and recorded in the maintenance log.

[0042] The initial waiting position is a fixed and safe preparation point in three-dimensional space for the end of the screw fastening tool (screw tip) before the fastening process begins. The initial waiting position is set by a safety height and an alignment preparation position. The safety height is higher than all possible worktables, fixtures and workpieces to prevent collisions. The alignment preparation position is usually located directly above the screw feeder outlet or a fixed pick-up point to facilitate the next screw pick-up.

[0043] The setting of the correction direction and angle is based on the normal vector analysis of the current screw head tilt plane. The current screw tilt angle and its maximum tilt direction are obtained from the multimodal fusion features. For example, by analyzing the projection direction of the head plane normal vector on the XY plane, the correction direction is set to the opposite direction of the maximum tilt direction. The correction angle is set to an empirical, small fixed value, such as 2° or -5°, as the magnitude of a single trial compensation. At the same time, the system will preset a maximum correction angle, such as 10°, as a safety limit to prevent overcompensation.

[0044] Arc compensation motion refers to the movement of the end of the locking tool along a small arc trajectory in a plane perpendicular to the axis, oscillating in the direction of correction, with the current axis of the tool as the center and the correction angle as the central angle, without releasing the axial pressure on the screw head. The execution of arc compensation motion includes the following process: First, upon receiving the third type of control command, the correction direction vector V and correction angle α are parsed out. The motion controller plans a circular interpolation path in the XY plane. The starting point of the circular interpolation path is the current position P0 of the tool, the ending point is the position P1 after angle α compensation, and the center of the circle is the center of the tool axis and an imaginary circle perpendicular to V and passing through P0. Then, the control system drives the electric screwdriver spindle (usually through a two-dimensional miniature cross slide or compliant mechanism) to precisely execute the arc motion while maintaining constant pressure on the Z-axis; after the motion is completed, the screw head point cloud is immediately collected again to calculate the new real-time tilt angle. Finally, determine whether the new real-time tilt angle has returned to within the safe process window: if yes, the correction is considered successful and the subsequent locking process continues; if no, and the cumulative correction angle has not exceeded the maximum correction angle, the correction can be attempted again; if the maximum correction angle has been reached or the tilt has worsened, immediately switch to executing the first type of control command.

[0045] Step S600: After the screw head and workpiece plane are fastened, the final three-dimensional point cloud is collected, the final height and final flatness of the screw head are calculated, and the panoramic data package of the screw fastening event is generated by combining the multimodal fusion features and diagnostic results generated during the fastening process; the panoramic data package is stored in the process knowledge base.

[0046] When the fastening process is completed normally (reaching the target torque / angle) or stopped by the control command, the control system issues a command to fully lift the fastening tool away from the screw head area, and controls the three-dimensional contour sensor to perform a final static, high-precision all-round scan of the station where the screw fastening is completed (or use the data of the previously synchronized sensor at the final moment) to obtain and reconstruct the complete three-dimensional point cloud data of the screw head and the surrounding workpiece surface at this moment, which is the final three-dimensional point cloud. The final height is calculated using the same method as in step S201. The center point of the screw head is calculated based on the final three-dimensional point cloud, and the vertical distance from it to the preset workpiece reference surface is calculated. This distance is the final height (embedding depth) of the screw head. When calculating the final flatness, the point set of the top surface of the screw head is segmented from the final 3D point cloud, and the final head plane is obtained by fitting. The distance from all points in the point set to the fitted plane is calculated, and the sum of the absolute values ​​of the maximum positive and the maximum negative values ​​of these distances, or their standard deviation, is taken as the flatness error value of the top surface of the head, which is used to determine whether the head is deformed due to over-tightening or material problems. The panoramic data package is a structured data file that includes process data, diagnostic records, final result data, and contextual information. The process data contains a complete, time-aligned multimodal fusion feature sequence. The diagnostic records contain all anomaly identifiers generated during real-time analysis, associated pattern recognition results, and control command history. The final result data contains the final 3D point cloud, calculated final height, and flatness. The contextual information includes screw type, workpiece ID, fastening tool ID, timestamp, and total time. The panoramic data package establishes a complete electronic record for each screw fastening event for quality traceability, root cause analysis, and subsequent process optimization. The process knowledge base is a dedicated database system used to systematically store, manage, and analyze a comprehensive data package of all screw fastening events. The process knowledge base includes a raw data warehouse, a feature parameter library, a rule and model library, and a process parameter library. The raw data warehouse contains comprehensive data packages indexed by product model, time, etc. The feature parameter library contains key feature statistics extracted from qualified and unqualified events. The rule and model library stores and version-manages currently used association rules, diagnostic models, and threshold parameters. The process parameter library stores baseline fastening process parameters for each product model, such as target torque, speed, and pressing speed. The process knowledge base contains data from the assembly process, supporting historical data retrieval, statistical process control analysis, and providing a data foundation for self-learning optimization in step S700.

[0047] Step S700: Perform cluster analysis on the panoramic data package of M qualified locking events of the same product model stored in the process knowledge base, where M is an integer, to mine the optimal range of parameters in the multimodal fusion features; update the benchmark locking process parameters of the product model according to the optimal range of parameters.

[0048] First, the locking time is determined. A qualified locking event must meet two conditions simultaneously: result acceptance and process control. Result acceptance means that the final height and final flatness are within the tolerance range defined in the drawings or specifications. Process control means that the corresponding panoramic data package records show that the multimodal fusion features did not trigger any first-level anomaly indicators during the entire locking process, and the diagnostic model did not identify any preset association patterns. Secondly, cluster analysis is performed on the panoramic data package. M qualified locking events for the same product model are extracted from the process knowledge base. From the multimodal fusion feature sequence of each event, a set of steady-state features that can stably characterize process quality are extracted, such as the mean of torque rise slope, the mean of torque peak value, the final height value, the mean of overall smoothness of the sinking trajectory, and the maximum tilt angle, forming a feature vector. Unsupervised clustering algorithms (such as K-means or DBSCAN) are used to cluster these M feature vectors. The goal is to find a core cluster containing the largest number of qualified samples. Samples within this cluster are considered the most stable and optimal samples. The cluster analysis process and algorithm can be directly calculated and analyzed using any of the following computer software: SPSS data analysis software, R language, or Python's sklearn library. Then, the statistical distribution of each key characteristic parameter of all samples in the core cluster is calculated, such as the torque rise slope and final height. An interval is defined for each parameter, which covers the parameter values ​​of the vast majority of samples (e.g., 95%) in the cluster. This set of intervals is the parameter optimal interval. The parameter optimal interval refers to the distribution range of each characteristic parameter value of all samples in the core cluster. For example, the minimum and maximum values ​​or the mean ± k times the standard deviation of the torque rise slope of all samples in the core cluster are taken. The parameter optimal interval defines the most stable and reliable process parameter operating range for this product model, which is more stringent and ideal than the safety process window set based on all qualified samples (which may include some marginal qualified samples). Finally, the basic locking process parameters are updated based on the optimal parameter range. The existing benchmark locking process parameters (such as the target torque of 300 cN·m) are compared with the optimal parameter range obtained from the core cluster analysis (such as the optimal range for peak torque of [295, 305] cN·m). If the benchmark value of the existing benchmark locking process parameter is not within the optimal range or deviates from the center of the range, it is adjusted to be near the center value of the optimal range (for example, the target torque is adjusted from 300 to 300, or if it was originally 310, it is adjusted to 300). For process parameters such as rotational speed, the speed value corresponding to the sample with the smoothest dip in the core cluster can be analyzed and set as the new benchmark. The updated benchmark parameters are verified in small-batch production. After confirming their stability and yield improvement effect, they are officially released and updated to the process parameter library in the process knowledge base for all production line workstations of this product model to call. This realizes the closed loop of benchmark locking process parameters from experience setting to data-driven optimization.

[0049] Among them, the reference clamping process parameters are the set of direct control parameters that drive the clamping tool to perform actions. The reference clamping process parameters include target torque value, target angle value, rotational speed (RPM), pressing speed, contact detection threshold, etc. The initial reference clamping process parameters are based on product design requirements and the initial process debugging settings of engineers.

[0050] like Figure 4 As shown, a machine vision-based multi-dimensional inspection and control system for screw assembly quality includes: The multi-dimensional visual perception module simultaneously acquires a time-series 3D point cloud sequence of the screw head and surrounding area during the screw fastening process; A high-precision process sensing module is used to synchronously acquire real-time torque and angle data streams of the fastening tool; The real-time quality analysis and decision control module includes a data fusion unit, an online intelligent anomaly detection unit, and an instruction generation unit; The data fusion unit is used to extract dynamic visual feature sequences representing screw posture and sinking process from a time-series 3D point cloud sequence in real time; and to perform spatiotemporal alignment and feature fusion with real-time torque and angle data streams to generate multimodal fusion features of the fastening process. The online intelligent anomaly detection unit is used to identify preset abnormal patterns in real time based on multimodal fusion features, through preset association rules and diagnostic models. The instruction generation unit is used to generate corresponding diagnostic results and hierarchical control instructions based on the identification of preset abnormal patterns. The motion control and execution module is used to execute corresponding real-time control actions according to the hierarchical control instructions. The real-time control actions include generating early warnings, adjusting the process parameters of the locking tool, executing adaptive correction motion, and stopping the locking process. The process data management and optimization module includes a process knowledge base and an optimization analysis unit. The process knowledge base is used to store panoramic data packets with locking time. The optimization analysis unit is connected to the process knowledge base and is used to analyze historical panoramic data packets and perform process parameter optimization.

[0051] In this embodiment, by deploying a three-dimensional contour sensor that moves synchronously with the electric screwdriver spindle, the entire screw fastening process is monitored in a time-series three-dimensional visual manner. This allows for continuous acquisition and reconstruction of the three-dimensional morphological changes of the screw head and surrounding area in the form of a point cloud sequence. Furthermore, it enables the real-time extraction of two key dynamic visual features: the screw sinking trajectory curve and the screw head tilt angle change curve. Compared to existing static single-point measurement or 2D image analysis, this provides a three-dimensional visual perspective of the dynamic physical process of fastening. It not only provides the final state of the screw but also fully records the path and posture evolution process leading to that final state. Before torque anomalies occur, early fault signs such as misaligned entry holes and thread obstruction can be identified by analyzing abnormal smoothness of the sinking trajectory or abnormal increase in the tilt angle. This shifts the quality inspection point from result acceptance to process monitoring, advancing the inspection time, achieving dynamic three-dimensional process inspection, reducing the scrap rate, thereby improving the screw assembly qualification rate and reducing assembly costs.

[0052] This invention constructs a multimodal fusion feature data layer and hierarchical intelligent diagnosis, fusing temporal three-dimensional visual features with high-synchronization-precision torque angle curves, and running preset cross-modal association rules on this fused data. This changes the single-parameter threshold judgment mode and can accurately diagnose complex root causes such as "gradual increase in tilt caused by bit wear" and "fluctuating jamming caused by foreign objects in the thread". Based on the accurate diagnosis, hierarchical control commands are generated, such as triggering millimeter-precision adaptive online correction motion to attempt to repair slight tilt, or decisively stopping in an emergency to prevent serious defects. This forms a continuous decision chain from problem occurrence to cause analysis to solution, giving the assembly system the intelligent ability to autonomously cope with complex working conditions and actively maintain the process window without human intervention, thus improving the system's robustness and process consistency.

[0053] This application also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, can perform the machine vision-based multi-dimensional detection and control method and system for screw assembly quality as described above.

[0054] The methods and systems according to the embodiments of this application can also be implemented using the architecture of the electronic device shown in this application. The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output, a hard disk, etc. The storage device in the electronic device, such as a ROM or hard disk, may store the machine vision-based multi-dimensional detection and control method and system for screw assembly quality provided in this application. Furthermore, the electronic device may also include a user interface. Of course, the architecture shown in this application is merely exemplary; when implementing different devices, one or more components in the electronic device shown in this application may be omitted according to actual needs.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-dimensional detection and control method for screw assembly quality based on machine vision, characterized in that, The method includes: During the screw fastening process, a time-series 3D point cloud sequence of the screw head and surrounding area is simultaneously acquired, along with real-time torque and angle data streams of the fastening tool. Dynamic visual feature sequences representing screw posture and sinking process are extracted in real time from a temporal 3D point cloud sequence. The dynamic visual feature sequence is spatiotemporally aligned and feature fused with the real-time torque and angle data stream to generate multimodal fusion features for the locking process; Based on the multimodal fusion features, real-time analysis is performed using preset association rules and diagnostic models. If a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated. According to the hierarchical control instructions, the corresponding real-time control actions are executed, including generating early warnings, adjusting the process parameters of the locking tool, executing adaptive correction motion, and stopping the locking process.

2. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 1, characterized in that, The synchronously acquired temporal 3D point cloud sequence of the screw head and surrounding area includes: Deploy a three-dimensional profile sensor that maintains a fixed offset relationship with the spindle of the fastening tool, and ensure that the scanning axis of the three-dimensional profile sensor is synchronously tracked with the screw head; Throughout the entire process of pressing down and rotating the fastening tool, the three-dimensional contour sensor is controlled to continuously scan the screw head and its surrounding area in contact with the workpiece. The data obtained from continuous scanning is reconstructed into a time-series 3D point cloud sequence in chronological order of acquisition time. Each frame of the 3D point cloud in the time-series 3D point cloud sequence carries timestamp information.

3. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 2, characterized in that, The process of extracting dynamic visual feature sequences representing the screw's posture and sinking process in real time from a temporally sequenced 3D point cloud sequence includes: Acquire each frame of the 3D point cloud in the time-series 3D point cloud sequence, calculate the real-time height of the center point of the screw head relative to the preset workpiece reference surface, and form the screw sink trajectory curve. Acquire each frame of the 3D point cloud in the time-series 3D point cloud sequence, calculate the real-time angle between the screw head plane and the preset workpiece reference plane, and form the screw tilt angle change curve. Calculate the smoothness index of the screw sinking trajectory curve and the rate of change index of the screw tilt angle change curve; The real-time height, real-time angle, smoothness index, and rate of change index are used as visual feature data and arranged in chronological order to form a dynamic visual feature sequence.

4. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 3, characterized in that, The step of spatiotemporally aligning and fusing the dynamic visual feature sequence with the real-time torque and angle data stream to generate multimodal fusion features for the locking process includes: Based on a unified time base, a synchronization timestamp is applied to each visual feature data in the dynamic visual feature sequence and the torque angle data acquired at each sampling point in the real-time torque angle data stream. Visual feature data with the same timestamp is paired and associated with torque angle data to form a set of multidimensional feature data, which includes torque value, rotation angle, real-time height of screw head, and real-time tilt angle of screw head. Multidimensional feature data arranged in chronological order are combined to generate multimodal fusion features for the locking process.

5. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 4, characterized in that, The real-time analysis based on the multimodal fusion features, using preset association rules and diagnostic models, includes: The multimodal fusion features are monitored in real time to determine whether the multidimensional feature data exceeds the preset safety process window. If any multidimensional feature data exceeds the window, a first-level anomaly identifier is generated. Identify cross-modal association patterns in multimodal fusion features, wherein the association patterns include a first association pattern, a second association pattern, and a third association pattern; If the upward slope of the torque curve extracted from the real-time torque angle data stream is lower than the first preset threshold, and the change in height of the screw sinking trajectory curve near the angle corresponding to the torque peak is lower than the fourth preset threshold, then it is judged as the first association mode. If the standard deviation of the torque curve is higher than the fifth preset threshold and the smoothness index of the screw sinking trajectory curve is lower than the second preset threshold, it is judged as the second association mode. If the rate of change of the real-time tilt angle of the screw head exceeds the third preset threshold, it is determined to be the third association mode; When the first or second association pattern is identified, it is determined that a specific abnormal condition has occurred in the locking process, and a diagnostic result containing the abnormality type code is generated, with the abnormality type code corresponding to the identified association pattern.

6. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 5, characterized in that, If a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated, including: The preset anomaly modes include a first association mode, a second association mode, and a third association mode; If a first-level anomaly identifier is generated, or a first association pattern is identified, then a first-class control instruction is generated. If the second association pattern is identified, a second type of control instruction is generated; If the pattern is identified as the third association pattern, then a third type of control instruction is generated; If the multimodal fusion features of N consecutive locking cycles show that the real-time tilt angle change rate of the screw head exhibits a monotonically increasing trend, then a fourth type of control command is generated.

7. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 6, characterized in that, The step of executing corresponding real-time control actions according to the hierarchical control instructions includes: If a first-class control command is received, the pressing and rotating actions of the locking tool will be stopped immediately, and it will be raised to the initial waiting position before the locking process begins, while triggering an audible and visual alarm. If a third type of control instruction is received, the screw head tilt angle is corrected in real time according to the third type of control instruction while the fastening tool is under downward pressure. The change of the screw head tilt angle in real time is monitored. If the screw head tilt angle returns to within the safe process window, the correction is considered successful and the fastening process continues. Otherwise, the first type of control instruction is executed. If a second or fourth type of control command is received, a warning message will be displayed on the human-machine interface and recorded in the maintenance log.

8. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 7, characterized in that, The method further includes: After the screw is fastened, the final three-dimensional point cloud of the screw head and the workpiece plane is collected, the final height and final flatness of the screw head are calculated, and the panoramic data package of the screw fastening event is generated by combining the multimodal fusion features and diagnostic results generated during the fastening process. The panoramic data package is stored in the process knowledge base.

9. The multi-dimensional detection and control method for screw assembly quality based on machine vision according to claim 8, characterized in that, The method further includes: Cluster analysis is performed on the panoramic data package of M qualified locking events of the same product model stored in the process knowledge base to mine the optimal range of parameters in the multimodal fusion features; Update the baseline fastening process parameters for this product model based on the optimal range of the parameters.

10. A multi-dimensional detection and control system for screw assembly quality based on machine vision, characterized in that, The system includes: The multi-dimensional visual perception module simultaneously acquires a time-series 3D point cloud sequence of the screw head and surrounding area during the screw fastening process; A high-precision process sensing module is used to synchronously acquire real-time torque and angle data streams of the fastening tool; The real-time quality analysis and decision control module is used to extract dynamic visual feature sequences representing screw posture and sinking process from a time-series 3D point cloud sequence in real time; the dynamic visual feature sequence is spatiotemporally aligned and feature fused with real-time torque and angle data streams to generate multimodal fusion features of the fastening process; based on the multimodal fusion features, real-time analysis is performed through preset association rules and diagnostic models; if a preset abnormal pattern is identified, corresponding diagnostic results and hierarchical control instructions are generated. The motion control and execution module is used to execute corresponding real-time control actions according to the hierarchical control instructions. The real-time control actions include generating early warnings, adjusting the process parameters of the locking tool, executing adaptive correction motion, and stopping the locking process. The process data management and optimization module is used to store panoramic data packages of locking time and to optimize process parameters based on these panoramic data packages.

Images