Comprehensive determination method and system for pose and screw quality before and after industrial product fastening process

Abstract

The invention relates to the technical field of industrial robot visual control, and discloses a method and a system for comprehensively judging poses and screw quality before and after an industrial product fastening process. The problems that in the prior art, detection processes before and after the fastening process are split, pose judgment robustness is insufficient, abnormity processing lacks priority, small sample adaptability is poor, and quality tracing visualization is weak are solved. According to the method, when a product is in a procedure before fastening, fixture mark points and product screw holes are positioned through a template detection model, the reference geometrical relationship between the mark points and the screw holes is stored, overall offset measurement is constructed through line segment distance changes from the screw holes to the mark points during operation, and the product grabability is judged by adopting a double-threshold strategy; when the product is in a post-fastening process, generating a screw region cutting block for the target screw hole, and performing quality classification by using the screw classification model; and at the end of each period, writing the key data into an asynchronous governance queue for subsequent statistical analysis and incremental learning.

Description

Technical Field

[0001] This invention relates to the field of industrial robot vision control technology, specifically to a method and system for comprehensively judging the posture and screw quality before and after the fastening process of industrial products. Background Technology

[0002] In industrial assembly lines, fasteners such as screws are widely used for connecting structural components and installing functional components. To ensure assembly quality and production safety, it is usually necessary to confirm whether the product's position relative to the fixture meets the conditions for gripping and fastening before the fastening process. After the fastening process, it is further determined whether there are any problems such as missing screws, incorrect installation, floating screws, tilting screws, or damage.

[0003] Existing placement determination methods mostly rely on a single detection box or a small number of feature points, judging placement based on the coordinate deviation between the center of the detection box and a preset reference point. These methods have the following shortcomings:

[0004] (1) It is more sensitive to occlusion, changes in lighting and the presence of additional objects. When the product outline is complex or local features are weak, it is easy to misjudge or miss the judgment.

[0005] (2) There is a lack of explicit modeling of the overall geometric relationship between product holes and fixture holes, making it difficult to provide indicators with engineering significance such as "total point deviation" and "distribution of cross-hole deviation".

[0006] (3) Placement judgment and screw quality inspection are generally in different work stations and different systems. There is a lack of integrated modeling of the preceding and following processes, and it is impossible to output a comprehensive conclusion of "grabable + screw qualified" in a single cycle.

[0007] (4) The system generally lacks comprehensive pre-image judgment of the status of the robotic arm, screw machine, fixture and human-machine safety, and also lacks an abnormal priority short circuit mechanism, making it difficult to balance safety and efficiency under high-frequency conditions.

[0008] (5) Existing solutions rely heavily on large-scale labeled data and lack a mechanism for incremental benchmark updates and drift adaptation using online qualified samples, making it difficult to guarantee performance in small sample scenarios.

[0009] (6) The visualization capability of the test results is limited. The screw cutting block and the global status display are not consistent, which brings greater costs to quality traceability and long-term trend analysis.

[0010] Therefore, there is an urgent need for a structured, scalable, and stable comprehensive judgment method and system that can perform workstation imaging readiness judgment, pre-tightening pose and graspability judgment, and post-tightening screw quality judgment under a unified framework, and support multi-state output and long-term benchmark adaptive update. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a method and system for comprehensively judging the position and screw quality before and after the fastening process of industrial products, which solves the problems of fragmented detection process before and after the fastening process, insufficient robustness of position judgment, lack of priority in abnormality handling, poor adaptability to small samples and weak visualization of quality traceability in the prior art.

[0012] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0013] On the one hand, the present invention provides a method for comprehensively judging the position and screw quality before and after the fastening process of industrial products, including the following steps:

[0014] A. Acquire workstation images and equipment status signals within the preset imaging time window. If a serious abnormality is detected, output the abnormal status and terminate the detection of this cycle. When all ready conditions are met, proceed to the next step.

[0015] B. When the product is in the pre-fastening process, a lightweight target detection model is used to locate the fixture markers and the product screw holes. During the template creation stage, the template coordinates of the markers and screw holes and the template pixel distance between them are stored and written into the template JSON file. During the running stage, the distance change from each screw hole to each marker is calculated based on the running detection results and the template JSON file, and an overall offset metric is constructed accordingly to classify and determine the relative pose and gripping ability of the product in the fixture.

[0016] C. After the product has undergone the fastening process, a screw area cutting block is generated for each target screw hole. The cutting block is normalized and the presence or absence of screws and the fastening quality of each screw hole are classified using a screw classification model.

[0017] D. Based on the pre-tightening pose offset judgment result and the post-tightening screw classification result, a unified quality visual canvas is constructed. Each screw clipping block and its classification information are displayed in the canvas, and multi-dimensional judgment status statistics are output.

[0018] E. Write the original image, marker coordinates, screw hole coordinates, distance change, and screw classification results into the asynchronous management queue. Within a set period, statistically analyze the new qualified samples, update the placement benchmark, distance threshold, and screw appearance baseline, and distribute them in an online hot update manner.

[0019] Furthermore, in step A, the ready conditions include:

[0020] Check whether the end effector of the robotic arm is in a safe standby position or a safe position before grasping;

[0021] Check whether the screw-operating head has left the product surface and is at a safe height;

[0022] Check whether the fixture has completed the clamping or releasing action and reached a stable state;

[0023] Check whether there are any unauthorized personnel or extra products mixed in in the testing station area;

[0024] Check whether the output of the detection light source is stable within the preset working range;

[0025] In step A, if any of the above conditions are not met within the preset imaging time window, the status of this cycle is determined to be "none" or "no_grab", the detection of this cycle is terminated and the subsequent steps are not entered.

[0026] Furthermore, in step B, the step of using a lightweight target detection model to locate the fixture markers and product screw holes, and storing the template coordinates of the markers and screw holes and the template pixel distance between them in the template creation stage and writing them into the template JSON file, specifically includes:

[0027] Multiple representative images were acquired in calibration mode. A lightweight target detection model was used to obtain the template coordinate set of fixture marker points and screw holes. By averaging and filtering the multiple detection results, the template coordinates of the marker points and screw holes were obtained. The template distance from each screw hole to each marker point was calculated. The coordinates and template distances were written into a template JSON file in a structured form.

[0028] The process of calculating the distance change from each screw hole to each marker point based on the runtime detection results and the template JSON file during the runtime phase, and constructing an overall offset metric accordingly, specifically includes:

[0029] Within each production cycle, the lightweight target detection model is invoked on the current image to obtain the runtime marker coordinates and screw hole coordinates. The corresponding template distance is read from the template JSON file, and the difference between the runtime distance and the template distance is calculated to obtain the distance change from each screw hole to each marker and the overall offset metric.

[0030] Furthermore, the overall offset metric is a weighted sum of the changes in distance between each screw hole, and in step B, the grabbability is graded based on a preset first threshold and a second threshold:

[0031] When the overall deviation statistic is lower than the first threshold, the placement status is determined to be grabbable;

[0032] When the overall deviation statistic is between the first threshold and the second threshold, the placement status is determined to be grabbable but the deviation is too large.

[0033] When the overall deviation statistic is higher than the second threshold, the placement status is determined to be ungrabable;

[0034] The second threshold is greater than the first threshold and can be dynamically adjusted based on historical offset statistics and on-site process requirements.

[0035] Furthermore, in step C, the trimmed blocks are normalized, specifically including:

[0036] Perform geometric normalization, brightness and contrast normalization on the cropped blocks respectively, and then perform noise reduction or sharpening processing.

[0037] Furthermore, in step C, the output categories of the screw classification model include:

[0038] Screws are present and properly tightened;

[0039] The screws are present but have defects such as floating, tilting, biasing, or head damage.

[0040] No screws;

[0041] The screw area is severely obscured or contaminated, making reliable identification impossible.

[0042] Furthermore, in step D, a unified quality visualization canvas is constructed based on the pre-tightening pose offset determination result and the post-tightening screw classification result, specifically including:

[0043] Arrange all screw cutting blocks on the same canvas according to the pre-set screw hole number and physical space layout, so that the position index on the canvas corresponds one-to-one with the actual hole position.

[0044] Next to each screw cutting block, label the corresponding hole index, category label and confidence value, and mark the current product number, timestamp, overall placement status and comprehensive judgment result of screw quality on the edge of the canvas.

[0045] Furthermore, in step D, the multi-dimensional determination status statistics include at least one of the following statuses: "success", "fail", "no_grab", "none", and "skip_screw_det", and corresponding defect statistics; wherein,

[0046] "success" indicates that the screws are grippable and of generally acceptable quality.

[0047] "Fail" indicates that there is at least one improper placement or a defective screw.

[0048] "no_grab" indicates that the data cannot be grabbed due to excessive relative offset or failure to meet security conditions.

[0049] "none" indicates that no effective detection was performed or no effective images were collected in this cycle;

[0050] "skip_screw_det" indicates that screw detection is skipped due to an anomaly-priority short-circuit strategy or configuration settings, and only pose-related conclusions are output.

[0051] Furthermore, in step E, new qualified samples are statistically analyzed within a set period to update the placement benchmark, distance threshold, and screw appearance baseline. This specifically includes:

[0052] Qualified samples, confirmed manually or by rules, are selected from the asynchronous governance queue according to a preset cycle.

[0053] Based on the qualified samples, the average coordinates and covariance of each screw hole in the current time window and under different product models and batches are statistically analyzed, and the total cross-category geometric distance deviation at the hole position level and globally is calculated.

[0054] The template coordinates, offset threshold, and screw classification score weights are automatically updated based on statistical results.

[0055] On the other hand, the present invention also provides a comprehensive judgment system for the position and screw quality before and after the fastening process of industrial products, including:

[0056] The image acquisition unit, including an industrial camera and a light source, is used to acquire images of the workstation and provide stable lighting;

[0057] The status acquisition unit is used to acquire equipment status signals from the robotic arm, screw machine, fixture, and safety sensors.

[0058] The image processing and determination unit is used to execute the above-mentioned method for comprehensive determination of the pose and screw quality before and after the fastening process of industrial products, and to generate pose offset determination results, screw quality determination results and multi-state output.

[0059] Run the visualization unit to display the quality visualization canvas, multi-state output, defect list, and historical statistics;

[0060] The asynchronous governance and incremental update unit is used to perform offline governance, statistical analysis and model / benchmark updates on online detection results, and send the updated results to the image processing and judgment unit.

[0061] Furthermore, the image processing and determination unit includes:

[0062] The imaging readiness determination module is used to detect imaging safety and readiness conditions within a preset time window;

[0063] The marker point and screw hole detection module is used to identify the fixture marker point set and screw hole set using a lightweight target detection model;

[0064] The template management and distance calculation module is used to manage template JSON files, calculate template distance and runtime distance, and obtain the distance change.

[0065] The offset metric and dual-threshold grading module is used to construct an overall offset metric based on the distance change and to determine the crawlability grading based on dual thresholds.

[0066] The screw cutting and classification module is used to generate screw cutting blocks and perform quality classification using a screw classification model.

[0067] The visualization canvas generation module is used to construct a unified quality visualization canvas and overlay global judgment information;

[0068] The multi-state decision and result output module is used to fuse pose offset and screw classification results and output multi-state judgment and defect statistics information.

[0069] Furthermore, the asynchronous governance and incremental update unit includes:

[0070] The log management module is used to receive and store structured logs and image data generated by online detection;

[0071] The sample screening module is used to screen qualified samples and typical defective samples from the log;

[0072] The statistical analysis module is used to perform statistical analysis on screw hole coordinates, distance changes, and screw classification results, and generate drift and trend indicators.

[0073] The model / benchmark update module is used to update the benchmark position and distance thresholds, adjust the scoring weights, and incrementally train the screw classification model based on statistical results, and then send the updated parameters or model to the online system.

[0074] The beneficial effects of this invention are:

[0075] (1) Comprehensive judgment and multi-state output:

[0076] The system enables workstation readiness determination, pre-tightening posture determination, and post-tightening screw quality determination within a single system, and outputs results in a multi-state manner, distinguishing between various situations such as "unable to image", "unable to grasp", and "screw abnormality", significantly improving production line scheduling and abnormal response capabilities.

[0077] (2) Fine pose quantization based on marker points and screw holes:

[0078] The position of fixture markers and product screw holes is extracted by a lightweight target detection model. The overall offset metric is constructed by using the baseline line segment distance in the template JSON and the change of line segment distance during runtime. This approach balances computational efficiency with the ability to accurately calculate the relative pose change of the product in the fixture.

[0079] (3) Safety priority mechanism under high-frequency operation:

[0080] By employing a multi-stage readiness determination before imaging and an anomaly-priority short-circuit strategy, subsequent calculations in the current cycle are immediately terminated when a serious anomaly is detected, thus avoiding invalid actions and ensuring human-machine safety.

[0081] (4) High performance and transferability under small sample conditions:

[0082] By using few-sample training, data augmentation, and asynchronous incremental benchmark updates, this invention can still achieve high placement accuracy and missing nail detection rate even with fewer labeled samples. Furthermore, it can automatically identify and correct long-term drift through indicators such as the total cross-category geometric distance deviation.

[0083] (5) Unified visualization and traceability:

[0084] By using a quality visualization canvas to uniformly display batch screw cutting blocks, location indexes, tags, and confidence levels, and combining timestamps, product numbers, offset measurements, and comprehensive scores, the cost of quality problem investigation and long-term trend analysis is significantly reduced.

[0085] (6) Wide applicability and easy scalability:

[0086] This invention, through a unified "target instance set" modeling method, is not only applicable to screw fastening processes, but can also be extended to various assembly or connection processes such as solder joints, patches, connectors, snaps, and structural studs, and has broad application prospects for improving overall assembly quality and equipment utilization. Attached Figure Description

[0087] Figure 1 This is a flowchart of the overall process for determining the overall position and screw quality of industrial products before and after the fastening process in Embodiment 1 of the present invention.

[0088] Figure 2 This is a schematic diagram of the workstation imaging readiness and anomaly priority short circuit determination process in Embodiment 1 of the present invention.

[0089] Figure 3 This is a schematic diagram of fixture marker point and screw hole detection and template JSON establishment in Embodiment 1 of the present invention.

[0090] Figure 4 This is a schematic diagram of the offset measurement and grabbability dual threshold determination based on line segment distance change in Embodiment 1 of the present invention.

[0091] Figure 5 This is a schematic diagram of the batch screw quality judgment process performed on the product image after tightening in Embodiment 1 of the present invention.

[0092] Figure 6 This is a schematic diagram of the unified visualization canvas and multi-state output of screw quality in Embodiment 1 of the present invention.

[0093] Figure 7 This is a schematic diagram of the asynchronous governance and incremental baseline update process in Embodiment 1 of the present invention.

[0094] Figure 8 This is a block diagram showing the hardware composition and software module division of the integrated judgment system for the pre- and post-fastening posture and screw quality of industrial products in Embodiment 2 of the present invention. Detailed Implementation

[0095] This invention aims to provide a method and system for comprehensively judging the pose and screw quality before and after the fastening process of industrial products, solving the problems of fragmented detection processes before and after the fastening process, insufficient robustness of pose judgment, lack of priority in anomaly handling, poor adaptability to small samples, and weak visualization of quality traceability in existing technologies. Its core ideas are: First, establishing a "pre-emptive safety check + anomaly short-circuit" mechanism to complete equipment and environmental status checks before the detection process starts, ensuring a balance between safety and efficiency in high-cycle scenarios; Second, based on the geometric relationship of "marker point-screw hole" set, constructing an overall offset metric through the distance change between the template reference and runtime data to achieve robust pose judgment, breaking through the limitations of single feature points; Third, integrating online detection and offline management, using incremental updates of qualified samples to the reference and model, giving the system small sample adaptability and drift self-correction capabilities; Fourth, designing a unified quality visualization canvas and multi-state output, transforming scattered detection data into intuitive and traceable comprehensive conclusions, while supporting multi-assembly process expansion through a generalized framework design, ultimately achieving accurate, efficient, and scalable judgment of the entire process before and after the fastening process.

[0096] Specifically, the workflow of this invention for comprehensively determining the position and screw quality before and after the fastening process of industrial products is as follows:

[0097] 1. First determine "whether it can be photographed and whether it can be calculated": Through multi-stage judgment of the workstation imaging readiness and safety status, ensure that visual detection is triggered only when the robot arm, screw machine, fixture and human-machine safety conditions are met.

[0098] 2. Next, determine whether the product is placed correctly and whether it can be gripped (or locked with screws): Use a lightweight target detection model to detect the fixture markers and the product screw holes. Use a template JSON to store the reference geometric relationship between the markers and screw holes. During runtime, construct an overall offset metric by the change in the line segment distance from the screw hole to the marker. Use a dual threshold strategy to determine the product's gripability (or lockability).

[0099] 3. Then determine whether the screws are good or not: After the fastening process, each screw hole area is trimmed and normalized. A small-sample friendly screw classification model is used to determine the presence and quality of screws, and the screw pass rate, defect rate and other indicators are statistically analyzed.

[0100] 4. Finally, output a unified "understandable overall conclusion": Project the pose offset and screw quality results onto the quality visualization canvas and give multiple state outputs such as "success", "fail", "no_grab", "none" and "skip_screw_det". At the same time, send the key data to the asynchronous governance module to support long-term drift monitoring and benchmark updates.

[0101] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0102] Example 1

[0103] This embodiment provides a method for comprehensively judging the position and screw quality before and after the fastening process of industrial products. See [link to relevant documentation]. Figure 1 It includes the following implementation process:

[0104] 1. Industrial Condition Acquisition and Imaging

[0105] During each generation cycle, the system acquires images of the workstation using industrial cameras and light sources, while simultaneously reading status signals from the robotic arm, screw machine, and other components from the PLC or controller. This data will serve as the basis for subsequent processing and analysis.

[0106] 2. Workstation Imaging Readiness Determination and Anomaly Handling

[0107] The purpose of this process is to verify whether the workstation meets the safety and imaging requirements before starting the core testing, to avoid invalid calculations and safety risks, and to ensure the reliability and safety of the testing process.

[0108] In this embodiment, the system predefines an imaging ready condition set, including but not limited to:

[0109] Is the robotic arm in a safe standby position or a safe position before grasping?

[0110] Check whether the screw machine's actuator head has left the product surface and is at a safe height.

[0111] Check whether there are any unauthorized personnel entering the work area or any additional products or tools mixed in.

[0112] Is the light source in a stable output state?

[0113] In this embodiment, the process for readiness determination and exception handling is as follows: Figure 2 ,include:

[0114] At the beginning of each operation cycle, the system starts a timer to continuously acquire images and status signals within the imaging time window, and makes judgments according to the following logic:

[0115] (1) If serious abnormalities such as personnel intrusion or fixture not being locked are detected, the safety-related abnormal status (such as “fail” or “skip_screw_det” status) will be output immediately, and the subsequent detection in this cycle will be short-circuited, waiting for the next cycle to be triggered.

[0116] (2) If there is no serious abnormality within the time window, but the ready conditions are not fully met, output "none" or "no_grab" status when the window times out, end the current cycle, and wait for the next beat to be triggered.

[0117] (3) If all ready conditions are met within the time window, the “imaging ready” status is output and the image processing flow enters the pre-fastening or post-fastening detection stage.

[0118] 3. Forward fastening posture and graspability determination

[0119] This process precisely quantifies the product's placement posture relative to the fixture to determine whether it meets the requirements for robotic arm gripping and fastening operations, providing reliable pose data for subsequent processes. The basic idea is: using a lightweight target detection model to simultaneously detect fixture markers and product screw holes in the same image, ensuring detection speed and deployment cost; during the template creation stage, the template coordinates of markers and screw holes, as well as the template pixel distances between all "screw holes and markers," are structured and written into a template JSON file; during the runtime stage, the same model is used to detect runtime coordinates, calculating the difference between runtime distances and template distances one by one to obtain the distance change for each line segment, and then summarizing at the screw hole level and globally to construct an overall offset metric; through a dual-threshold strategy, the overall offset metric is mapped to levels such as "gripable," "gripable but with large deviation," and "ungripable"; the total point deviation can serve as an auxiliary indicator for external debugging and trend analysis, but the overall offset metric constructed based on line segment distance changes is truly used for grippability determination.

[0120] In this embodiment, the implementation process of the pre-tightening pose and graspability determination, including the template creation stage and the runtime determination stage, is described in detail below:

[0121] (1) Template creation stage

[0122] See Figure 3 In calibration mode, the system acquires multiple representative images and uses a lightweight object detection model for automatic localization:

[0123] 1) The pixel coordinates of the fixture marker point, denoted as ;

[0124] 2) The pixel coordinates of the screw holes to be fastened on the product are denoted as... .

[0125] After averaging and filtering the results of multiple detections, a stable set of template marker point coordinates is obtained. Set of coordinates of template screw holes Based on this, the template distance between each screw hole and each marker point is calculated and denoted as the screw hole-marker distance in the template (4):

[0126] ;

[0127] The coordinates and template distances mentioned above are written together into the template JSON file for quick reading and comparison during runtime.

[0128] (2) Operational Judgment Phase

[0129] 1) Calculation of point offset

[0130] During operation, within each production cycle, when "imaging ready" and the product is in the pre-fastening stage, the system calls the same lightweight object detection model for the current image to obtain the runtime marker coordinates. and runtime screw hole coordinates .

[0131] For each marker point and screw hole, calculate the Euclidean distance between the runtime coordinates and the template coordinates, denoted as the marker point offset in formula (1):

[0132] ;

[0133] The offset of the screw hole position is denoted as formula (2):

[0134] ;

[0135] Based on the above offset, the total positional deviation can be defined. (Mainly used for external debugging and trend analysis), denoted as the total deviation of the point in formula (3):

[0136] ;

[0137] in, and The weighting coefficients are for the marker points and screw holes.

[0138] 2) Calculation of the change in distance between screw hole and marker point line segments and the overall offset measurement

[0139] See Figure 4 Read template distance from template JSON file And calculate the runtime distance based on the runtime coordinates, denoted as formula (5) runtime screw hole – marker point distance:

[0140] ;

[0141] Define the relative pixel distance change of a single line segment as formula (6) line segment distance change:

[0142] ;

[0143] Based on this, the distance changes between each screw hole and all marker points are weighted and summarized to obtain the total distance change at the screw hole level, denoted as the total distance change of a single screw hole (7):

[0144] ;

[0145] in The weights are those of the marker points. Further, a weighted summation is performed on all screw holes to obtain the overall offset metric, denoted as Formula (8): Overall Offset Metric:

[0146] ;

[0147] in, This is the screw hole weight, used to highlight the impact of critical screw holes.

[0148] 3) Dual threshold crawlability determination

[0149] The system performs hierarchical judgment based on the relationship between the overall offset metric and the first and second thresholds, denoted as Formula (9) Dual-threshold crawlability judgment rule:

[0150] (9);

[0151] When a hole is determined to be "grabable but with a large deviation" or "ungrabable", the system can highlight the screw hole with a large distance change in the visualization interface to guide on-site personnel to make adjustments.

[0152] 4. Quality assessment and visualization of rear-fastening screws

[0153] The purpose of this process is to accurately detect the presence and quality grade of screws in each screw hole, and to statistically analyze overall quality indicators, providing data support for quality traceability and process optimization. The basic idea is as follows: Based on the screw hole coordinates obtained in the pre-tightening stage, the screw area is directly cropped from the post-tightening image at a uniform size; geometric normalization, brightness / contrast normalization, and necessary noise reduction or sharpening operations are used to improve the visual comparability of different screw hole areas; the normalized screw cropped blocks are input into a small-sample-friendly classification model, outputting classification results and confidence levels for each screw hole, such as "screw present - qualified," "screw present - suspicious," "no screw," and "occlusion / contamination"; based on the classification results and the expected total number of screws, the screw pass rate and defect rate are calculated; all screw cropped blocks and their labels are arranged on a uniform quality canvas and visualized along with the pre-tightening offset.

[0154] In this embodiment, the process for determining and visualizing the quality of the rear-fastening screw is described in detail below:

[0155] 1) After the fastening process is completed, the system performs a batch quality assessment of the screws on the product image after fastening. See [link / reference]. Figure 5 :

[0156] Based on the screw hole coordinates saved in the previous fastening step, the local area of ​​the screw is cropped from the image after fastening at a uniform size to form a screw cropping block sequence.

[0157] For each cropping block, geometric normalization, brightness and contrast normalization, and necessary noise reduction or sharpening are performed to make the screw areas of different hole positions comparable in pixel scale and brightness.

[0158] Input the normalized cut-out block into the screw classification model, and output the classification result (such as "with screw - qualified", "with screw - suspicious", "no screw", "obstruction / contamination", etc.) and confidence level for each screw hole;

[0159] Given the expected total number of screws Under the premise of counting the number of qualified screws and the number of defective screws The screw pass rate and defect rate are defined as follows, denoted by formula (13): Screw pass rate:

[0160] ;

[0161] Formula (14) Screw defect rate:

[0162] ;

[0163] Furthermore, the overall offset metric can be... Screw pass rate Combined with screw defect rate For a quality rating The comprehensive quality score is denoted as formula (15) (example):

[0164] ;

[0165] in To normalize the offset metric to Functions on intervals These are the weighting coefficients.

[0166] 2) Quality visualization canvas and multi-state decision output

[0167] See Figure 6Based on the classification results and the above statistical indicators, the system generates a quality visualization canvas, arranges each screw cutting block according to the hole position spatial layout, and labels the hole position number, classification label and confidence level next to each cutting block. At the same time, the product number, timestamp, overall status and quality score are displayed on the edge of the canvas.

[0168] 5. Asynchronous governance and incremental baseline updates

[0169] The purpose of this process is to use qualified samples accumulated online to adaptively update benchmark and model parameters, offset the effects of equipment geometric drift and product batch variations, and maintain long-term testing accuracy.

[0170] This embodiment decouples online detection from offline treatment to achieve adaptive support for long-term drift and multi-model data. See [link to documentation]. Figure 7 ,include:

[0171] (1) During the online detection process, the system writes the original image, marker coordinates, screw hole coordinates, distance change, screw classification results, etc. of each cycle into the asynchronous governance queue or log system in a structured manner.

[0172] (2) The offline governance task reads data from the queue according to the preset cycle and filters qualified samples confirmed by manual or high confidence rules.

[0173] (3) For each product category (model) and each screw hole Let the set of qualified samples be denoted as Define its average coordinates, denoted as the average coordinates of the category – borehole position, as in formula (10):

[0174] ;

[0175] (4) For each well location, the total cross-category geometric distance deviation is defined as the sum of the paired distances between the average coordinates of each category, denoted as formula (11) Total cross-category geometric distance deviation of a single well location:

[0176] ;

[0177] Furthermore, the total cross-category geometric distance deviation for all holes is defined as denoted by formula (12):

[0178] ;

[0179] in The hole position weight.

[0180] (5) If discovered Or certain key hole positions If the value continues to increase, it indicates that there is geometric drift in the equipment or a change in the assembly process. The system will then automatically adjust the template coordinates and offset threshold accordingly. , Weighting coefficients , , After manual review, the updated parameters are sent to the online system via hot update.

[0181] (6) For the screw classification model, after a sufficient number of new samples have been accumulated, the model weights can be updated by incremental training or fine-tuning. After verification by A / B testing, the old model can be gradually replaced to improve the overall detection performance and adaptability.

[0182] Example 2

[0183] This embodiment provides a comprehensive judgment system for the position and screw quality before and after the fastening process of industrial products. The system architecture is described below. Figure 8 .

[0184] The on-site hardware includes industrial cameras, robotic arms, light sources, screw machines, and status acquisition units. The industrial cameras, in conjunction with the light sources, capture clear images of the workstation, providing raw image data for visual inspection. Meanwhile, the status acquisition units are used to collect real-time signals such as equipment position, operating status, and human-machine safety from the robotic arms, screw machines, fixtures, and safety sensors, providing status basis for imaging readiness determination.

[0185] The software includes modules for imaging readiness and safety determination, hole location detection and matching, template management and distance calculation, deviation calculation and dual threshold grading, screw trimming and classification, visualization canvas generation, multi-state decision-making and result output, and asynchronous governance and incremental update.

[0186] Imaging readiness and safety determination module: Receives images and status signals acquired by the hardware layer, detects safety and readiness conditions within a preset time window, and outputs "imaging ready" or abnormal status.

[0187] Hole location detection and matching module: Used to identify the set of fixture marker points and the set of screw holes using a lightweight target detection model.

[0188] Template Management and Distance Calculation Module: Manages template JSON files, reads the baseline coordinates and template distance, and calculates runtime distance and distance change.

[0189] Deviation Calculation and Dual Threshold Grading Module: Used to construct an overall offset metric based on the distance change and to determine the grabbability grading based on dual thresholds.

[0190] Screw trimming and classification module: Trimming and normalizing the screw hole area, and using a classification model to determine the screw quality category and confidence level.

[0191] Visualization Canvas Generation Module: Integrates pose offset and screw quality results to construct a unified quality visualization canvas, overlaying hole position index, labels and other information.

[0192] Multi-state decision and result output module: used to fuse pose offset and screw classification results and output multi-state judgment and defect statistics information.

[0193] The asynchronous governance and incremental update module specifically includes a log management submodule, a sample screening submodule, a statistical analysis submodule, and a model / benchmark update submodule. The log management submodule receives and stores structured logs and image data generated by online detection. The sample screening submodule filters qualified samples and typical defect samples from the logs. The statistical analysis submodule performs statistical analysis on screw hole coordinates, distance changes, and screw classification results to generate drift and trend indicators. The model / benchmark update submodule updates the benchmark position and distance thresholds, adjusts scoring weights, and incrementally trains the screw classification model based on the statistical results, then sends the updated parameters or model to the online system.

[0194] It should be noted that the above embodiments use screws and screw holes as objects to illustrate the implementation of the present invention. In practical applications, the invention can be easily extended to other assembly and connection processes by abstracting "screw hole + screw" into a "target instance set," for example:

[0195] For solder joints and surface mount devices, "solder joint center" or "critical surface mount location" can be used as the equivalent of screw hole, and "whether the soldering is good" or "whether the fit is in place" can replace the screw quality category.

[0196] For objects such as connectors, snaps, and structural studs, corresponding feature points and classification labels can be defined, and the same templates, JSON, creation methods, offset measurement mechanisms for changes in line segment distance of multiple marker points, and unified quality visualization canvases can be reused.

[0197] With a unified framework, the system of this invention can support the expansion of multiple processes without major hardware modifications, thus shortening the engineering deployment cycle.

[0198] Although embodiments of the present invention have been described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present invention, and all such changes and alterations shall not depart from the protection scope of the present invention.

Claims

1. A method for comprehensively judging the position and screw quality before and after the fastening process of industrial products, characterized in that, Includes the following steps: A. Acquire workstation images and equipment status signals within the preset imaging time window. If a serious abnormality is detected, output the abnormal status and terminate the detection of this cycle. When all ready conditions are met, proceed to the next step. B. When the product is in the pre-fastening process, a lightweight target detection model is used to locate the fixture markers and the product screw holes. During the template creation stage, the template coordinates of the markers and screw holes and the template pixel distance between them are stored and written into the template JSON file. During the running stage, the distance change from each screw hole to each marker is calculated based on the running detection results and the template JSON file, and an overall offset metric is constructed accordingly to classify and determine the relative pose and gripping ability of the product in the fixture. C. After the product has undergone the fastening process, a screw area cutting block is generated for each target screw hole. The cutting block is normalized and the presence or absence of screws and the fastening quality of each screw hole are classified using a screw classification model. D. Based on the pre-tightening pose offset judgment result and the post-tightening screw classification result, a unified quality visual canvas is constructed. Each screw clipping block and its classification information are displayed in the canvas, and multi-dimensional judgment status statistics are output. E. Write the original image, marker coordinates, screw hole coordinates, distance change, and screw classification results into the asynchronous management queue. Within a set period, statistically analyze the new qualified samples, update the placement benchmark, distance threshold, and screw appearance baseline, and distribute them in an online hot update manner.

2. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 1, characterized in that, In step A, the ready conditions include: Check whether the end effector of the robotic arm is in a safe standby position or a safe position before grasping; Check whether the screw-operating head has left the product surface and is at a safe height; Check whether the fixture has completed the clamping or releasing action and reached a stable state; Check whether there are any unauthorized personnel or extra products mixed in in the testing station area; Check whether the output of the detection light source is stable within the preset working range; In step A, if any of the above conditions are not met within the preset imaging time window, the status of this cycle is determined to be "none" or "no_grab", the detection of this cycle is terminated and the subsequent steps are not entered.

3. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 1, characterized in that, In step B, the step of using a lightweight target detection model to locate the fixture markers and product screw holes, and storing the template coordinates of the markers and screw holes, as well as the template pixel distance between them, and writing them into a template JSON file during the template creation stage, specifically includes: Multiple representative images were acquired in calibration mode. A lightweight target detection model was used to obtain the template coordinate set of fixture marker points and screw holes. By averaging and filtering the multiple detection results, the template coordinates of the marker points and screw holes were obtained. The template distance from each screw hole to each marker point was calculated. The coordinates and template distances were written into a template JSON file in a structured form. The process of calculating the distance change from each screw hole to each marker point based on the runtime detection results and the template JSON file during the runtime phase, and constructing an overall offset metric accordingly, specifically includes: Within each production cycle, the lightweight target detection model is invoked on the current image to obtain the runtime marker coordinates and screw hole coordinates. The corresponding template distance is read from the template JSON file, and the difference between the runtime distance and the template distance is calculated to obtain the distance change from each screw hole to each marker and the overall offset metric.

4. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 3, characterized in that, The overall offset metric is a weighted sum of the changes in distance between each screw hole. In step B, the grabbability is graded based on a preset first threshold and a second threshold. When the overall deviation statistic is lower than the first threshold, the placement status is determined to be grabbable; When the overall deviation statistic is between the first threshold and the second threshold, the placement status is determined to be grabbable but the deviation is too large. When the overall deviation statistic is higher than the second threshold, the placement status is determined to be ungrabable; The second threshold is greater than the first threshold and can be dynamically adjusted based on historical offset statistics and on-site process requirements.

5. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 1, characterized in that, In step C, the cut-out blocks are normalized, specifically including: Perform geometric normalization, brightness and contrast normalization on the cropped blocks respectively, and then perform noise reduction or sharpening processing.

6. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 1, characterized in that, In step C, the output categories of the screw classification model include: Screws are present and properly tightened; The screws are present but have defects such as floating, tilting, biasing, or head damage. No screws; The screw area is severely obscured or contaminated, making reliable identification impossible.

7. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 1, characterized in that, In step D, a unified quality visualization canvas is constructed based on the pre-tightening pose offset determination result and the post-tightening screw classification result, specifically including: Arrange all screw cutting blocks on the same canvas according to the pre-set screw hole number and physical space layout, so that the position index on the canvas corresponds one-to-one with the actual hole position. Next to each screw cutting block, label the corresponding hole index, category label and confidence value, and mark the current product number, timestamp, overall placement status and comprehensive judgment result of screw quality on the edge of the canvas.

8. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 7, characterized in that, In step D, the multi-dimensional determination status statistics include at least one of the following: "success", "fail", "no_grab", "none", and "skip_screw_det" and corresponding defect statistics; wherein, "success" indicates that the screws are grippable and of generally acceptable quality. "fail" indicates that there is at least one improper placement or a defective screw. "no_grab" indicates that the data cannot be grabbed due to excessive relative offset or failure to meet security conditions. "none" indicates that no effective detection was performed or no effective images were collected in this cycle; "skip_screw_det" indicates that screw detection is skipped due to an anomaly-priority short-circuit strategy or configuration settings, and only pose-related conclusions are output.

9. The method for comprehensively judging the position and screw quality before and after the fastening process of industrial products as described in claim 1, characterized in that, In step E, new qualified samples are statistically analyzed within a set period, and the placement benchmark, distance threshold and screw appearance baseline are updated. Specifically, qualified samples confirmed by manual or rule-based methods are selected from the asynchronous management queue according to a preset period. Based on the qualified samples, the average coordinates and covariance of each screw hole in the current time window and under different product models and batches are statistically analyzed, and the total cross-category geometric distance deviation at the hole position level and globally is calculated. The template coordinates, offset threshold, and screw classification score weights are automatically updated based on statistical results.

10. A comprehensive judgment system for the position and screw quality before and after the fastening process of industrial products, characterized in that it includes: The image acquisition unit, including an industrial camera and a light source, is used to acquire images of the workstation and provide stable lighting; The status acquisition unit is used to acquire equipment status signals from the robotic arm, screw machine, fixture, and safety sensors. The image processing and determination unit is used to execute the comprehensive determination method of pose and screw quality before and after fastening process of industrial products as described in any one of claims 1 to 9, and generate pose offset determination result, screw quality determination result and multi-state output. Run the visualization unit to display the quality visualization canvas, multi-state output, defect list, and historical statistics; The asynchronous governance and incremental update unit is used to perform offline governance, statistical analysis and model / benchmark updates on online detection results, and send the updated results to the image processing and judgment unit.

11. The industrial product fastening process pre- and post-fastening position and screw quality comprehensive judgment system as described in claim 10, characterized in that, The image processing and determination unit includes: The imaging readiness determination module is used to detect imaging safety and readiness conditions within a preset time window; The marker point and screw hole detection module is used to identify the fixture marker point set and screw hole set using a lightweight target detection model; The template management and distance calculation module is used to manage template JSON files, calculate template distance and runtime distance, and obtain the distance change. The offset metric and dual-threshold grading module is used to construct an overall offset metric based on the distance change and to determine the crawlability grading based on dual thresholds. The screw cutting and classification module is used to generate screw cutting blocks and perform quality classification using a screw classification model. The visualization canvas generation module is used to construct a unified quality visualization canvas and overlay global judgment information; The multi-state decision and result output module is used to fuse pose offset and screw classification results and output multi-state judgment and defect statistics information.

12. The industrial product fastening process pre- and post-fastening position and screw quality comprehensive judgment system as described in claim 11, characterized in that, The asynchronous governance and incremental update unit includes: The log management module is used to receive and store structured logs and image data generated by online detection; The sample screening module is used to screen qualified samples and typical defective samples from the log; The statistical analysis module is used to perform statistical analysis on screw hole coordinates, distance changes, and screw classification results, and generate drift and trend indicators. The model / benchmark update module is used to update the benchmark position and distance thresholds, adjust the scoring weights, and incrementally train the screw classification model based on statistical results, and then send the updated parameters or model to the online system.

Images