A visual-based automatic screw detection and relocking method and all-in-one machine

Abstract

This invention discloses a vision-based automatic screw detection and replacement method and an integrated machine. The method includes: acquiring images; correcting offsets and locking hole positions through template matching and affine transformation; extracting metal features through adaptive threshold segmentation; matching the screw head pixel height and the distance between the screw head and the hole edge with a qualified depth threshold; determining defects using multi-feature fusion based on feature existence and conformity; converting the defect image coordinates into physical coordinates to drive an actuator for replacement. The integrated machine includes a frame, a vision component, an air-blowing locking module, a motion platform, and a controller. This invention integrates detection and replacement on the same platform, effectively resists interference through multi-algorithm fusion, adapts to product placement deviations, and is suitable for high-precision automatic detection and replacement of screws on the bottom of a machine.

Description

Technical Field

[0001] This invention relates to the field of automated inspection and assembly technology, specifically to a vision-based automatic screw inspection and replacement method and integrated machine. Background Technology

[0002] In the assembly process of electronic products (such as small household appliances and controller housings), the quality of the screw fastening at the bottom directly affects the structural strength and reliability of the product. Traditional methods often rely on manual visual inspection to check for missing or loose screws, which suffers from low efficiency, fatigue, and a high rate of missed detections. While some existing equipment can perform visual inspection, the detection and re-fastening processes are separated, requiring secondary positioning, which is inefficient and increases equipment costs. In addition, slight product placement deviations, as well as metal reflections and dust interference from different screw hole positions, lead to a high false detection rate for traditional image algorithms. Summary of the Invention

[0003] Purpose of the invention: To provide an automatic screw detection and repair method and equipment with high detection accuracy, strong anti-interference ability, and integrated detection and repair.

[0004] Technical solution: I. Methodology A vision-based automatic screw detection and replacement method includes the following steps: Step 1: Image Acquisition The vision inspection component is moved to the bottom of the product and, under the vertical illumination of a ring light source and in an optical darkroom environment formed by a light shield, a single frame image containing all screw holes is captured.

[0005] Step 2: Hole Positioning and Offset Correction Load a pre-stored product-specific hole location template library (containing standard ROI regions and features for each screw hole). Use a template matching algorithm to search for overall product features across the entire image to obtain the product's actual position and angular deviation in the image. Then, use affine transformations (including translation, rotation, and scaling) to map the standard ROI coordinates to the corresponding positions in the current image, accurately locating the detection areas for the eight screw holes and eliminating the influence of product placement deviations.

[0006] Step 3: Metal Feature Extraction and Segmentation For each locked hole area, an adaptive local threshold segmentation algorithm (such as the Otsu or Sauvola algorithm) is used to automatically calculate the segmentation threshold based on the grayscale distribution within the area, separating the metal screw head area from the background. The following core features are extracted: Area of ​​connected domains Outline perimeter Average gray level Step 4: Deep Feature Matching The real-time extracted pixel height of the screw head (reflecting the screw insertion depth) and the pixel spacing between the screw head edge and the hole edge are matched item by item with a pre-calibrated threshold library of qualified screw insertion depth features. This threshold library is determined by statistically analyzing the above feature values ​​from multiple qualified insertion samples to determine the normal range.

[0007] Step 5: Multi-feature fusion determination A two-tiered decision-making strategy of "feature existence + feature conformity" is adopted: Feature Existence: If the area of ​​the metal feature region is smaller than the set threshold or the average gray level deviates from the template mean by more than the allowable range, it is determined as "missing screw". Feature compliance: If the feature exists, but the pixel height of the screw head exceeds the acceptable range or the distance between the edge of the head and the edge of the hole exceeds the acceptable range, it is judged as "not properly locked (floating lock or locked)". A designation is considered "qualified" only if the feature exists and all features meet the threshold.

[0008] Step Six: Lock Replacement Execution If a defect is identified, the image coordinates of the defective hole are converted into the actual physical coordinates of the motion platform based on a pre-defined mapping relationship between image coordinates and actual physical coordinates (such as the nine-point calibration method or homography transformation matrix). The controller drives the motion platform to move the air-blowing locking module to that coordinate position, initiating air-blowing locking from bottom to top to supplement the missing screw in the hole or to perform a secondary tightening of the screws on the floating lock.

[0009] Step 7: Re-inspection (optional) After the lock is re-locked, control the visual inspection component to move to the shooting position again, and repeat steps two through five for re-inspection. If it is confirmed to be qualified, output a completion signal and stop the alarm; if it is still unqualified, record the alarm or perform a second lock-locking.

[0010] II. Equipment Section A vision-based automatic screw detection and replacement machine for implementing the above method includes: The frame consists of an upper cabinet and a lower cabinet, with a product positioning mold between them. The top of the upper cabinet is equipped with a clamping cylinder and an upper pressure mold for clamping and securing the product to be tested. Through-beam photoelectric sensors are located on both sides of the product positioning mold to detect whether a product has been placed inside.

[0011] Motion platform: Located in the lower cabinet, it is a precision servo XY two-axis or XYZ three-axis linear module.

[0012] Visual inspection components: including an industrial area scan camera and a ring light source, mounted on a motion platform with the optical axis pointing vertically upwards, forming a closed optical environment with the surrounding light shields.

[0013] Execution component: includes an air-cushioning module, which is mounted on a motion platform and moves synchronously with the vision detection component in a fixed relative position, with the locking direction being from bottom to top.

[0014] Controller: An industrial computer, electrically connected to the above components, used to execute the methods described in steps one through seven. It is also connected to indicator lights and a buzzer to indicate whether the detection is successful or the lockout status is active.

[0015] Beneficial effects: 1. The detection and locking are integrated into the same motion platform, eliminating the need for secondary positioning, resulting in fast cycle time and high accuracy.

[0016] 2. The "template matching + affine transformation" method is used to correct product placement deviations and adapt to positional fluctuations in actual production.

[0017] 3. By adopting "adaptive threshold segmentation + multi-feature fusion judgment", it effectively distinguishes metal reflection, dust interference and real defects, and reduces the false detection rate.

[0018] 4. The method is highly portable and can be adapted to different product models (only the template library and threshold library need to be changed). Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the all-in-one machine of the present invention; Figure 2 This is a side view of the overall structure of the present invention.

[0020] Figure 3 This is an overall flowchart of the method of the present invention; Detailed Implementation Example 1 like Figure 1 and Figure 2 As shown, a vision-based automatic screw detection and repair machine includes an upper cabinet 15, a lower cabinet 16, and an electrical control cabinet 14.

[0021] I. Mechanical Structure The product limiting mold 12 is fixed to the partition between the upper and lower cabinets, and adopts a contoured groove that matches the shape of the product. The through-beam photoelectric sensor 11 is installed on both sides of the product limiting mold 12. When the product is placed in, the light path is blocked, triggering a start signal.

[0022] The clamping cylinder 2 is installed on the top of the upper cabinet 15, and the piston rod is connected to the upper pressure mold 4. The lower surface of the upper pressure mold 4 is provided with a soft pressure head to avoid damaging the product.

[0023] A precision servo platform 13 (using an XY dual-axis linear module) is fixed inside the lower cabinet 16. A camera mounting plate and a locking mounting plate are installed on it, maintaining a fixed relative position between the vision camera 7 and the air-blowing locking module 8. The vision camera is a 5-megapixel industrial area array camera with a 16mm macro fixed-focus lens. A ring light source 9 is coaxially mounted in front of the lens, with its emitting surface facing the bottom of the product. Light-shielding plates 10, made of black acrylic, are provided around the lower cabinet 16, forming a closed darkroom.

[0024] II. Electrical Control System Industrial PC (i.e., controller): An industrial-grade embedded industrial PC is installed in a separate electrical control cabinet 14, with built-in image acquisition card, motion control card, and I / O control card. The industrial PC serves as the core processing unit, running a Windows or Linux operating system, and deploying vision inspection and motion control software developed based on C++ / Python.

[0025] Display Screen 1: A 15.6-inch industrial touch screen is installed on the top of the electrical control cabinet 14 and is electrically connected to the industrial control computer. Display Screen 1 is used to display images captured by the vision camera in real time, mark the location and quantity of defective screws, display the current inspection results (pass / fail), and provide a human-machine interface (such as parameter settings, template calibration, historical data query, etc.).

[0026] Signal Light 5: A tri-color LED signal light (red, yellow, green) is installed in a prominent position on the top of the control cabinet and electrically connected to the I / O output of the industrial control computer. Its working logic is as follows: A solid green light indicates that all screws have passed inspection. Yellow light flashing: Device is in standby or initializing; A solid red light indicates a screw defect has been detected and repair is underway. Flashing red light: Equipment malfunction or alarm.

[0027] Buzzer 3: Installed inside the control cabinet or in the integrated signal light assembly, it is electrically connected to the I / O output of the industrial control computer. Its working logic is as follows: When a screw defect is detected: the buzzer emits an intermittent "beep-beep-beep" alarm sound; After the lock replacement is completed and the re-inspection is passed: the buzzer will stop ringing; When equipment malfunctions: the buzzer emits a continuous long beep.

[0028] Through-beam photoelectric sensor 11: Installed on both sides of the product limiting mold 12, with the transmitting end and receiving end facing each other. Its signal output end is connected to the I / O input end of the industrial control computer. When the product is placed into the product limiting mold 12, the through-beam light path is blocked, the sensor output level changes, and the industrial control computer detects this change and automatically triggers the vision inspection process.

[0029] III. Specific Implementation Procedures for Detection and Lock Repair Methods Step 0: System Initialization After the equipment is powered on, the industrial control computer automatically starts the detection software, the indicator light shows white (initializing), the motion platform (precision servo platform 13) performs a homing operation, the vision camera 7 automatically adjusts its exposure, and the light source brightness performs a self-check. After initialization is complete, the indicator light switches to yellow (standby), and the display shows "Equipment ready".

[0030] Step 1: Product Placement and Triggering The operator places the product into the product limiting mold 12. After the photoelectric sensor 11 detects the product, it sends a trigger signal to the industrial control computer. The industrial control computer records the trigger time and displays "Product placed, start detection" on the screen.

[0031] Step 2: Press firmly The industrial control computer controls the solenoid valve via I / O to drive the clamping cylinder 2 to press down. After clamping is completed, the pressure sensor (optional) sends a feedback signal, and the industrial control computer confirms that the product has been fixed.

[0032] Step 3: Image Acquisition and Detection Specifically, it includes: Step 31: Image Acquisition During inspection, the vision inspection component is moved to the bottom of the product and, under the vertical illumination of a ring light source and the optical darkroom environment formed by the light shield, a single frame image containing all screw holes is acquired.

[0033] Step 32: Hole Positioning and Offset Correction The industrial control computer loads a pre-stored product-specific hole position template library (containing standard ROI areas and features for each screw hole). A template matching algorithm is used to search for overall product features across the entire image, obtaining the product's actual position and angular deviation in the image. Then, affine transformations (including translation, rotation, and scaling) are used to map the standard ROI coordinates to their corresponding positions in the current image, accurately locating the eight screw hole detection areas and eliminating the impact of product placement deviations.

[0034] Step 33: Metal Feature Extraction and Segmentation For each locked hole area, an adaptive local threshold segmentation algorithm (such as the Otsu or Sauvola algorithm) is used to automatically calculate the segmentation threshold based on the grayscale distribution within the area, separating the metal screw head area from the background. The following core features are extracted: connected region area, contour perimeter, and average grayscale.

[0035] Step 34: Deep Feature Matching The real-time extracted pixel height of the screw head (reflecting the screw insertion depth) and the pixel spacing between the screw head edge and the hole edge are matched item by item with a pre-calibrated threshold library of qualified screw insertion depth features. This threshold library is determined by statistically analyzing the above feature values ​​from multiple qualified insertion samples to determine the normal range.

[0036] Step 3: Multi-feature fusion determination A two-tiered decision-making strategy of "feature existence + feature conformity" is adopted: Feature Existence: If the area of ​​the metal feature region is smaller than the set threshold or the average gray level deviates from the template mean by more than the allowable range, it is determined as "missing screw". Feature compliance: If the feature exists, but the pixel height of the screw head exceeds the acceptable range or the distance between the edge of the head and the edge of the hole exceeds the acceptable range, it is judged as "not properly locked (floating lock or locked)". A designation is considered "qualified" only if the feature exists and all features meet the threshold.

[0037] Step 4: Result Output and Display The industrial control computer displays the inspection results on the screen in real time, labeling each screw hole with its serial number and status (e.g., qualified, missing, floating lock), as well as information such as inspection time, total number of screws, number of qualified screws, number of defects, and defect location. Simultaneously, indicator lights change color based on the overall results, and the buzzer emits corresponding alarm sounds according to the defect status.

[0038] Step 5: Lock replacement execution If a defect is identified, the image coordinates of the defective hole are converted into the actual physical coordinates of the motion platform based on a pre-defined mapping relationship between image coordinates and actual physical coordinates (such as the nine-point calibration method or homography transformation matrix). The controller drives the motion platform to move the air-blowing locking module to that coordinate position, initiating air-blowing locking from bottom to top to replace the missing screw in the hole or to re-tighten the screws of the floating lock. This operation requires no operator intervention; the industrial computer automatically performs the locking.

[0039] Step 6: Re-inspection (optional) After the lock replacement is completed, the industrial control computer automatically performs a re-inspection. If the re-inspection passes: The traffic light switched to green; The buzzer has stopped ringing. The display screen shows "Lock replacement complete, final inspection passed"; Generate an inspection report, which includes the product ID (can be entered using an external barcode scanner), inspection time, original image, defect image, and repair record.

[0040] Step 7: Remove the product The industrial control computer resets the clamping cylinder 2, allowing the operator to remove the product. Once the photoelectric sensor detects the product's departure, the system automatically resets to standby mode (yellow light flashing), awaiting the next product.

[0041] IV. Supplementary Explanation of Human-Computer Interaction The touch screen in this embodiment provides the following operation buttons: Start detection: Manually trigger the detection process (use when the photoelectric sensor fails); Stop / Reset: Emergency stop and reset of the current action; Parameter settings: Enter the threshold adjustment, servo parameters, and camera parameters interface; Calibration: Enter the nine-point calibration, template registration, and qualified threshold learning interface; History: View testing data for the past 7 days, with filtering options by time and product model; User management: Hierarchical administrator / operator permissions to prevent accidental modification of core parameters.

[0042] Example 2 This embodiment is basically the same as Embodiment 1, except that: The industrial control computer integrates a wireless communication module (4G / 5G or Wi-Fi), which uploads the test data to the cloud server or the factory MES system in real time, enabling remote monitoring and quality traceability.

[0043] The display screen is a 10.8-inch handheld tablet that connects to the industrial computer via wireless projection or remote desktop, allowing operators to move around the equipment while holding the tablet.

[0044] Add a voice prompt function to announce the test results through an external speaker: "Test passed", "2 screw defects found, start repair", "Repair completed", etc.

[0045] Industrial applicability This invention can be widely applied to assembly lines for 3C electronic products, home appliances, automotive electronics, communication equipment, etc., which require the inspection and replacement of screws on the bottom of the entire machine.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual-based automatic screw detection and re-tightening method, characterized in that, Includes the following steps: Step 1: Acquire an image containing all screw holes on the product; Step 2: Based on the pre-stored product-specific hole position template library, the product placement offset in the image is corrected through template matching and affine transformation algorithms to lock the detection area of ​​each screw hole; Step 3: Perform adaptive local threshold segmentation on each locked hole location area to extract the area, contour, and grayscale mean features of the metal feature area; Step 4: Match the real-time extracted screw head pixel height and pixel spacing with the hole edge with the pre-stored qualified screw fastening depth feature threshold library; Step 5: Determine whether there are screw defects in each hole position based on the multi-feature fusion strategy of "feature existence + feature conformity"; Step 6: If a defect exists, convert the image coordinates of the defective hole position into actual physical coordinates, and drive the actuator to move to that coordinate position to perform a repair lock.

2. The method of claim 1, wherein, The affine transformation described in step two is used to correct the translation and rotation offset of the product in the image.

3. The method of claim 1, wherein, The features of the metal feature region mentioned in step three also include the area of ​​the connected domain, the perimeter of the outline, and the average gray level.

4. The method of claim 1, wherein, The qualified depth feature threshold library mentioned in step four is obtained by pre-locking qualified samples for calibration.

5. The method of claim 1, wherein, The conversion from image coordinates to physical coordinates in step six uses the nine-point calibration method or homography transformation based on the actual size ratio of the product.

6. The method of claim 1, wherein, Step six also includes: after the lock is repaired, the image is acquired again for re-inspection, and a completion signal is output after it is confirmed to be qualified.

7. A visual-based automatic screw detection and relocking integrated machine for implementing the method of any one of claims 1-6, characterized in that, include: Upper cabinet and lower cabinet; The product positioning mold is set between the upper cabinet and the lower cabinet to position the product to be tested. The product positioning mold has a detection opening. A clamping assembly, located at the top of the upper cabinet, is used to clamp and fix the product in the product limiting mold; A vision inspection component, including an industrial camera and a light source, is mounted on a motion platform and located inside the lower cabinet. The vision inspection component is located below the product limiting mold, with its optical axis pointing vertically upward, and it acquires images of the screw hole positions on the bottom of the product through the inspection opening. An execution component, including an air-locking module, is mounted on the motion platform and moves synchronously with the vision detection component; The controller is electrically connected to the vision detection component, the execution component, and the motion platform, respectively, and is used to execute the method according to any one of claims 1-6.

8. The all-in-one machine of claim 7, wherein, The light source is a ring light source, and the lower cabinet is equipped with a light shield to form a closed or semi-closed optical darkroom.

9. The all-in-one machine of claim 7, wherein, It also includes photoelectric sensors, which are set on both sides of the product limiting mold to detect whether the product has been placed in.

10. The all-in-one machine of claim 7, wherein, It also includes indicator lights and a buzzer, which are electrically connected to the controller and are used to indicate whether the detection is qualified or the lock is replenished.

Images