Automatic adjusting method of adjustable component based on machine vision

Abstract

The invention relates to an adjustable component automatic adjusting method and device based on machine vision. The method comprises the following steps: firstly, carrying out image acquisition and data enhancement processing on a circuit to be identified, and manufacturing a data set containing adjustable components and key point marks of adjusting holes of the adjustable components; and a suitable target detection algorithm is selected to train the data set to obtain a target detection model. A depth camera is used for acquiring a circuit image and converting the circuit image into a depth map for depth detection, depth segmentation is carried out on the image according to a set depth value range, and a part matched with the depth of the adjustable component is reserved. And then detecting the adjustable component by using a target detection algorithm, carrying out key point detection on the detected component, and outputting a two-dimensional coordinate of a key point. And the angle of the adjusting hole is calculated according to the coordinates, and finally the adjustable component is automatically adjusted by the mechanical arm according to the key point coordinates and angle information.

Description

Technical Field

[0001] The invention relates to the technical field of automatic circuit detection and correction, and in particular to an automatic adjustment method for adjustable components based on machine vision. Background Art

[0002] In circuit production, adjustable components are often used for circuit adjustment and correction. However, due to their small size and complex structure, the adjustment of adjustable components is currently done manually, which is inefficient and costly. With the development of industrial automation, realizing the automatic adjustment of adjustable components is an urgent problem to be solved in my country's circuit production line, which can effectively reduce the cost of circuit production correction.

[0003] Existing machine vision technology has made significant progress in target detection and key point detection. At present, target detection algorithms are mainly divided into one-stage and two-stage. The one-stage target detection method is fast but relatively low in accuracy, while the two-stage target detection algorithm is highly accurate but slow. In addition, the existing key point detection method is not very accurate for key point detection of small-sized targets, especially when it is applied to key point detection of electronic components. The existing target detection and key point detection algorithms are based on two-dimensional planes for detection, and are often used to detect key points of the human body, and cannot be directly applied to the automatic adjustment of adjustable components. Especially in the angle measurement of the adjustment holes of electronic components, the existing technology is still difficult to meet the requirements of high-precision and high-efficiency automated processing. Summary of the invention

[0004] In order to overcome the above-mentioned technical defects, the main purpose of the present invention is to provide an automatic adjustment method and device for adjustable components based on machine vision. The present invention proposes an automatic adjustment method for adjustable components based on machine vision. Specifically, the coordinates of the adjustable components and the angles of their adjustment holes are identified by machine vision to provide accurate adjustment information for the robot arm, so as to realize automatic correction in circuit production and improve the efficiency of circuit detection and adjustment.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for automatically adjusting adjustable components based on machine vision, comprising the following steps:

[0007] 1) Collect the circuit image to be identified;

[0008] 2) Performing depth detection on the collected circuit images;

[0009] 3) Perform depth segmentation on the image after depth detection according to the set depth value range, and retain the image part that matches the depth of the adjustable component;

[0010] 4) Use the target detection algorithm to detect adjustable components in the image after depth segmentation and determine the position of the adjustable components;

[0011] 5) Perform key point detection on the detected adjustable components;

[0012] 6) Output the two-dimensional coordinates of key points;

[0013] 7) Calculate the adjustment hole angle according to the two-dimensional coordinates of the key points;

[0014] 8) The robotic arm automatically adjusts the adjustable components according to the key point coordinates and the adjustment hole angle information.

[0015] Preferably, the method may further include the following steps:

[0016] Dataset creation: Image acquisition and data enhancement are performed on the circuit to be identified, and key points of the adjustable components and their adjustment holes to be detected are marked to form a data set;

[0017] Model training: Select relevant target detection algorithms, use the data set for training, and establish a model that can identify key points of adjustable components and their adjustment holes.

[0018] The present invention also provides an automatic adjustment device for adjustable components based on machine vision, which includes an image acquisition module, a depth detection module, a depth segmentation module, a target detection module, a key point detection module, an angle calculation module and a mechanical arm control module, wherein:

[0019] The image acquisition module is used to acquire the circuit image to be identified;

[0020] The depth detection module is used to perform depth detection on the collected circuit image;

[0021] The depth segmentation module is used to perform depth segmentation on the image after depth detection according to the set depth value range, and retain the image part that matches the depth of the adjustable component;

[0022] The target detection module is used to detect adjustable components on the image after depth segmentation using a target detection algorithm to determine the position of the adjustable components. The target detection algorithm includes but is not limited to the YOLO algorithm and the Faster RCNN algorithm, and is optimized to adapt to the detection of adjustable components.

[0023] The key point detection module is used to detect key points of the detected adjustable components. The key point detection adopts the method of direct regression of key point coordinates and combines depth information to improve the detection accuracy.

[0024] The angle calculation module is used to calculate the angle of the adjustment hole according to the two-dimensional coordinates of the key points, and uses mathematical methods such as high-precision inverse tangent function to calculate the angle;

[0025] The robot control module is used to control the robot to automatically adjust the adjustable components according to the key point coordinates and the adjustment hole angle information.

[0026] The automatic adjustment device for adjustable components based on machine vision can implement any automatic adjustment method of the present invention.

[0027] Preferably, the data set preparation in step 1) also includes feature enhancement processing of the image, and the data enhancement methods include rotation, cropping, flipping, adding Gaussian noise, adjusting HSV contrast and adjusting brightness; the depth detection in step 2) can be implemented using a binocular camera. When using a depth camera to obtain a circuit image, the camera is calibrated before use to ensure the alignment of the color image and the depth image so that the pixel coordinates of the color image and the depth image after depth segmentation are consistent. The target detection algorithm in step 4) can be selected from one or more of the first-stage target detection algorithm and the second-stage target detection algorithm, and the selection range includes YOLO, SSD, R-CNN and Faster R-CNN.

[0028] Preferably, the key point detection in step 5) adopts a method of directly regressing the key point coordinates and combines depth information to improve detection accuracy.

[0029] Preferably, the method of calculating the angle of the adjustment hole using two points to determine a straight line is to calculate the slope of the straight line according to the coordinates of the two key points, and then find the angle of the straight line, that is, the angle of the adjustment hole. And the image acquisition module transmits the acquired image to the data enhancement submodule for data enhancement processing, and the data enhancement method includes rotation, cropping, flipping, adding Gaussian noise, adjusting HSV contrast and adjusting brightness.

[0030] Preferably, the depth detection module uses a binocular camera accurately calibrated by Zhang's calibration method or other calibration methods to improve the accuracy of depth detection.

[0031] Compared with the prior art, the automatic adjustment method of adjustable components based on machine vision of the present invention combines depth information with key point detection algorithms to achieve high-precision detection and angle calculation of key points of adjustment holes of adjustable circuit components. The automatic adjustment method of adjustable components based on machine vision of the present invention has at least the following beneficial effects:

[0032] 1. Combining depth information for detection improves the accuracy of key point detection of small-sized adjustable components and achieves high-precision key point detection: To address the difficulty in key point detection of small-sized electronic components, the present invention adopts a method of direct regression of key point coordinates to perform accurate key point detection on images after depth segmentation and target detection, thereby achieving high-precision positioning of the key points of the adjustment holes.

[0033] 2. Based on traditional two-dimensional target detection, the present invention innovatively introduces a depth segmentation method, uses a depth camera to obtain the depth information of the circuit, and achieves accurate segmentation of the image within the depth range of the adjustable hole of the adjustable component by setting the depth threshold range, thereby improving the accuracy and reliability of target detection.

[0034] 3. Use visual methods to calculate the angle of the adjustment hole, providing accurate angle information for automated adjustment.

[0035] 4. The whole process is automated, which improves the efficiency of circuit detection and adjustment and reduces production costs.

[0036] Based on the above high-precision detection results, the present invention can provide accurate adjustment hole angle information for actuators such as robotic arms, realize automatic adjustment of adjustable components, and greatly improve the efficiency and accuracy of circuit production correction.

[0037] The automatic adjustment method of the adjustable component based on machine vision of the present invention is further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A basic flow chart of the automatic adjustment method of adjustable components based on machine vision of the present invention;

[0039] Figure 2 It is a schematic diagram of key points of adjustable components;

[0040] Figure 3 The figure is a schematic diagram of the steps of reading the original image in the automatic adjustment method of the adjustable component based on machine vision of the present invention;

[0041] Figure 4 It is a schematic diagram of the depth map of the circuit after alignment of the automatic adjustment method of adjustable components based on machine vision of the present invention;

[0042] Figure 5 It is a schematic diagram of an image after depth segmentation of the automatic adjustment method of adjustable components based on machine vision of the present invention.

[0043] Among them, 1-the first key point 2-the second key point. DETAILED DESCRIPTION

[0044] like Figure 1The figure shows a flow chart of the automatic adjustment method of adjustable components based on machine vision of the present invention. The automatic adjustment method of adjustable components based on machine vision of the present invention is an automatic detection and adjustment process based on a depth camera. In this process, the system can automatically detect the adjustable components in the circuit and measure the angle of its adjustment hole, thereby improving the efficiency and accuracy of circuit production and detection. The present invention adds a depth segmentation method to the traditional two-dimensional target detection, uses the depth information to segment the image within the depth threshold range of the adjustable hole of the adjustable component, and then detects the adjustable components on the image after depth segmentation, uses the direct regression key point coordinate method to perform key point detection on the detected adjustable components, and uses the key point coordinates to calculate the angle of the adjustment hole.

[0045] The method of the present invention mainly includes: 1. Model building and training process; 2. Automatic detection process; 3. Adjustment process.

[0046] in:

[0047] 1. Model building and training process

[0048] Use the camera to collect the circuit images to be inspected, perform feature enhancement and annotation, and create a data set; then select a suitable target detection algorithm to train the prepared data set to obtain a target detection model that can detect adjustable components. Specifically include:

[0049] (1) Dataset creation

[0050] Capture images of the circuit to be identified, and collect circuit images under different lighting, angles, and other conditions to increase the diversity of the data set.

[0051] Data enhancement techniques such as rotation, cropping, flipping, adding Gaussian noise, adjusting HSV contrast, adjusting brightness, etc. are used to expand the dataset.

[0052] The processed images are annotated with target bounding boxes and key points to create a data set.

[0053] (2) Target detection algorithm selection and model training

[0054] According to actual needs, select a suitable target detection algorithm, such as the YOLO algorithm in the first-stage target detection algorithm or the Faster R-CNN in the second-stage target detection algorithm.

[0055] The target detection model is trained using the prepared data set so that the model can accurately identify the adjustable components in the circuit.

[0056] 2. Automatic detection process

[0057] It mainly uses a depth camera to obtain circuit images and converts the images into depth maps to obtain the depth information of the circuit; then the image is preprocessed, the depth range required for detection is set, the images within the depth range are retained, the remaining images are set to white, and the image after depth segmentation is output; the target detection model is used to perform adjustable component detection on the image after depth segmentation, and the image part within the detection frame is retained; the method of direct regression of key point coordinates is used to output the key point coordinates of the component image after depth segmentation and target detection.

[0058] The main steps may include:

[0059] (3) Circuit image acquisition and depth detection

[0060] A depth camera is used to obtain a circuit image, and the image is converted into a depth map to obtain the depth value of each pixel.

[0061] (4) Deep segmentation and object detection

[0062] Set the depth value range that matches the adjustable resistor adjustment hole to perform depth segmentation on the circuit image. Retain image information with depth values ​​within the threshold range and remove background or unnecessary interference.

[0063] The target detection algorithm is applied to scan the deep segmented image and identify the location of the adjustable components.

[0064] (5) Key point detection

[0065] In the retained target bounding box image, key point detection is performed. The key point coordinates of the detected adjustable components are directly regressed to obtain the precise position of the key point of the adjustment hole.

[0066] (6) Output of two-dimensional coordinates of key points and calculation of adjustment hole angles

[0067] Output the detected key point coordinates;

[0068] According to the output coordinates of the two first and second key points (1, 2), the slope of the straight line is calculated using the method of determining a straight line with two points, and then the angle of the straight line is calculated using the slope, which is the angle of the adjustment hole.

[0069] 3. Adjustment process

[0070] The adjustment hole angle is calculated based on the output key point coordinates, and the robot arm performs the adjustment. During the adjustment process, the robot arm can accurately adjust the adjustable components based on the key point coordinates and the adjustment hole angle information, thus achieving automatic adjustment.

[0071] Embodiment 1:

[0072] Automatic adjustment of adjustable resistors in an electronic circuit board.

[0073] In an electronic equipment production workshop, small adjustable resistors on a batch of electronic circuit boards need to be precisely adjusted to ensure optimal circuit performance.

[0074] Implementation steps:

[0075] 1. Model building and training process

[0076] (1) Image acquisition and enhancement: A high-resolution camera was used to capture images of various types of electronic circuit boards containing adjustable resistors. A total of 1,000 images of circuit boards at different angles and lighting conditions were collected. These images were subjected to data enhancement processing, including random rotation (angle between -30 degrees and 30 degrees), cropping (cropping off 10% to 20% of the image edge), horizontal flipping, adding Gaussian noise (the noise intensity was moderate and did not affect the main circuit features), adjusting HSV contrast (the contrast adjustment range was ±30%), and adjusting brightness (the brightness adjustment range was ±20%).

[0077] (2) Annotation and dataset production: Professional electronic engineers annotate the processed images with target bounding boxes and key points. For each adjustable resistor in the image, annotate its circumscribed rectangular box as the target bounding box, and annotate the two endpoints of the adjustment hole as key points. A dataset containing 1,000 annotated images is produced.

[0078] (3) Target detection model training: The YOLOv5 target detection algorithm is used to train the prepared data set. After multiple iterations of training and adjustment of hyperparameters such as the learning rate, a target detection model that can accurately detect adjustable resistors on electronic circuit boards is finally obtained.

[0079] 2. Automatic detection process

[0080] (1) Image acquisition and depth detection: Use a binocular depth camera to acquire images of electronic circuit boards and convert them into depth maps. Perform depth measurement on each pixel to obtain depth information of different parts of the circuit board. (See Figure 3-5 , respectively, the target original image of each step of image acquisition and processing, the circuit depth map after alignment, and the image after depth segmentation)

[0081] (2) Depth segmentation: According to the pre-measured depth range of the adjustable resistor adjustment hole, the threshold range of depth segmentation is set. The image is pre-processed, and the areas with depth values ​​exceeding the range are set to white in the image, and the image parts with depth values ​​within the threshold range are retained, and the image after depth segmentation is output. For example, if the depth range of the adjustable resistor adjustment hole is 10mm to 15mm, the areas with depth values ​​less than 10mm and greater than 15mm are set to white.

[0082] (3) Object detection: Use the trained object detection model to detect the adjustable resistor in the image after depth segmentation. If the adjustable resistor is detected, a bounding box is drawn for it and the image information within the bounding box is retained. If the adjustable resistor is not detected, the image acquisition and depth detection steps are repeated.

[0083] (4) Key point detection: In the retained bounding box image, the key point detection of the detected adjustable resistor is performed by directly regressing the key point coordinates. Through model prediction, the two-dimensional coordinates of the two endpoints of the adjustment hole are output.

[0084] 3. Adjustment process

[0085] (1) Angle calculation: Based on the output coordinates of the two key points, the slope of the straight line is calculated using the formula for determining a straight line from two points. Assuming that the coordinates of the two key points are (x1, y1) and (x2, y2), the slope k = (y2-y1) / (x2-x1). Then the inverse tangent function is used to calculate the angle of the straight line, that is, the adjustment hole angle θ = arctan(k).

[0086] (2) Automatic adjustment: The calculated adjustment hole angle information and key point coordinates are transmitted to the robot control system. The robot accurately locates the adjustment hole position of the adjustable resistor based on this information, and automatically adjusts the adjustable resistor according to the preset adjustment strategy so that the parameters of the adjustable resistor meet the design requirements.

[0087] After testing, in this embodiment, the automatic detection and precise adjustment of small adjustable resistors on electronic circuit boards are successfully achieved. The method greatly improves production efficiency, reduces errors caused by manual operation, and can adapt to different types of electronic circuit boards and adjustable resistors, with high versatility and reliability.

[0088] Example 2: Automatic adjustment of adjustable capacitance in a communication device

[0089] During the production process of communication equipment, the adjustable capacitor in the equipment needs to be accurately adjusted to optimize the frequency response of the communication signal.

[0090] Implementation steps:

[0091] 1. Model building and training process

[0092] (1) Image acquisition and enhancement: For the adjustable capacitors in communication equipment, a special industrial camera was used to collect 800 images of the internal parts of the communication equipment under different working conditions. These images were subjected to data enhancement processing, including vertical flipping, random cropping (cropping off 15% to 25% of the image edge), adding slight Gaussian noise, adjusting HSV contrast (contrast adjustment range is ±25%), and adjusting brightness (brightness adjustment range is ±15%).

[0093] (2) Annotation and dataset creation: Communication equipment engineers annotate the adjustable capacitors in the images, marking the two key points of the external rectangular frame and the adjustment hole. A dataset containing 800 annotated images is created.

[0094] (3) Target detection model training: The Faster R-CNN target detection algorithm is used to train the data set. After multiple training and optimization, a target detection model that can accurately detect adjustable capacitors in communication equipment is obtained.

[0095] 2. Automatic detection process

[0096] (1) Image acquisition and depth detection: Use a high-precision depth camera to acquire images of the inside of the communication device and generate a depth map. Measure the depth value of each pixel to obtain depth information of different parts inside the device.

[0097] (2) Depth segmentation: According to the depth characteristics of the adjustable capacitor adjustment hole, the threshold range of depth segmentation is set. The image is preprocessed to retain the image part with depth value within the threshold range and set the other parts to white.

[0098] (3) Object detection: Use the trained object detection model to detect the adjustable capacitance of the image after depth segmentation. If the adjustable capacitance is detected, the image information within its bounding box is retained.

[0099] (4) Key point detection: Perform key point detection in the bounding box image and output the two-dimensional coordinates of the two key points of the adjustment hole.

[0100] 3. Adjustment process

[0101] (1) Angle calculation: The angle of the adjustment hole is calculated according to the coordinates of the key points, and the method is the same as that of the first embodiment.

[0102] (2) Automatic adjustment: The robotic arm automatically adjusts the adjustable capacitor based on the angle information of the adjustment hole and the coordinates of the key points, so that the signal frequency response of the communication equipment reaches the optimal state.

[0103] This embodiment is successfully applied to the automatic adjustment of adjustable capacitors in communication equipment, improving the production quality and efficiency of communication equipment and reducing production costs. At the same time, this method also provides a reference for the automatic adjustment of other similar electronic components.

[0104] The embodiments described above are merely descriptions of preferred implementation modes of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for automatic adjustment of adjustable components based on machine vision, characterized in that: The following steps are involved: 1) Collect the circuit image to be identified; 2) Performing depth detection on the collected circuit images; 3) Perform depth segmentation on the image after depth detection according to the set depth value range, and retain the image part that matches the depth of the adjustable component; 4) Use the target detection algorithm to detect adjustable components in the image after depth segmentation and determine the position of the adjustable components; 5) Perform key point detection on the detected adjustable components; 6) Output the two-dimensional coordinates of key points; 7) Calculate the adjustment hole angle according to the two-dimensional coordinates of the key points; 8) The robotic arm automatically adjusts the adjustable components according to the key point coordinates and the adjustment hole angle information.

2. The method for automatic adjustment of adjustable components based on machine vision according to claim 1, characterized in that: The following steps are involved: Dataset creation: Image acquisition and data enhancement are performed on the circuit to be identified, and key points of the adjustable components and their adjustment holes to be detected are marked to form a data set; Model training: Select relevant target detection algorithms, use the data set for training, and establish a model that can identify key points of adjustable components and their adjustment holes.

3. The method for automatic adjustment of adjustable components based on machine vision according to claim 1, characterized in that: The data set preparation in step 1) also includes feature enhancement processing of the image, and the data enhancement methods include rotation, cropping, flipping, adding Gaussian noise, adjusting HSV contrast and adjusting brightness.

4. The method for automatic adjustment of adjustable components based on machine vision according to claim 1, characterized in that: The target detection algorithm in step 4) is selected from one or more of a one-stage target detection algorithm and a two-stage target detection algorithm, and the selection range includes YOLO, SSD, R-CNN and Faster R-CNN.

5. The method for automatic adjustment of adjustable components based on machine vision according to claim 1, characterized in that: The key point detection in step 5) adopts the method of directly regressing the key point coordinates and combines the depth information to improve the detection accuracy.

6. The method for automatic adjustment of adjustable components based on machine vision according to claim 1, characterized in that: The depth detection in step 2) is implemented using a binocular camera. When using a depth camera to obtain a circuit image, the camera is calibrated before use to ensure the alignment of the color image and the depth image so that the pixel coordinates of the color image and the depth image after depth segmentation are consistent.

7. The method for automatic adjustment of adjustable components based on machine vision according to claim 1, characterized in that: The adjustment hole angle calculation in step 7) uses the method of determining a straight line with two points, and calculates the slope of the straight line according to the coordinates of the two key points, and then finds the angle of the straight line, that is, the angle of the adjustment hole.

8. An automatic adjustment device for adjustable components based on machine vision, characterized in that: It includes image acquisition module, depth detection module, depth segmentation module, target detection module, key point detection module, angle calculation module and robotic arm control module, among which: The image acquisition module is used to acquire the circuit image to be identified; The depth detection module is used to perform depth detection on the collected circuit image; The depth segmentation module is used to perform depth segmentation on the image after depth detection according to the set depth value range, and retain the image part that matches the depth of the adjustable component; The target detection module is used to detect adjustable components on the image after depth segmentation using a target detection algorithm to determine the position of the adjustable components. The target detection algorithm includes but is not limited to the YOLO algorithm and the Faster RCNN algorithm, and is optimized to adapt to the detection of adjustable components. The key point detection module is used to detect key points of the detected adjustable components. The key point detection adopts the method of direct regression of key point coordinates and combines depth information to improve the detection accuracy. The angle calculation module is used to calculate the angle of the adjustment hole according to the two-dimensional coordinates of the key points, and uses mathematical methods such as high-precision inverse tangent function to calculate the angle; The robot control module is used to control the robot to automatically adjust the adjustable components according to the key point coordinates and the adjustment hole angle information.

9. The automatic adjustment device for adjustable components based on machine vision according to claim 8, characterized in that: The image acquisition module transmits the acquired image to the data enhancement submodule for data enhancement processing, and the data enhancement methods include rotation, cropping, flipping, adding Gaussian noise, adjusting HSV contrast and adjusting brightness.

10. The automatic adjustment device for adjustable components based on machine vision according to claim 8, characterized in that: The depth detection module uses a binocular camera accurately calibrated by Zhang's calibration method to improve the accuracy of depth detection.

Images