A machine vision-based door trim stitching position detection method and system
By using a machine vision-based detection method and deep learning algorithms and image processing technology, the problem of accurate detection of the stitching position on the trim panel of the car door was solved. It achieved fast and accurate measurement of the distance from the stitching to the black edge, adapts to different targets and complex scenarios, and has good generalization performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGSU RUNMO AUTOMOBILE TESTING EQUIP
- Filing Date
- 2023-06-29
- Publication Date
- 2026-07-14
Smart Images
Figure CN116580082B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automotive door trim panel seam position detection technology, specifically to a machine vision-based method and system for automotive door trim panel seam position detection. Background Technology
[0002] Machine vision measurement technology uses computer vision and image processing algorithms to accurately measure the size of objects. Specifically, it involves capturing images of objects using cameras or other sensing devices, then applying image processing algorithms to analyze and process the images, extracting key features and geometric information, and finally inferring the size and shape of the object.
[0003] Currently, after the production of the upper trim panel of a car door is completed, it is necessary to inspect the distance from each seam to the black edge on the upper trim panel. Existing inspection methods are generally difficult to accurately detect the distance from each seam to the black edge, which leads to the finished upper trim panel of the car door having unqualified seam positions. Moreover, the inspection process is also easily affected by the external environment, workpiece position and workpiece type. Therefore, it is necessary to design a machine vision-based method and system for detecting the seam position of upper trim panels of car doors. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of existing technologies. To better address the problem that existing inspection schemes generally struggle to accurately detect the distance from each seam to the black edge, leading to substandard seam positions on finished car door trim panels, and are also susceptible to influences from the external environment, workpiece position, and workpiece type during inspection, this invention provides a machine vision-based method and system for detecting seam positions on car door trim panels. This invention achieves the ability to quickly and accurately detect targets of multiple scales and sizes in images, adapts to different target categories and complex scenarios, exhibits good generalization performance, and also enables rapid measurement of the distance from the seam to the black edge with high accuracy.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] A machine vision-based method for detecting the position of stitching on a car door trim panel includes the following steps:
[0007] Step (A): Collect training images and create a dataset, then use the dataset to train a deep learning algorithm to complete the construction of the localization measurement model;
[0008] Step (B): Collect sample images and input them into the positioning and measurement model, and then use the positioning and measurement model to locate the measurement position;
[0009] Step (C): Crop the measured positions and output the cropped image;
[0010] Step (D): Convert the cropped image to grayscale and locate the seam positions, then separate the seam, then locate the black border area and obtain the area skeleton;
[0011] Step (E): Calculate the distance using sutures and the regional skeleton to obtain calculation data;
[0012] Step (F) involves determining the results of the calculation data and completing the inspection of the stitching position on the door trim panel.
[0013] The aforementioned machine vision-based method for detecting the position of seams on car door trim panels includes step (A), which involves acquiring training images and creating a dataset, then using the dataset to train a deep learning algorithm to complete the construction of a localization and measurement model. The specific steps are as follows.
[0014] Step (A1): Collect training images and create a dataset. The collected training images are clear images of the workpiece surface. 80% of the dataset is used as the training set and 20% as the validation set. The training set is used to provide training sample data and enable the deep learning algorithm to learn the features and shapes of different target categories. The validation set is used to test the performance of the deep learning algorithm.
[0015] Step (A2) involves training a deep learning algorithm using the dataset. The deep learning algorithm used is the YOLOv5s method, which consists of a backbone network, a feature pyramid, and a prediction head. The specific steps for training the deep learning algorithm using the dataset are as follows.
[0016] In step (A21), the images in the dataset are processed by the backbone network to extract image features and output three feature maps. The three feature maps are then processed by the feature pyramid to output three predictive feature layers.
[0017] Step (A22) involves generating anchor boxes of different sizes from three prediction feature layers via a prediction head, and using anchor boxes of different sizes to predict targets of different sizes.
[0018] Step (A23) outputs the confidence score, class score and predicted bounding box coordinates for each anchor box, and filters some anchor boxes based on the confidence score, discarding those with a confidence score below 0.5 and retaining the rest.
[0019] Step (A24) involves filtering the remaining anchor boxes using non-maximum suppression. The specific steps are as follows:
[0020] Step (A241), Non-maximum suppression, sorts the input anchor boxes according to their confidence scores, and then sorts them in descending order of their confidence scores.
[0021] Step (A242): Select the highest-scoring bounding box. Select the bounding box with the highest confidence score from the sorted bounding box list, set it as the best detection result, and then add it to the selection list.
[0022] Step (A243): Calculate the overlapping regions. For the remaining bounding boxes, calculate their overlapping regions with the highest-scoring bounding box.
[0023] Step (A244): Remove overlapping bounding boxes. For bounding boxes with an overlapping area greater than a preset threshold, remove them from the list and repeat the step until all bounding boxes have been processed.
[0024] Step (A245), Output: The output is a list of bounding boxes after nonmaximum suppression, where each bounding box represents a unique target.
[0025] The aforementioned machine vision-based method for detecting the position of seams on a car door trim panel includes step (B): acquiring sample images and inputting them into a positioning measurement model, then using the positioning measurement model to locate the measurement position. The positioning measurement model returns the upper left corner coordinates, lower right corner coordinates, confidence score, and category score of the seam position, while the upper left corner coordinates and lower right corner coordinates of the seam position are used to locate the measurement position.
[0026] The aforementioned machine vision-based method for detecting the seam position of a car door trim panel includes step (C), which involves cropping the located measurement position and outputting a cropped image. Specifically, the measurement position is cropped using a slicing method based on the returned coordinates of the upper left and lower right corners of the seam position.
[0027] The aforementioned machine vision-based method for detecting the position of seams on a car door trim panel includes step (D), which involves converting the cropped image to grayscale and locating the seam position, then separating the seam, locating the black border area, and obtaining the region skeleton. The specific steps are as follows.
[0028] Step (D1) converts the cropped image to grayscale. Grayscale conversion specifically involves converting a three-channel image into a single-channel grayscale image.
[0029] Step (D2) Locate the suture position. After the image is grayscaled, the suture position is highlighted in white, while other areas are gray or darker black areas. Locating the suture position is done by using threshold segmentation to obtain a brighter suture area with a grayscale value of 120-255.
[0030] Step (D3): Separate the sutures, use the opening operation to divide the area of each suture segment into an independent region, and then obtain the center point coordinates of each region.
[0031] Step (D4): Locate the black border area. The specific steps are as follows:
[0032] Step (D41): After cropping the image and converting it to grayscale, the black edge position will be in a black or gray state with a low grayscale value. Then, use threshold segmentation to obtain the area with a grayscale value of 0-40.
[0033] Step (D42): Use opening operation to remove noise, and then use closing operation to close unconnected regions;
[0034] Step (D43): Make a judgment on the closing operation. If the region of the connected component is equal to 1, continue to the next step. If the region of the connected component is greater than 1, increase the convolution kernel of the closing operation and continue to perform the closing operation until only one connected component remains.
[0035] Step (D5) involves obtaining the region skeleton, specifically by using the thinning() operator of the opencv extension module ximgproc in the Zhang-Suen algorithm to thin the image region and obtain the region skeleton.
[0036] The aforementioned machine vision-based method for detecting the position of stitching on a car door trim panel includes step (E), which involves calculating the distance using the stitching and the region skeleton to obtain calculation data. Specifically, the distance calculation is performed by using the coordinates of the midline point of each stitching segment to calculate the distance from each point to the black edge region skeleton, and the resulting calculation data is the distance from the stitching to the black edge.
[0037] The aforementioned machine vision-based method for detecting the position of seams on a car door trim panel includes step (F), which involves determining the results of the calculated data to complete the detection of the seam position on the car door trim panel. Specifically, an upper and lower limit is set according to the requirements, and then it is determined whether the distance from each seam segment to the black edge is within the set range. If each distance is within the set range, ok is returned; otherwise, ng is returned.
[0038] A machine vision-based system for detecting the position of seams on a car door trim panel includes a model building module, a model localization module, a position cropping module, a morphological analysis module, a distance calculation module, and a data judgment module. The model building module collects training images and creates a dataset, then uses the dataset to train a deep learning algorithm to construct a localization measurement model. The model localization module collects sample images and inputs them into the localization measurement model, then uses the model to locate the measurement position. The position cropping module crops the located measurement position and outputs a cropped image. The morphological analysis module converts the cropped image to grayscale, locates the seam position, separates the seam, locates the black border area, and obtains the region skeleton. The distance calculation module calculates the distance using the seam and the region skeleton to obtain calculated data. The data judgment module judges the calculated data to complete the detection of the seam position on the car door trim panel.
[0039] The beneficial effects of this invention are as follows: This invention provides a machine vision-based method and system for detecting the position of seams on a car door trim panel. First, training images are collected and compiled into a dataset. The training set within this dataset provides sample data, which helps the deep learning algorithm learn the features and shapes of different target categories, thereby improving the accuracy and robustness of the localization measurement model. Then, the performance of the deep learning algorithm is tested using a validation set. Finally, the deep learning algorithm employs the YOLOv5s method, composed of a backbone network, feature pyramid, and prediction head, to quickly and accurately detect targets of multiple scales and sizes in images. Furthermore, the YOLOv5s method can adapt to different target categories. This method is suitable for complex scenarios and exhibits good generalization performance. Based on the top-left and bottom-right coordinates of the seam position returned by the positioning measurement model, the measurement position is cropped using a slicing method. The cropped image is then converted to grayscale, and the seam position is located. The seam is then separated, and the black border area is located. The region skeleton is then obtained, and finally, the distance is calculated using the seam and the region skeleton. The calculated data is then evaluated to complete the detection of the seam position on the car door trim panel. This effectively realizes the function of quickly measuring the distance from the seam to the black border, with high accuracy. It boasts advantages such as a scientifically sound method, strong applicability, and excellent results. Attached Figure Description
[0040] Figure 1 This is an overall flowchart of the present invention;
[0041] Figure 2 This is a flowchart illustrating the specific process of locating the black border area in this invention;
[0042] Figure 3 This is a diagram showing the positioning measurement results of the positioning measurement model of the present invention;
[0043] Figure 4 This is a schematic diagram illustrating the separation of the suture and regional skeleton in this invention. Detailed Implementation
[0044] The present invention will now be further described with reference to the accompanying drawings.
[0045] like Figure 1 As shown, the present invention provides a machine vision-based method and system for detecting the position of seams on a car door trim panel, comprising the following steps:
[0046] Step (A) involves collecting training images and creating a dataset, then using the dataset to train a deep learning algorithm to complete the construction of the localization measurement model. The specific steps are as follows:
[0047] Step (A1): Collect training images and create a dataset. The collected training images are clear images of the workpiece surface. 80% of the dataset is used as the training set and 20% as the validation set. The training set is used to provide training sample data and enable the deep learning algorithm to learn the features and shapes of different target categories. The validation set is used to test the performance of the deep learning algorithm.
[0048] Specifically, 500 training images are collected. The training set in the dataset provides sample data, which helps the deep learning algorithm learn the features and shapes of different target categories, thereby improving the accuracy and robustness of the localization and measurement model. The performance of the deep learning algorithm can then be tested through the validation set.
[0049] Step (A2) involves training a deep learning algorithm using the dataset. The deep learning algorithm used is the YOLOv5s method, which consists of a backbone network, a feature pyramid, and a prediction head. The specific steps for training the deep learning algorithm using the dataset are as follows.
[0050] In step (A21), the images in the dataset are processed by the backbone network to extract image features and output three feature maps. The three feature maps are then processed by the feature pyramid to output three predictive feature layers.
[0051] Step (A22) involves generating anchor boxes of different sizes from three prediction feature layers via a prediction head, and using anchor boxes of different sizes to predict targets of different sizes.
[0052] Step (A23) outputs the confidence score, class score and predicted bounding box coordinates for each anchor box, and filters some anchor boxes based on the confidence score, discarding those with a confidence score below 0.5 and retaining the rest.
[0053] Step (A24) involves filtering the remaining anchor boxes using non-maximum suppression. The specific steps are as follows:
[0054] Step (A241), Non-maximum suppression, sorts the input anchor boxes according to their confidence scores, and then sorts them in descending order of their confidence scores.
[0055] Step (A242): Select the highest-scoring bounding box. Select the bounding box with the highest confidence score from the sorted bounding box list, set it as the best detection result, and then add it to the selection list.
[0056] Step (A243): Calculate the overlapping regions. For the remaining bounding boxes, calculate their overlapping regions with the highest-scoring bounding box.
[0057] Step (A244): Remove overlapping bounding boxes. For bounding boxes with an overlapping area greater than a preset threshold, remove them from the list and repeat the step until all bounding boxes have been processed.
[0058] Step (A245), Output: The output is a list of bounding boxes after nonmaximum suppression, where each bounding box represents a unique target.
[0059] Among them, the YOLOV5s method, which uses deep learning algorithms and consists of a backbone network, a feature pyramid, and a prediction head, can quickly and accurately detect targets of different scales and sizes in images. Furthermore, the YOLOV5s method can adapt to different target categories and complex scenes, and has good generalization performance.
[0060] like Figure 3 As shown, in step (B), sample images are collected and input into the positioning measurement model, and then the positioning measurement model is used to locate the measurement position. The positioning measurement model will return the upper left corner coordinates, lower right corner coordinates, confidence score and category score of the suture position, while the upper left corner coordinates and lower right corner coordinates of the suture position are used to locate the measurement position.
[0061] Step (C) involves cropping the located measurement position and outputting the cropped image. Specifically, the measurement position is cropped using a slicing method based on the returned coordinates of the upper left and lower right corners of the seam position.
[0062] Step (D) involves converting the cropped image to grayscale and locating the seam positions, then separating the seam. Next, the black border area is located, and the region skeleton is obtained. The specific steps are as follows:
[0063] Step (D1) converts the cropped image to grayscale. Grayscale conversion specifically involves converting a three-channel image into a single-channel grayscale image.
[0064] Step (D2) Locate the suture position. After the image is grayscaled, the suture position is highlighted in white, while other areas are gray or darker black areas. Locating the suture position is done by using threshold segmentation to obtain a brighter suture area with a grayscale value of 120-255.
[0065] Step (D3): Separate the sutures, use the opening operation to divide the area of each suture segment into an independent region, and then obtain the center point coordinates of each region.
[0066] like Figure 2 As shown, step (D4) involves locating the black border area. The specific steps are as follows:
[0067] Step (D41): After cropping the image and converting it to grayscale, the black edge position will be in a black or gray state with a low grayscale value. Then, use threshold segmentation to obtain the area with a grayscale value of 0-40.
[0068] Step (D42): Use opening operation to remove noise, and then use closing operation to close unconnected regions;
[0069] Step (D43): Make a judgment on the closing operation. If the region of the connected component is equal to 1, continue to the next step. If the region of the connected component is greater than 1, increase the convolution kernel of the closing operation and continue to perform the closing operation until only one connected component remains.
[0070] Step (D5) involves obtaining the region skeleton, specifically by using the thinning() operator of the opencv extension module ximgproc in the Zhang-Suen algorithm to thin the image region and obtain the region skeleton.
[0071] like Figure 4 As shown, in step (E), the distance is calculated using the stitches and the area skeleton to obtain the calculated data. Specifically, the distance is calculated by using the coordinates of the midline points of each stitch to calculate the distance from each point to the black edge area skeleton. The calculated data is the distance from the stitch to the black edge.
[0072] Step (F) involves judging the calculated data and completing the inspection of the stitching position on the door trim panel. Specifically, an upper and lower limit is set according to the requirements, and then it is determined whether the distance from each stitching segment to the black edge is within the set range. If each distance is within the set range, ok is returned; otherwise, ng is returned.
[0073] A machine vision-based system for detecting the position of seams on a car door trim panel includes a model building module, a model localization module, a position cropping module, a morphological analysis module, a distance calculation module, and a data judgment module. The model building module collects training images and creates a dataset, then uses the dataset to train a deep learning algorithm to construct a localization measurement model. The model localization module collects sample images and inputs them into the localization measurement model, then uses the model to locate the measurement position. The position cropping module crops the located measurement position and outputs a cropped image. The morphological analysis module converts the cropped image to grayscale, locates the seam position, separates the seam, locates the black border area, and obtains the region skeleton. The distance calculation module calculates the distance using the seam and the region skeleton to obtain calculated data. The data judgment module judges the calculated data to complete the detection of the seam position on the car door trim panel.
[0074] In summary, the present invention provides a machine vision-based method and system for detecting the position of seams on a car door trim panel. First, training images are collected and compiled into a dataset. Then, a deep learning algorithm is trained using this dataset, effectively constructing a localization and measurement model. Next, sample images are collected and input into the localization and measurement model, which is then used to locate the measurement position. The located measurement position is then cropped and output as a cropped image. The cropped image is then converted to grayscale, and the seam position is located. The seam is then separated, and the black border region is located. A region skeleton is then obtained. Finally, the distance between the seam and the region skeleton is calculated, and the calculated data is obtained. The results are then evaluated, effectively completing the detection of the seam position on the car door trim panel. This invention achieves the ability to quickly and accurately detect targets of multiple scales and sizes in images, adapts to different target categories and complex scenes, exhibits good generalization performance, and also enables rapid measurement of the distance from the seam to the black border with high detection accuracy.
[0075] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. A machine vision-based method for detecting the position of stitching on a car door trim panel, characterized in that: Includes the following steps, Step (A): Collect training images and create a dataset, then use the dataset to train a deep learning algorithm to complete the construction of the localization measurement model; Step (B): Collect sample images and input them into the positioning measurement model. Then, use the positioning measurement model to locate the measurement position. The positioning measurement model will return the upper left corner coordinates, lower right corner coordinates, confidence score and category score of the suture position. The upper left corner coordinates and lower right corner coordinates of the suture position are used to locate the measurement position. Step (C): Crop the measured positions and output the cropped image; Step (D) involves converting the cropped image to grayscale and locating the seam positions, then separating the seam. Next, the black border area is located, and the skeleton of the black border area is obtained. The specific steps are as follows: Step (D1) converts the cropped image to grayscale. Grayscale conversion specifically involves converting a three-channel image into a single-channel grayscale image. Step (D2) locate the suture position. After the image is grayscaled, the suture position is highlighted in white, while other areas are gray or black. Locating the suture position is done by using threshold segmentation to obtain a brighter suture area with a grayscale value of 120-255. Step (D3): Separate the sutures, use the opening operation to divide the area of each suture segment into an independent area, and then obtain the center point coordinates of the area of each suture segment. Step (D4): Locate the black border area. The specific steps are as follows: Step (D41): After cropping the image and converting it to grayscale, the black edge position will be in a black or gray state with a low grayscale value. Then, use threshold segmentation to obtain the area with a grayscale value of 0-40. Step (D42): Use opening operation to remove noise, and then use closing operation to close unconnected regions; Step (D43): Make a judgment on the closing operation. If the region of the connected component is equal to 1, continue to the next step. If the region of the connected component is greater than 1, increase the convolution kernel of the closing operation and continue to perform the closing operation until only one connected component remains. Step (D5) Obtain the skeleton of the black border region, specifically by using the thinning() operator of the opencv extension module ximgproc in the Zhang-Suen algorithm to refine the black border region and obtain the skeleton of the black border region. Step (E): Calculate the distance using the stitches and the skeleton of the black edge area to obtain calculation data. Specifically, the distance calculation involves calculating the distance from the coordinates of the center point of each stitch to the skeleton of the black edge area, and using the obtained distance as the calculation data. Step (F) involves judging the calculated data and completing the detection of the stitching position on the door trim panel. Specifically, an upper and lower limit is set according to the requirements, and then it is determined whether the distance from the center point coordinate of each stitching segment to the skeleton of the black edge area is within the set upper and lower limit range. If each distance is within the set upper and lower limit range, ok is returned; otherwise, ng is returned.
2. The method for detecting the position of stitching on a car door trim panel based on machine vision according to claim 1, characterized in that: Step (A) involves collecting training images and creating a dataset, then using the dataset to train a deep learning algorithm to complete the construction of the localization measurement model. The specific steps are as follows: Step (A1): Collect training images and create a dataset. The collected training images are clear images of the surface of the trim panel on the car door. 80% of the dataset is used as the training set and 20% as the validation set. The training set is used to provide training sample data and enable the deep learning algorithm to learn the features and shapes of different target categories. The validation set is used to test the performance of the deep learning algorithm. Step (A2) uses the dataset to train a deep learning algorithm, which uses the YOLOv5s method, and the YOLOv5s method is composed of a backbone network, a feature pyramid and a prediction head.
3. The method for detecting the position of seam on a car door trim panel based on machine vision according to claim 2, characterized in that: Step (C) involves cropping the located measurement position and outputting the cropped image. Specifically, the measurement position is cropped using a slicing method based on the returned coordinates of the upper left and lower right corners of the seam position.
4. A machine vision-based system for detecting the position of stitching on a car door trim panel, wherein the system employs the method described in any one of claims 1-3, characterized in that: It includes a model building module, a model localization module, a location cropping module, a morphological analysis module, a distance calculation module, and a data determination module. The model building module is used to collect training images and create a dataset, and then use the dataset to train a deep learning algorithm to complete the construction of the localization measurement model. The model positioning module is used to collect sample images and input them into the positioning measurement model, and then use the positioning measurement model to locate the measurement position; The location cropping module is used to crop the located measurement position and output the cropped image; The morphological analysis module is used to convert the cropped image to grayscale and locate the seam position, then separate the seam, then locate the black border area and obtain the skeleton of the black border area. The distance calculation module is used to calculate distances using the stitching and black border area skeleton to obtain calculation data; The data determination module is used to determine the results of the calculation data and complete the detection of the stitching position on the door trim panel.