Online visual detection method and system for virtual plugging of television plug connector

Abstract

The invention relates to the technical field of automatic detection in television production, and discloses an online visual detection method and system for virtual plugging of a television plug connector. The method comprises the following steps: firstly, acquiring an image of a television to be detected on a television production line in real time, and preprocessing to obtain a spliced image; then performing template matching on the spliced image to obtain a circuit board ROI region, and positioning a connector ROI region in the circuit board ROI region; and classifying the image of the ROI area of the plug connector through a pre-trained classification model so as to judge whether the plug connector exists in the ROI area of the plug connector, if so, carrying out semantic segmentation on the image of the ROI area of the plug connector by adopting a pre-trained segmentation model, and judging whether the plug connector is in virtual plug connection according to a semantic segmentation result. According to the invention, virtual plugging of the television plug connector can be detected in real time, the labor cost is effectively reduced, the error and omission ratio is reduced, the damage caused by manual product contact is reduced, and the production line detection efficiency is greatly improved at the same time.

Description

Technical Field

[0001] The invention relates to the technical field of automated detection in television production, in particular to an online visual detection method and system for virtual plugging of television connectors. Background Art

[0002] With the continuous advancement of television technology and the continuous expansion of production scale, the degree of automation and detection accuracy of television production lines have become key factors in improving production efficiency and product quality. In the traditional television production process, whether the connectors are correctly plugged in is a crucial quality control link. At present, the common detection method relies on manual judgment by pulling the free end of the wire harness. This method is not only inefficient, but also overly dependent on manual experience, resulting in difficulty in ensuring detection accuracy. In addition, some existing automated detection equipment has limitations in its ability to detect virtual plug-ins of connectors. In the actual production process, workers often block the camera, resulting in inaccurate detection results, which needs to be solved urgently. Summary of the invention

[0003] In order to solve the technical problems existing in the prior art, the present invention provides an online visual detection method and system for virtual connection of television connectors. The present invention can effectively reduce labor costs, reduce the false detection rate, and greatly improve the production line detection efficiency.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] The present invention discloses an online visual detection method for virtual plug-in of a television connector, comprising:

[0006] S1. Real-time acquisition of images of the TV to be tested on the TV production line, and obtaining a spliced ​​image after preprocessing;

[0007] S2. Performing template matching on the stitched image to obtain a circuit board ROI area, and locating a connector ROI area in the circuit board ROI area;

[0008] S3. Classifying the image of the connector ROI region by a pre-trained classification model to determine whether there is a connector in the connector ROI region, and if so, executing step S4;

[0009] S4. Using a pre-trained segmentation model to perform semantic segmentation on the image of the connector ROI area, and judging whether the connector is a virtual connector according to the semantic segmentation result.

[0010] As a further improvement of the above scheme, in step S1, N visual detection sensors installed directly above the TV production line are used to collect images, N>1; wherein the N visual detection sensors are arranged in a straight line at equal intervals, and the arrangement direction is perpendicular to the transmission direction of the TV to be detected, that is, the length direction of the TV, and the total imaging width is greater than the production line width, and there is a set imaging overlap area between adjacent visual detection sensors.

[0011] As a further improvement of the above solution, in step S1, the preprocessing includes the following specific processes:

[0012] S11. performing distortion correction on the image collected by each visual detection sensor to eliminate the geometric distortion caused by the lens;

[0013] S12. Establishing a transformation matrix required for stitching based on the overlapping imaging area, thereby stitching the images collected by all visual detection sensors;

[0014] S13. Use a lookup table method to adjust the contrast and brightness of the image spliced ​​in step S12.

[0015] As a further improvement of the above scheme, in step S2, a predefined template is used to match the stitched image, and the corresponding circuit board ROI area in the stitched image is obtained by edge extraction and shape feature comparison; wherein, the generation process of the predefined template includes: according to the TV model, a custom circuit board ROI area is drawn through a software graphical interface, calibration is performed according to the provided image sample, and a circuit board area image in the image sample is generated to create a template for the corresponding circuit board, and then a custom connector ROI area is drawn in the circuit board area image to determine the specific position and shape of each connector on the circuit board, thereby creating the predefined template.

[0016] As a further improvement of the above scheme, in step S3, the classification model adopts the YOLOv8 model, and the training process includes: batch collecting image samples of connectors, dividing the image samples into two categories and adding corresponding category labels, one category indicating that there are connectors that meet the requirements in the sample, and the other category is the opposite; using the classified image samples to train and test the YOLOv8 model to generate a classification model that can classify images in the connector ROI area.

[0017] As a further improvement of the above scheme, in step S4, the segmentation model adopts the YOLOv8 model, and the training process includes: batch collecting image samples of connectors, marking the boundaries of connectors in the image samples, training and testing the YOLOv8 model using the labeled image samples, and generating a segmentation model that can segment the connectors in the image of the connector ROI area.

[0018] As a further improvement of the above scheme, in step S4, the segmentation model is used to perform pixel-level classification on the image of the connector ROI area, and learn and distinguish the fine-grained features of the image of the connector ROI area by performing multi-level feature extraction and fusion on the image, and generate a semantic label corresponding to each pixel based on global and local information, thereby segmenting the connectors in the connector ROI area.

[0019] As a further improvement of the above solution, in step S4, judging whether the connector is virtually plugged in according to the semantic segmentation result includes:

[0020] Through the semantic segmentation results, the shape of each connector is analyzed to obtain the length and width dimensions of the connector, and compared with the preset standard length and width dimensions. If the dimensions are within the preset error range, the connector is judged to be normal, otherwise the connector is judged to be a virtual connection.

[0021] As a further improvement of the above solution, in step S4, if it is determined that the connector is a virtual connection, an alarm signal is sent to an interactive end, and the virtual connection event is recorded in the detection log.

[0022] The invention also discloses an online visual detection system for virtual plug-in of a television plug-in, which applies the detection method described above; the detection system comprises an image acquisition module and a data processing module.

[0023] The image acquisition module is used to acquire images of the TV to be inspected on the TV production line in real time;

[0024] The data processing module is used to pre-process the image captured by the image acquisition module to obtain a stitched image; perform template matching on the stitched image to obtain the circuit board ROI area, and locate the connector ROI area in the circuit board ROI area; classify the image of the connector ROI area through a pre-trained classification model to determine whether there is a connector in the connector ROI area, and if so, perform semantic segmentation on the image of the connector ROI area using a pre-trained segmentation model, and determine whether the connector is a virtual connection based on the semantic segmentation result.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The present invention can detect the virtual connection of TV connectors in real time without changing the TV production line. It has significant advantages over traditional manual detection and can effectively reduce labor costs, reduce the rate of false detection and missed detection, and reduce damage caused by manual contact with products, while greatly improving the production line detection efficiency.

[0027] 2. The visual inspection sensor in the present invention can cover the entire width of the production line and detect the TV in sections along the length direction. The entire inspection process only requires a simple drawing of the ROI area in advance, and can be directly applied, and is not limited by factors such as circuit boards, connector models, and position directions. At the same time, it can avoid false detections caused by occlusion during on-site operation.

[0028] 3. The present invention directly locates small target connectors through a large-area field of view and measures their sizes to convert them into priority positioning of circuit board positions, and then locates the connectors based on their relative positions on various circuit boards and then divides and measures their sizes, which greatly improves the positioning accuracy of small target connectors and improves the detection performance of virtual plug-ins of connectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The flowchart of the online visual detection method of virtual plug-in of the TV connector in embodiment 1 of the present invention.

[0030] Figure 2 This is a logic block diagram of the online visual detection method in Example 1 of the present invention.

[0031] Figure 3 It is a top view schematic diagram of multiple visual detection sensors arranged above a television production line in Example 1 of the present invention.

[0032] Figure 4 for Figure 3 Schematic diagram of the front view of the viewing range of multiple visual detection sensors located above the TV production line.

[0033] Figure 5 This is a schematic diagram of the process of configuring the ROI region in Embodiment 1 of the present invention.

[0034] In the figure: 1. Camera; 2. Imaging overlap area; 3. TV to be tested; 4. Imaging lens; 5. Ring light source; 6. TV internal circuit board structure; 7. Circuit board ROI area projection; 8. Connector structure; 9. Connector ROI area projection. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Example 1

[0037] See also Figure 1 and Figure 2This embodiment provides an online visual detection method for virtual plug-in of a television connector, including steps S1 to S4.

[0038] S1. The images of the TV to be inspected on the TV production line are collected in real time, and a spliced ​​image is obtained after preprocessing.

[0039] See also Figure 3 and Figure 4 In this embodiment, three visual detection sensors are installed directly above the TV production line to form an array, facing directly below for image acquisition; wherein, each visual detection sensor is arranged in a straight line with equal distances, and the arrangement direction is perpendicular to the transmission direction of the TV 3 to be detected, that is, the length direction of the TV, and the total imaging width is greater than the production line width, and there is a set imaging overlap area 2 between adjacent visual detection sensors 1. As the TV moves, the camera continuously shoots, and there is a certain overlap area between the front and rear frames to ensure that there is no missed shooting area of ​​the TV.

[0040] Each visual detection sensor includes a camera 1 , an imaging lens 4 , and an annular light source 5 . The annular light source 5 can illuminate the television 3 to be detected. Each annular light source 5 is installed at the front end of the camera 1 and is adapted to the position of the imaging lens 4 .

[0041] The pretreatment includes the following specific processes:

[0042] S11. Perform distortion correction on the images collected by each visual detection sensor. In a multi-camera system, distortion correction can improve the quality and accuracy of the stitched image. The process includes camera intrinsic parameter calibration and distortion model estimation to eliminate the geometric distortion caused by the lens.

[0043] S12. Establishing a transformation matrix required for stitching based on the imaging overlap area 2, thereby stitching images collected by all visual detection sensors.

[0044] S13. Use a lookup table method (LUT method) to adjust the contrast and brightness of the image spliced ​​in step S12.

[0045] S2. Perform template matching on the stitched image to obtain a circuit board ROI (Region of Interest) area, and locate a connector ROI area in the circuit board ROI area.

[0046] In step S2, a predefined template is used to match the stitched image, and the corresponding circuit board ROI area in the stitched image is obtained by edge extraction and shape feature comparison.

[0047] Before executing the method of the present invention, ROI area configuration is required to ensure accurate detection according to the requirements of specific TV models and production lines. The configuration process is as follows:

[0048] (1) Draw the ROI area of ​​the TV circuit board: Figure 5 As shown, first, according to the TV model, the approximate area of ​​the circuit board is drawn through the software graphical interface (that is, the circuit board ROI area projection 7, the circuit board structure 6 can be completely covered by the area and leave some space), and calibration is performed according to the provided image sample to determine the area of ​​the circuit board in the image sample, and form an image of the circuit board area in the image sample (that is, generate a circuit board template).

[0049] (2) Draw the ROI area of ​​the connector: Figure 5 As shown, after the circuit board area is demarcated, the specific area of ​​the connector on the circuit board needs to be further demarcated. The specific position and shape of each connector on the circuit board can be clarified by drawing a rectangular frame of the connector area or a polygonal frame of other shapes in the circuit board area image. Figure 5 In the figure, the range of the connector ROI area projection 9 is larger than the connector structure 8 .

[0050] This step lays the foundation for subsequent template matching and image analysis. Through these area calibration information, the connector area can be accurately identified during the detection process, thereby making further judgments and analyses.

[0051] Under the premise that the template configuration is completed, step S2 can perform automatic detection. The image processing process mainly includes the following steps:

[0052] (1) Template matching: Based on traditional image processing technology, the template matching algorithm is first used. According to the circuit board template calibrated in the above configuration process, the standard template matching method is adopted to calculate the matching degree by comparing the known circuit board template and the real-time image, so as to accurately identify the exact position of the circuit board.

[0053] (2) Determine the ROI area: Determine the circuit board area information based on the template matching results and the ROI area of ​​the connector, and further lock the specific position of the connector on the circuit board.

[0054] In the traditional TV production process, whether the connector is plugged in properly represents whether the internal circuit of the TV can be connected, so it is a key quality control link. However, the TV models at the TV production site are different, and their internal circuit board combinations and positions are different. At the same time, whether the interfaces on the same model of circuit board are plugged in and whether they are plugged in properly brings great difficulties to the on-site connector size detection; the present invention can draw the ROI area according to the TV model and obtain all the information of the model for subsequent detection.

[0055] S3. Classify the image of the connector ROI region using a pre-trained classification model to determine whether there is a connector in the connector ROI region, and if so, execute step S4.

[0056] In the actual production process, workers often block the camera. Therefore, before semantic segmentation, it is necessary to classify the image in the connector ROI area to determine whether it is a connector.

[0057] In step S3, the classification model adopts the YOLOv8 model, and the training process includes: batch collecting image samples of the connectors, dividing the image samples into two categories and adding corresponding category labels, one category indicating that there are connectors that meet the requirements in the sample, and the other category is the opposite; using the classified image samples to train and test the YOLOv8 model to generate a classification model that can classify images in the connector ROI area.

[0058] S4. Using a pre-trained segmentation model to perform semantic segmentation on the image of the connector ROI area, and judging whether the connector is a virtual connector according to the semantic segmentation result.

[0059] After locating the connector area, the connectors on the circuit board are further analyzed through deep learning and semantic segmentation technology.

[0060] In step S4, the segmentation model also uses the YOLOv8 model, but the application direction is different from that of the aforementioned S3. The training process of the segmentation model includes: batch collecting image samples of the connector, marking the connector boundaries in the image samples, training and testing the YOLOv8 model using the labeled image samples, and generating a segmentation model that can segment the connector in the image of the connector ROI area.

[0061] It should be noted that the above method for marking the boundaries of connectors in image samples is: using the labelme tool, drawing polygons according to the edges of connectors, so that the polygons can cover the connector area more accurately, and making the coordinates of each corner point of the polygon into corresponding labels, which can be used for model training.

[0062] The segmentation model can separate each connector on the circuit board from the background and accurately identify the boundaries of the connector. This process uses a convolutional neural network to perform pixel-level classification on the image of the connector ROI area, thereby separating the connector on the circuit board from other background parts. By performing multi-level feature extraction and fusion on the image, the fine-grained features of the image in the connector ROI area are learned and distinguished, and the semantic label corresponding to each pixel is generated based on global and local information. Even in complex scenes and backgrounds, and in the case of multiple changes in the appearance, position, shape, angle, etc. of the connectors on the circuit board, changes in lighting or different viewing angles, the connectors can still be effectively identified and separated with high accuracy.

[0063] The step of judging whether the connector is virtually plugged in according to the semantic segmentation result includes:

[0064] Through the semantic segmentation results, the shape of each connector is analyzed, and the edge coordinates of the connector in the segmentation results are calculated to obtain the length and width dimensions of the connector, and compared with the preset standard length and width dimensions. If the dimensions are within the preset error range, the connector is judged to be normal, otherwise the connector is judged to be a virtual connection.

[0065] If the connector is determined to be a virtual connection, an alarm signal is sent to an interactive end, and the virtual connection event is recorded in a detection log.

[0066] Example 2

[0067] This embodiment provides an online visual detection system for virtual plug-in of a television connector, and applies the detection method in Embodiment 1; the detection system includes: an image acquisition module and a data processing module.

[0068] The image acquisition module is used to acquire images of the TV to be inspected on the TV production line in real time;

[0069] The data processing module is used to pre-process the image captured by the image acquisition module to obtain a stitched image; perform template matching on the stitched image to obtain the circuit board ROI area, and locate the connector ROI area in the circuit board ROI area; classify the image of the connector ROI area through a pre-trained classification model to determine whether there is a connector in the connector ROI area, and if so, perform semantic segmentation on the image of the connector ROI area using a pre-trained segmentation model, and determine whether the connector is a virtual connection based on the semantic segmentation result.

[0070] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An online visual detection method for virtual plug-in of a TV connector, characterized in that: include: S1. Real-time acquisition of images of the TV to be tested on the TV production line, and obtaining a spliced ​​image after preprocessing; S2. Performing template matching on the stitched image to obtain a circuit board ROI area, and locating a connector ROI area in the circuit board ROI area; S3. Classifying the image of the connector ROI region by a pre-trained classification model to determine whether there is a connector in the connector ROI region, and if so, executing step S4; S4. Using a pre-trained segmentation model to perform semantic segmentation on the image of the connector ROI area, and judging whether the connector is a virtual connector according to the semantic segmentation result.

2. The online visual detection method for virtual plug-in of a TV connector according to claim 1, characterized in that: In step S1, N visual detection sensors installed directly above the TV production line are used to collect images, where N>1; wherein the N visual detection sensors are arranged in a straight line at equal intervals, and the arrangement direction is perpendicular to the transmission direction of the TV to be detected, i.e., the length direction of the TV, and the total imaging width is greater than the production line width, and there is a set imaging overlap area between adjacent visual detection sensors.

3. The online visual detection method for virtual plug-in of a TV connector according to claim 2, characterized in that: In step S1, the preprocessing includes the following specific processes: S11. performing distortion correction on the image collected by each visual detection sensor to eliminate the geometric distortion caused by the lens; S12. Establishing a transformation matrix required for stitching based on the overlapping imaging area, thereby stitching the images collected by all visual detection sensors; S13. Use a lookup table method to adjust the contrast and brightness of the image spliced ​​in step S12.

4. The online visual detection method for virtual plug-in of a TV connector according to claim 1, characterized in that: In step S2, a predefined template is used to match the stitched image, and the corresponding circuit board ROI area in the stitched image is obtained by edge extraction and shape feature comparison; wherein, the generation process of the predefined template includes: according to the TV model, a custom circuit board ROI area is drawn through a software graphical interface, calibration is performed according to the provided image sample, and a circuit board area image in the image sample is generated to create a template for the corresponding circuit board, and then a custom connector ROI area is drawn in the circuit board area image to determine the specific position and shape of each connector on the circuit board, thereby creating the predefined template.

5. The online visual detection method for virtual plug-in of a TV connector according to claim 1, characterized in that: In step S3, the classification model adopts the YOLOv8 model, and the training process includes: batch collecting image samples of the connectors, dividing the image samples into two categories and adding corresponding category labels, one category indicating that there are connectors that meet the requirements in the sample, and the other category is the opposite; using the classified image samples to train and test the YOLOv8 model to generate a classification model that can classify images in the connector ROI area.

6. The online visual detection method for virtual plug-in of a TV connector according to claim 1, characterized in that: In step S4, the segmentation model adopts the YOLOv8 model, and the training process includes: batch collecting image samples of the connector, marking the connector boundaries in the image samples, training and testing the YOLOv8 model using the labeled image samples, and generating a segmentation model that can segment the connector in the image of the connector ROI area.

7. The online visual detection method for virtual plug-in of a TV connector according to claim 6, characterized in that: In step S4, the segmentation model is used to perform pixel-level classification on the image of the connector ROI area, learn and distinguish the fine-grained features of the image of the connector ROI area by performing multi-level feature extraction and fusion on the image, and generate a semantic label corresponding to each pixel based on global and local information, thereby segmenting the connectors in the connector ROI area.

8. The online visual detection method for virtual plug-in of a TV connector according to claim 7, characterized in that: In step S4, judging whether the connector is virtually plugged in according to the semantic segmentation result includes: Through the semantic segmentation results, the shape of each connector is analyzed to obtain the length and width dimensions of the connector, and compared with the preset standard length and width dimensions. If the dimensions are within the preset error range, the connector is judged to be normal, otherwise the connector is judged to be a virtual connection.

9. The online visual detection method for virtual plug-in of a TV connector according to claim 1, characterized in that: In step S4, if it is determined that the connector is a virtual connection, an alarm signal is sent to an interactive end, and the virtual connection event is recorded in the detection log.

10. An online visual inspection system for virtual plug-in of TV connectors, characterized in that: An online visual detection method for virtual plugging of a TV connector as claimed in any one of claims 1 to 9 is used; the detection system comprises: An image acquisition module is used to acquire images of the TV to be inspected on the TV production line in real time; The data processing module is used to pre-process the image captured by the image acquisition module to obtain a stitched image; perform template matching on the stitched image to obtain a circuit board ROI area, and locate a connector ROI area in the circuit board ROI area; classify the image of the connector ROI area by a pre-trained classification model to determine whether there is a connector in the connector ROI area, and if so, perform semantic segmentation on the image of the connector ROI area by a pre-trained segmentation model, and determine whether the connector is a virtual connection based on the semantic segmentation result.

Images