Connector detection method based on multi-camera dynamic cooperation and deep learning fusion
Through the detection method of multi-camera dynamic collaboration and deep learning, the problem that traditional detection methods are difficult to detect complex connectors is solved, and efficient and automated connector detection is achieved to output the precise three-dimensional coordinates and dimension data of defects.
Patent Information
- Application Number
- CN202510592251.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2025-08-12
AI Technical Summary
Traditional detection methods are difficult to efficiently and accurately detect the multi-planar, multi-angle, and refined structures of complex connectors. The manual detection efficiency is low and easy to misjudgment, and mechanical detection cannot cover the internal structural defects.
The detection method of multi-camera dynamic collaboration and deep learning is adopted to filter suspected defects through initial screening of lightweight networks, and the improved deep neural network performs accurate detection, combining optical fiber sensors and PID control to adjust the lighting, outputting the three-dimensional position and size data of the defect.
It realizes efficient and automated inspection of complex connectors, improves detection accuracy and efficiency, reduces labor costs, and adapts to the inspection needs of diverse products.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine vision detection and intelligent manufacturing technology, and in particular to a connector detection method based on multi-camera dynamic collaboration and deep learning fusion. Background Art
[0002] As an essential component of electronic devices, connector quality directly impacts their performance and reliability. As electronic devices evolve towards miniaturization, integration, and higher performance, connector structures and functions are becoming increasingly complex. Modern connectors are no longer confined to a simple, single-plane form; instead, they feature a multi-dimensional, three-dimensional interlacing of multiple planes, the precise superposition of micro-features, and refined surface structures. Some connectors even incorporate complex curves and special-shaped terminals. Traditional connector inspection methods are unable to meet the challenges of such complex structures. Manual visual inspection of connectors with multi-plane, multi-angle, and intricate structures not only requires repeated adjustments of viewing angle, resulting in extremely low inspection efficiency, but also leads to visual fatigue caused by prolonged and intense work. The subjective experience and attention span of the inspector can also affect the frequent occurrence of missed inspections and misjudgments. Simple mechanical inspection devices can only measure a limited number of physical parameters on a single plane of the connector. They are completely unable to effectively detect complex issues such as surface scratches, solder defects, dimensional deviations, and hidden defects within the connector's internal structure. With the continuous development of connector production technology and the continuous improvement of product quality requirements, the development of an efficient, accurate and automated connector intelligent detection system and method has important practical significance. Summary of the Invention
[0003] To overcome the above problems, the present invention proposes a connector detection method based on multi-camera dynamic collaboration and deep learning fusion, which is characterized by comprising the following steps: S1. Configure the number of cameras and observation positions to collect data for the connector to be inspected; S2 uses a lightweight network to perform a first-stage detection on the image captured by the camera in step S1, screening defective connectors, and outputting a first detection result; S3. According to the first test results, for the suspected defective connection machine, re-camera number, adjust the camera observation posture, and conduct a secondary collection of the connector; S4. Use an improved deep neural network to perform second-stage detection on the secondary collected images, outputting the precise three-dimensional position coordinates and dimensional quantification data of the defects as the basis for connector quality assessment.
[0004] Furthermore, the basis for configuring the number of cameras and observation postures in step S1 is: Determine the number of cameras based on the connector's geometry, dimensions, structural complexity, and preset inspection accuracy standards; According to the characteristic parts and defect-prone areas of the connector, the observation posture of each camera is adjusted to fully cover all surfaces of the connector to be inspected and obtain clear and complete image data.
[0005] Furthermore, the first-stage detection of the image captured by the camera in step S1 using a lightweight network described in step 2 is specifically as follows: The image data collected by the camera is input into a pre-trained lightweight network model for feature extraction, analysis and recognition to determine whether the connector has suspected defects. If there are suspected defects, the defective connector is recorded, the category of the suspected defect is identified, and the specific location of the suspected defect in the image is determined, which is output as the first-stage detection result.
[0006] Furthermore, the lightweight network model adopts a lightweight convolutional neural network architecture, removes redundant network connections through a pruning algorithm, and uses quantization technology to convert network parameters from high bits to low bits to reduce the number of network parameters and the amount of calculation; during the model training process, data enhancement technology is used, including random rotation, scaling, translation, and noise addition operations on the image, to expand the training data set and improve the generalization ability of the model.
[0007] Furthermore, step S3 also includes a fill light device and an optical fiber sensor, which monitors the light intensity and uniformity of the connector surface in real time through the light sensor, compares the monitoring data with the preset lighting standard, and uses the PID control algorithm to dynamically adjust the fill light angle and brightness, thereby adjusting the light intensity of the defective part of the connector.
[0008] Furthermore, the improved deep neural network described in step S4 includes a feature extraction module, a multimodal fusion module, a three-dimensional coordinate regression module, and a size quantization module, which work together to achieve refined defect analysis.
[0009] Furthermore, the feature extraction module adopts ResNet-50 as the basic CNN architecture, and uses an independent ResNet-50 branch to process each camera input to fully capture the image features of the connector under different perspectives, and uses multi-scale feature fusion and spatial pyramid pooling to enhance the detection of defects of different sizes.
[0010] Furthermore, the multimodal fusion module adopts a spatial-channel dual attention fusion mechanism to splice the feature maps output by each CNN branch in the channel dimension, compress and fuse them through a series of convolutional layers, and use the spatial attention mechanism and channel attention mechanism to adaptively integrate features from different perspectives.
[0011] Furthermore, the three-dimensional coordinate regression module converts the fused feature map into a one-dimensional vector through global average pooling, connects multiple fully connected layers to construct a regression network, uses mean square error as the loss function, introduces coordinate normalization and angle constraints, and adopts a learning rate decay strategy and data transformation training optimization strategy.
[0012] Furthermore, the size quantization module shares fusion features with the three-dimensional coordinate regression module, but has an independent output layer and adopts a fully connected network structure. The output dimension corresponds to the number of size parameters that need to be measured. The mean square error is also used as the loss function, and each size parameter is normalized.
[0013] Beneficial effects of the present invention: The present invention proposes a connector detection method based on the dynamic collaboration of multiple cameras and the fusion of deep learning. It uses a lightweight network for rapid screening, greatly improving detection efficiency and meeting the high-speed detection needs of the production line. For suspected defects initially screened out, an improved deep neural network is used to accurately output the three-dimensional coordinates and size data of the defects through multi-scale feature extraction, attention enhancement and other algorithms and a cross-validation mechanism. This solution can flexibly adjust the detection parameters according to the geometric shape and structural characteristics of the connector and adapt to a variety of products. The fully automated process effectively reduces labor costs, optimizes the closed-loop of hardware and models, and takes into account both efficiency and accuracy. It is suitable for high-precision fully automatic detection scenarios of connectors with complex shapes. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0015] Figure 1 Flowchart of the present invention; Figure 2 Flowchart of the camera quantity and position scheme implemented in the present invention; Figure 3 This is a flowchart of the deep neural network module in the implementation of the present invention. DETAILED DESCRIPTION
[0016] The following is combined with Figure 1-3 The present application is described with specific embodiments: Example
[0017] This embodiment provides a connector detection method based on multi-camera dynamic collaboration and deep learning fusion, characterized by comprising the following steps: S1. Configure the number of cameras and observation positions to collect data on the connector to be inspected.
[0018] The required number of cameras is determined based on the specific connector's geometry, dimensions, and structural complexity. Using a three-dimensional coordinate system, each camera's observation position is precisely adjusted, including horizontal rotation angle, vertical pitch angle, shooting distance, and lens focal length. During this adjustment process, the impact of environmental factors such as ambient light intensity and uniformity on image acquisition is fully considered. A lighting system tailored to each connector type and scenario is designed to ensure high-quality image capture.
[0019] When the ambient light intensity is lower than a preset threshold, an auxiliary light source is used to enhance the ambient light to ensure that the camera can obtain image information covering the surface of the connector to be inspected.
[0020] S2. Use a lightweight network to perform a first-stage inspection on the image captured by the camera in step S1, screen out defective connectors, and output a first inspection result.
[0021] The connector image obtained in step 1 is fed into a pre-trained lightweight network model. This lightweight network model uses a lightweight convolutional neural network architecture, removes redundant network connections through a pruning algorithm, and uses quantization technology to convert network parameters from high bits to low bits, effectively reducing the number of network parameters and the amount of computation required.
[0022] During model training, data augmentation techniques are employed, including random image rotation, scaling, translation, and noise addition, to expand the training dataset and improve the model's generalization capabilities. The lightweight network model sequentially extracts, analyzes, and recognizes features from the input image. Multiple convolutional layers extract underlying features such as image edges and textures. After dimensionality reduction using pooling layers, fully connected layers map these features to a defect category space to determine whether the connector contains suspected defects. If a suspected defect is present, the model further identifies the category of the suspected defect, such as scratches, holes, deformations, stains, or coating loss. An object detection algorithm is then used to determine the specific location coordinates of the suspected defect in the image, outputting the first-stage detection results, including the suspected defect category and location information.
[0023] S3. Based on the first detection result, for connectors with suspected defects, the number of cameras is adjusted, the observation posture of the cameras is adjusted, and the connector is sampled again.
[0024] The number of cameras is dynamically adjusted based on the first-stage inspection results (i.e., the type and location of suspected defects) output from step 2. If multiple suspected defects of varying types or complex locations are detected in the connector, the number of cameras is increased to capture detailed images of the defects from more angles. If the suspected defect locations are clear and the connector structure is simple, the number of cameras is appropriately reduced to improve inspection efficiency.
[0025] For different types of suspected defects, the observation posture of each camera is reconfigured. For example, when the suspected defect is a surface scratch, the camera posture is adjusted so that the shooting direction is perpendicular to the extension direction of the scratch, and the scratch is completely within the camera's effective field of view. When the suspected defect is a hole, the camera posture is adjusted so that it is vertically aligned with the center of the hole, and the lens focal length is adjusted according to the size and depth of the hole. In addition, the fill light angle and brightness of the fill light device are adjusted according to the optical characteristics of the suspected defect. For example, for surface scratches, side light fill light is used, and the scratch contrast is enhanced by adjusting the angle between the fill light and the connector surface and the fill light intensity. For holes, top light fill light is used, and a ring fill light is combined to highlight the hole outline. During the process of adjusting the fill light device, the light intensity and uniformity of the connector surface are monitored in real time through a light sensor. The monitoring data is compared with the preset lighting standard. The PID control algorithm is used to dynamically adjust the fill light angle and brightness to ensure clear and accurate image details of the suspected defect area, providing high-quality image data for the second stage of detection.
[0026] S4. Use an improved deep neural network to perform second-stage detection on the secondary collected images, outputting the precise three-dimensional position coordinates and dimensional quantification data of the defects as the basis for connector quality assessment.
[0027] In the connector intelligent inspection system, the second phase employs an improved deep neural network for high-precision inspection, primarily to output precise 3D location coordinates and dimensional data for defects. This phase builds on the rapid screening achieved with the lightweight network in the first phase, performing detailed analysis of suspected defect areas.
[0028] In the second phase of inspection, the improved deep neural network uses multiple core modules to work together to output the precise 3D position coordinates and dimensional quantification data of the defect. Specifically, the network includes four core modules: feature extraction module, multimodal fusion module, 3D coordinate regression module, and dimensional quantification module. The following is a detailed workflow for each module: (1) Feature extraction module Feature extraction is the foundation of the entire system and directly affects the accuracy of subsequent detection and positioning. In view of the characteristics of multi-camera data in the connector intelligent detection system, an independent CNN branch structure is designed for feature extraction, as follows: ResNet-50 is used as the base CNN architecture. ResNet effectively addresses the vanishing gradient problem in deep networks by introducing residual connections, enabling the training of deeper networks to extract more complex features. A separate ResNet-50 branch is used to process each camera input, fully capturing the image characteristics of the connector from different viewpoints.
[0029] Using a multi-branch design, each CNN branch is trained independently, preserving the unique features of images from different viewpoints. For example, a top camera might capture the top-down features of a connector, while a side camera can provide information about edges and side structures. Independent branch processing prevents interference between different viewpoints during early fusion.
[0030] During the feature extraction process, a variety of enhancement techniques are used: Multi-scale feature fusion, in addition to the final feature map, also extracts features from the intermediate layers of ResNet and fuses information at different scales, which helps detect defects of different sizes.
[0031] Spatial pyramid pooling extracts multi-scale features by applying pooling windows of different sizes on the feature map, enhancing the network's adaptability to defects of different sizes.
[0032] (2) Multimodal fusion module Multimodal fusion is a key step in integrating features extracted from multiple cameras into a unified representation. We propose a spatial-channel dual attention fusion mechanism that can adaptively integrate features from different viewpoints, as follows: First, the feature maps output by each CNN branch are concatenated along the channel dimension. For example, if each branch outputs a 2048-channel feature map, the three cameras will be concatenated to produce a 6144-channel feature map. These concatenated features are then compressed and fused through a series of 1×1 and 3×3 convolutional layers, reducing the number of parameters while preserving key information.
[0033] The spatial attention module enhances the feature representation of important regions by calculating the spatial distribution weights of feature maps. Specifically, it performs average pooling and max pooling on the input feature map in the channel dimension to obtain two spatial feature maps. These are then concatenated and processed through convolution and sigmoid activation functions to generate a spatial attention map. Finally, the attention map is multiplied with the original feature map to enhance the features of important regions.
[0034] The above Sigmoid activation function expression is:
[0035] The channel attention module focuses on the relationship between channels in the feature map and adaptively adjusts the weights of each channel. The feature map is compressed into channel descriptors through global average pooling and maximum pooling. The channel attention weights are then generated through a fully connected layer and activation function, and finally applied to the original feature map.
[0036] During the fusion process, a multi-scale fusion strategy is adopted to consider both local details and global context information. By applying the attention mechanism on feature maps at different levels, feature information at different scales can be better integrated.
[0037] (3) Three-dimensional coordinate regression module The goal of the 3D coordinate regression module is to predict the precise 3D position coordinates (x, y, z) of the defect from the fused features. This is a typical regression problem, but due to the high precision requirements of 3D positioning, a specialized network structure and training strategy need to be designed, as follows: The fused feature map is converted into a one-dimensional vector through global average pooling, and then multiple fully connected layers are connected to build a regression network. To improve regression accuracy, a deeper network structure is adopted, and dropout layers are added between fully connected layers to prevent overfitting.
[0038] The mean square error (MSE) is used as the loss function to calculate the square of the Euclidean distance between the predicted coordinates and the true coordinates. In order to further improve the accuracy, coordinate normalization and angle constraints are introduced: The loss function MSE formula is:
[0039] Normalize the 3D coordinates to the range [0, 1] to reduce the impact of dimensional differences on training. For certain directional defects, introduce angle constraints to ensure that the predicted direction is consistent with the true direction.
[0040] Training optimization strategy: Add L2 regularization term to the loss function to constrain model parameters and prevent overfitting. The L2 regularization formula is:
[0041] A learning rate decay strategy is adopted, using a larger learning rate for rapid convergence in the early stage of training, and reducing the learning rate in the later stage to obtain a more accurate solution.
[0042] By performing random rotation, translation, and scaling on the training data, the diversity of the data is increased and the generalization ability of the model is improved.
[0043] (4) Size quantization module The size quantization module is used to predict the precise size parameters of the defect, such as length, width, and height. This module shares fusion features with the 3D coordinate regression module but has an independent output layer, forming a multi-task learning architecture.
[0044] Similar to the 3D coordinate regression module, the size quantization module also uses a fully connected network structure. The difference lies in the design of the output layer. The output dimension of the size quantization module corresponds to the number of size parameters to be measured.
[0045] The mean square error (MSE) is also used as the loss function to calculate the error between the predicted size and the actual size. In order to deal with the dimensional differences of different size parameters, each size parameter is normalized.
[0046] To simultaneously optimize detection, classification, localization, and quantization, the network adopts a multi-task learning framework. By designing a reasonable loss function and weighting the losses of each task, the network can balance the learning of each task during training, thereby improving overall detection performance.
[0047] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0048] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations (such as quantity, shape, position, etc.) can be made to the technical solution of the present invention, and these equivalent transformations are all protected by the present invention.
Claims
1. A connector detection method based on multi-camera dynamic collaboration and deep learning fusion, characterized in that: The following steps are involved: S1. Configure the number of cameras and observation positions to collect data for the connector to be inspected; S2 uses a lightweight network to perform a first-stage detection on the image captured by the camera in step S1, screening defective connectors, and outputting a first detection result; S3. According to the first test results, for the suspected defective connection machine, re-camera number, adjust the camera observation posture, and conduct a secondary collection of the connector; S4. Use an improved deep neural network to perform second-stage detection on the secondary collected images, outputting the precise three-dimensional position coordinates and dimensional quantification data of the defects as the basis for connector quality assessment.
2. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 1 is characterized in that ,The basis for configuring the number of cameras and the observation ,pose in step S1 is: Determine the number of cameras based on the connector's geometry, dimensions, structural complexity, and preset inspection accuracy standards; According to the characteristic parts and defect-prone areas of the connector, the observation posture of each camera is adjusted to fully cover all surfaces of the connector to be inspected and obtain clear and complete image data.
3. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 1 is characterized in that ,The first stage detection of the image captured by the camera in step S1 using a lightweight network as described in step 2 is specifically as follows: The image data collected by the camera is input into a pre-trained lightweight network model for feature extraction, analysis and recognition to determine whether the connector has suspected defects. If there are suspected defects, the defective connector is recorded, the category of the suspected defect is identified, and the specific location of the suspected defect in the image is determined, which is output as the first-stage detection result.
4. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 3 is characterized in that : The lightweight network model adopts a lightweight convolutional neural network architecture, removes redundant network connections through a pruning algorithm, and uses quantization technology to convert network parameters from high bits to low bits to reduce the number of network parameters and the amount of calculation; during the model training process, data enhancement technology is used, including random rotation, scaling, translation, and noise addition operations on the image to expand the training data set and improve the generalization ability of the model.
5. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 1 is characterized in that: Step S3 also includes a fill light device and an optical fiber sensor, which monitors the light intensity and uniformity of the connector surface in real time through the light sensor, compares the monitoring data with the preset lighting standard, and uses the PID control algorithm to dynamically adjust the fill light angle and brightness, thereby adjusting the light intensity of the defective part of the connector.
6. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 1 is characterized in that: The improved deep neural network described in step S4 includes a feature extraction module, a multimodal fusion module, a three-dimensional coordinate regression module, and a size quantization module, which work together to achieve refined defect analysis.
7. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 6 is characterized in that: The feature extraction module uses ResNet-50 as the basic CNN architecture and uses an independent ResNet-50 branch to process each camera input to fully capture the image features of the connector from different perspectives. It also uses multi-scale feature fusion and spatial pyramid pooling to enhance the detection of defects of different sizes.
8. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 7 is characterized in that: The multimodal fusion module adopts a spatial-channel dual attention fusion mechanism to splice the feature maps output by each CNN branch in the channel dimension, compress and fuse them through a series of convolutional layers, and use the spatial attention mechanism and channel attention mechanism to adaptively integrate features from different perspectives.
9. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 8 is characterized in that: The three-dimensional coordinate regression module converts the fused feature map into a one-dimensional vector through global average pooling, connects multiple fully connected layers to build a regression network, uses mean square error as the loss function, introduces coordinate normalization and angle constraints, and adopts a learning rate decay strategy and data transformation training optimization strategy.
10. The connector detection method based on multi-camera dynamic collaboration and deep learning fusion according to claim 9 is characterized in that: The size quantization module shares fusion features with the three-dimensional coordinate regression module, but has an independent output layer and adopts a fully connected network structure. The output dimension corresponds to the number of size parameters that need to be measured. The mean square error is also used as the loss function, and each size parameter is normalized.