A Method and System for Detecting Welding Defects in Flat Wire Stator Based on Cross-Modal Feature Mapping

Abstract

This invention discloses a method and system for detecting welding defects in flat wire stators based on cross-modal feature mapping. The method simultaneously acquires two-dimensional images and three-dimensional point cloud data of the weld area using a structured light camera, and establishes pixel-level spatial alignment based on preset calibration parameters. The two-dimensional images are input into a first feature extraction network to obtain pixel-level two-dimensional feature maps, and the three-dimensional point cloud data is input into a second feature extraction network to obtain pixel-level three-dimensional feature maps. A trained first mapping network maps the two-dimensional features to a predicted three-dimensional feature map, and a second mapping network maps the three-dimensional features to a predicted two-dimensional feature map. The deviation between the predicted and actual feature maps is calculated to generate an anomaly scoring map, thereby locating the welding defect area. The method and system of this application can achieve high-precision detection of complex and unknown defects such as cold welds and porosity without relying on defect annotation data, reducing dependence on defect annotation samples.

Description

Technical Field

[0001] This invention relates to the field of industrial defect detection technology, and in particular to a method and system for detecting welding defects in flat wire stators based on cross-modal feature mapping. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing technologies, electric motors, as core power drive components, are widely used in new energy vehicles, industrial robots, precision equipment, and other fields. In the new energy vehicle industry, the performance of the drive motor directly determines the vehicle's power, economy, and reliability. Flat wire motors, due to their significant advantages such as high slot fill factor, good heat dissipation, and high power density, are gradually replacing traditional round wire motors as the mainstream technology for drive motors in new energy vehicles.

[0003] As a key component of an electric motor, the stator's manufacturing quality directly affects the motor's efficiency, lifespan, and operational stability. In the production of flat wire motor stators, the connection between the stator winding ends and the lead ends requires welding. The welding quality directly determines the conductivity and long-term reliability of the motor windings. Due to the large number and dense arrangement of weld points in flat wire stators, and the extremely high requirements for weld consistency and appearance quality, the stator welding process has become one of the key control links in motor manufacturing. Traditional manual visual inspection methods are inefficient, subjective, and prone to fatigue, and can no longer meet the speed and accuracy requirements of modern large-scale production. Automated appearance defect inspection based on machine vision has become an inevitable trend in the industry.

[0004] Currently, the technologies used for visual inspection of flat wire stator welding mainly include the following categories: (1) Detection method based on 2D machine vision; The detection method based on 2D machine vision is currently the most widely used visual inspection technology for flat wire stator welding. Its basic principle is to use an industrial camera to acquire images of the weld area and to analyze and judge the welding quality through digital image processing algorithms. Commonly used image processing methods include: using edge detection operators (such as Canny operator and Sobel operator) to extract weld contour features; using threshold segmentation methods based on grayscale threshold or color space to separate the target area from the background; using morphological processing (such as opening operation and closing operation) to remove image noise and optimize structural connectivity; and accurately locking the weld position through template matching or region of interest localization technology.

[0005] This method can effectively detect surface defects such as weld point misalignment, abnormal weld contour morphology, welding spatter contamination, and obvious weld point loss. Its advantages include low system cost, relatively simple implementation, and fast detection speed. However, this method has inherent limitations: firstly, it is extremely sensitive to changes in lighting; fluctuations in ambient light or reflections from the workpiece surface can lead to false or missed detections; secondly, because it relies solely on two-dimensional planar information, its ability to identify internal or minute complex defects such as porosity and cold welds is severely insufficient, making it difficult to meet the high reliability requirements of detection scenarios.

[0006] (2) 3D vision-based inspection method; The 3D vision-based inspection method belongs to the mid-to-high-end appearance inspection technology solution. It mainly obtains the three-dimensional morphological information of the weld area through laser contour scanning or structured light three-dimensional reconstruction, and performs quantitative analysis of welding quality based on three-dimensional geometric parameters. This method can accurately measure key geometric feature parameters such as the height, width, cross-sectional area and volume of the weld, thereby accurately judging whether the weld is full and whether there are structural defects such as collapse or protrusion.

[0007] Compared to 2D vision inspection methods, 3D vision technology does not rely on the color and grayscale information of an object's surface, possesses inherent robustness to changes in illumination, and offers higher detection accuracy, making it particularly suitable for applications with strict requirements on three-dimensional geometric dimensions. However, this type of technology also has significant shortcomings: First, high-precision 3D vision systems are expensive, have complex structures, and require stringent precision during on-site installation and debugging; second, 3D vision systems are not sensitive to two-dimensional information such as surface texture, color distribution, and subtle grayscale changes, making it difficult to effectively identify appearance defects that rely on image texture features, such as minor burns, surface oxidation, color anomalies, and small splashes; furthermore, the resolution of 3D point cloud data is relatively limited, and minute defects may be smoothed or ignored during 3D reconstruction, leading to a loss of detection completeness.

[0008] (3) AI deep learning-based detection method; AI deep learning-based detection method is a current research hotspot in the field of appearance inspection of flat wire stator welding. This method relies on the powerful feature self-learning ability of deep neural networks. By constructing convolutional neural networks or target detection models (such as YOLO series, Faster R-CNN, etc.), it automatically learns and extracts deep discriminative features of the weld area from a large number of labeled samples, thereby achieving high-precision identification and classification of complex defects such as porosity, cracks, and cold welds.

[0009] While these methods demonstrate significant advantages in detection accuracy and generalization ability, their practical application in industrial scenarios faces two major challenges: First, obtaining defect samples is difficult and the annotation cost is extremely high. To ensure good generalization ability, it is usually necessary to collect and annotate massive amounts of weld point image data covering various defect types and different operating conditions. However, in actual production lines, defect samples are low-probability events, making collection difficult, and the detailed pixel-level annotation work is time-consuming, labor-intensive, and has huge labor costs. Second, model maintenance costs are high. When production line conditions change (such as lighting adjustments, material batch differences, equipment replacement), the detection performance of the original model may significantly decrease, often requiring data re-collection and model retraining or fine-tuning, further increasing the complexity and cost of deployment and maintenance.

[0010] In summary, existing technologies for detecting welding defects in flat wire stators each have unavoidable shortcomings: 2D vision methods are insufficient for recognizing complex defects; 3D vision methods are insensitive to texture-based defects; and AI deep learning methods are overly reliant on labeled data and have high maintenance costs. Therefore, there is an urgent need for a novel detection method that can integrate the advantages of multimodal information, reduce reliance on data labeling, and possess high robustness to complex operating conditions to meet the practical industrial application needs of post-weld defect detection in flat wire motor stators for new energy vehicles. Summary of the Invention

[0011] To address the technical problems existing in the background art, this invention proposes a method and system for detecting welding defects in flat wire stators based on cross-modal feature mapping.

[0012] The proposed method for detecting welding defects in flat wire stators based on cross-modal feature mapping includes the following steps: S1. Acquire two-dimensional images and three-dimensional point cloud data of the weld point area of ​​the flat wire stator to be detected, and project the three-dimensional point cloud data onto the two-dimensional image plane based on preset calibration parameters to obtain three-dimensional projection data that is pixel-level aligned with the two-dimensional image. S2. Input the two-dimensional image into the trained first feature extraction network to obtain a pixel-level two-dimensional feature map, and input the three-dimensional point cloud data into the trained second feature extraction network to obtain a pixel-level three-dimensional feature map. S3. Input the pixel-level two-dimensional feature map into the trained first mapping network to generate the predicted three-dimensional feature map, and input the pixel-level three-dimensional feature map into the trained second mapping network to generate the predicted two-dimensional feature map. S4. Based on the first deviation between the predicted 3D feature map and the pixel-level 3D feature map, and the second deviation between the predicted 2D feature map and the pixel-level 2D feature map, generate an anomaly score map representing the degree of anomaly. S5. Determine the location of defects in the flat wire stator weld area based on the anomaly scoring diagram.

[0013] Preferably, the first feature extraction network specifically includes: The backbone network layer is used to receive two-dimensional images and perform spatial resolution compression and semantic feature extraction on the two-dimensional images through multi-level convolution and pooling operations, outputting a low-resolution feature map with a spatial resolution lower than that of the two-dimensional image. The upsampling recovery layer, connected to the output of the backbone network layer, is used to receive low-resolution feature maps and use a bilinear interpolation algorithm to restore the spatial resolution of the low-resolution feature maps to be consistent with the two-dimensional image. This ensures that each pixel position in the restored feature map corresponds to a high-dimensional semantic feature vector of a preset dimension, thereby obtaining a pixel-level two-dimensional feature map.

[0014] Preferably, the second feature extraction network specifically includes: The three-dimensional backbone network layer is used to receive three-dimensional point cloud data and perform key point filtering and local geometric feature encoding on the three-dimensional point cloud data through multi-level point cloud sampling and feature aggregation operations, and output a set of sparse key point feature vectors. The number of sparse key point feature vectors is less than the number of original point clouds in the three-dimensional point cloud data. The feature propagation interpolation layer, connected to the output of the 3D backbone network layer, is used to receive sparse keypoint feature vectors and propagate the sparse keypoint feature vectors to each original point cloud location in the 3D point cloud data based on spatial distance weights, thereby outputting a set of point-level feature vectors. The second feature extraction network also includes a projection smoothing layer, which is connected to the output of the feature propagation interpolation layer. The projection smoothing layer is used to project the set of point-level feature vectors onto the two-dimensional image plane based on preset calibration parameters, and to perform zero-filling and convolution smoothing on the hole regions generated after projection, thereby outputting a pixel-level three-dimensional feature map.

[0015] Preferably, the first mapping network specifically includes: The first input flattening layer is used to receive the two-dimensional feature vector corresponding to each pixel position in the pixel-level two-dimensional feature map, convert the two-dimensional feature vector into a one-dimensional sequence structure, and output the first flattened feature vector. The first feature transformation layer, connected to the first input flattening layer, consists of multiple fully connected processing units stacked sequentially. Each fully connected processing unit contains a linear transformation operation and a nonlinear activation operation. The first feature transformation layer is used to receive the first flattened feature vector and transform the first flattened feature vector from the original two-dimensional feature dimension space to the target three-dimensional feature dimension space through step-by-step linear mapping and nonlinear activation, and output the first cross-modal mapping feature vector. The first output reconstruction layer, connected to the first feature transformation layer, is used to receive the first cross-modal mapping feature vector and restore the first cross-modal mapping feature vector into a feature map structure with the same size as the pixel-level three-dimensional feature map according to the spatial arrangement rules of the pixel-level three-dimensional feature map, and output the predicted three-dimensional feature map.

[0016] Preferably, the second mapping network specifically includes: The second input flattening layer is used to receive the three-dimensional feature vector corresponding to each pixel position in the pixel-level three-dimensional feature map, convert the three-dimensional feature vector into a one-dimensional sequence structure, and output the second flattened feature vector. The second feature transformation layer, connected to the second input flattening layer, consists of multiple fully connected processing units stacked sequentially. Each fully connected processing unit contains a linear transformation operation and a nonlinear activation operation. The second feature transformation layer is used to receive the second flattened feature vector and transform the second flattened feature vector from the original three-dimensional feature dimension space to the target two-dimensional feature dimension space through step-by-step linear mapping and nonlinear activation, and output the second cross-modal mapping feature vector. The second output reconstruction layer, connected to the second feature transformation layer, is used to receive the second cross-modal mapping feature vector and restore the second cross-modal mapping feature vector into a feature map structure with the same size as the pixel-level two-dimensional feature map according to the spatial arrangement rules of the pixel-level two-dimensional feature map, and output the predicted two-dimensional feature map.

[0017] Preferably, step S1 specifically includes: A structured light camera is used to simultaneously trigger the acquisition of two-dimensional grayscale images and three-dimensional point cloud data of the flat wire stator welding point area. The resolution of the two-dimensional grayscale image is a preset height multiplied by a preset width, and the three-dimensional point cloud data contains three-dimensional spatial point coordinates in a number equal to the number of original point clouds. Based on the internal parameter matrix and hand-eye calibration matrix obtained by pre-calibration of the structured light camera, the coordinates of each three-dimensional point in the three-dimensional point cloud data are converted into pixel coordinates in the pixel coordinate system of the two-dimensional grayscale image. Based on the pixel coordinates corresponding to each three-dimensional spatial point coordinate, the geometric information of each three-dimensional spatial point coordinate is mapped to the corresponding pixel position in the two-dimensional grayscale image. For pixel positions in the two-dimensional grayscale image that are not covered by any three-dimensional spatial point coordinate mapping, the geometric information corresponding to the pixel position is set to zero. The mapped geometric information is organized into three-dimensional projection data according to the spatial arrangement rules of the two-dimensional grayscale image. Each pixel position in the three-dimensional projection data corresponds to a geometric feature representation of a preset dimension, thereby achieving pixel-level alignment between the three-dimensional projection data and the two-dimensional grayscale image.

[0018] Preferably, step S4 specifically includes: For each pixel position in the pixel-level 3D feature map and the predicted 3D feature map, extract the real 3D feature vector and the predicted 3D feature vector corresponding to the pixel position, and calculate the first cosine distance between the real 3D feature vector and the predicted 3D feature vector as the first deviation. For each pixel position in the pixel-level two-dimensional feature map and the predicted two-dimensional feature map, extract the real two-dimensional feature vector and the predicted two-dimensional feature vector corresponding to the pixel position, and calculate the second cosine distance between the real two-dimensional feature vector and the predicted two-dimensional feature vector as the second deviation. Multiply the first cosine distance corresponding to each pixel position by the preset three-dimensional weight coefficient to obtain the three-dimensional anomaly contribution value, multiply the second cosine distance by the preset two-dimensional weight coefficient to obtain the two-dimensional anomaly contribution value, and add the three-dimensional anomaly contribution value and the two-dimensional anomaly contribution value to obtain the anomaly score of the pixel position. The anomaly scores of all pixel locations are organized according to the spatial arrangement rules of the two-dimensional image to generate an anomaly score map with the same size as the two-dimensional image. The value of each pixel location in the anomaly score map represents the degree of probability that a welding defect exists at that location.

[0019] Preferably, step S5 specifically includes: The abnormal score map is binarized using a preset segmentation threshold. Pixels with abnormal scores greater than or equal to the preset segmentation threshold are marked as suspected defect pixels, and pixels with abnormal scores less than the preset segmentation threshold are marked as background pixels, thus obtaining a binarized abnormal mask map. Connectivity analysis is performed on the binarized anomaly mask image to extract all connected regions composed of suspected defect pixels in the binarized anomaly mask image, and the boundary coordinates and area information of each connected region are obtained. Based on the area information of each connected region, connected regions with an area smaller than the preset minimum defect area are filtered out, and the remaining connected regions are determined as valid defect regions. Based on the boundary coordinates of each valid defect region, a corresponding defect location marker box is generated on the two-dimensional image, and the defect location detection results of the flat wire stator weld area are output.

[0020] Preferably, the training process for the first mapping network and the second mapping network includes: Construct a training dataset containing only normal welding samples, wherein the normal welding samples include pairs of two-dimensional training images and three-dimensional training point clouds; Two-dimensional training images and three-dimensional training point clouds are input into the first feature extraction network and the second feature extraction network, respectively, to obtain pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps for training. Using the training pixel-level two-dimensional feature map and the training pixel-level three-dimensional feature map as supervision signals, the first mapping network and the second mapping network are jointly trained using the cosine similarity loss function until the loss value converges.

[0021] The flat wire stator welding defect detection system proposed in this invention based on cross-modal feature mapping includes: The data acquisition and alignment module is used to simultaneously acquire two-dimensional images and three-dimensional point cloud data of the flat wire stator welding point area, and project the three-dimensional point cloud data onto the two-dimensional image plane according to the built-in preset calibration parameters, and output pixel-level aligned three-dimensional projection data and corresponding mapping relationships. The feature extraction module is used to receive two-dimensional images and three-dimensional point cloud data, and extract features based on the mapping relationship using the built-in trained first feature extraction network and second feature extraction network respectively, and output pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps. The cross-modal prediction module receives pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps, and performs feature mapping using the built-in trained first mapping network and second mapping network respectively, outputting predicted three-dimensional feature maps and predicted two-dimensional feature maps. An anomaly scoring module is used to calculate the first deviation between the predicted 3D feature map and the pixel-level 3D feature map, and the second deviation between the predicted 2D feature map and the pixel-level 2D feature map, and to generate an anomaly scoring map based on the first deviation and the second deviation. The defect determination module is used to perform threshold segmentation on the anomaly scoring map to determine and output the defect location in the flat wire stator weld area.

[0022] This invention presents a method and system for detecting welding defects in flat wire stators based on cross-modal feature mapping. By constructing a bidirectional cross-modal mapping mechanism between two-dimensional image features and three-dimensional point cloud features, and using only normal samples for unsupervised training, the method can accurately locate and identify welding defects during the detection stage by leveraging inter-modal consistency deviations. This method integrates the complementary advantages of two-dimensional texture information and three-dimensional geometric information, improving the detection capability for complex and hidden defects such as cold welds, porosity, and inclusions. It also exhibits excellent generalization performance for novel defects not previously seen in the training data. Furthermore, it reduces the dependence on labeled defect samples, avoiding the high data labeling costs and model retraining overhead of traditional supervised learning methods, and demonstrates good robustness and stability under complex industrial conditions such as lighting fluctuations and surface reflections. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the workflow of the flat wire stator welding defect detection method based on cross-modal feature mapping proposed in this invention. Figure 2This is a schematic diagram of the model training and inference process of the first and second mapping networks of the flat wire stator welding defect detection method based on cross-modal feature mapping proposed in this invention; Figure 3 This is a schematic diagram of the system architecture of the flat wire stator welding defect detection system based on cross-modal feature mapping proposed in this invention. Detailed Implementation

[0024] Reference Figure 1 and Figure 2 The present invention proposes a method for detecting welding defects in flat wire stators based on cross-modal feature mapping, comprising the following steps: S1. Acquire two-dimensional images and three-dimensional point cloud data of the weld point area of ​​the flat wire stator to be inspected, and project the three-dimensional point cloud data onto the two-dimensional image plane based on preset calibration parameters to obtain three-dimensional projection data that is pixel-level aligned with the two-dimensional image.

[0025] In this embodiment, step S1 specifically includes: A structured light camera is used to simultaneously trigger the acquisition of two-dimensional grayscale images and three-dimensional point cloud data of the flat wire stator welding point area. The resolution of the two-dimensional grayscale image is the preset height multiplied by the preset width, and the three-dimensional point cloud data contains the coordinates of three-dimensional spatial points in the original point cloud. Based on the internal parameter matrix and hand-eye calibration matrix obtained by pre-calibration of the structured light camera, the coordinates of each three-dimensional point in the three-dimensional point cloud data are converted into pixel coordinates in the pixel coordinate system of the two-dimensional grayscale image. Based on the pixel coordinates corresponding to each three-dimensional spatial point coordinate, the geometric information of each three-dimensional spatial point coordinate is mapped to the corresponding pixel position in the two-dimensional grayscale image. For pixel positions in the two-dimensional grayscale image that are not covered by any three-dimensional spatial point coordinate mapping, the geometric information corresponding to the pixel position is set to zero. The mapped geometric information is organized into three-dimensional projection data according to the spatial arrangement rules of the two-dimensional grayscale image. Each pixel position in the three-dimensional projection data corresponds to a geometric feature representation of a preset dimension, thereby achieving pixel-level alignment between the three-dimensional projection data and the two-dimensional grayscale image.

[0026] Specifically, this embodiment uses a Keyence XT60 series camera to simultaneously trigger and acquire two-dimensional grayscale images and three-dimensional point cloud data of the flat wire stator welding point area. The resolution of the two-dimensional grayscale image is set to a preset height H × preset width W, which is 2048 × 2048 pixels in this embodiment. The three-dimensional point cloud data contains N three-dimensional spatial point coordinates, which in this embodiment is approximately 100,000 to 500,000 spatial points.

[0027] Based on the internal parameter matrix and hand-eye calibration matrix obtained by pre-calibrating the structured light camera before deployment, the coordinates of each 3D spatial point in the 3D point cloud data are transformed to the pixel coordinate system of the 2D grayscale image to obtain the corresponding pixel coordinates. According to the pixel coordinates corresponding to each 3D spatial point coordinate, the geometric information of the 3D spatial point coordinate is mapped to the corresponding pixel position in the 2D grayscale image. For pixel positions in the 2D grayscale image that are not covered by any 3D spatial point coordinate mapping due to sparse point cloud or viewpoint occlusion, the geometric information corresponding to the pixel position is set to zero, i.e., zero-filling processing is performed.

[0028] Finally, the mapped geometric information is organized into three-dimensional projection data according to the spatial arrangement rules of the two-dimensional grayscale image. Each pixel position in the three-dimensional projection data corresponds to a geometric feature representation of a preset dimension, thereby achieving strict alignment between the three-dimensional projection data and the two-dimensional grayscale image at the pixel level. In this embodiment, the geometric feature representation of the preset dimension is consistent with the subsequent three-dimensional feature dimension.

[0029] S2. Input the two-dimensional image into the trained first feature extraction network to obtain a pixel-level two-dimensional feature map, and input the three-dimensional point cloud data into the trained second feature extraction network to obtain a pixel-level three-dimensional feature map.

[0030] In this embodiment, the first feature extraction network specifically includes: The backbone network layer is used to receive two-dimensional images and perform spatial resolution compression and semantic feature extraction on the two-dimensional images through multi-level convolution and pooling operations, outputting a low-resolution feature map with a spatial resolution lower than that of the two-dimensional image. The upsampling recovery layer, connected to the output of the backbone network layer, is used to receive low-resolution feature maps and use a bilinear interpolation algorithm to restore the spatial resolution of the low-resolution feature maps to be consistent with the two-dimensional image. This ensures that each pixel position in the restored feature map corresponds to a high-dimensional semantic feature vector of a preset dimension, thereby obtaining a pixel-level two-dimensional feature map.

[0031] In this embodiment, the second feature extraction network specifically includes: The three-dimensional backbone network layer is used to receive three-dimensional point cloud data and perform key point filtering and local geometric feature encoding on the three-dimensional point cloud data through multi-level point cloud sampling and feature aggregation operations, and output a set of sparse key point feature vectors. The number of sparse key point feature vectors is less than the number of original point clouds in the three-dimensional point cloud data. The feature propagation interpolation layer, connected to the output of the 3D backbone network layer, is used to receive sparse keypoint feature vectors and propagate the sparse keypoint feature vectors to each original point cloud location in the 3D point cloud data based on spatial distance weights, thereby outputting a set of point-level feature vectors. The second feature extraction network also includes a projection smoothing layer, which is connected to the output of the feature propagation interpolation layer. It is used to project the set of point-level feature vectors onto the two-dimensional image plane based on preset calibration parameters, and to perform zero-filling and convolution smoothing on the hole regions generated after projection, thereby outputting a pixel-level three-dimensional feature map.

[0032] Specifically, the first feature extraction network can employ a two-dimensional convolutional neural network or a visual Transformer network pre-trained on a large image dataset. For example, the two-dimensional convolutional neural network can be the ResNet series, WideResNet series, etc.; the visual Transformer network can be ViT, Swing Transformer, etc. In this embodiment, the backbone network layer specifically uses a WideResNet-50 network pre-trained on the ImageNet dataset, whose network parameters are frozen during subsequent training and inference. The upsampling recovery layer is connected to the output of the backbone network layer, receives the low-resolution feature map, and uses a bilinear interpolation algorithm to restore the spatial resolution of the low-resolution feature map to be consistent with the two-dimensional image, so that each pixel position in the recovered feature map corresponds to a high-dimensional semantic feature vector of a preset dimension D_2D, thereby obtaining a pixel-level two-dimensional feature map. In this embodiment, the preset dimension D_2D is set to 1024.

[0033] Specifically, the second feature extraction network can be a 3D feature extraction network pre-trained on a 3D dataset. For example, PointNet++, DGCNN, or 3D-GCN can be used. In this embodiment, the 3D backbone network layer specifically uses a PointNet++ network pre-trained on the ShapeNet dataset, and its network parameters are also frozen. The number of sparse keypoint feature vectors, N_f, is significantly less than the number of original point clouds, N; in this embodiment, N_f is set to 4096. The feature propagation interpolation layer is connected to the output of the 3D backbone network layer, receives the sparse keypoint feature vectors, and propagates them to each original point cloud location in the 3D point cloud data based on spatial distance weights, thereby outputting a set of point-level feature vectors. In this embodiment, the interpolation operation selects the three nearest neighboring points to the current point for inverse distance weighted averaging. The projection smoothing layer is connected to the output of the feature propagation interpolation layer. Based on the preset calibration parameters used in step S1, it projects the set of point-level feature vectors onto the two-dimensional image plane and performs zero-filling and convolutional smoothing on the resulting hole regions, thereby outputting a pixel-level three-dimensional feature map aligned with the pixel level of the two-dimensional image. Each pixel position corresponds to a three-dimensional feature vector of a preset dimension D_3D; in this embodiment, D_3D is set to 512. The convolutional smoothing process uses a 3×3 convolution kernel to enhance the spatial continuity of the feature map and reduce noise interference.

[0034] S3. Input the pixel-level two-dimensional feature map into the trained first mapping network to generate a predicted three-dimensional feature map, and input the pixel-level three-dimensional feature map into the trained second mapping network to generate a predicted two-dimensional feature map.

[0035] In this embodiment, the first mapping network specifically includes: The first input flattening layer is used to receive the two-dimensional feature vector corresponding to each pixel position in the pixel-level two-dimensional feature map, convert the two-dimensional feature vector into a one-dimensional sequence structure, and output the first flattened feature vector. The first feature transformation layer, connected to the first input flattening layer, consists of multiple fully connected processing units stacked sequentially. Each fully connected processing unit contains a linear transformation operation and a nonlinear activation operation. The first feature transformation layer is used to receive the first flattened feature vector and transform the first flattened feature vector from the original two-dimensional feature dimension space to the target three-dimensional feature dimension space through step-by-step linear mapping and nonlinear activation, and output the first cross-modal mapping feature vector. The first output reconstruction layer, connected to the first feature transformation layer, is used to receive the first cross-modal mapping feature vector and restore the first cross-modal mapping feature vector into a feature map structure with the same size as the pixel-level three-dimensional feature map according to the spatial arrangement rules of the pixel-level three-dimensional feature map, and output the predicted three-dimensional feature map.

[0036] In this embodiment, the first feature transformation layer adopts a 4-layer MLP structure, the hidden layer dimension is set to 512, and the non-linear activation function is ReLU.

[0037] In this embodiment, the second mapping network specifically includes: The second input flattening layer is used to receive the three-dimensional feature vector corresponding to each pixel position in the pixel-level three-dimensional feature map, convert the three-dimensional feature vector into a one-dimensional sequence structure, and output the second flattened feature vector. The second feature transformation layer, connected to the second input flattening layer, consists of multiple fully connected processing units stacked sequentially. Each fully connected processing unit contains a linear transformation operation and a nonlinear activation operation. The second feature transformation layer is used to receive the second flattened feature vector and transform the second flattened feature vector from the original three-dimensional feature dimension space to the target two-dimensional feature dimension space through successive linear mapping and nonlinear activation, and output the second cross-modal mapping feature vector. The second output reconstruction layer, connected to the second feature transformation layer, is used to receive the second cross-modal mapping feature vector and restore the second cross-modal mapping feature vector into a feature map structure with the same size as the pixel-level two-dimensional feature map according to the spatial arrangement rules of the pixel-level two-dimensional feature map, and output the predicted two-dimensional feature map.

[0038] Specifically, the training process of the first mapping network and the second mapping network includes: Construct a training dataset containing only normal welding samples, which include pairs of two-dimensional training images and three-dimensional training point clouds; Two-dimensional training images and three-dimensional training point clouds are input into the first feature extraction network and the second feature extraction network, respectively, to obtain pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps for training. Using the training pixel-level two-dimensional feature map and the training pixel-level three-dimensional feature map as supervision signals, the first mapping network and the second mapping network are jointly trained using the cosine similarity loss function until the loss value converges.

[0039] Specifically, the training process for the first and second mapping networks is completed offline before the model is deployed to the production line. A flowchart illustrating this process can be found here. Figure 2 The training phase section.

[0040] The training data sources and construction for the first and second mapping networks specifically include: The training data for the first and second mapping networks were simultaneously acquired on the production line using the same structured light camera, consisting of two-dimensional training images and three-dimensional training point clouds. Unlike supervised learning methods, this application uses only normal welding samples to construct the training dataset, without including any defective samples. Normal welding samples refer to flat wire stator weld point data that have passed both manual visual inspection and electrical performance testing.

[0041] In this embodiment, 200 normal weld point samples from different stator workpieces were collected and screened from continuous production batches. Each sample contained a pair of two-dimensional training images and a three-dimensional training point cloud. Since only normal samples need to be collected and no manual annotation of defective areas is required, the data preparation cycle and cost are significantly reduced.

[0042] Before training begins, the pre-trained weights of the first and second feature extraction networks are loaded. In this embodiment, the first feature extraction network uses WideResNet-50 pre-trained on the ImageNet dataset, and the second feature extraction network uses PointNet++ pre-trained on the ShapeNet dataset. Throughout the training process, the parameters of both the first and second feature extraction networks are frozen and do not participate in gradient updates; they are used solely as fixed feature encoders to extract multimodal features. This setup ensures the stability and consistency of feature extraction while significantly reducing the number of trainable parameters.

[0043] The parameters of the fully connected layers in the first and second mapping networks are initialized using a random initialization strategy. In this embodiment, the Xavier uniform initialization method is used for weight initialization, and the bias term is initialized to zero.

[0044] This invention employs a bidirectional cosine similarity loss function to jointly train the first and second mapping networks. For any training sample in the training dataset, the two-dimensional training image is first input into the first feature extraction network to obtain a pixel-level two-dimensional feature map for training. The 3D training point cloud is input into the second feature extraction network to obtain pixel-level 3D feature maps for training. Subsequently, with and As a monitoring signal, the total loss value L is calculated using the following loss function: ; in, The summation symbol Σ represents the total loss value, which is calculated by summing the values ​​at all valid pixel locations in the feature map. For training, use pixel-level two-dimensional feature maps. For training, use pixel-level 3D feature maps; The second mapping network has the following input: The output is a two-dimensional feature predicted from the three-dimensional feature; The first mapping network has the following input: The output is the 3D features predicted from the 2D features; CosSim represents the cosine similarity operation. The first term of this loss function constrains the consistency between the predicted 3D-to-2D mapping and the orientation of the true 2D features in the feature space, and the second term constrains the consistency between the predicted 2D-to-3D mapping and the orientation of the true 3D features in the feature space. By minimizing this loss function, the network gradually learns the inherent correlation between the 2D texture modality and the 3D geometric modality in normal solder joint samples.

[0045] During training, the Adam optimizer was selected, with an initial learning rate of 0.0001 and a first-order momentum decay coefficient. Set to 0.9, second-order momentum decay coefficient Set to 0.999. Batch size is set to 4. All training samples are iterated once per training epoch. The total loss value L is continuously monitored during training; when the loss value... When the loss value no longer decreases within 20 consecutive training cycles, it is determined that the loss value has converged. Training is terminated, and the network parameters at this point are saved as the parameters of the first and second mapping networks for subsequent online inference deployment.

[0046] S4. Based on the first deviation between the predicted 3D feature map and the pixel-level 3D feature map, and the second deviation between the predicted 2D feature map and the pixel-level 2D feature map, generate an anomaly score map representing the degree of anomaly.

[0047] In this embodiment, step S4 specifically includes: For each pixel position in the pixel-level 3D feature map and the predicted 3D feature map, extract the real 3D feature vector and the predicted 3D feature vector corresponding to the pixel position, and calculate the first cosine distance between the real 3D feature vector and the predicted 3D feature vector as the first deviation. For each pixel position in the pixel-level two-dimensional feature map and the predicted two-dimensional feature map, extract the real two-dimensional feature vector and the predicted two-dimensional feature vector corresponding to the pixel position, and calculate the second cosine distance between the real two-dimensional feature vector and the predicted two-dimensional feature vector as the second deviation. Multiply the first cosine distance corresponding to each pixel position by the preset three-dimensional weight coefficient to obtain the three-dimensional anomaly contribution value, multiply the second cosine distance by the preset two-dimensional weight coefficient to obtain the two-dimensional anomaly contribution value, and add the three-dimensional anomaly contribution value and the two-dimensional anomaly contribution value to obtain the anomaly score of the pixel position. The anomaly scores of all pixel locations are organized according to the spatial arrangement rules of the two-dimensional image to generate an anomaly score map with the same size as the two-dimensional image. The value of each pixel location in the anomaly score map represents the degree of probability that a welding defect exists at that location.

[0048] Specifically, in this embodiment, a three-dimensional weighting coefficient is preset. =0.6, preset two-dimensional weighting coefficient =0.4; In the anomaly rating chart, the higher the value of each pixel, the greater the probability of an anomaly.

[0049] S5. Determine the location of defects in the flat wire stator weld area based on the anomaly scoring diagram.

[0050] In this embodiment, step S5 specifically includes: The abnormal score map is binarized using a preset segmentation threshold. Pixels with abnormal scores greater than or equal to the preset segmentation threshold are marked as suspected defect pixels, and pixels with abnormal scores less than the preset segmentation threshold are marked as background pixels, thus obtaining a binarized abnormal mask map. Connectivity analysis is performed on the binarized anomaly mask image to extract all connected regions composed of suspected defect pixels in the binarized anomaly mask image, and the boundary coordinates and area information of each connected region are obtained. Based on the area information of each connected region, connected regions with an area smaller than the preset minimum defect area are filtered out, and the remaining connected regions are determined as valid defect regions. Based on the boundary coordinates of each valid defect region, a corresponding defect location marker box is generated on the two-dimensional image, and the defect location detection results of the flat wire stator weld area are output.

[0051] Reference Figures 1-3The present invention proposes a flat wire stator welding defect detection system based on cross-modal feature mapping, comprising: The data acquisition and alignment module is used to simultaneously acquire two-dimensional images and three-dimensional point cloud data of the flat wire stator welding point area, and project the three-dimensional point cloud data onto the two-dimensional image plane according to the built-in preset calibration parameters, and output pixel-level aligned three-dimensional projection data and corresponding mapping relationships. The feature extraction module is used to receive two-dimensional images and three-dimensional point cloud data, and extract features based on the mapping relationship using the built-in trained first feature extraction network and second feature extraction network respectively, and output pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps. The cross-modal prediction module receives pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps, and performs feature mapping using the built-in trained first mapping network and second mapping network respectively, outputting predicted three-dimensional feature maps and predicted two-dimensional feature maps. An anomaly scoring module is used to calculate the first deviation between the predicted 3D feature map and the pixel-level 3D feature map, and the second deviation between the predicted 2D feature map and the pixel-level 2D feature map, and to generate an anomaly scoring map based on the first deviation and the second deviation. The defect determination module is used to perform threshold segmentation on the anomaly scoring map to determine and output the defect location in the flat wire stator weld area.

[0052] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting welding defects in flat wire stators based on cross-modal feature mapping, characterized in that, Includes the following steps: S1. Acquire two-dimensional images and three-dimensional point cloud data of the weld point area of ​​the flat wire stator to be detected, and project the three-dimensional point cloud data onto the two-dimensional image plane based on preset calibration parameters to obtain three-dimensional projection data that is pixel-level aligned with the two-dimensional image. S2. Input the two-dimensional image into the trained first feature extraction network to obtain a pixel-level two-dimensional feature map, and input the three-dimensional point cloud data into the trained second feature extraction network to obtain a pixel-level three-dimensional feature map. S3. Input the pixel-level two-dimensional feature map into the trained first mapping network to generate the predicted three-dimensional feature map, and input the pixel-level three-dimensional feature map into the trained second mapping network to generate the predicted two-dimensional feature map. S4. Based on the first deviation between the predicted 3D feature map and the pixel-level 3D feature map, and the second deviation between the predicted 2D feature map and the pixel-level 2D feature map, generate an anomaly score map representing the degree of anomaly. S5. Determine the location of defects in the flat wire stator weld area based on the anomaly scoring diagram.

2. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, The first feature extraction network specifically includes: The backbone network layer is used to receive two-dimensional images and perform spatial resolution compression and semantic feature extraction on the two-dimensional images through multi-level convolution and pooling operations, outputting a low-resolution feature map with a spatial resolution lower than that of the two-dimensional image. The upsampling recovery layer, connected to the output of the backbone network layer, is used to receive low-resolution feature maps and use a bilinear interpolation algorithm to restore the spatial resolution of the low-resolution feature maps to be consistent with the two-dimensional image. This ensures that each pixel position in the restored feature map corresponds to a high-dimensional semantic feature vector of a preset dimension, thereby obtaining a pixel-level two-dimensional feature map.

3. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, The second feature extraction network specifically includes: The three-dimensional backbone network layer is used to receive three-dimensional point cloud data and perform key point filtering and local geometric feature encoding on the three-dimensional point cloud data through multi-level point cloud sampling and feature aggregation operations, and output a set of sparse key point feature vectors. The number of sparse key point feature vectors is less than the number of original point clouds in the three-dimensional point cloud data. The feature propagation interpolation layer, connected to the output of the 3D backbone network layer, is used to receive sparse keypoint feature vectors and propagate the sparse keypoint feature vectors to each original point cloud location in the 3D point cloud data based on spatial distance weights, thereby outputting a set of point-level feature vectors. The second feature extraction network also includes a projection smoothing layer, which is connected to the output of the feature propagation interpolation layer. The projection smoothing layer is used to project the set of point-level feature vectors onto the two-dimensional image plane based on preset calibration parameters, and to perform zero-filling and convolution smoothing on the hole regions generated after projection, thereby outputting a pixel-level three-dimensional feature map.

4. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, The first mapping network specifically includes: The first input flattening layer is used to receive the two-dimensional feature vector corresponding to each pixel position in the pixel-level two-dimensional feature map, convert the two-dimensional feature vector into a one-dimensional sequence structure, and output the first flattened feature vector. The first feature transformation layer, connected to the first input flattening layer, consists of multiple fully connected processing units stacked sequentially. Each fully connected processing unit contains a linear transformation operation and a nonlinear activation operation. The first feature transformation layer is used to receive the first flattened feature vector and transform the first flattened feature vector from the original two-dimensional feature dimension space to the target three-dimensional feature dimension space through step-by-step linear mapping and nonlinear activation, and output the first cross-modal mapping feature vector. The first output reconstruction layer, connected to the first feature transformation layer, is used to receive the first cross-modal mapping feature vector and restore the first cross-modal mapping feature vector into a feature map structure with the same size as the pixel-level three-dimensional feature map according to the spatial arrangement rules of the pixel-level three-dimensional feature map, and output the predicted three-dimensional feature map.

5. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, The second mapping network specifically includes: The second input flattening layer is used to receive the three-dimensional feature vector corresponding to each pixel position in the pixel-level three-dimensional feature map, convert the three-dimensional feature vector into a one-dimensional sequence structure, and output the second flattened feature vector. The second feature transformation layer, connected to the second input flattening layer, consists of multiple fully connected processing units stacked sequentially. Each fully connected processing unit contains a linear transformation operation and a nonlinear activation operation. The second feature transformation layer is used to receive the second flattened feature vector and transform the second flattened feature vector from the original three-dimensional feature dimension space to the target two-dimensional feature dimension space through step-by-step linear mapping and nonlinear activation, and output the second cross-modal mapping feature vector. The second output reconstruction layer, connected to the second feature transformation layer, is used to receive the second cross-modal mapping feature vector and restore the second cross-modal mapping feature vector into a feature map structure with the same size as the pixel-level two-dimensional feature map according to the spatial arrangement rules of the pixel-level two-dimensional feature map, and output the predicted two-dimensional feature map.

6. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, Step S1 specifically includes: A structured light camera is used to simultaneously trigger the acquisition of two-dimensional grayscale images and three-dimensional point cloud data of the flat wire stator welding point area. The resolution of the two-dimensional grayscale image is a preset height multiplied by a preset width, and the three-dimensional point cloud data contains three-dimensional spatial point coordinates in a number equal to the number of original point clouds. Based on the internal parameter matrix and hand-eye calibration matrix obtained by pre-calibration of the structured light camera, the coordinates of each three-dimensional point in the three-dimensional point cloud data are converted into pixel coordinates in the pixel coordinate system of the two-dimensional grayscale image. Based on the pixel coordinates corresponding to each three-dimensional spatial point coordinate, the geometric information of each three-dimensional spatial point coordinate is mapped to the corresponding pixel position in the two-dimensional grayscale image. For pixel positions in the two-dimensional grayscale image that are not covered by any three-dimensional spatial point coordinate mapping, the geometric information corresponding to the pixel position is set to zero. The mapped geometric information is organized into three-dimensional projection data according to the spatial arrangement rules of the two-dimensional grayscale image. Each pixel position in the three-dimensional projection data corresponds to a geometric feature representation of a preset dimension, thereby achieving pixel-level alignment between the three-dimensional projection data and the two-dimensional grayscale image.

7. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, Step S4 specifically includes: For each pixel position in the pixel-level 3D feature map and the predicted 3D feature map, extract the real 3D feature vector and the predicted 3D feature vector corresponding to the pixel position, and calculate the first cosine distance between the real 3D feature vector and the predicted 3D feature vector as the first deviation. For each pixel position in the pixel-level two-dimensional feature map and the predicted two-dimensional feature map, extract the real two-dimensional feature vector and the predicted two-dimensional feature vector corresponding to the pixel position, and calculate the second cosine distance between the real two-dimensional feature vector and the predicted two-dimensional feature vector as the second deviation. Multiply the first cosine distance corresponding to each pixel position by the preset three-dimensional weight coefficient to obtain the three-dimensional anomaly contribution value, multiply the second cosine distance by the preset two-dimensional weight coefficient to obtain the two-dimensional anomaly contribution value, and add the three-dimensional anomaly contribution value and the two-dimensional anomaly contribution value to obtain the anomaly score of the pixel position. The anomaly scores of all pixel locations are organized according to the spatial arrangement rules of the two-dimensional image to generate an anomaly score map with the same size as the two-dimensional image. The value of each pixel location in the anomaly score map represents the degree of probability that a welding defect exists at that location.

8. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, Step S5 specifically includes: The abnormal score map is binarized using a preset segmentation threshold. Pixels with abnormal scores greater than or equal to the preset segmentation threshold are marked as suspected defect pixels, and pixels with abnormal scores less than the preset segmentation threshold are marked as background pixels, thus obtaining a binarized abnormal mask map. Connectivity analysis is performed on the binarized anomaly mask image to extract all connected regions composed of suspected defect pixels in the binarized anomaly mask image, and the boundary coordinates and area information of each connected region are obtained. Based on the area information of each connected region, connected regions with an area smaller than the preset minimum defect area are filtered out, and the remaining connected regions are determined as valid defect regions. Based on the boundary coordinates of each valid defect region, a corresponding defect location marker box is generated on the two-dimensional image, and the defect location detection results of the flat wire stator weld area are output.

9. The method for detecting welding defects in flat wire stators based on cross-modal feature mapping according to claim 1, characterized in that, The training process for the first mapping network and the second mapping network includes: Construct a training dataset containing only normal welding samples, wherein the normal welding samples include pairs of two-dimensional training images and three-dimensional training point clouds; Two-dimensional training images and three-dimensional training point clouds are input into the first feature extraction network and the second feature extraction network, respectively, to obtain pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps for training. Using the training pixel-level two-dimensional feature map and the training pixel-level three-dimensional feature map as supervision signals, the first mapping network and the second mapping network are jointly trained using the cosine similarity loss function until the loss value converges.

10. A flat wire stator welding defect detection system based on cross-modal feature mapping, characterized in that, include: The data acquisition and alignment module is used to simultaneously acquire two-dimensional images and three-dimensional point cloud data of the flat wire stator welding point area, and project the three-dimensional point cloud data onto the two-dimensional image plane according to the built-in preset calibration parameters, and output pixel-level aligned three-dimensional projection data and corresponding mapping relationships. The feature extraction module is used to receive two-dimensional images and three-dimensional point cloud data, and extract features based on the mapping relationship using the built-in trained first feature extraction network and second feature extraction network respectively, and output pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps. The cross-modal prediction module receives pixel-level two-dimensional feature maps and pixel-level three-dimensional feature maps, and performs feature mapping using the built-in trained first mapping network and second mapping network respectively, outputting predicted three-dimensional feature maps and predicted two-dimensional feature maps. An anomaly scoring module is used to calculate the first deviation between the predicted 3D feature map and the pixel-level 3D feature map, and the second deviation between the predicted 2D feature map and the pixel-level 2D feature map, and to generate an anomaly scoring map based on the first deviation and the second deviation. The defect determination module is used to perform threshold segmentation on the anomaly scoring map to determine and output the defect location in the flat wire stator weld area.

Images