High-precision multi-view image recognition and detection method based on CCD (Charge Coupled Device) camera

Abstract

The invention provides a high-precision multi-view image recognition detection method based on a CCD camera, and the method comprises the steps: employing a convolutional neural network to extract the noise perception features of each view image through a multi-view image set shot by a CCD camera array; carrying out semantic segmentation by adopting a self-adaptive multi-view semantic segmentation network; constructing final feature representation of the target area based on the enhanced features and the key features; performing view angle consistency matching based on the final feature representation of the target region by adopting similarity calculation; and based on the final target feature map, combining a classification module to identify the defect type, and outputting the type and position information of the defective product. According to the method, the noise removal of the multi-view image is realized through a self-supervised learning method, and the image features under different views are optimized in combination with a self-adaptive semantic segmentation technology, so that the accurate target recognition of the multi-view image is realized.

Description

Technical Field

[0001] The present invention belongs to the field of high-precision multi-view image recognition, and particularly relates to a high-precision multi-view image recognition and detection method based on a CCD camera. Background Art

[0002] With the development of industrial automation and precision detection technology, CCD (Charge-Coupled Device) high-definition detection equipment has been widely used in the electronic manufacturing industry, especially in the defect detection of motor components. Defects in motors may manifest as problems such as missing soldering, missing glue coating, and foreign object attachment, which have a direct impact on the performance and safety of products. Therefore, precise and efficient defect detection has become an important link in improving product quality and reducing the defective rate.

[0003] Although current CCD detection technology can already provide high-resolution images and detect some common defects, there are still many challenges in practical applications. First, due to the complex and variable surface texture of the motor, the CCD detection equipment may not be able to fully capture all defects from certain perspectives, resulting in missed detections and false detections. In addition, due to environmental noise, light changes, and the perspective limitations of the equipment itself, the manifestations of motor defects in different images may vary significantly, which greatly affects the accuracy of defect detection.

[0004] Currently, most traditional methods for motor defect detection rely on manual feature extraction and rule-based judgment. These methods are insufficient in terms of efficiency and accuracy for defect recognition in high-complexity and large-scale production lines. At the same time, although some automated detection methods based on convolutional neural networks (CNNs) have made preliminary progress, these methods are usually affected by factors such as light, angle, and noise, and still perform poorly in the detection of some small defects or minor differences.

[0005] Therefore, there is an urgent need for a new type of CCD detection technology that can improve the detection accuracy of motor defective products (such as missing soldering, missing glue coating, foreign object attachment, etc.) through more precise image processing and feature optimization, and overcome problems such as perspective differences, light changes, and noise interference. Summary of the Invention

[0006] The purpose of the present invention is to propose a high-precision multi-view image recognition and detection method based on a CCD camera, which provides an innovative multi-view image denoising and precise recognition solution through the combination of self-supervised learning, multi-view adaptive segmentation, and image feature enhancement.

[0007] To achieve the above purpose, the present invention provides a high-precision multi-view image recognition and detection method based on a CCD camera, and the method includes:

[0008] A multi-view image set is captured by a CCD camera array. The noise-aware features of each view image are extracted using a convolutional neural network. Based on the noise-aware features of each view image, a global noise estimate is fused through weighted averaging, and the global noise estimate is denoised based on the convolutional neural network to obtain a denoised multi-view image set;

[0009] For the denoised multi-view image set, an adaptive multi-view semantic segmentation network is used for semantic segmentation to obtain a semantically segmented multi-view image set. Each image in the multi-view image set is the result of semantic segmentation of its corresponding view image and contains semantic label information for each pixel. The adaptive multi-view semantic segmentation network is used to jointly model the features of the denoised multi-view images and achieve semantic segmentation through shared feature learning and perspective feature weighting;

[0010] The target region features are adaptively enhanced according to the semantic label information of each pixel to obtain enhanced features, and a self-attention mechanism is used to extract the key features of the target region. The final feature representation of the target region is constructed based on the enhanced features and the key features;

[0011] Similarity calculation is used to perform perspective consistency matching based on the final feature representation of the target region, and feature fusion is performed based on the final feature representation of the target region. The final target feature map is generated based on convolutional operations and pooling operations;

[0012] Based on the final target feature map, the defective product region is located through feature enhancement and bounding box regression, and the defect type is identified by combining a classification module, and the category and location information of the defective product are output.

[0013] Furthermore, the noise-aware features are extracted by a convolutional neural network from the gradients and local texture features of the multi-view images. The weighted average fusion includes:

[0014] The noise-aware features of each view image are given different weights according to their quality, and the noise-aware features are fused through weighted averaging to obtain a global noise estimate;

[0015] The denoising of the global noise estimate based on the convolutional neural network includes:

[0016] Denoising processing is performed using a convolutional neural network, and the denoised image is restored from each multi-view image by removing the global noise estimate. At the same time, the smoothness constraint of the image is introduced during the restoration process to ensure that the structure and texture of the image are preserved.

[0017] Further, the convolutional neural network adopts a self-supervised learning method to minimize the reconstruction error after image denoising; then the loss function of the convolutional neural network includes the image reconstruction error and the noise estimation error, and the network weights are adjusted through self-supervised learning to gradually approximate the true noise.

[0018] Further, the adaptive multi-view semantic segmentation network includes a deep convolutional neural network; then, when using the adaptive multi-view semantic segmentation network for semantic segmentation, a multi-view image set after semantic segmentation is obtained, including:

[0019] Based on the similarity between each perspective image, the features of each perspective are weighted and fused to adaptively adjust the contribution of different perspective images to the segmentation result, and the features of the weighted and fused perspective are obtained;

[0020] The features of the weighted and fused perspective of each perspective image are adjusted through an adaptive weighting mechanism to obtain the final global features; wherein, the adaptive weighting mechanism calculates the weighted sum of the features of the weighted and fused perspective by using the weighted sum of the similarities between the multi-view images and other images as the weight;

[0021] The final global features are input into a deep convolutional neural network for final semantic segmentation to obtain a multi-view image set after semantic segmentation.

[0022] Further, the adaptive enhancement performs a weighted sum calculation based on the distance from the i-th pixel to the center of the target area and the speech label information to obtain the weighting coefficient of each pixel in the target area. According to the weighting coefficient of each pixel in the target area, the weights of each pixel can be dynamically adjusted according to the semantic category and position of the target to enhance the representation of the target features; the self-attention mechanism performs a weighted sum calculation based on semantic suppression on the weighting coefficient of each pixel in the target area and the semantic features of the corresponding pixel to obtain the key features; wherein, the semantic suppression suppresses background interference by calculating the contrast between the background area and the target area.

[0023] Further, the use of similarity calculation for perspective consistency matching of the final feature representation based on the target area includes:

[0024] The perspective consistency loss function is determined through the target features from the i-th and j-th perspectives;

[0025] Based on the perspective consistency loss function, the target features from different perspectives are optimized so that the target features with similar appearances are consistent with each other.

[0026] Furthermore, the feature fusion based on the final feature representation of the target region is determined by weighted fusion based on the feature difference metric between the features from the i-th perspective and the features from all other perspectives, and the fused target feature map is obtained.

[0027] Furthermore, the feature enhancement includes high-frequency texture enhancement and edge strengthening of the target feature map, and amplifies the texture features of the areas with solder missing, glue missing, and foreign object attachment.

[0028] Furthermore, the bounding box regression is performed through the target feature map to perform bounding box regression and accurately locate the defects of each motor;

[0029] Among them, the smooth L1 loss is used to optimize the positioning accuracy, and at the same time, a spatial transformation regularization term is introduced, and the spatial transformation regularization term is used to handle the positioning error caused by the characteristics of the CCD device.

[0030] Furthermore, the classification module introduces a fuzzy adversarial loss function as a regularization term to simulate the fuzzy scenes of low-resolution or noisy images and improve the robustness of the classification module to fuzzy images.

[0031] The beneficial technical effects of the present invention are at least as follows:

[0032] The present invention effectively solves the problems of noise interference, perspective difference, and information inconsistency existing in the existing multi-perspective image processing technology. Different from the existing technology, the present invention realizes noise removal of multi-perspective images through a self-supervised learning method, and at the same time combines an adaptive semantic segmentation technology to optimize the image features under different perspectives, so as to realize accurate target recognition of multi-perspective images. The specific invention points are as follows:

[0033] Multi-perspective image noise modeling and denoising self-supervised learning:

[0034] The present invention constructs a multi-perspective image noise modeling and denoising network through a self-supervised learning mechanism. By extracting gradient information and local texture features from each perspective image and combining an adaptive weighted fusion strategy, the noise in the multi-perspective image is accurately estimated and removed, ensuring that key details and image structures are retained during the denoising process. This innovation enables the denoising network to not only adapt to different noise types but also adaptively adjust the denoising intensity in multi-perspective data, thereby improving the denoising effect.

[0035] Adaptive multi-perspective semantic segmentation:

[0036] In multi-view image processing, it is often difficult to utilize complementary information from different views by directly applying traditional semantic segmentation networks. To address this issue, the present invention proposes an adaptive multi-view semantic segmentation network, which calculates the similarity between images from each view and adaptively adjusts the contribution of each view to the segmentation result according to the view differences. This method can improve the accuracy of semantic segmentation, especially in the case of large view differences, ensuring accurate extraction of key semantic information in the images.

[0037] Target Feature Enhancement and Key Feature Extraction:

[0038] Regarding the feature expression of the target region, the present invention enhances the expression ability of target features through an adaptive feature enhancement and key feature extraction mechanism. After image semantic segmentation, by combining the semantic information of the target region, dynamic weights are assigned to the features of the target region, thereby improving the distinctiveness of target features. At the same time, a semantic suppression mechanism is introduced to reduce background interference, ensuring the accurate extraction of target features in complex environments and providing high-quality feature support for subsequent target recognition and localization.

[0039] Feature Fusion with Perspective Consistency Optimization:

[0040] The present invention proposes a feature fusion strategy based on perspective consistency optimization. By performing consistency optimization and weighted fusion on target features from different views, it ensures that the features of the target can be accurately retained in multi-view images. The introduction of perspective consistency matching further improves the quality of feature fusion, suppresses the interference of redundant information, thereby generating a more accurate and efficient feature map and enhancing the accuracy of target recognition and localization. Description of the Drawings

[0041] The present invention is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the following drawings.

[0042] Figure 1 It is a flowchart of a high-precision multi-view image recognition and detection method based on a CCD camera according to the present invention. Detailed Description of the Embodiment

[0043] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0044] Such as Figure 1As shown, a high-precision multi-view image recognition and detection method based on a CCD camera provided by an embodiment of the present invention includes the following steps S1-S5:

[0045] S1. For the multi-view image set captured by the CCD camera array, use a convolutional neural network to extract the noise perception features of each view image, fuse them into a global noise estimate value through weighted average based on the noise perception features of each view image, and denoise the global noise estimate value based on the convolutional neural network to obtain a denoised multi-view image set.

[0046] Specifically, the input of step S1 is the multi-view image set captured by the CCD camera array. Assume these image sets are I1, I2, …, I n , where each image I i comes from the i-th CCD camera in the array. Different cameras may generate noise interference due to factors such as shooting angle, lighting, and lens distortion. The goal is to build a system for modeling and denoising the image noise characteristics of these CCD cameras through self-supervised learning, eliminating noise and maintaining the key details of the image. Since the CCD camera array often shows noise differences caused by different hardware conditions and environmental factors in multi-view images, the present invention designs a noise perception model that combines the gradient information and local texture of the image to model these noise characteristics. Specifically, the present invention uses a convolutional neural network (CNN) to extract the noise perception features i of each view image I This process not only captures the high-frequency noise in the image but also considers texture distortion. The present invention defines the following feature extraction formula:

[0047]

[0048] where, represents the gradient of image I i , which is used to capture the noise and high-frequency components in the image, represents the local texture feature of image I i , usually extracted through methods such as gray-level co-occurrence matrix, and θ is the trainable parameter of the CNN. The goal of this step is to extract the potential noise features from the CCD camera images and provide basic information for the denoising stage.

[0049] Further, in the multi-view image, since the noise characteristics of each camera view are not exactly the same, the present invention adopts a strategy based on adaptive weighted fusion to calculate the global noise estimate by combining the noise characteristics of each view. Specifically, the noise characteristics of each view are assigned different weights w i according to their quality (including clarity, noise intensity, etc.), and these noise characteristics are fused through weighted average. The fused global noise estimate value Obtained by the following formula:

[0050]

[0051] Wherein, is the noise feature extracted from the i-th perspective image, and w i is the adaptive weight of the i-th perspective, and the specific calculation method is:

[0052]

[0053] The weight w i is calculated in such a way that images with less noise or clearer images have lower weights, while images with stronger noise (such as images taken under insufficient light or with large lens distortion) have higher weights, thus ensuring the effectiveness of noise fusion.

[0054] Furthermore, after obtaining the global noise estimate, the present invention uses a convolutional neural network (CNN) for denoising. By removing the estimated noise from each perspective image I i to recover the clear image I cleaned . The denoising operation is as follows:

[0055]

[0056] Wherein, I cleaned is the denoised image, is the global noise estimate.

[0057] To prevent loss of image details during the denoising process, the present invention introduces a regularization constraint to ensure that the structure and texture of the image are preserved. The specific loss function is as follows:

[0058]

[0059] Wherein, λ is the weight of the regularization term, controlling the smoothness degree, is the smoothness constraint of the image, ensuring detail preservation.

[0060] Furthermore, to train the denoising network, the present invention adopts a self-supervised learning method to minimize the reconstruction error after image denoising. The loss function includes the image reconstruction error and the noise estimation error, and the network weights are adjusted through self-supervised learning to gradually approximate the true noise. The training loss function is as follows:

[0061]

[0062] Wherein, α is a tuning hyperparameter, controlling the balance between the denoising error and the noise estimation error, and ∈ true is the true noise.

[0063] The final output is the denoised multi-view image set I cleaned ={I1′, I2′, …, I n ′}, and these denoised images will be used for subsequent multi-view image recognition and detection tasks.

[0064] S2. For the denoised multi-view image set, an adaptive multi-view semantic segmentation network is used for semantic segmentation to obtain the multi-view image set after semantic segmentation. Each image in the multi-view image set is the result of semantic segmentation of its corresponding view image and contains semantic label information of each pixel. Among them, the adaptive multi-view semantic segmentation network is used to jointly model the features of the denoised multi-view images and achieve semantic segmentation by means of shared feature learning and view feature weighting.

[0065] Specifically, due to significant differences that may exist in aspects such as view, illumination, and occlusion in multi-view images, directly inputting the denoised images into a traditional semantic segmentation network may not fully utilize the view information between the images. Therefore, the present invention designs an adaptive multi-view semantic segmentation network, which can make adaptive adjustments according to the features of each view image and its association with other view images. Specifically, the present invention jointly models the features of multi-view images and achieves more accurate semantic segmentation by means of shared feature learning and view feature weighting. The main structure of this network includes the following parts:

[0066] Furthermore, the present invention designs an adaptive feature fusion mechanism for fusing information from different view images. By calculating the similarity S ij between each pair of view images, the features of each view are weighted and fused using this similarity. This mechanism can adaptively adjust the contribution of different view images to the segmentation result. The similarity S ij is calculated as follows:

[0067]

[0068] where cosine_similarity(I i ′, I j ′) is the cosine similarity between the denoised images I i ′ and I j ′, which is used to measure the similarity between two view images, and ∥I i ′∥2 and ∥I j ′∥2 are the L2 norms of the images I i ′ and I j ′ respectively, which are used to normalize the image features. This similarity metric value S ijIt will be used as a weight to weighted-fuse the features of each perspective image, so as to better reflect the similarities and differences between perspectives.

[0069] Furthermore, after feature fusion, the present invention adjusts the features of each perspective image through an adaptive weighting mechanism to obtain an optimal segmentation effect. The features of each perspective image are weighted to obtain the final global features The formula is as follows:

[0070]

[0071] Wherein, is the feature of the i-th perspective image obtained through the perspective feature fusion module, and w i is the adaptive weighting coefficient, which is adjusted based on image quality, perspective difference, and the complexity of image content. The calculation method of w i is as follows:

[0072]

[0073] Wherein, is the weighted sum of the similarity between image I i ′ and other images, which is used to measure the importance of this image in the overall segmentation.

[0074] Through this weighting mechanism, the images of each perspective are dynamically adjusted according to their relationships and contribution degrees with the images of other perspectives, so as to more accurately utilize multi-perspective information during semantic segmentation.

[0075] Furthermore, after obtaining the globally weighted-fused features , the present invention inputs them into a deep convolutional neural network (CNN) for final semantic segmentation. The present invention uses structures such as U-Net, which has been widely applied to image segmentation tasks and can perform accurate boundary segmentation while retaining image details. The loss function of the segmentation task is:

[0076]

[0077] Wherein, y k is the pixel label of the k-th class of the ground truth label image, is the pixel label of the k-th class predicted by the network, K is the number of classes, is the regularization term of image I cleaned to prevent the segmentation result from being too rough.

[0078] The final output is the multi-perspective image set after semantic segmentation each image It is the result after semantic segmentation of the corresponding perspective image. These segmented images will provide precise semantic information for subsequent image recognition, detection, and other analysis tasks.

[0079] S3. Adaptive enhancement of the target region features is performed based on the semantic label information of each pixel to obtain enhanced features, and a self-attention mechanism is used to extract the key features of the target region. The final feature representation of the target region is constructed based on the enhanced features and the key features.

[0080] Specifically, the input of step 3 is the output of step 2, which is an image set processed by denoising and an adaptive multi-perspective semantic segmentation network. Each image has undergone semantic segmentation and contains the semantic label information of each pixel.

[0081] Furthermore, the goal of this step is to improve the feature representation ability of the target based on semantic information, combined with an adaptive feature enhancement and target key feature extraction mechanism, and further improve the accuracy of subsequent target recognition and detection. The core strategy is to enhance the features of the target region by assigning dynamic weights to the target region, and at the same time extract key target features through semantic information.

[0082] Furthermore, according to the semantic label information output in step 2, the present invention adaptively enhances the features of the target region. After semantic segmentation of the image, the present invention can clarify the semantic category of each pixel, and enhance the expression ability of the target region by weighting the pixel features of the target region. For the weighting coefficient w of each pixel in the target region i , its calculation not only takes into account the distance d of the pixel from the target center i , but also introduces a weighting factor α based on the target category i to adapt to the target feature differences of different semantic categories. The specific calculation formula is as follows:

[0083]

[0084] where d i is the distance from the i-th pixel to the center of the target region, and σ is a hyperparameter used to control the smoothness of the weight; α i is the weighting factor based on the semantic label (for example, for the background region, α i is smaller, and for the target region, α i is larger). By combining d i and α i , the present invention can dynamically adjust the weights of each pixel according to the semantic category and position of the target, and optimize the representation of the target features.

[0085] Furthermore, after enhancing the features of the target region, the present invention further extracts the key features of the target. To ensure accurate identification of the target under complex backgrounds and different perspectives, the present invention introduces a feature suppression mechanism based on semantic information to eliminate the interference of background information on target features. The key feature extraction process incorporates a self-attention mechanism, enabling the model to focus on important regions in the image. The present invention achieves feature extraction through the following calculation method:

[0086]

[0087] Among them, is the semantic feature of the i-th pixel, and w i is the weighting coefficient of the target region (as described above), is the semantic suppression factor, which suppresses background interference by calculating the contrast between the background region and the target region. The calculation method of

[0088]

[0089] Among them, is the feature vector of the background region, and β is a balance factor used to adjust the contrast between the target and background features. Through this suppression mechanism, the model can focus more on the key information of the target region, suppress background interference, and thus improve the recognition accuracy of the target.

[0090] Furthermore, finally, based on the enhanced features and the extracted key features, the present invention constructs the final feature representation of the target. This representation combines the adaptively enhanced target region features and the key features extracted from semantic information, and is weighted and fused to obtain the final target feature representation:

[0091]

[0092] Among them, is the final feature representation of the target region, is the weighting coefficient matrix, which is dynamically adjusted based on the semantic labels and the relative importance of the target region. This feature representation contains the semantic information and geometric features of the target, and can effectively support subsequent target recognition and localization.

[0093] By introducing the target feature enhancement based on semantic information and the adaptive key feature extraction mechanism, this step can effectively improve the feature representation ability of the target. In the multi-perspective and complex background of the image, the solution of the present invention can automatically focus on the target region, reduce background interference through semantic suppression, and finally construct a more accurate target feature representation. This process plays a key role in improving the target detection accuracy and localization ability.

[0094] S4. Use similarity calculation to perform perspective consistency matching for the final feature representation based on the target region, perform feature fusion based on the final feature representation of the target region, and generate the final target feature map based on convolutional operations and pooling operations.

[0095] Specifically, the main objective of this step is to perform effective feature fusion by optimizing perspective consistency and combining target features from different perspectives, thereby generating a final high-quality feature map for subsequent target recognition and localization. To achieve this goal, the present invention designs a feature fusion strategy based on adaptive perspective consistency matching and optimizes the multi-perspective feature maps to ensure consistency and accuracy across different perspectives.

[0096] Furthermore, under multiple perspectives, the appearance of the target may vary. To ensure feature consistency, the present invention first introduces a perspective consistency optimization mechanism to optimize the similarity between features by calculating the feature changes of the target under different perspectives. In this process, the present invention performs matching based on the geometric features and semantic information of the target.

[0097] The present invention uses a perspective consistency loss function to measure the similarity of features under different perspectives, which is specifically defined as:

[0098]

[0099] where, and are the target features from the i-th and j-th perspectives respectively; is the perspective consistency weight, which is calculated based on the perspective difference and is used to adjust the feature matching strength from different perspectives. ∥·∥ 2 is the Euclidean distance metric used to measure the difference between feature vectors. Through this loss function, the present invention can optimize the target features from different perspectives, making the target features with similar appearances consistent, thereby enhancing the accuracy of the target features.

[0100] Furthermore, based on the perspective consistency matching, the present invention then performs feature fusion to effectively synthesize the target features from multiple perspectives. Feature fusion should not only retain the important information from each perspective but also suppress redundant or noisy information. The present invention designs an adaptive feature fusion function to perform weighted fusion on the features from different perspectives to obtain a unified target feature map. The fusion process is as follows:

[0101]

[0102] where, is the fused target feature map; σ iis the weighting coefficient from the i-th perspective, representing the contribution of this perspective to the final feature map. σ i The value of σ is dynamically adjusted according to perspective consistency and feature importance. The weighting coefficient σ i is calculated as follows:

[0103]

[0104] where D i is the measure of the difference between the features from the i-th perspective and the features from all other perspectives, usually calculated using the Euclidean distance. γ is a hyperparameter used to control the smoothness of the weighting coefficient. Through this weighted fusion strategy, the present invention can ensure that the most valuable perspective information contributes the most to the final feature map, thereby enhancing the recognition ability of the target.

[0105] Furthermore, after completing the feature fusion, the present invention further optimizes the generated feature map to ensure its more prominent performance in the target detection task. By performing convolutional operations and pooling operations on the fused feature map, the present invention can extract high-level semantic features and further compress the feature dimension to generate the final target feature map Its calculation formula is:

[0106]

[0107] where: Conv(·) represents the convolutional operation for extracting high-level features; Pool(·) represents the pooling operation for reducing the dimension of the feature map and retaining the most important information; represents the concatenation operation of features.

[0108] Finally, the generated target feature map will provide precise feature support for subsequent target detection and localization.

[0109] Through perspective consistency optimization and feature fusion, this step effectively improves the consistency and accuracy of target features under different perspectives. Using an adaptive perspective consistency loss function and a weighted fusion mechanism, the present invention can optimize the feature representation of the target under different perspectives and generate a more precise target feature map. This process is of great significance for the multi-perspective detection and localization of the target and can effectively improve the robustness and accuracy of the system in practical applications.

[0110] S5. Based on the final target feature map, locate the defective product area through feature enhancement and bounding box regression, and combine the classification module to identify the defect type, and output the category and location information of the defective product.

[0111] Specifically, the input is the output of step S4, that is, the final target feature map set after perspective consistency optimization, feature fusion, and enhancement processing

[0112]

[0113] Each is a feature map containing various defective products on the motor surface (such as solder leakage, glue leakage, foreign object attachment, etc.), providing a deep understanding of the image content and texture. Our goal is to perform accurate defect detection and localization based on these optimized feature maps.

[0114] Furthermore, although the feature map in step S4 has been basically optimized, to further enhance the saliency of the target defect and ensure efficient recognition, we add a feature enhancement module. By using special convolution operations and enhancement functions, we further amplify the features of defective products. Through such enhancement, we can better reveal those defects that may be due to camera perspective changes or incomplete displays.

[0115] The operation of the feature enhancement module is as follows:

[0116]

[0117] Among them, ε represents the feature enhancement operation. It enhances the defect features such as solder leakage, glue leakage, and foreign object attachment on the motor surface through methods such as high-frequency texture enhancement filters, edge enhancement, and blur intensification, making them more prominent and easy to identify. Specifically, is the enhanced target feature map, where the texture features of the defect area are more obvious, facilitating subsequent processing.

[0118] Furthermore, after obtaining the enhanced feature map we input it into the target localization module. The core task of this module is to locate the defective product area according to the texture features in the image. Through the optimized feature map, we perform bounding box regression and accurately locate the defects of each motor.

[0119] For each defect (such as solder leakage, glue leakage, foreign object attachment, etc.), we not only predict its position but also calculate its actual size and direction in the image. To this end, the smooth L1 loss is used to optimize the localization accuracy, and an additional spatial transformation regularization term is introduced to help better handle the distortion and perspective changes in CCD images.

[0120] The calculation formula for localization optimization is as follows:

[0121]

[0122] Among them, is the true bounding box of target i (i.e., the actual area position of solder leakage, glue leakage, or foreign object attachment); is the predicted bounding box; is the localization loss, which calculates the difference between the predicted bounding box and the ground truth bounding box using smooth L1 loss; is the spatial transformation loss, which is used to correct the localization error caused by CCD device distortion or imperfect perspective; λ spatial is the weight of the spatial transformation regularization term, which controls its impact on model training.

[0123] This spatial transformation loss is specifically designed for the distortion that appears on the surface of the defective product by the motor. It can help handle the localization error caused by the characteristics of the CCD device (such as optical distortion, perspective change, etc.).

[0124] Furthermore, once the accurate localization of the defective product area is completed, the next step is to classify each defect area to determine its specific type. For example: missing soldering, missing glue coating, foreign object attachment, etc. To improve the detection accuracy, we introduce a deep convolutional feature separation module, which extracts fine-grained texture features from the localization area and classifies and identifies them.

[0125] The formula for the classification process is as follows:

[0126]

[0127] Among them, is the classification module, which is used to classify the target category (missing soldering, missing glue coating, foreign object attachment, etc.) according to the enhanced feature map; is the predicted class label.

[0128] To cope with the possible blurry and unclear situations in the CCD device images, we add the Blur Adversarial Loss as a regularization term to ensure that the model can correctly classify defective products even in low-resolution or noisy images. The calculation method of the Blur Adversarial Loss is as follows:

[0129]

[0130] Among them, is the standard classification loss (such as mean squared error); is the blur loss based on adversarial training, which is used to simulate different degrees of blur in the image, so as to improve the robustness of the model to blurry or noisy images; β is the weight hyperparameter of the Blur Adversarial Loss.

[0131] Training and loss function design:

[0132] To ensure the overall performance of the entire model in localization and classification tasks, we combined multiple loss functions such as localization loss, classification loss, and fuzzy adversarial loss for multi-objective training. The final total loss function is as follows:

[0133]

[0134] Where: and are the classification and localization losses respectively; is the fuzzy adversarial loss, which helps improve the recognition ability for low-quality images; is the spatial transformation loss, which solves the problems caused by device perspective or optical distortion; λ1 and λ2 are the weight hyperparameters of each loss.

[0135] Output and Results:

[0136] The final output will be:

[0137] The predicted category of each defective area (such as missing soldering, missing glue coating, foreign object attachment, etc.);

[0138] The exact bounding box position of each defective area.

[0139] These outputs can be directly applied to the CCD detection device to help achieve efficient and accurate automated detection and classification, thereby greatly reducing the risk of defective products flowing out and ensuring the quality of the products.

[0140] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is made here.

[0141] In addition, for the technical details not described in detail in this embodiment, reference can be made to the parameter operation method provided in any embodiment of the present invention, which will not be elaborated here.

[0142] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0143] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory / random access memory, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0145] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A high-precision multi-view image recognition and detection method based on a CCD camera, characterized in that, The method includes: Taking a multi-view image set captured by a CCD camera array, using a convolutional neural network to extract the noise-aware features of each view image, fusing the noise-aware features of each view image into a global noise estimate through weighted averaging, and denoising the global noise estimate based on the convolutional neural network to obtain a denoised multi-view image set; For the denoised multi-view image set, using an adaptive multi-view semantic segmentation network for semantic segmentation to obtain a multi-view image set after semantic segmentation, where each image in the multi-view image set is the result of semantic segmentation of its corresponding view image and contains semantic label information of each pixel; wherein, the adaptive multi-view semantic segmentation network is used to jointly model the features of the denoised multi-view images and achieve semantic segmentation by means of shared feature learning and perspective feature weighting; Adaptive enhancement of the target region features according to the semantic label information of each pixel to obtain enhanced features, and using a self-attention mechanism to extract the key features of the target region, and constructing the final feature representation of the target region based on the enhanced features and the key features; Using similarity calculation to perform perspective consistency matching based on the final feature representation of the target region, and performing feature fusion based on the final feature representation of the target region, and generating the final target feature map based on convolutional operations and pooling operations; Based on the final target feature map, locating the defective product region through feature enhancement and bounding box regression, combining a classification module to identify the defect type, and outputting the category and location information of the defective product.

2. The high-precision multi-view image recognition and detection method based on a CCD camera according to claim 1, characterized in that, The noise-aware features are extracted by a convolutional neural network from the gradients and local texture features of the multi-view images; the weighted average fusion includes: The noise-aware features of each view image are assigned different weights according to their quality, and the noise-aware features are fused through weighted averaging to obtain a global noise estimate; The denoising of the global noise estimate based on the convolutional neural network includes: Using a convolutional neural network for denoising processing, and by removing the global noise estimate, restoring the denoised image from each multi-view image, and introducing a smoothness constraint of the image during the restoration process to ensure that the structure and texture of the image are retained.

3. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 2, characterized in that, The convolutional neural network adopts a self-supervised learning method to minimize the reconstruction error after image denoising; then the loss function of the convolutional neural network includes the image reconstruction error and the noise estimation error, and the network weights are adjusted through self-supervised learning to gradually approximate the true noise.

4. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 1, characterized in that The adaptive multi-view semantic segmentation network includes a deep convolutional neural network; then using the adaptive multi-view semantic segmentation network for semantic segmentation to obtain a multi-view image set after semantic segmentation includes: Based on the similarity between each view image, weighting and fusing the features of each view to adaptively adjust the contribution of different view images to the segmentation result, and obtaining the features of the weighted-fused view. Adjust the features of the weighted fusion perspective of each perspective image through an adaptive weighting mechanism to obtain the final global features; wherein, the adaptive weighting mechanism calculates the weighted sum of the features of the weighted fusion perspective by using the weighted sum of the similarities between the multi-perspective image and other images as the weight. Input the final global features into a deep convolutional neural network for final semantic segmentation to obtain a multi-perspective image set after semantic segmentation.

5. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 1, characterized in that The adaptive enhancement calculates the weighted sum based on the distance from the i-th pixel to the center of the target area and the voice label information to obtain the weighting coefficient of each pixel in the target area. According to the weighting coefficient of each pixel in the target area, the weights of each pixel can be dynamically adjusted according to the semantic category and position of the target to enhance the representation of the target features. The self-attention mechanism calculates the weighted sum based on semantic suppression between the weighting coefficient of each pixel in the target area and the semantic features of the corresponding pixel to obtain the key features; wherein, the semantic suppression suppresses background interference by calculating the contrast between the background area and the target area.

6. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 1, characterized in that, The use of similarity calculation for perspective consistency matching of the final feature representation based on the target area includes: Determine the perspective consistency loss function through the target features from the i-th and j-th perspectives. Optimize the target features from different perspectives based on the perspective consistency loss function so that the target features with similar appearances are consistent with each other.

7. A high-precision multi-view image recognition and detection method based on a CCD camera according to any one of claims 5 to 6, characterized in that, The feature fusion based on the final feature representation of the target area is determined by weighted fusion based on the feature difference metric between the features from the i-th perspective and the features from all other perspectives to obtain the fused target feature map.

8. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 7, characterized in that, The feature enhancement includes high-frequency texture enhancement and edge strengthening of the target feature map, and amplifies the texture features of the areas with missing soldering, missing glue application, and foreign object attachment.

9. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 7, characterized in that, The bounding box regression performs bounding box regression through the target feature map and accurately locates the defects of each motor. Among them, the smooth L1 loss is used to optimize the positioning accuracy, and at the same time, a spatial transformation regularization term is introduced, and the spatial transformation regularization term is used to handle the positioning error caused by the characteristics of the CCD device.

10. A high-precision multi-view image recognition and detection method based on a CCD camera according to claim 8, characterized in that, The classification module introduces a fuzzy adversarial loss function as a regularization term to simulate the fuzzy scene of low-resolution or noisy images and improve the robustness of the classification module to fuzzy images.

Images