Mobile phone shell defect detection method based on fusion of multi-scale feature pyramid network and ViT

Abstract

The invention provides a mobile phone shell surface defect detection method based on fusion of a multi-scale feature pyramid network and ViT, and belongs to the field of artificial intelligence image processing. The method comprises the steps of analyzing defect characteristics of a mobile phone shell and determining defect types; a mobile phone shell image is collected and preprocessed; pixel-level labeling is carried out on the image, label information is added, and a data set is constructed; performing data enhancement through cutting, overturning, noise adding and the like to generate a training set, a verification set and a test set; constructing a defect detection model fusing the multi-scale feature pyramid network and ViT; inputting the training set, the verification set and the test set into a model for training and evaluation to obtain a trained model; and inputting a to-be-detected image into the model, and outputting a defect type and a result image marked with a defect position. According to the method, multi-scale target detection and semantic segmentation are realized through the feature pyramid network, global information and position relevance are acquired by fusing ViT, and the accuracy and positioning capability of defect detection are improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning industrial product defect detection, and in particular relates to a mobile phone shell defect detection method integrating ViT with a multi-scale feature pyramid network. Background Art

[0002] The popularity and increasing demand for mobile smart devices have made the quality of mobile phone casings an important factor in user selection. However, traditional manual quality inspection methods have problems such as low efficiency and strong subjectivity, and cannot meet the needs of large-scale production. With the development of deep learning and machine vision technology, vision-based defect detection methods have gradually become the mainstream method for mobile phone casing quality inspection. Among them, multi-scale feature pyramid networks and visual ViT are two important technical means that can effectively extract and fuse the defect features of mobile phone casings.

[0003] At present, some studies have proposed multi-scale feature pyramid networks and ViT models as directions for improvement. Multi-scale feature pyramid networks can capture detailed information at different scales and extract global context through feature fusion. The ViT model can learn global feature representation by modeling the relationship between different positions in the image through the self-attention mechanism. However, current methods often apply these two technologies independently to mobile phone shell defect detection, lacking an effective fusion strategy to fully utilize their advantages.

[0004] In the existing technology, there are still some challenges in the fusion of multi-scale feature pyramid network and ViT model. First, how to match the different scale features in the multi-scale feature pyramid network with the input of the ViT model requires the design of a suitable feature alignment mechanism. Secondly, how to effectively fuse these multi-scale features and global features to extract more expressive feature representations while retaining detail information. In addition, how to effectively optimize the parameters of the entire multi-scale feature pyramid network fused with the ViT model during the training process is also a problem worth exploring. Therefore, it is necessary to propose an innovative method for mobile phone shell defect detection that can overcome these technical bottlenecks and improve detection accuracy and robustness. Summary of the invention

[0005] The present invention provides a mobile phone shell defect detection method integrating a multi-scale feature pyramid network and ViT, which overcomes the shortcomings of the above-mentioned multi-scale feature pyramid network and ViT technologies being independently applied to mobile phone shell defect detection. The method can effectively capture various scale and shape features of mobile phone shell defects and improve the detection accuracy, and also helps to capture the global context information of mobile phone shell defects and provide richer semantic expressions, thereby improving the robustness of detection, reducing the false detection rate and missed detection rate, and improving the overall detection performance.

[0006] In order to achieve the above object, the main technical solution adopted by the present invention is realized by the following measures: a mobile phone shell defect detection method integrating multi-scale feature pyramid network with ViT, comprising the following steps:

[0007] Analyze the defect types of mobile phone shells: collect a certain number of mobile phone shell samples, including defective and non-defective samples; observe and record defect characteristics; specify defect classification standards and determine defect types;

[0008] Acquire a mobile phone shell image data set: collect mobile phone shell images; pre-process the collected mobile phone shell image data;

[0009] After preprocessing, the image can be annotated. The preprocessed image can be annotated at the pixel level and the corresponding label information can be added. The image annotation tool can be used to annotate the defect type, location, size and other information. The annotation information can be matched with the image to construct a mobile phone shell image dataset.

[0010] Data enhancement: The constructed mobile phone shell image dataset is enhanced by cropping, flipping, adding noise, etc., and the dataset is divided into a training set, a validation set, and a test set;

[0011] Build a defect detection model: select a multi-scale feature pyramid network and a ViT model;

[0012] Inputting the training set, validation set and test set into the constructed defect detection model for training and evaluation to obtain a defect detection model that has completed training;

[0013] The mobile phone shell image to be inspected is input into the trained defect detection model for defect detection. The model processes and analyzes the image, identifies the defect area in the image and outputs a defect type label indicating the defect type detected in the image.

[0014] Furthermore, the types of defects of the mobile phone shell are: scratches, cracks, deformation, and oil stains.

[0015] Furthermore, the collecting of the mobile phone shell image is performed using an optical projection measurement device, a structured light projector, and specifically includes:

[0016] Make sure the structured light projector is working properly and connected to the camera device. Perform necessary calibration and settings according to the device's instructions;

[0017] Place the mobile phone shell: Place the mobile phone shell to be collected in an appropriate position, making sure its surface faces the structured light projector;

[0018] Projection structured light: Start the structured light projector and select fringe projection. It is usually used to capture the details of the surface shape of an object. Make sure the light source is strong enough to produce clearly visible structured light. Then project the structured light onto the surface of the phone case through the fringe to form light and shadow changes.

[0019] Video or image acquisition: Use a camera device to capture images of the surface of the phone case. Ensure synchronization between the camera and the structured light projector to ensure a clear image under the projected light;

[0020] Furthermore, the collected mobile phone shell image is preprocessed, specifically including:

[0021] Converting the collected mobile phone shell image into a grayscale image;

[0022] Adjust the size and ratio of the mobile phone shell image and perform normalization preprocessing on the image size.

[0023] Furthermore, the preprocessed mobile phone shell image is annotated at the pixel level, and the annotation information is matched with the image to obtain the constructed mobile phone shell image dataset, including:

[0024] The collected mobile phone shell images are imported using the labeling tool Labelme, and the locations of defects are marked on the images. After the defect locations in the images are marked and the labels are selected, the marking results are saved in the format of a JSON file to obtain a mobile phone shell image dataset.

[0025] Furthermore, the specific steps of data enhancement include:

[0026] In order to generate mobile phone case images with different sizes and positions, a part of the original image can be randomly cropped. By randomly selecting the size and position of the cropped area, a variety of mobile phone case images can be obtained;

[0027] To increase the diversity of the dataset, 50% of the mobile phone case images can be flipped horizontally and vertically;

[0028] To simulate the noise in an image, Gaussian noise can be added to the image.

[0029] Furthermore, by processing the data set after the data enhancement, the data set can be divided into a training set, a validation set and a test set in a certain ratio for training the model, verifying the model performance and performing model evaluation.

[0030] Furthermore, the mobile phone casing defect detection model specifically includes:

[0031] The multi-scale feature pyramid segmentation network and the classification network fused with ViT, the fused multi-scale feature map is passed to the ViT model as input for processing. The ViT model uses the self-attention mechanism to pay attention to different positions in the feature map and generate a global feature representation. In this way, the multi-scale feature pyramid network and the ViT model are combined to complete the task of mobile phone shell defect detection, making full use of multi-scale information and the ability to model global relationships.

[0032] The advantages of the present invention are:

[0033] By introducing a multi-scale feature pyramid network, the mobile phone shell can be inspected at different scales, thereby improving the accuracy and robustness of the detection. Features of different scales can capture defects of different sizes, making the detection results more comprehensive. ViT is a Transformer-based visual model with strong feature extraction and representation capabilities. Applying ViT to mobile phone shell defect detection can effectively extract the features of the mobile phone shell and use them in defect detection tasks, thereby improving the accuracy and effect of detection. This method fuses the multi-scale feature pyramid network and ViT to make full use of their respective advantages. By fusing features of different scales and features extracted by ViT, more comprehensive and accurate mobile phone shell defect detection results can be obtained, which enhances the robustness of mobile phone shell defect detection and improves the efficiency of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a method for detecting defects in a mobile phone casing by integrating a multi-scale feature pyramid network with ViT;

[0035] Figure 2 This is the model structure diagram for mobile phone shell defect detection;

[0036] Figure 3 A diagram of the multi-scale feature pyramid network model used in the mobile phone shell defect detection model;

[0037] Figure 4 This is the ViT model diagram in the mobile phone shell defect detection model. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] In the present invention, for the convenience of description, the present invention will be described in more detail below in conjunction with the accompanying drawings and embodiments.

[0040] like Figure 1As shown, the present invention discloses a mobile phone shell defect detection method integrating a multi-scale feature pyramid network with ViT, which specifically includes the following:

[0041] Classify and analyze the defects of mobile phone shells, and determine the types of mobile phone shell defects that need to be detected by observing and analyzing the characteristics of mobile phone shell defects;

[0042] Collecting images of mobile phone shells, preprocessing the image data, and constructing a mobile phone shell defect data set;

[0043] Data augmentation by cropping, flipping, and adding noise;

[0044] Divide the data-augmented dataset into training set, validation set, and test set;

[0045] Build a mobile phone casing defect detection model;

[0046] Sending the training set, validation set and test set to the constructed model for training and evaluation to obtain a trained model;

[0047] The mobile phone shell image to be inspected is input into the trained defect detection model for defect detection. The model processes and analyzes the image, identifies the defect area in the image and outputs a defect type label indicating the defect type detected in the image.

[0048] Example

[0049] like Figure 1 , 2, 3, and 4, a method for detecting defects in a mobile phone casing by integrating a multi-scale feature pyramid network with ViT is provided in this embodiment, and specifically comprises the following steps:

[0050] By observing the shape, size, color, texture and other characteristics of mobile phone shell defects, and combining knowledge and experience in the field of defect detection, we can further analyze the commonalities and differences between different defect types. Different types of defects can be identified and classified to determine the type of mobile phone shell defects that need to be detected.

[0051] Use a camera to capture images of the phone case and ensure that the captured images contain different types of defects, such as scratches, cracks, deformation, and oil stains.

[0052] Capture images of mobile phone shells using the structured light projector, an optical projection measurement device. First, make sure the structured light projector is working properly and connected to the camera device. Perform necessary calibration and settings according to the device's instructions; then place the mobile phone shell to be captured in an appropriate position, making sure its surface faces the structured light projector; then start the structured light projector and select appropriate parameters and settings. Project structured light onto the surface of the mobile phone shell through stripes to create light and shadow changes; finally, use the camera device to capture images of the surface of the mobile phone shell. Ensure that the camera or capture software is synchronized with the structured light projector to ensure a clear image under the projected light.

[0053] The collected mobile phone shell image is preprocessed, and the image size is normalized to enhance the visual effect and details of the image. The grayscale image is used to remove the color information and retain the brightness information to reflect the morphological characteristics of the object in the image. The grayscale formula is as follows:

[0054] F = 0. 3*R + 0. 59*G + 0. 11*B (1)

[0055] Among them, R, G, B represent the red, green, and blue channel values ​​of the pixel respectively, and F represents the grayscale value.

[0056] At the same time, the size and proportion of the mobile phone shell image are adjusted, and the image size is normalized and preprocessed.

[0057] The collected mobile phone shell images are imported using the labeling tool Labelme, and the locations of defects are marked on the images. After the defect locations in the images are marked and the labels are selected, the marking results are saved in the format of a JSON file to obtain a mobile phone shell image dataset.

[0058] Data enhancement is performed on the constructed dataset. Part of the mobile phone case image is randomly cropped to generate mobile phone case images of different sizes and positions. 50% of the mobile phone case images are flipped horizontally and vertically. Gaussian noise is added to the image to obtain more training data sets to prevent overfitting.

[0059] The data set after data enhancement is divided into a training set, a validation set and a test set according to a certain ratio.

[0060] The training set, validation set, and test set are input into the constructed multi-scale feature pyramid network fused with ViT, such as Figure 2 , 3, 4, specifically including:

[0061] Take a part of the input image adjusted to 640×640×1 as an example, input it into ResNet-50 to extract features to obtain 320×320×64 feature map C1, 160×160×256 feature map C2, 80×80×512 feature map C3, 40×40×1024 feature map C4, 20×20×2048 feature map C5; the feature maps {C2, C2, C3, C4, C5} are processed using 256 1×1 convolutions respectively, and the obtained feature map channels are all 256; then each layer is upsampled by 2 times using the nearest neighbor interpolation method, and the semantic information of the feature map is retained to the greatest extent during the upsampling process, and the feature map is added and fused with the corresponding feature map with rich spatial information in the bottom-up process to obtain the fusion. The feature maps after fusion are further extracted through 256 3×3 convolutions to obtain 160×160×256 feature maps P2, 80×80×256 feature maps P3, 40×40×256 feature maps P4, and 20×20×256 feature maps P5; the feature maps {P2, P3, P4, P5} are respectively subjected to the maximum pooling operation to obtain 10×10×256 feature maps {F2, F3, F4, F5}, and the feature maps {F2, F3, F4, F5} are concat-fused to obtain a new 10×10×1024 feature map F6; then the output has two branches, one branch performs a 1×1 convolution on F5 to compress the channel to 1, and outputs a 10×10×1 segmentation feature map F7. After the multi-scale feature pyramid network training is completed, the weights are frozen, and the ViT model is trained again;

[0062] The other branch concatenates the feature map F6 with the segmentation feature map to obtain a 10×10×1025 feature map F8, and then obtains a 10×10×32 feature map F9 through 32 1×1 convolutions as input to the ViT model, as shown in Figure 4 As shown, specifically including:

[0063] Feature block embedding: The 10×10×32 feature map is divided into 100 1×1×32 feature blocks, and linear projection is applied to each feature block to convert it into a 100×32 two-dimensional matrix. In order to pass the sequence format input to the Transformer model, an additional 1×32 token is added to change the dimension of the feature block embedding matrix to 101×32.

[0064] Position encoding: The position encoding is a 101×32 matrix that contains the position information of each position. The position encoding matrix is ​​added to the feature block embedding matrix to fuse the position information with the feature embedding vector.

[0065] Multi-head Attention: Perform layer normalization on the data output by the positional encoding, and input the normalized data into the multi-head attention module in the Transformer encoder. The multi-head attention module uses 8 attention heads to capture important information in the input sequence. In addition, the output of the multi-head attention is residually connected to the input to obtain a 101×32 output. The residual connection formula is:

[0066]

[0067] in, represents output, MSA represents multi-head self-attention mechanism, LN represents layer normalization, Represents the output of the previous layer.

[0068] Input MLP: Perform layer normalization on the output of the multi-head attention to further enhance the stability of the model. The normalized data is input into the MLP module, which consists of two linear transformation layers and a nonlinear activation function. The output of the MLP module is residually connected with the output of the multi-head attention to obtain a 101×32 output.

[0069] The multi-head attention and input MLP operations are repeated for a total of 8 iterations of the Transformer encoder layer to enhance the feature extraction and modeling capabilities of the model.

[0070] Output:

[0071] The content of the token is extracted from the 101×32 output to obtain a 1×32 vector as the final output result. On the other hand, the segmented image is converted into 1 value through global maximum pooling and global average pooling, and the scalar value obtained by global maximum pooling and the scalar value obtained by global average pooling are connected to obtain a 2-dimensional vector. The 1×32 vector output by the ViT model is concatenated with the obtained 2-dimensional vector to obtain a 34-dimensional vector, which is input into a fully connected layer for processing. Finally, the output of the fully connected layer passes through a softmax function to convert the output into a probability distribution, and then the cross entropy loss function is used to calculate the difference between the model output and the true label to obtain the classification result of the mobile phone shell defect detection.

[0072] Among them, the expression of the softmax function is:

[0073]

[0074] Where S i Indicates the probability value of the category corresponding to the i-th element in the output result of the Softmax function; zi Represents the i-th element in the input value vector; n represents the length of the input value vector, that is, the number of categories.

[0075] For multi-classification problems, the expression of the cross entropy loss function is:

[0076]

[0077] Where x represents the output vector of the model, x[class] is the model's predicted score for the true category, and x[j] represents the score of the jth category.

[0078] Finally, the image of the mobile phone case to be tested is input into the trained mobile phone case defect detection model to obtain the classification result.

[0079] The above is only a preferred embodiment of the present invention, which has a good implementation effect, but is not limited to the present invention. Ordinary technicians in this field can make changes and substitutions to the above embodiments within the scope of the present invention without departing from the principle of the present invention, which should be regarded as the protection scope of the present invention.

Claims

1. A mobile phone shell surface defect detection method based on a multi-scale feature pyramid network fused with ViT, characterized in that include: By observing and analyzing various problems and undesirable features on the surface of mobile phone shells, we can determine the types of mobile phone shell defects that can be clearly identified and classified; Collect images of mobile phone casing; Preprocessing the collected mobile phone shell image data to improve image quality and consistency; Annotate the preprocessed image at the pixel level and add corresponding label information, use a dedicated image annotation tool to annotate the defect type, location, and size information, correspond the annotation information to the image, and construct a mobile phone shell image dataset; The data set is enhanced by cropping, flipping, adding noise, etc., and then divided into a training set, a validation set, and a test set; Build defect detection models; Inputting the training set, validation set and test set into the constructed defect detection model for training and evaluation to obtain a trained efficient defect detection model; Inputting the mobile phone shell image to be inspected into the trained defect detection model for defect detection, and obtaining a defect type label and a result image corresponding to the mobile phone shell image; The result image is an image in which the defect position is marked in the image of the mobile phone shell to be inspected; Among them, the constructed mobile phone shell defect detection model integrates the multi-scale feature pyramid segmentation network and the ViT model to achieve more accurate detection.

2. The method according to claim 1, characterized in that: The types of defects of the mobile phone shell are: scratches, cracks, deformation, and oil stains.

3. The method according to claim 1, characterized in that: The method of collecting the image of the mobile phone shell is performed by using an optical projection measurement device, a structured light projector, and specifically includes: Make sure the structured light projector is working properly and connected to the camera device. Perform necessary calibration and settings according to the device's instructions; Place the mobile phone shell: Place the mobile phone shell to be collected in an appropriate position, making sure its surface faces the structured light projector; Projection structured light: Start the structured light projector and select fringe projection. It is usually used to capture the details of the surface shape of an object. Make sure the light source is strong enough to produce clearly visible structured light. Then project the structured light onto the surface of the phone case through the fringe to form light and shadow changes. Video or image acquisition: Use a camera device to capture images of the surface of the phone housing. Ensure synchronization between the camera and the structured light projector to ensure clear images under the projected light.

4. The method according to claim 1, characterized in that: The collected mobile phone shell image is subjected to data preprocessing by graying the image and adjusting the image size.

5. The method according to claim 1, characterized in that The collected mobile phone shell images are imported using the labeling tool Labelme, and the locations of defects are marked on the images. After the defect locations in the images are marked and the labels are selected, the marking results are saved in the format of a JSON file to obtain a mobile phone shell image dataset.

6. The method according to claim 1, characterized in that: The specific steps of data enhancement include: Randomly crop a part of the mobile phone case image to generate mobile phone case images of different sizes and positions; Flip 50% of the phone case images horizontally or vertically; Add Gaussian noise to the image.

7. The method according to claim 1, characterized in that: The data set after data enhancement is divided into a training set, a validation set and a test set according to a certain ratio.

8. The method according to claim 1, characterized in that: The mobile phone casing defect detection model specifically includes: The multi-scale feature pyramid segmentation network and the classification network fused with ViT, the feature map of each layer obtained by the feature pyramid network is pooled to obtain a feature map of the same dimension, the new feature map obtained by concat splicing is concat spliced, and then a segmentation map is obtained by a 1×1 convolution. The new feature map obtained by concat splicing and the segmentation map are concat spliced ​​again to obtain a comprehensive multi-scale feature map as input and passed to the ViT model for processing. The ViT model uses the self-attention mechanism to pay attention to different positions in the feature map and generate a global feature representation. In this way, the multi-scale feature pyramid network and the ViT model are combined to jointly complete the task of mobile phone shell defect detection, making full use of multi-scale information and the ability to model global relationships.

9. The method according to claim 8, characterized in that: The ViT model is specifically: The feature map F8 is cut into a series of fixed-size image blocks, which are linearly projected to convert them into lower-dimensional vector representations for passing to the ViT module. In the vector representation, the positional encoding embeds the positional information of the image blocks to preserve the spatial relationship in the image. These vector sequences are then passed to ViT's multi-layer Transformer encoder. Each Transformer encoder uses a self-attention mechanism to learn the dependencies between image blocks and performs feature representation through a feedforward neural network. Finally, after passing through multiple Transformer encoders, the resulting feature sequence is fed into a global pooling layer to average pool the features at all positions to obtain a global feature representation of the entire mobile phone shell. This global feature representation will be used for subsequent classification tasks by connecting a fully connected layer to predict the category of the mobile phone shell defect.

Images