Method and system for detecting multi-class faults of sun-rain umbrella based on machine vision

Abstract

The invention discloses an umbrella multi-class fault detection method and system based on machine vision. The method comprises the following steps: acquiring an image of a sun-rain umbrella in a completely unfolded state by using a camera; a composite image processing method is adopted to process the collected images, and a sun-rain umbrella multi-class fault classification data set is constructed; the method comprises the following steps: constructing a multi-class fault classification model based on improved EfficientNetV2S, inputting a sun-rain umbrella multi-class fault data set into the multi-class fault classification model for training to obtain a trained model, and finally inputting a sun-rain umbrella completely-unfolded image collected in real time in a loop test into the trained model for fault classification. And finally judging whether the product meets the quality requirement according to the classification result and the cycle test frequency of the to-be-tested umbrella. According to the multi-class fault classification model and the composite image processing method for constructing the data set, high-precision classification is realized in real-time fault detection of the sun-rain umbrella, and the detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue testing of sunshades, and in particular to a sunshade multi-category fault detection method and system based on machine vision. Background Art

[0002] During manufacturing and use, umbrellas are subject to repeated opening and closing, as well as exposure to diverse environmental conditions. Umbrellas manufactured in accordance with the national standard "GB / T 23147-2018" and user requirements undergo fatigue testing. Fatigue testing for umbrellas requires high standards and high efficiency, and can simulate fatigue testing under a variety of operating conditions. Currently, failures during fatigue testing of umbrellas rely primarily on manual detection and entry of fault types. Leveraging machine vision to more accurately and efficiently detect failures would improve fatigue testing efficiency.

[0003] Furthermore, ordinary umbrellas are opened and closed 400-600 times, which is very demanding. During fatigue testing, umbrellas may experience problems such as misaligned sashes, broken thread, loose seams, detached seams, and broken ribs. Failure to detect these problems and stop testing in a timely manner will reduce the efficiency of fatigue testing. These types of failures are prone to misjudgment and missed detection due to subjective human factors. Therefore, research on machine vision-based umbrella fault classification methods can help fill the gaps between the technical testing requirements of national standards and existing testing equipment. Summary of the Invention

[0004] In order to solve the problems and defects in the background technology, the present invention provides a multi-category fault detection method and system for sun umbrellas based on machine vision, which uses a machine learning algorithm to perform real-time multi-category fault detection on sun umbrellas during fatigue testing, to make up for the lack of intelligent fault classification algorithm in the current field of sun umbrella fatigue testing.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] 1. A multi-category fault detection method for umbrellas based on machine vision, comprising the following steps:

[0007] S1. Use a camera to capture images of fully unfolded umbrellas, perform label processing on the captured images, and generate composite images to obtain a multi-category fault classification dataset for umbrellas.

[0008] S2. Build a multi-category fault classification model based on the improved EfficientNetV2_S model. Input the obtained umbrella multi-category fault classification dataset into the multi-category fault classification neural network model for training to obtain a trained multi-category fault classification model.

[0009] S3. Generate a composite image of the fully expanded umbrella to be tested, and input it into the trained multi-category fault classification model to determine whether the umbrella has a fault and the type of fault. Based on the classification results of the model and the number of times the umbrella is opened and closed, determine whether the umbrella is of qualified quality.

[0010] S11. Use a camera to capture images of a fully expanded umbrella in a normal state and in multiple fault states as basic sample images of the umbrella. Label the captured images and classify them into the following categories: good image, image of a clamping and shifting fault, image of a broken rib fault, image of a broken thread fault, image of a seam eyelet disconnection fault, and image of a seam separation fault.

[0011] S12, pairing each good grayscale image of the umbrella with each grayscale image of a different fault category in sequence, and simultaneously pairing each good grayscale image with other good grayscale images to obtain a corresponding grayscale image pair set;

[0012] S13. Perform image segmentation processing on all collected images of good umbrellas and various types of faulty umbrellas to separate the umbrellas from the background; perform image preprocessing, threshold segmentation, image summing, histogram enhancement, and gradient calculation on the segmented umbrella images in sequence to obtain a gradient image of the umbrella structure with significant edge features;

[0013] S14, performing composite image generation processing on the grayscale image of a good umbrella, the grayscale image of a faulty umbrella, and the corresponding umbrella structure gradient image obtained in S13, to obtain a plurality of composite images of the umbrella;

[0014] The labels after the composite image generation and processing are determined according to the categories of good images and faulty images, specifically: the composite umbrella image generated by composite image generation of a good image and a good image is marked as a normal umbrella sample; the composite umbrella image generated by composite image generation of a good image and a clamp shift fault is marked as a clamp shift fault type composite image; the composite umbrella image generated by composite image generation of a good image and a string wire breakage fault image is marked as a string wire breakage fault type composite image; the composite umbrella image generated by composite image generation of a good image and a rib breakage fault image is marked as a rib breakage fault type composite image; the composite umbrella image generated by composite image generation of a good image and a seam eye derailment fault image is marked as a seam eye derailment fault type composite image; the composite umbrella image generated by composite image generation of a good image and a seam detachment fault image is marked as a seam detachment fault type composite image.

[0015] S15, combining all the composite images corresponding to each good image of the same umbrella in step S14 to form a multi-category fault classification dataset of the umbrella;

[0016] S16, using the same method as steps S11 to S15 to construct a multi-category fault classification dataset for multiple umbrellas;

[0017] S17: Combine the multi-category fault classification datasets of all the umbrellas obtained in step S16 to finally form a multi-category fault classification dataset of the umbrellas.

[0018] The multi-category fault classification model is constructed based on the improved EfficientNetV2_S model, and is connected in sequence to a multi-scale convolutional feature extraction module, a feature extraction backbone network, and a classification layer; wherein the multi-scale convolutional feature extraction module is used to replace the first convolution layer in the EfficientNetV2_S network model; the feature extraction backbone network is composed of improved modules, specifically: replacing the Fused-MBConv1 module with the AsymFused-MBConv1 module; replacing the first Fused-MBConv4 module with the first AsymFused-MBConv4 module; replacing the second Fused-MBConv4 module with the second AsymFused-MBConv4 module; replacing the MBConv4 module with The first SSESA-MBConv4 module; the second MBConv6 module is replaced by the second SSESA-MBConv4 module; the third MBConv6 module is replaced by the third SSESA-MBConv4 module; and the first AsymFused-MBConv4 module, the second AsymFused-MBConv4 module, the first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module in the feature extraction backbone network are streamlined to reduce the number of model parameters, and a CAA attention mechanism is inserted between the second AsymFused-MBConv4 module and the first SSESA-MBConv4 module. The classification layer mainly consists of a fully connected layer.

[0019] The multi-category fault dataset of the sunny umbrella obtained in step S1 is input to the input end of the multi-scale convolution feature extraction module, the output end of the multi-scale convolution feature extraction module is connected to the input end of the asymmetric convolution in the AsymFused-MBConv1 module, the output end of the AsymFused-MBConv1 module is connected to the input end of the asymmetric convolution in the first AsymFused-MBConv4 module, the output end of the first AsymFused-MBConv4 module is connected to the input end of the asymmetric convolution layer of the second AsymFused-MBConv4 module, and the output end of the second AsymFused-MBConv4 module is connected to the input end of the asymmetric convolution layer. The first end is connected to the input end of the CAA attention mechanism layer, the output end of the CAA attention mechanism layer is connected to the first 1×1 convolutional layer in the first SSESA-MBConv4 module, the output end of the first SSESA-MBConv4 module is connected to the first 1×1 convolutional layer of the second SSESA-MBConv4 module, the output end of the second SSESA-MBConv4 module is connected to the first 1×1 convolutional layer of the third SSESA-MBConv4 module, the output end of the third SSESA-MBConv4 module is connected to the input end of the last convolutional layer, the output end of the last convolutional layer is connected to the input end of the classification layer, and the classification layer outputs the classification result of the fault type.

[0020] The AsymFused-MBConv1 module connected to the output end of the multi-scale convolutional feature extraction module has 1 layer and a step size of 1; the first AsymFused-MBConv4 module and the second AsymFused-MBConv4 module connected to the output end of the AsymFused-MBConv1 module have 4 layers and a step size of 2; the first SSESA-MBConv4 module connected to the output end of the second AsymFused-MBConv4 module has 3 layers and a step size of 2; the second SSESA-MBConv4 module connected to the output end of the first SSESA-MBConv4 module has 5 layers and a step size of 2; the third SSESA-MBConv4 module connected to the output end of the second SSESA-MBConv4 module has 8 layers and a step size of 2; and Conv1×1&Pooling&FC is connected to the output end of the third SSESA-MBConv4 module, and its layer number is 1.

[0021] The multi-scale convolution feature extraction module includes four parallel splicing branches, the first parallel splicing branch includes a first 1×1 convolution layer, the second parallel splicing branch includes a second 1×1 convolution layer, a second 1×3 convolution layer, and a second 3×1 convolution layer connected in series, the third parallel splicing branch includes a third 1×1 convolution layer, a third 1×5 convolution layer, and a third 5×1 convolution layer connected in series, and the fourth parallel splicing branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolution layer; the input end of the first 1×1 convolution layer, the second 1×3 convolution layer, and the second 3×1 convolution layer are connected in series. The input end of the first × 1 convolutional layer, the input end of the third 1 × 1 convolutional layer, and the input end of the fourth 3 × 3 pooling layer are connected in parallel as the input end of the multi-scale convolutional feature extraction module; the output end of the first 1 × 1 convolutional layer, the output end of the second 3 × 1 convolutional layer, the output end of the third 5 × 1 convolutional layer, and the output end of the fourth 1 × 1 convolutional layer are connected in parallel as the output end of the multi-scale convolutional feature extraction module. The output end of the multi-scale convolutional feature extraction module is connected to the 2 × 2 maximum pooling layer, and its output is the final output of the multi-scale convolutional feature extraction module.

[0022] Each convolution layer in the multi-scale convolution feature extraction module is connected to a LeakyReLU activation function.

[0023] The AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module are improved based on the Fused-MBConv1 module, the first Fused-MBConv4 module, and the second Fused-MBConv4 module. Specifically, the 3×3 convolutional layers in the Fused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module are replaced with asymmetric convolutional layers; the asymmetric convolutional layers are connected in parallel. The network consists of a 1×3 convolutional layer, a 3×1 convolutional layer and a 3×3 convolutional layer; the input of the asymmetric convolutional layer is connected to the output of the previous layer; the output of the asymmetric convolutional layer is the result of adding the outputs of the 1×3 convolutional layer, the 3×1 convolutional layer and the 3×3 convolutional layer; the output of the asymmetric convolution is connected in series with the Swish activation function and the input of the 1×1 convolutional layer in sequence; the output of the 1×1 convolutional layer is residually connected to the input of the asymmetric convolution, and finally serves as the output of the AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module.

[0024] The first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module are improved based on the MBConv4, the first MBConv6 module, and the second MBConv6 module. Specifically, the SE channel attention mechanism layer in the MBConv4, the first MBConv6 module, and the second MBConv6 module is replaced by a channel grouping-dual attention-shuffle fusion SSESA attention mechanism layer.

[0025] The first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module include a first convolutional layer, a depth convolutional layer, an SSESA attention mechanism layer, and a second convolutional layer connected in sequence; the first convolutional layer and the second convolutional layer are both 1×1 convolutional layers, the depth convolutional layer is a depth 3×3 expanded convolutional layer, and the output end of the second convolutional layer is residually connected to the input end of the first convolutional layer, and its output serves as the output end of the first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module.

[0026] The SSESA attention mechanism is a neural network model structure that combines channel grouping, dual attention, and shuffling. The SSESA attention mechanism layer is set according to the following formula:

[0027] X=X1+X2

[0028] M C1 =σ{W[GAP(X1)]}

[0029] M S1 =σ{f 7×7 [H(GAP(X1),GMP(X1))]}

[0030] M C2 =σ{W[GAP(X2)]}

[0031] M S2 =σ{f 7×7 [H(GAP(X2),GMP(X2))]}

[0032] X out1 =X1·M C1 +X1·M S1

[0033] X out2 =X2·M C2 +X2·M S2

[0034] Y=S[H(X out1 ,Xout2 )]

[0035] Where X1 and X2 are the channel splits of input X, W is the fully connected operation in the channel attention mechanism, and f 7×7 is the 7×7 convolution layer operation in the spatial attention mechanism, M C1 is the channel attention mechanism of X1, M S1 is the X1 spatial attention mechanism, M C2 is the channel attention mechanism of X2, M S2 is the X2 spatial attention mechanism, · is the channel multiplication, σ is the Sigmoid activation function, GAP is the global average pooling, GMP is the global maximum pooling, H is the channel fusion operation, S is the channel mixing operation, X out1 is the output of the X1 attention mechanism, X out2 is the attention mechanism output of X2, and Y is the final output of the SSESA attention mechanism module.

[0036] The CAA attention mechanism layer is a neural network model structure that combines multi-scale feature extraction and local feature enhancement. The CAA attention mechanism layer is set according to the following formula:

[0037] F pool =Conv 1×1 (P avgpool (F))

[0038] F w =DWConv 1×11 (F pool )

[0039] F H =DWConv 11×1 (F w )

[0040] M CAA =Sigmoid(Conv 1×1 (F H ))

[0041] Where F is the input feature map, Conv 1×1 is a 1×1 convolutional layer, P avgpool is global average pooling, DWConv 1×11 A 1×11 depthwise separable convolutional layer, DWConv 11×1 is a 11×1 depthwise separable convolutional layer, M CAA is the output of the CAA attention mechanism layer.

[0042] S31, collecting an image of the umbrella to be tested in the first fully expanded state as a basic image;

[0043] S32, collecting the next fully unfolded image of the umbrella to be tested as a matching image;

[0044] S33, performing image segmentation, image preprocessing, and image AND operations on the matching image to obtain an umbrella structure gradient image with significant edge features;

[0045] S34, combining the base image, the matching image, and the umbrella structure gradient image into a three-channel image;

[0046] S35, inputting the obtained composite image into the multi-category fault classification model obtained in step S2 to obtain the classification result of the model;

[0047] S36. Determine whether the umbrella is qualified based on the classification result of the model and the number of cycle tests of the umbrella to be tested;

[0048] If the fault classification result is normal, the number of cycle tests of the umbrella to be tested is increased by one, and the obtained number of cycle tests of the umbrella to be tested is compared with the standard number of cycle tests: if the total number of cycle tests of the umbrella to be tested is equal to the standard number of cycle tests, the test is stopped, indicating that the umbrella is of qualified quality; if the number of cycle tests of the umbrella to be tested is less than the standard number of cycle tests, the above steps S32-S35 are repeated;

[0049] If the fault diagnosis result is a certain fault type, the detection is stopped and the fault type of the umbrella is given, indicating that the umbrella has not passed the fatigue test and the quality of the umbrella is unqualified.

[0050] The composite image generation method of S14 and S34 is specifically as follows: grayscale processing is performed on the two images in the image pair respectively to obtain two corresponding grayscale images, then the second original image in the good image pair and the faulty original image in the different faulty image pairs are selected, and image segmentation processing is performed on them respectively to separate the umbrella area, and image preprocessing, threshold segmentation, image and processing, histogram enhancement, and gradient calculation are performed on the original image and the corresponding umbrella area image in sequence to obtain an umbrella structure gradient image with significant edge features; the original grayscale image in the image pair and the corresponding umbrella gradient image are fused in the channel dimension to obtain a composite umbrella image, and the composite umbrella image obtained after the fusion processing is a three-channel image.

[0051] The image preprocessing performed on the segmented umbrella image in sequence includes grayscale conversion, Gaussian denoising, and image scaling.

[0052] 2. A Sunny Umbrella Fault Detection System

[0053] It includes an image acquisition camera, an industrial control computer, a programmable logic controller and a fatigue test device; the image acquisition camera is arranged below the umbrella for detecting sunny or rainy weather; the industrial control computer includes a memory, a central processing unit and an input / output interface; the programmable logic controller controls the fatigue test device to execute the umbrella's cyclic opening and closing; the programmable logic controller is electrically connected to the fatigue test device, and the industrial control computer is electrically connected to the image acquisition camera and the programmable logic controller.

[0054] The image acquisition cameras are arranged on the left and right sides below the umbrella for detection; the memory is used to store computer programs and operating systems; the central processing unit is used to execute the computer programs to implement the aforementioned umbrella multi-category fault detection method; the input / output interface is an Ethernet interface for connecting the industrial control computer with the image acquisition camera and the programmable logic controller.

[0055] The innovation of the present invention lies in the improvement of the EfficientNetV2_S network, the design and introduction of a multi-scale convolution feature extraction module based on asymmetric convolution and LeakyReLU activation function, the use of asymmetric convolution technology to improve the AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module, the design of the channel grouping-dual attention-shuffle fusion SSESA attention mechanism, the introduction of the CAA attention mechanism combining multi-scale feature extraction and local feature enhancement, and the use of the improved multi-category fault classification model to achieve higher accuracy than the original model and other common network models, and bring the advantage of meeting real-time requirements.

[0056] The innovation of the present invention lies in constructing a composite umbrella image dataset that fuses original grayscale information and umbrella structure gradient images. The composite image generation method is also used to process images during real-time fault detection of clear umbrellas, so that the image data input into the network model can provide more image difference information and umbrella structure feature information.

[0057] Beneficial effects of the present invention:

[0058] 1. The improved multi-category fault classification network and composite image generation method in the present invention realize multi-category fault classification of umbrellas, with an accuracy rate of 97.25%, which is 5% higher than the original EfficientNetV2_S network model.

[0059] 2. This method constructs a composite image by combining grayscale images of a healthy umbrella, grayscale images of a faulty umbrella, and a gradient image of the umbrella structure. This composite image preserves the texture details and edge structure information of the umbrella, solving the problem of insufficient feature extraction caused by background interference and unclear umbrella shape changes in traditional methods, and significantly improving the ability to identify different types of faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Flowchart for the construction of the umbrella dataset, network construction, and real-time detection;

[0061] Figure 2 This is a flow chart of the multi-category fault classification method for umbrellas;

[0062] Figure 3 The flowchart of the gradient image processing of the umbrella structure with significant edges;

[0063] Figure 4 This is the network structure diagram for multi-category fault classification of umbrellas;

[0064] Figure 5 This is the structural diagram of the CAA attention mechanism layer;

[0065] Figure 6 This is the structural diagram of the multi-scale convolutional feature extraction module;

[0066] Figure 7 This is the general structure diagram of the AsymFused-MBConv module;

[0067] Figure 8 This is the general structure diagram of the SSESA-MBConv module;

[0068] Figure 9 This is the structural diagram of the SSESA attention mechanism layer;

[0069] Figure 10 This is the neural network structure diagram of the dual attention mechanism;

[0070] Figure 11 This is the overall module diagram of the umbrella fault detection system; DETAILED DESCRIPTION

[0071] The present invention will be further described below with reference to the accompanying drawings and examples.

[0072] like Figure 1 and Figure 2 As shown, the specific embodiments of the present invention are as follows:

[0073] S1. Use a camera to capture images of fully unfolded umbrellas, perform label processing on the captured images, and generate composite images to obtain a multi-category fault classification dataset for umbrellas.

[0074] S11. Use a camera to capture images of a fully expanded umbrella in a normal state and in multiple fault states as basic sample images of the umbrella. Label the captured images and classify them into the following categories: good image, image of a clamping and shifting fault, image of a broken rib fault, image of a broken thread fault, image of a seam eyelet disconnection fault, and image of a seam separation fault.

[0075] The fault-free state refers to a fault-free image when the umbrella is fully opened with random rotation angle and random brightness adjustment.

[0076] Different fault states refer to random rotation angles and random brightness adjustments. The umbrella is subjected to a cyclic opening and closing fatigue test until the umbrella exhibits one of the following faults: displacement of the ribs, breakage of the ribs, breakage of the thread strings, derailment of the seams, or detachment of the seams. Single fault morphological images of the umbrella are collected when it is fully unfolded at random angles and brightness.

[0077] S12, pairing each good grayscale image of the umbrella with each grayscale image of a different fault category in sequence, and simultaneously pairing each good grayscale image with other good grayscale images to obtain a corresponding grayscale image pair set;

[0078] S13. Perform image segmentation processing on all collected images of good umbrellas and various types of faulty umbrellas to separate the umbrellas from the background; perform image preprocessing, threshold segmentation, image summing, histogram enhancement, and gradient calculation on the segmented umbrella images in sequence to obtain a gradient image of the umbrella structure with significant edge features;

[0079] S14, performing composite image generation processing on the grayscale image of a good umbrella, the grayscale image of a faulty umbrella, and the corresponding umbrella structure gradient image obtained in S13, to obtain a plurality of composite images of the umbrella;

[0080] The specific method for generating a composite image is as follows: grayscale processing is performed on the two images in the image pair respectively to obtain two corresponding grayscale images, and then the second original image in the good image pair and the faulty original image in the different faulty image pairs are selected, and image segmentation processing is performed on them respectively to separate the umbrella area, and image preprocessing, threshold segmentation, image and processing, histogram enhancement, and gradient calculation are performed on the original image and the corresponding umbrella area image in sequence to obtain an umbrella structure gradient image with significant edge features, and the original grayscale image in the image pair and the corresponding umbrella gradient image are fused in the channel dimension to obtain a composite umbrella image. The composite umbrella image obtained after fusion processing is a three-channel image.

[0081] The image preprocessing performed on the segmented umbrella image in sequence includes grayscale conversion, Gaussian denoising, and image scaling.

[0082] like Figure 3 Figure 4 shows the process of processing an umbrella structure gradient image with significant edge features. Specifically, the following steps are performed: inputting the reduced umbrella image into the image segmentation model, outputting a binary classification image of the umbrella and background, performing image preprocessing on the reduced original umbrella image and the binary classification image in sequence, then performing threshold segmentation on the binary classification image, performing an image AND operation on the resulting threshold image and the preprocessed original umbrella image to obtain an umbrella image with the background removed, then performing histogram enhancement and gradient calculation in sequence, and outputting a gradient image of the umbrella structure with significant edge features.

[0083] The labels after composite image generation and processing are determined according to the categories of good images and faulty images, specifically: the composite umbrella image generated by composite image generation of good images and good images is marked as a normal umbrella sample; the composite umbrella image generated by composite image generation of good images and clamp shift fault is marked as a clamp shift fault type composite image; the composite umbrella image generated by composite image generation of good images and string wire breakage fault image is marked as a string wire breakage fault type composite image; the composite umbrella image generated by composite image generation of good images and rib breakage fault image is marked as a rib breakage fault type composite image; the composite umbrella image generated by composite image generation of good images and seam eye derailment fault image is marked as a seam eye derailment fault type composite image; the composite umbrella image generated by composite image generation of good images and seam detachment fault image is marked as a seam detachment fault type composite image.

[0084] S15, combining all the composite images corresponding to each good image of the same umbrella in step S14 to form a multi-category fault classification dataset of the umbrella;

[0085] S16, using the same method as steps S11 to S15 to construct a multi-category fault classification dataset for multiple umbrellas;

[0086] S17: Combine the multi-category fault classification datasets of all the umbrellas obtained in step S16 to finally form a multi-category fault classification dataset of the umbrellas.

[0087] This example randomly selected 80 umbrellas of different shapes from a sample of 150 umbrellas to be tested and subjected them to a cyclic opening and closing test. A binocular industrial camera was used as the image acquisition device to capture images of the inner surfaces of fully opened umbrellas. These images were obtained, along with images of umbrellas with different fault types, to construct a basic dataset. The original images were preprocessed. Because the industrial camera's native resolution was 3072 × 2048 pixels, the original image size was too large, making direct input into the model for training and inference very time-consuming. Therefore, the original images were first downscaled to 680 × 480 pixels using bilinear interpolation. Image segmentation and image AND operations were performed on the 680 × 480 pixel images to obtain a gradient map of the umbrellas with significant edges. Bilinear interpolation was then used to uniformly downscale the three images to be fused to 224 × 224 pixels. Using this downscaled image not only improves the training and inference speed of image segmentation and fault classification, but also ensures image segmentation accuracy and fault classification accuracy, reducing the time cost of dataset training and real-time inference.

[0088] Since the patterns and colors of umbrellas vary from one umbrella to another, the reduced image data needs to be converted into a grayscale image and Gaussian denoising performed to remove the interference of color and pattern. This can enable the neural network to pay more attention to the texture details and edge structure information of the faulty parts of the composite umbrella image, thereby improving the accuracy of the model.

[0089] After careful screening and composite image generation, the resulting dataset of umbrella images totaled 12,680 pairs. These included 1,740 pairs with broken ribs, 2,247 pairs with misaligned ribs, 2,101 pairs with good condition, 2,245 pairs with detached seams, 2,211 pairs with unthreaded eyelets, and 2,136 pairs with broken string threads. The training set to validation set image ratio was 8:2.

[0090] S2. Build a multi-category fault classification model based on the improved EfficientNetV2_S model. Input the obtained umbrella multi-category fault classification dataset into the multi-category fault classification neural network model for training to obtain a trained multi-category fault classification model.

[0091] like Figure 4As shown, the multi-category fault classification model is constructed based on the improved EfficientNetV2_S model, which is connected in sequence with a multi-scale convolutional feature extraction module, a feature extraction backbone network, and a classification layer; wherein the multi-scale convolutional feature extraction module is used to replace the first convolution layer in the EfficientNetV2_S network model; the feature extraction backbone network is composed of improved modules, specifically: replacing the Fused-MBConv1 module with the AsymFused-MBConv1 module; replacing the first Fused-MBConv4 module with the first AsymFused-MBConv4 module; replacing the second Fused-MBConv4 module with the second AsymFused-MBConv4 module; replacing the MBConv4 module with the first SSESA-MBConv4 module; and replacing the second MBConv6 The module is replaced by the second SSESA-MBConv4 module; the third MBConv6 module is replaced by the third SSESA-MBConv4 module; and the first AsymFused-MBConv4 module, the second AsymFused-MBConv4 module, the first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module in the feature extraction backbone network are streamlined to reduce the number of model parameters, and a layer of CAA attention mechanism is inserted between the second AsymFused-MBConv4 module and the first SSESA-MBConv4 module. The classification layer is mainly composed of a fully connected layer; the numbers after each module represent the convolution expansion ratio, Layer represents the number of layers at each level, and 0.25 represents the weight of the SSESA channel attention mechanism.

[0092] The multi-category fault dataset of the sunny umbrella obtained in step S1 is input to the input end of the multi-scale convolution feature extraction module, the output end of the multi-scale convolution feature extraction module is connected to the input end of the asymmetric convolution in the AsymFused-MBConv1 module, the output end of the AsymFused-MBConv1 module is connected to the input end of the asymmetric convolution in the first AsymFused-MBConv4 module, the output end of the first AsymFused-MBConv4 module is connected to the input end of the asymmetric convolution layer of the second AsymFused-MBConv4 module, and the output end of the second AsymFused-MBConv4 module is connected to the input end of the asymmetric convolution layer. The first end is connected to the input end of the CAA attention mechanism layer, the output end of the CAA attention mechanism layer is connected to the first 1×1 convolutional layer in the first SSESA-MBConv4 module, the output end of the first SSESA-MBConv4 module is connected to the first 1×1 convolutional layer of the second SSESA-MBConv4 module, the output end of the second SSESA-MBConv4 module is connected to the first 1×1 convolutional layer of the third SSESA-MBConv4 module, the output end of the third SSESA-MBConv4 module is connected to the input end of the last convolutional layer, the output end of the last convolutional layer is connected to the input end of the classification layer, and the classification layer outputs the classification result of the fault type.

[0093] like Figure 5 The figure shows the module structure of the CAA attention mechanism. The CAA attention mechanism layer is a neural network model structure that combines multi-scale feature extraction and local feature enhancement. The CAA attention mechanism layer is set according to the following formula:

[0094] F pool =Conv 1×1 (P avgpool (F))

[0095] F w =DWConv 1×11 (F pool )

[0096] F H =DWConv 11×1 (F w )

[0097] M CAA =Sigmoid(Conv 1×1 (F H ))

[0098] Where F is the input feature map, Conv 1×1 is a 1×1 convolutional layer, P avgpool is global average pooling, DWConv 1×11A 1×11 depthwise separable convolutional layer, DWConv 11×1 is a 11×1 depthwise separable convolutional layer, M CAA is the output of the CAA attention mechanism layer.

[0099] The CAA attention mechanism is used to perform global average pooling on the input feature map to extract the global feature information of each channel and capture the overall semantics of the image. Then, a 1×1 convolution layer is used to fuse the information between channels and generate preliminary feature anchors. Then, 1×11 and 11×1 depthwise separable convolution layers are used to extract spatial image features in the horizontal and vertical directions respectively, enhancing the model's perception of horizontal and vertical structures. Then, a 1×1 convolution layer is used to integrate features and extract key feature information. Finally, a layer of Sigmoid activation function is used to generate the attention feature map M. CAA .

[0100] like Figure 6 The figure shows a multi-scale convolution feature extraction module, which uses the receptive field of asymmetric convolution and multiple parallel splicing branches to achieve the final multi-scale convolution feature fusion.

[0101] The multi-scale convolution feature extraction module includes four parallel splicing branches, the first parallel splicing branch includes a first 1×1 convolution layer, the second parallel splicing branch includes a second 1×1 convolution layer, a second 1×3 convolution layer, and a second 3×1 convolution layer connected in series, the third parallel splicing branch includes a third 1×1 convolution layer, a third 1×5 convolution layer, and a third 5×1 convolution layer connected in series, and the fourth parallel splicing branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolution layer; the input end of the first 1×1 convolution layer, the second 1×1 The input end of the convolution layer, the input end of the third 1×1 convolution layer, and the input end of the fourth 3×3 pooling layer are connected in parallel as the input end of the multi-scale convolution feature extraction module; the output end of the first 1×1 convolution layer, the output end of the second 3×1 convolution layer, the output end of the third 5×1 convolution layer, and the output end of the fourth 1×1 convolution layer are connected in parallel as the output end of the multi-scale convolution feature extraction module. The output end of the multi-scale convolution feature extraction module is connected to the input end of the 2×2 maximum pooling layer, and its output is the final output of the multi-scale convolution feature extraction module.

[0102] The multi-scale convolutional feature extraction module replaces the original first convolutional layer in the EfficientNetV2_S network model. It uses asymmetric convolution kernels of different sizes to capture local details and global image differences in parallel, then performs parallel fusion processing to extract more detailed feature information. The maximum pooling layer connected to this module is designed to reduce the resolution of the image input feature extraction module, thereby improving the model's computational speed and accuracy.

[0103] like Figure 7The structure of the AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module is shown. The AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module are improved based on the Fused-MBConv1 module, the first Fused-MBConv4 module, and the second Fused-MBConv4 module. Specifically, the 3×3 convolutional layers in the Fused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module are replaced with asymmetric convolutional layers; the asymmetric convolutional layers are arranged in parallel. The network consists of a 1×3 convolutional layer, a 3×1 convolutional layer and a 3×3 convolutional layer; the input of the asymmetric convolutional layer is connected to the output of the previous layer; the output of the asymmetric convolutional layer is the result of adding the outputs of the 1×3 convolutional layer, the 3×1 convolutional layer and the 3×3 convolutional layer; the asymmetric convolution output is connected in series with the Swish activation function and the input of the 1×1 convolutional layer in sequence; the output of the 1×1 convolutional layer is residually connected to the input of the asymmetric convolution, and finally serves as the output of the AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module.

[0104] like Figure 8 The figure shows the structure of the first, second, and third SSESA-MBConv4 modules. The first, second, and third SSESA-MBConv4 modules include a first convolutional layer, a depthwise convolutional layer, an SSESA attention mechanism layer, and a second convolutional layer connected in sequence. The first and second convolutional layers are both 1×1 convolutional layers, and the depthwise convolutional layer is a depthwise 3×3 dilated convolutional layer. The output of the second convolutional layer is residually connected to the input of the first convolutional layer, and its output serves as the output of the first, second, and third SSESA-MBConv4 modules.

[0105] like Figure 9 and Figure 10 The following figure shows the network structure of the SSESA attention mechanism. The SSESA attention mechanism is a neural network model structure that combines channel grouping, dual attention, and shuffling. The SSESA attention mechanism layer is set according to the following formula:

[0106] X=X1+X2

[0107] M C1 =σ{W[GAP(X1)]}

[0108] M S1 =σ{f 7×7 [H(GAP(X1),GMP(X1))]}

[0109] M C2 =σ{W[GAP(X2)]}

[0110] M S2 =σ{f 7×7 [H(GAP(X2),GMP(X2))]}

[0111] X out1 =X1·M C1 +X1·M S1

[0112] X out2 =X2·M C2 +X2·M S2

[0113] Y=S[H(X out1 ,X out2 )]

[0114] Where X1 and X2 are the channel splits of input X, W is the fully connected operation in the channel attention mechanism, and f 7×7 is the 7×7 convolution layer operation in the spatial attention mechanism, M C1 is the channel attention mechanism of X1, M S1 is the X1 spatial attention mechanism, M C2 is the channel attention mechanism of X2, M S2 is the X2 spatial attention mechanism, · is the channel multiplication, σ is the Sigmoid activation function, GAP is the global average pooling, GMP is the global maximum pooling, H is the channel fusion operation, S is the channel mixing operation, X out1 is the output of the X1 attention mechanism, X out2 is the attention mechanism output of X2, and Y is the final output of the SSESA attention mechanism module.

[0115] This channel grouping, dual attention, and shuffling fusion scheme splits the input channel into two, applies channel attention and spatial attention separately, and then achieves cross-dimensional interaction through channel shuffling. This design captures both channel importance and spatial key regions without increasing computational complexity, improving edge and skeleton perception while maintaining the lightweight advantages of the SE attention mechanism. This makes it more suitable for mobile visual inspection scenarios with limited computing resources.

[0116] The AsymFused-MBConv1 module connected to the output end of the multi-scale convolutional feature extraction module has 1 layer and a step size of 1; the first AsymFused-MBConv4 module and the second AsymFused-MBConv4 module connected to the output end of the AsymFused-MBConv1 module have 4 layers and a step size of 2; the first SSESA-MBConv4 module connected to the output end of the second AsymFused-MBConv4 module has 3 layers and a step size of 2; the second SSESA-MBConv4 module connected to the output end of the first SSESA-MBConv4 module has 5 layers and a step size of 2; the third SSESA-MBConv4 module connected to the output end of the second SSESA-MBConv4 module has 8 layers and a step size of 2; and Conv1×1&Pooling&FC is connected to the output end of the third SSESA-MBConv4 module, and its layer number is 1.

[0117] Since the number of layers of the Fused-MBConv1 module before scaling is 2, the number of layers of the first Fused-MBConv4 module is 4, the number of layers of the second Fused-MBConv4 module is 4, the number of layers of the MBConv4 module is 6, the number of layers of the second MBConv6 module is 9, and the number of layers of the third MBConv6 module is 15.

[0118] Therefore, this proportional scaling of the number of layers not only reduces the total number of model parameters, but also ensures the accuracy of network model reasoning, improves the speed of model training and real-time reasoning, and saves energy.

[0119] S3. Generate a composite image of the fully expanded umbrella to be tested, and input it into the trained multi-category fault classification model to determine whether the umbrella has a fault and the type of fault. Based on the classification results of the model and the number of times the umbrella is opened and closed, determine whether the umbrella is of qualified quality.

[0120] S31, collecting an image of the umbrella to be tested in the first fully expanded state as a basic image;

[0121] S32, collecting the next fully unfolded image of the umbrella to be tested as a matching image;

[0122] S33, performing image segmentation, image preprocessing, and image AND operations on the matching image to obtain an umbrella structure gradient image with significant edge features;

[0123] S34, combining the base image, the matching image, and the umbrella structure gradient image into a three-channel image;

[0124] S35, inputting the obtained composite image into the multi-category fault classification model obtained in step S2 to obtain the classification result of the model;

[0125] S36. Determine whether the umbrella is qualified based on the classification result of the model and the number of cycle tests of the umbrella to be tested;

[0126] If the fault classification result is normal, the number of cycle tests of the umbrella to be tested is increased by one, and the obtained number of cycle tests of the umbrella to be tested is compared with the standard number of cycle tests: if the total number of cycle tests of the umbrella to be tested is equal to the standard number of cycle tests, the test is stopped, indicating that the umbrella is of qualified quality; if the number of cycle tests of the umbrella to be tested is less than the standard number of cycle tests, the above steps S32-S35 are repeated;

[0127] If the fault diagnosis result is a certain fault type, the detection is stopped and the fault type of the umbrella is given, indicating that the umbrella has not passed the fatigue test and the quality of the umbrella is unqualified.

[0128] The present invention uses a composite image generation method to process the data and then input it into various neural networks. The comparison results are shown in Table 1. The ratio of the training set to the validation set is 8:2, and the number of iterations is 50.

[0129] Table 1

[0130]

[0131] As can be seen from Table 1, the composite image generation and processing method proposed in the present invention is applicable to common classification networks. Combined with the improved multi-category fault classification model of the present invention, its performance is significantly better than the classic neural network model.

[0132] Compared with the training results of classic neural network models such as Swin Transformer, the improved multi-category fault classification model of the present invention has a very small difference in accuracy and loss value between the training set and the validation set. The precision and accuracy curves of the validation set show a smooth upward trend, which is significantly better than other comparison models and improves the overfitting phenomenon.

[0133] Table 2

[0134]

[0135] In this embodiment, an ablation experiment was also conducted on the present invention. As shown in Table 2, Model 1 is compared with the present invention in that only the Fused-MBConv1 module is replaced with the AsymFused-MBConv1 module, the first Fused-MBConv4 module is replaced with the first AsymFused-MBConv4 module, and the second Fused-MBConv4 module is replaced with the second AsymFused-MBConv4 module, while the rest of the model structure remains unchanged. Model 2 is compared with the present invention in that only the multi-scale convolutional feature extraction module is replaced with the original first convolution layer of the EfficientNetV2_S network model, while the rest of the original model structure remains unchanged. Model 3 is compared with the present invention in that only the SSESA attention mechanism is added, the MBConv4 module is replaced with the first SSESA-MBConv4 module, the second MBConv6 module is replaced with the second SSESA-MBConv4 module, and the third MBConv6 module is replaced with the third SSESA-MBConv4 module, while the rest of the original model structure remains unchanged. Model 4, compared to the present invention, adds a CAA attention mechanism between the second AsymFused-MBConv4 module and the first SSESA-MBConv4 module, based on Model 3. Model 5, without any improvements compared to the present invention, is the original EfficientNetV2_S model. This ablation experiment clearly demonstrates that the improved neural network model structure of the present invention performs exceptionally well.

[0136] like Figure 11 As shown, the present invention provides a sunshade umbrella fault detection system, which specifically includes: an image acquisition camera 10, an industrial control computer 20, a programmable logic controller 30, and a fatigue testing device 40. The industrial control computer 20 comprises a memory 201, a central processing unit 202, and an input / output interface 203. The memory 201 primarily stores an operating system and computer programs, which are loaded and executed by the central processing unit 202 to implement the relevant steps of the aforementioned sunshade umbrella multi-category fault detection method.

[0137] The memory 201 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored therein include an operating system and computer programs, and the storage method can be temporary storage or permanent storage.

[0138] The programmable logic controller 30 is used to control the fatigue test device to perform the umbrella cyclic opening and closing test. It can be a PLC of various models and is electrically connected to the fatigue test device.

[0139] The image acquisition camera 10 is arranged below the umbrella to be inspected, and is kept on the same horizontal line as the fixture for clamping the umbrella handle, and is located on the left and right sides of the umbrella handle.

[0140] The input / output interface 203 is used for the electrical connection between the industrial control computer 20 and the programmable logic controller 30 and the electrical connection between the image acquisition camera 10 and the industrial control computer 20. Its physical interface is an Ethernet interface, and the communication protocol it follows is the industrial-grade TCP / IP protocol.

[0141] The operating system is used to manage and control various hardware devices and computer programs on the industrial control computer 20, and can be Windows Server, Linux, Unix, etc.

[0142] This invention applies multi-category fault classification technology to the real-time multi-category fault classification task for umbrellas, eliminating the need for human decision-making and significantly improving detection efficiency and accuracy. The multi-fault classification algorithm of the present invention provides real-time diagnosis of whether a fault has occurred during the opening and closing cycle of an umbrella, and can quickly determine the type of fault, improving the efficiency and accuracy of real-time fault detection. The detection method of the present invention is scalable and can be used to identify local fault locations in an umbrella, enabling targeted fault location detection.

Claims

1. A multi-category fault detection method for umbrellas based on machine vision, characterized in that: The following steps are involved: S1. Use a camera to capture images of fully unfolded umbrellas, perform label processing on the captured images, and generate composite images to obtain a multi-category fault classification dataset for umbrellas. S2. Build a multi-category fault classification model based on the improved EfficientNetV2_S model. Input the obtained umbrella multi-category fault classification dataset into the multi-category fault classification neural network model for training to obtain a trained multi-category fault classification model. S3. Generate a composite image of the fully expanded umbrella to be tested, and input it into the trained multi-category fault classification model to determine whether the umbrella has a fault and the type of fault. Based on the classification results of the model and the number of times the umbrella is opened and closed, determine whether the umbrella is of qualified quality.

2. The multi-category fault detection method for umbrellas based on machine vision according to claim 1 is characterized in that: The step S1 is specifically as follows: S11. Use a camera to capture images of a fully expanded umbrella in a normal state and in multiple fault states as basic sample images of the umbrella. Label the captured images and classify them into the following categories: good image, image of a clamping and shifting fault, image of a broken rib fault, image of a broken thread fault, image of a seam eyelet disconnection fault, and image of a seam separation fault. S12, pairing each good grayscale image of the umbrella with each grayscale image of a different fault category in sequence, and simultaneously pairing each good grayscale image with other good grayscale images to obtain a corresponding grayscale image pair set; S13. Perform image segmentation processing on all collected images of good umbrellas and various types of faulty umbrellas to separate the umbrellas from the background; perform image preprocessing, threshold segmentation, image summing, histogram enhancement, and gradient calculation on the segmented umbrella images in sequence to obtain a gradient image of the umbrella structure with significant edge features; S14, performing composite image generation processing on the grayscale image of a good umbrella, the grayscale image of a faulty umbrella, and the corresponding umbrella structure gradient image obtained in S13, to obtain a plurality of composite images of the umbrella; S15, combining all the composite images corresponding to each good image of the same umbrella in step S14 to form a multi-category fault classification dataset of the umbrella; S16, using the same method as steps S11 to S15 to construct a multi-category fault classification dataset for multiple umbrellas; S17: Combine the multi-category fault classification datasets of all the umbrellas obtained in step S16 to finally form a multi-category fault classification dataset of the umbrellas.

3. The multi-category fault detection method for umbrellas based on machine vision according to claim 1 is characterized by: The multi-category fault classification model is constructed based on the improved EfficientNetV2_S model, and is connected in sequence to a multi-scale convolutional feature extraction module, a feature extraction backbone network, and a classification layer; wherein the multi-scale convolutional feature extraction module is used to replace the first convolution layer in the EfficientNetV2_S network model; the feature extraction backbone network is composed of improved modules, specifically: replacing the Fused-MBConv1 module with the AsymFused-MBConv1 module; replacing the first Fused-MBConv4 module with the first AsymFused-MBConv4 module; replacing the second Fused-MBConv4 module with the second AsymFused-MBConv4 module; replacing the MBConv4 module with The first SSESA-MBConv4 module; the second MBConv6 module is replaced by the second SSESA-MBConv4 module; the third MBConv6 module is replaced by the third SSESA-MBConv4 module; and the first AsymFused-MBConv4 module, the second AsymFused-MBConv4 module, the first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module in the feature extraction backbone network are streamlined to reduce the number of model parameters, and a CAA attention mechanism is inserted between the second AsymFused-MBConv4 module and the first SSESA-MBConv4 module. The classification layer mainly consists of a fully connected layer.

4. The multi-category fault detection method for umbrellas based on machine vision according to claim 3 is characterized by: The multi-scale convolution feature extraction module includes four parallel splicing branches, the first parallel splicing branch includes a first 1×1 convolution layer, the second parallel splicing branch includes a second 1×1 convolution layer, a second 1×3 convolution layer, and a second 3×1 convolution layer connected in series, the third parallel splicing branch includes a third 1×1 convolution layer, a third 1×5 convolution layer, and a third 5×1 convolution layer connected in series, and the fourth parallel splicing branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolution layer; the input end of the first 1×1 convolution layer, the second 1×1 The input end of the convolution layer, the input end of the third 1×1 convolution layer, and the input end of the fourth 3×3 pooling layer are connected in parallel as the input end of the multi-scale convolution feature extraction module; the output end of the first 1×1 convolution layer, the output end of the second 3×1 convolution layer, the output end of the third 5×1 convolution layer, and the output end of the fourth 1×1 convolution layer are connected in parallel as the output end of the multi-scale convolution feature extraction module. The output end of the multi-scale convolution feature extraction module is connected to the input end of the 2×2 maximum pooling layer, and its output is the final output of the multi-scale convolution feature extraction module.

5. The multi-category fault detection method for umbrellas based on machine vision according to claim 3 is characterized by: The AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module are improved based on the Fused-MBConv1 module, the first Fused-MBConv4 module, and the second Fused-MBConv4 module. Specifically, the 3×3 convolutional layers in the Fused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module are replaced with asymmetric convolutional layers; the asymmetric convolutional layers are connected in parallel. The network consists of a 1×3 convolutional layer, a 3×1 convolutional layer and a 3×3 convolutional layer; the input of the asymmetric convolutional layer is connected to the output of the previous layer; the output of the asymmetric convolutional layer is the result of adding the outputs of the 1×3 convolutional layer, the 3×1 convolutional layer and the 3×3 convolutional layer; the output of the asymmetric convolution is connected in series with the Swish activation function and the input of the 1×1 convolutional layer in sequence; the output of the 1×1 convolutional layer is residually connected to the input of the asymmetric convolution, and finally serves as the output of the AsymFused-MBConv1 module, the first AsymFused-MBConv4 module, and the second AsymFused-MBConv4 module.

6. The multi-category fault detection method for umbrellas based on machine vision according to claim 3 is characterized by: The first SSESA-MBConv4 module, the second SSESA-MBConv4 module, and the third SSESA-MBConv4 module are improved based on the MBConv4, the first MBConv6 module, and the second MBConv6 module. Specifically, the SE channel attention mechanism layer in the MBConv4, the first MBConv6 module, and the second MBConv6 module is replaced by a channel grouping-dual attention-shuffle fusion SSESA attention mechanism layer.

7. The multi-category fault detection method for umbrellas based on machine vision according to claim 6 is characterized by: The SSESA attention mechanism is a neural network model structure that combines channel grouping, dual attention, and shuffling. The SSESA attention mechanism layer is set according to the following formula: X=X1+X2 M C1 =σ{W[GAP(X1)]} M S1 σ{f 7×7 [H(GAP(X1),GMP(X1))]} M C2 =σ{W[GAP(X2)]} M S2 =σ{f 7×7 [H(GAP(X2),GMP(X2))]} X out1 =X1·M C1 +X1·M S1 X out2 =X2·M C2 +X2·M S2 Y=S[H(X out1 ,X out2 )] Where X1 and X2 are the channel splits of input X, W is the fully connected operation in the channel attention mechanism, and f 7×7 is the 7×7 convolution layer operation in the spatial attention mechanism, M C1 is the channel attention mechanism of X1, M S1 is the X1 spatial attention mechanism, M C2 is the channel attention mechanism of X2, M S2 is the X2 spatial attention mechanism, · is the channel multiplication, σ is the Sigmoid activation function, GAP is the global average pooling, GMP is the global maximum pooling, H is the channel fusion operation, S is the channel mixing operation, X out1 is the output of the X1 attention mechanism, X out2 is the attention mechanism output of X2, and Y is the final output of the SSESA attention mechanism module.

8. The multi-category fault detection method for umbrellas based on machine vision according to claim 3 is characterized by: The number of layers of the feature extraction backbone network is simplified as follows: Set up a multi-scale convolution feature extraction module; The AsymFused-MBConv1 module connected to the output of the multi-scale convolutional feature extraction module has 1 layer and 1 stride; The first AsymFused-MBConv4 module and the second AsymFused-MBConv4 module connected to the output of the AsymFused-MBConv1 module have 4 layers and a stride of 2; The first SSESA-MBConv4 module connected to the output of the second AsymFused-MBConv4 module has 3 layers and a stride of 2; A second SSESA-MBConv4 module connected to the output of the first SSESA-MBConv4 module has 5 layers and a stride of 2; A third SSESA-MBConv4 module connected to the output of the second SSESA-MBConv4 module has 8 layers and a stride of 2; The output of the third SSESA-MBConv4 module is connected to Conv1×1&Pooling&FC, and the number of layers is 1.

9. The multi-category fault detection method for umbrellas based on machine vision according to claim 1 is characterized in that: The step S3 is specifically as follows: S31, collecting an image of the umbrella to be tested in the first fully expanded state as a basic image; S32, collecting the next fully unfolded image of the umbrella to be tested as a matching image; S33, performing image segmentation, image preprocessing, and image AND operations on the matching image to obtain an umbrella structure gradient image with significant edge features; S34, combining the base image, the matching image, and the umbrella structure gradient image into a three-channel image; S35, inputting the obtained composite image into the multi-category fault classification model obtained in step S2 to obtain the classification result of the model; S36. Determine whether the umbrella is qualified based on the classification result of the model and the number of cycle tests of the umbrella to be tested; If the fault classification result is normal, the number of cycle tests of the umbrella to be tested is increased by one, and the obtained number of cycle tests of the umbrella to be tested is compared with the standard number of cycle tests: if the total number of cycle tests of the umbrella to be tested is equal to the standard number of cycle tests, the test is stopped, indicating that the umbrella is of qualified quality; if the number of cycle tests of the umbrella to be tested is less than the standard number of cycle tests, the above steps S32-S35 are repeated; If the fault diagnosis result is a certain fault type, the detection is stopped and the fault type of the umbrella is given, indicating that the umbrella has not passed the fatigue test and the quality of the umbrella is unqualified.

10. A sunshade umbrella fault detection system used in the sunshade umbrella multi-category fault detection method according to any one of claims 1 to 9, characterized in that: It includes an image acquisition camera, an industrial control computer, a programmable logic controller and a fatigue test device; the image acquisition camera is arranged below the umbrella for detecting sunny or rainy weather; the industrial control computer includes a memory, a central processing unit and an input / output interface; the programmable logic controller controls the fatigue test device to execute the umbrella's cyclic opening and closing; the programmable logic controller is electrically connected to the fatigue test device, and the industrial control computer is electrically connected to the image acquisition camera and the programmable logic controller.

Images