Umbrella fault diagnosis method and system based on machine vision

Abstract

The invention discloses an umbrella fault diagnosis method and system based on machine vision. The method comprises the steps that a camera is used for shooting, collecting and processing to obtain a sun-rain umbrella fault data set, then a fault diagnosis model is constructed, the sun-rain umbrella fault data set is input into the fault diagnosis model for training, and a trained fault diagnosis model is obtained; and finally, inputting the image that the to-be-detected sun-rain umbrella is not opened continuously into the trained fault diagnosis model, and finally obtaining whether the complete opening frequency of the to-be-detected sun-rain umbrella meets the quality requirement or not. The system comprises a camera, a test device, a PLC control system and an upper computer. According to the invention, the image matching method and the constructed fault diagnosis model have good accuracy in sun-rain umbrella fault diagnosis, and more energy consumption and manpower consumption are saved.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue testing of sunshade umbrellas, and in particular to a sunshade umbrella fault diagnosis method and system based on machine vision. Background Art

[0002] During the production and use process, umbrellas will experience multiple openings and closings and various environmental conditions. According to national standards and user needs. The fatigue test of umbrellas needs to be high-precision, high-efficiency, and able to simulate various situations in actual use. The traditional umbrella fatigue test fault diagnosis method relies on manual judgment, which is inefficient, prone to false detection, missed detection, and difficult to standardize. The use of machine vision technology can provide more accurate and efficient fault detection. At present, the domestic umbrella fatigue test system does not have the automatic judgment of relevant mechanical indicators, cannot form standard test reports and quality traceability documents, and cannot realize intelligent real-time detection of umbrella faults. Therefore, the development of a machine vision-based umbrella fatigue test system will help fill the gaps in the technical testing requirements of national standards and existing testing equipment.

[0003] At present, the fault diagnosis during the fatigue test of the umbrella mainly relies on traditional infrared sensor monitoring and manual diagnosis. Infrared sensor monitoring requires that the number of sensors must be consistent with the number of umbrella ribs, and can only detect the failure of the broken umbrella ribs. This method is not only expensive in equipment cost and high in umbrella installation requirements, but also has poor adaptability. It is difficult for the manual diagnosis method to find the subtle faults of the umbrella during the fatigue test, and the general opening and closing times of the umbrella are 400 or 600 times. It is not only labor-intensive, but also prone to misjudgment and missed judgment due to people's subjective differences.

[0004] At present, machine vision and deep learning related technologies and methods have been widely used in some areas of industrial inspection. The existing classic neural network model and image data input method are difficult to identify the fault shape without obvious umbrella shape change in the test environment with complex background, which is difficult to meet the needs of fast and accurate fault diagnosis of sunny umbrellas during fatigue tests. Therefore, providing a sunny umbrella fault diagnosis method based on machine vision has become a technical problem to be solved urgently in the relevant technical field. Summary of the invention

[0005] In order to solve the problems and defects in the background technology, the present invention provides a sunshade umbrella fault diagnosis method and system based on machine vision, which uses a machine learning algorithm to perform real-time fault diagnosis on sunshade umbrellas during fatigue testing, so as to solve the current data management and intelligent diagnosis in the field of sunshade umbrella fatigue testing.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] 1. A method for diagnosing umbrella faults based on machine vision, comprising the following steps:

[0008] S1. Use a camera to capture images of a fully opened umbrella, and perform label processing and image pairing processing on the collected images in turn to obtain a dataset of umbrella failures;

[0009] S2, constructing a fault diagnosis model, inputting the obtained umbrella fault data set into the neural network diagnosis model for training, and obtaining a trained fault diagnosis model;

[0010] S3. Perform a cyclic full opening test on the sun-rain umbrella to be tested, and perform image pairing processing on the real-time collected images of the sun-rain umbrella to be tested after each full opening, and input them into the trained fault diagnosis model, so as to continuously obtain fault diagnosis results, and judge in real time whether the fault diagnosis results meet the requirements for stopping detection, so as to obtain whether the number of full opening times of the sun-rain umbrella to be tested meets the quality requirements.

[0011] S11, using a camera to capture images of a fully opened umbrella in different states as umbrella images, labeling the umbrella images to obtain good images and damaged images of the umbrella;

[0012] S12, performing image pairing processing and label processing on each good image of the umbrella and other images in the collected umbrella images in sequence to obtain a plurality of grayscale image pairs;

[0013] The label processing after the image pairing processing is specifically as follows: a grayscale image pair obtained by pairing a good image with a good image is used as a positive sample image pair, and a grayscale image pair obtained by pairing a good image with a damaged image is used as a negative sample image pair;

[0014] S13, combining a plurality of grayscale image pairs obtained from each good image of the umbrella to obtain a fault data set of the umbrella;

[0015] S14, using the same method as step S11 to step S13 to obtain a fault data set of several umbrellas of different types;

[0016] The different types of sunny and rainy umbrellas include hand-opened and folded umbrellas, self-opening straight-bone umbrellas, self-opening and self-folding umbrellas and hand-opened straight-bone umbrellas.

[0017] S15. Combine the fault data sets of several umbrellas of different types to obtain a sun-rain umbrella fault data set.

[0018] The fault diagnosis model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module and a classification module which are connected in series in sequence; the first feature extraction module is mainly composed of a first cross-channel convolutional layer, a first regularization layer 1, a first convolutional layer, a first regularization layer 2, a first maximum pooling layer and a first attention mechanism layer which are connected in series in sequence; the second feature extraction module is mainly composed of a second cross-channel convolutional layer, a second regularization layer 1, a second convolutional layer, a second regularization layer 2 and a second maximum pooling layer which are connected in series in sequence; the third feature extraction module is mainly composed of a third cross-channel convolutional layer, a third regularization layer 1, a third convolutional layer 1, a third regularization layer 2 and a third attention mechanism layer which are connected in series in sequence The fourth feature extraction module is mainly composed of a fourth cross-channel convolutional layer, a fourth regularization layer one, a fourth convolutional layer one, a fourth regularization layer two, a fourth convolutional layer two, a fourth regularization layer three, a fourth maximum pooling layer and a fourth attention mechanism layer connected in series in sequence; the fifth feature extraction module is mainly composed of a fifth cross-channel convolutional layer, a fifth regularization layer one, a fifth convolutional layer one, a fifth regularization layer two, a fifth convolutional layer two, a fifth regularization layer three, a fifth depth splicing module layer one, a fifth maximum pooling layer one, a fifth depth splicing module layer two and a fifth maximum pooling layer two connected in series in sequence; the classification layer is mainly composed of a fully connected layer;

[0019] The umbrella fault data set obtained in step S1 is input into the first cross-channel convolutional layer of the first feature extraction module, the output end of the first attention mechanism layer is connected to the input end of the second cross-channel convolutional layer, the output end of the second maximum pooling layer is connected to the input end of the third cross-channel convolutional layer, the output end of the third maximum pooling layer is connected to the input end of the fourth cross-channel convolutional layer, the output end of the fourth attention mechanism layer is connected to the input end of the fifth cross-channel convolutional layer, the output end of the fifth maximum pooling layer 2 is connected to the input end of the fully connected layer, and the output end of the fully connected layer outputs the classification result.

[0020] The first regularization layer one, the first regularization layer two, the second regularization layer one, the second regularization layer two, the third regularization layer one, the third regularization layer two, the third regularization layer three, the fourth regularization layer one, the fourth regularization layer two, the fourth regularization layer three, the fifth regularization layer one, the fifth regularization layer two and the fifth regularization layer three are all regularization layers, and all use a combination of regularization and ReLU activation function.

[0021] The first convolutional layer, the second convolutional layer, the third convolutional layer one, the third convolutional layer two, the fourth convolutional layer two, the fourth convolutional layer one, the fifth convolutional layer one and the fifth convolutional layer two are all 3*3 convolutional layers.

[0022] The first cross-channel convolution layer, the second cross-channel convolution layer, the third cross-channel convolution layer, the fourth cross-channel convolution layer and the fifth cross-channel convolution layer are all 1×1 convolution layers with a step size of 1 and a padding of 0; the convolution layer is a 3*3 layer. The fifth depth splicing module layer one and the fifth depth splicing module layer two both include four parallel splicing branches, the first parallel splicing branch includes a first 1×1 convolutional layer, the second parallel splicing branch includes a second 1×1 convolutional layer and a second 3×3 convolutional layer connected in series in sequence, the third parallel splicing branch includes a third 1×1 convolutional layer and a third 5×5 convolutional layer, and the fourth parallel splicing branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolutional layer; the input end of the first 1×1 convolutional layer, the input end of the second 1×1 convolutional layer, the input end of the third 1×1 convolutional layer, and the input end of the fourth 3×3 pooling layer are connected in parallel as the input end of the fifth depth splicing module layer one or the fifth depth splicing module layer two; the output end of the first 1×1 convolutional layer, the output end of the second 3×3 convolutional layer, the output end of the third 5×5 convolutional layer, and the output end of the fourth 1×1 convolutional layer are connected in parallel as the output end of the fifth depth splicing module layer one or the fifth depth splicing module layer two.

[0023] The attention mechanism layer uses a convolutional neural network with neurons as calculation factors, and the attention mechanism layer is set according to the following formula:

[0024]

[0025]

[0026] in, and are the mean and variance of all neurons, is the minimum energy function, t is the target neuron of a single channel in the input feature, x i is the i-th neuron other than the target neuron in a single channel in the input feature, i is the spatial dimension index, M is the number of neurons on the channel, λ is an adjustable parameter, and the present invention sets it to λ=0.0001.

[0027] S31, collecting an image of the sun-rain umbrella to be tested after it is fully opened for the first time as a basic image, and initializing the number of times the sun-rain umbrella to be tested is fully opened;

[0028] S32, collecting an image of the tested umbrella after it is fully opened next time as a matching image;

[0029] S33, performing image pairing processing on the basic image and the matching image to obtain a grayscale image pair to be tested;

[0030] S34, inputting the obtained grayscale image pair to be tested into the trained fault diagnosis model obtained in step S2 to obtain a fault diagnosis result;

[0031] S35, judging whether the fault diagnosis result meets the requirement of stopping detection:

[0032] If the fault diagnosis result is good, the number of times the sun-and-rain umbrella to be tested is fully opened is increased by one, and the obtained number of times the sun-and-rain umbrella to be tested is fully opened is compared with the preset threshold of the number of times of fully opening: if the total number of times the sun-and-rain umbrella to be tested is fully opened is equal to the threshold of the number of times of fully opening, the detection is stopped, and the number of times the sun-and-rain umbrella to be tested is fully opened meets the quality requirements; if the number of times the sun-and-rain umbrella to be tested is fully opened is less than the threshold of the number of times of fully opening, the above steps S32-S34 are repeated;

[0033] If the fault diagnosis result is damage, the test is stopped, and the number of times the umbrella is fully opened does not meet the quality requirements.

[0034] The image pairing method in step S12 or step S33 is specifically as follows: grayscale processing is performed on the two images respectively to obtain two grayscale images, and then the two grayscale images are fused in a single channel dimension to obtain a grayscale image pair. The grayscale image pair obtained by the fusion processing is a dual-channel image.

[0035] 2. A Sunshade Fault Diagnosis System Based on Machine Vision

[0036] It includes a camera, a test device, a PLC control system and a host computer; the sunshade umbrella to be tested is clamped on the test device, the camera is arranged below the sunshade umbrella to be tested, and collects and detects photos of the inner surface of the sunshade umbrella to be tested; the PLC control system is installed on the test device, and the PLC control system is electrically connected to the test device and the host computer respectively, and the host computer is also electrically connected to the camera.

[0037] The test device includes an opening and closing mechanism and a pressure sensor, and the PLC control module includes an opening and closing control module and a pressure measuring module; the sunshade umbrella to be tested is clamped on the opening and closing mechanism, the input end and the output end of the opening and closing control module are electrically connected to the host computer and the opening and closing mechanism respectively, and the input end and the output end of the pressure measuring module are electrically connected to the pressure sensor and the host computer respectively; the pressure sensor is used to measure the switch button force and the umbrella pole compression force of the sunshade umbrella to be tested.

[0038] The host computer includes an image acquisition module, a result evaluation module and a fault diagnosis module with the fault diagnosis model installed inside; the input end and the output end of the image acquisition module are respectively connected to the camera output end and the input end of the fault diagnosis module, the input end and the output end of the result evaluation module are respectively communicatively connected to the output end of the fault diagnosis module and the input end of the opening and closing control module, the input end of the result evaluation module is also communicatively connected to the output end of the pressure measurement module, and the output end of the result evaluation module is also electrically connected to the control end of the camera.

[0039] The innovation of the present invention lies in the way of using image pairing processing method to process images when preparing the umbrella fault data set and diagnosing the umbrella fault to be tested, which brings the beneficial effect of enabling the fault diagnosis model to extract more feature correlation information compared to the traditional RGB three-channel single color image network input method.

[0040] The innovation of the present invention is that it uses machine vision methods for the first time to diagnose the fault of a parasol, and the adopted fault diagnosis model achieves higher accuracy than the traditional common network model, and brings the advantage of still meeting the real-time requirements.

[0041] The beneficial effects of the present invention are:

[0042] 1. In the preparation of the umbrella fault data set, the present invention uses image pairing processing to obtain a dual-channel grayscale image pair on the collected images. Compared with the traditional RGB three-channel single color image network input method, its positive and negative sample images have better feature association information. Compared with the traditional RGB three-channel single color image network input method, in the fault diagnosis model of the present invention, its accuracy is improved by 26%, reaching 99.93%.

[0043] 2. The fault diagnosis model constructed by the present invention achieves a high accuracy while maintaining low energy consumption, and the performance of the network model structure designed by the present invention is better than that of common networks such as VGG16 and ResNet18.

[0044] 3. The umbrella fault diagnosis system of the present invention uses machine vision methods to diagnose umbrella faults for the first time, achieving a higher accuracy rate than traditional conventional methods, meeting real-time requirements while saving more energy and manpower consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the overall module diagram of the umbrella fault diagnosis system;

[0046] Figure 2 The flowchart of the umbrella fault diagnosis method is shown in FIG.

[0047] Figure 3 It is a positive sample image pair of the umbrella failure dataset;

[0048] Figure 4 It is the negative sample image pair of the umbrella failure dataset;

[0049] Figure 5 This is the network structure diagram of the fault diagnosis model;

[0050] Figure 6 This is the structural diagram of the attention mechanism layer;

[0051] Figure 7 This is the structural diagram of the deep stitching module. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] like Figure 1 As shown, the umbrella fault diagnosis system of the present invention includes a camera, a test device, a PLC control system and a host computer.

[0054] The umbrella to be tested is clamped on the test device, and the camera is arranged under the umbrella to be tested to collect and detect photos of the inner surface of the umbrella to be tested. The PLC control system is installed on the test device, and the PLC control system is electrically connected to the test device and the host computer respectively, and the host computer is also electrically connected to the camera; the camera takes a picture of the umbrella to be tested and transmits the photographed image to the host computer, and the host computer uses the internal fault diagnosis model to perform detection and evaluation processing on the image in sequence, and the host computer transmits the evaluation result to the PLC control system, and the PLC control system controls the test device to determine whether to perform the next opening and closing processing of the umbrella to be tested according to the evaluation result; the test device measures the pressure of the umbrella to be tested and transmits the measured pressure data to the PLC control system, the PLC control system processes the pressure data and transmits it to the host computer, and the host computer analyzes whether the pressure of the umbrella to be tested is qualified.

[0055] The test device includes an opening and closing mechanism and a pressure sensor, and the PLC control module includes an opening and closing control module and a pressure measuring module; the sunshade umbrella to be tested is clamped on the opening and closing mechanism, the input and output ends of the opening and closing control module are electrically connected to the host computer and the opening and closing mechanism respectively, and the input and output ends of the pressure measuring module are electrically connected to the pressure sensor and the host computer respectively; the pressure sensor is used to measure the switch button force and umbrella pole compression force of the sunshade umbrella to be tested.

[0056] The opening and closing control module controls the opening and closing mechanism to open and close the sun-ray umbrella to be tested. The pressure sensor transmits the measured pressure results of the switch button force and the umbrella pole compression force of the sun-ray umbrella to be tested to the pressure measurement module. The pressure measurement module processes the pressure data and transmits it to the host computer. The host computer uses the internal fault diagnosis model to detect and evaluate the image of the sun-ray umbrella to be tested after it is fully opened. The host computer transmits the evaluation result to the opening and closing control module. The opening and closing control module controls the opening and closing mechanism to decide whether to perform the next opening and closing process of the sun-ray umbrella to be tested according to the evaluation result. The host computer also evaluates the received pressure data to evaluate whether the quality of the switch button and the umbrella pole of the sun-ray umbrella to be tested is qualified.

[0057] The host computer includes an image acquisition module, a result evaluation module and a fault diagnosis module with a fault diagnosis model installed inside; the input and output ends of the image acquisition module are respectively communicatively connected to the camera output end and the input end of the fault diagnosis module, the input and output ends of the result evaluation module are respectively communicatively connected to the output end of the fault diagnosis module and the input end of the opening and closing control module, the input end of the result evaluation module is also communicatively connected to the output end of the pressure measurement module, and the output end of the result evaluation module is also electrically connected to the control end of the camera.

[0058] The camera collects the image of the sun umbrella to be tested and transmits the collected image to the image acquisition module. The image acquisition module transmits the image to the fault diagnosis module. The fault diagnosis module uses the internal fault diagnosis model to perform fault diagnosis processing on the sun umbrella to be tested to obtain a fault diagnosis result. The fault diagnosis module transmits the fault diagnosis result to the result evaluation module to determine whether it meets the requirements for stopping detection. The result evaluation module then transmits the judgment result to the opening and closing control module to determine whether to perform the next opening and closing process until the fault diagnosis result is damage and stops detection. The result evaluation module determines whether the number of full openings of the sun umbrella to be tested meets the quality requirements based on whether the total number of full openings of the sun umbrella to be tested is equal to the threshold number of full openings. The pressure measurement module processes the pressure data and transmits it to the result evaluation module. The result evaluation module also determines whether the quality of the switch button and the umbrella pole is qualified based on the obtained pressure of the sun umbrella to be tested, thereby determining whether the quality of the sun umbrella to be tested is finally qualified. The result evaluation module transmits the result to the control end of the camera based on whether the number of full openings of the sun umbrella to be tested meets the quality requirements, and controls the camera to shoot.

[0059] The test device also includes a temperature and humidity sensor and a pull-wire displacement sensor; the PLC control system also includes a temperature and humidity sensing module and a pull-wire displacement sensing module; the host computer also includes an information management module, a data storage module, a parameter configuration module and a real-time data display and analysis module.

[0060] The temperature and humidity sensor measures the temperature and humidity of the test environment, and then transmits the temperature and humidity to the temperature and humidity sensing module, which then transmits the processed data to the real-time data display and analysis module for analysis and display.

[0061] The pull-wire displacement sensor transmits the data of whether the sun-rain umbrella to be tested is fully opened to the pull-wire displacement sensing module. The pull-wire displacement sensing module is responsible for judging whether the sun-rain umbrella to be tested is fully opened. If it is fully opened, the pull-wire displacement sensing module transmits the full opening signal to the result evaluation module of the host computer. The result evaluation module of the host computer controls the camera to take pictures. If it is not fully opened, the detection is stopped and the sun-rain umbrella to be tested does not meet the quality requirements.

[0062] The parameter configuration module is responsible for the basic parameter configuration of the umbrella and the parameter configuration of the acquisition camera. The basic parameter configuration of the umbrella includes the sample number setting, inspection category setting, umbrella model setting, umbrella type setting, sample name setting, umbrella specification setting, qualified full opening times threshold setting, high and low value setting of the button force of the switch umbrella, and high and low value setting of the compression pole force. The basic parameters of the umbrella only need to be configured once, and will be automatically stored in the database after the configuration is completed. When testing, you only need to select the umbrella model, and the database will automatically filter the relevant data and automatically adjust all the parameters of the current test. The parameter configuration of the acquisition camera includes the camera's exposure and gain, image zoom parameters, and video storage address.

[0063] The information management module includes the employee basic information and authority management module and the detection historical data traceability module.

[0064] The real-time data display and analysis module includes the basic parameters of the umbrella, the current opening and closing times, the switch button force value, the compression pole force value, the current detection speed, the test date value, the real-time test progress, the real-time temperature and humidity values, the inspection results, the real-time inspection log, the image display of the outer surface of the umbrella, and the real-time image display of the inner surface of the umbrella.

[0065] The real-time fault diagnosis of the present invention is run in the form of a separate process, which is different from the traditional method of collecting images in the same thread and passing them into the fault diagnosis model. The method adopted by the present invention is that model reasoning is in the form of a process to process data in parallel with other modules, and the TCP / IP protocol is used to package data for communication, to ensure that when executing fault reasoning, the running speed of other modules is not affected. The traditional method of collecting images in a single thread and then performing fault reasoning must wait until the fault reasoning is completed before taking the next picture of a fully opened umbrella. Compared with the traditional reasoning method, the method of the present invention greatly improves the speed of model reasoning and meets the opening and closing speed of the sunny umbrella fatigue test required by the national standard.

[0066] like Figure 2 As shown, the specific embodiments of the present invention are as follows:

[0067] S1. Use a camera to capture images of a fully opened umbrella, perform label processing and image pairing processing on the collected images in turn, and obtain a dataset of umbrella faults.

[0068] S11. Use a camera to capture images of a fully opened umbrella in different states as umbrella images, perform label processing on the umbrella images, and obtain good images and damaged images of the umbrella.

[0069] Different states include fault-free images of the umbrella when fully opened at different angles, different brightness and the same brightness. The umbrella angle is then randomly rotated and the brightness is adjusted to perform continuous fatigue opening and closing tests on the sun-rain umbrella until the umbrella fails for the first time. Images of a single fault form when the umbrella is fully opened at different angles, different brightness and the same brightness are collected. Further fatigue opening and closing tests are performed on the sun-rain umbrella to cause multiple faults in the umbrella at the same time. Images of multiple fault forms when the umbrella is fully opened at different angles, different brightness and the same brightness are collected.

[0070] S12, performing image pairing processing and label processing on each good image of the umbrella with other images in the collected umbrella images in sequence to obtain a plurality of grayscale image pairs.

[0071] The image pairing method is specifically as follows: grayscale processing is performed on the two images respectively to obtain two grayscale images, and then the two grayscale images are fused in a single channel dimension to obtain a grayscale image pair. The grayscale image pair obtained by the fusion processing is a dual-channel image.

[0072] The label processing after the image pairing processing is specifically as follows: the grayscale image pair obtained by the image pairing processing of the good image and the good image is used as the positive sample image pair, and the grayscale image pair obtained by the image pairing processing of the good image and the damaged image is used as the negative sample image pair;

[0073] In a specific implementation, two duplicate images are removed to obtain a grayscale image pair after image pairing processing.

[0074] S13, combining several grayscale image pairs obtained from each good image of the umbrella to obtain a fault data set of the umbrella.

[0075] S14. The same method as step S11 to step S13 is used to obtain a fault data set of several umbrellas of different types.

[0076] Different types of umbrellas include hand-opening retracting umbrellas, self-opening straight-rib umbrellas, self-opening and self-retracting umbrellas, and hand-opening straight-rib umbrellas.

[0077] S15. Combine the fault data sets of several umbrellas of different types to obtain a sun-rain umbrella fault data set.

[0078] like Figure 3 and Figure 4As shown, this embodiment randomly selects 70 umbrellas from 100 umbrella samples, conducts continuous fatigue opening and closing tests on them, uses a binocular industrial camera as an image acquisition device to shoot the fully opened inner surface of the umbrella, obtains images of fault-free umbrellas and umbrellas with different fault forms, and constructs a basic data set. The original image is preprocessed. Since the original image size obtained by using an industrial camera is 3072×2048pixel, the original size of the image is too large, so directly inputting it into the model for training will result in a huge amount of calculation for the entire training process, and the training speed is too slow, so the width and height of the original image size are first reduced to 680×480pixel. Using the reduced image for training can not only ensure that the feature information loss is small, but also greatly improve the training speed and reduce the detection time cost.

[0079] Since the colors of umbrellas are diverse, it is necessary to convert the reduced image data into grayscale images to remove color interference, which can make the neural network pay more attention to the feature differences between image pairs, thereby better improving the model accuracy.

[0080] Finally, the umbrella fault dataset has a total of 20,026 grayscale image pairs, of which the training set has 12,016 grayscale image pairs, the validation set has 4,005 grayscale image pairs, and the test set has 4,005 grayscale image pairs. The image pairs in the training set are in the same proportion as the faulty and non-faulty datasets.

[0081] S2. Construct a fault diagnosis model, input the obtained umbrella fault data set into the neural network diagnosis model for training, and obtain a trained fault diagnosis model.

[0082] like Figure 5As shown, the fault diagnosis model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module and a classification module which are connected in series in sequence. The first feature extraction module is mainly composed of a first cross-channel convolutional layer, a first regularization layer one, a first convolutional layer, a first regularization layer two, a first maximum pooling layer and a first attention mechanism layer which are connected in series in sequence; the second feature extraction module is mainly composed of a second cross-channel convolutional layer, a second regularization layer one, a second convolutional layer, a second regularization layer two and a second maximum pooling layer which are connected in series in sequence; the third feature extraction module is mainly composed of a third cross-channel convolutional layer, a third regularization layer one, a third convolutional layer one, a third regularization layer two, a third convolutional layer two, a third regularization layer three and a third maximum pooling layer which are connected in series in sequence; the fourth feature extraction module is mainly composed of a first cross-channel convolutional layer, a first ... The feature extraction module is mainly composed of the fourth cross-channel convolution layer, the fourth regularization layer one, the fourth convolution layer one, the fourth regularization layer two, the fourth convolution layer two, the fourth regularization layer three, the fourth maximum pooling layer and the fourth attention mechanism layer connected in series in sequence; the fifth feature extraction module is mainly composed of the fifth cross-channel convolution layer, the fifth regularization layer one, the fifth convolution layer one, the fifth regularization layer two, the fifth convolution layer two, the fifth regularization layer three, the fifth deep splicing module layer one, the fifth maximum pooling layer one, the fifth deep splicing module layer two and the fifth maximum pooling layer two connected in series in sequence; the classification layer is mainly composed of a fully connected layer.

[0083] The umbrella fault data set obtained in step S1 is input into the first cross-channel convolutional layer of the first feature extraction module, the output end of the first attention mechanism layer of the first feature extraction module is connected to the input end of the second cross-channel convolutional layer of the second feature extraction module, the output end of the second maximum pooling layer of the second feature extraction module is connected to the input end of the third cross-channel convolutional layer of the third feature extraction module, the output end of the third maximum pooling layer of the third feature extraction module is connected to the input end of the fourth cross-channel convolutional layer of the fourth feature extraction module, the output end of the fourth attention mechanism layer of the fourth feature extraction module is connected to the input end of the fifth cross-channel convolutional layer of the fifth feature extraction module, the output end of the fifth maximum pooling layer 2 of the fifth feature extraction module is connected to the input end of the fully connected layer of the classification layer, and the output end of the fully connected layer of the classification layer outputs the classification result.

[0084] The first regularization layer one, the first regularization layer two, the second regularization layer one, the second regularization layer two, the third regularization layer one, the third regularization layer two, the third regularization layer three, the fourth regularization layer one, the fourth regularization layer two, the fourth regularization layer three, the fifth regularization layer one, the fifth regularization layer two and the fifth regularization layer three are all regularization layers, and all use a combination of regularization and ReLU activation function.

[0085] The first convolutional layer, the second convolutional layer, the third convolutional layer 1, the third convolutional layer 2, the fourth convolutional layer 2, the fourth convolutional layer 1, the fifth convolutional layer 1 and the fifth convolutional layer 2 are all 3*3 convolutional layers. The first cross-channel convolutional layer, the second cross-channel convolutional layer, the third cross-channel convolutional layer, the fourth cross-channel convolutional layer and the fifth cross-channel convolutional layer are all 1×1 convolutional layers with a step size of 1 and a padding of 0. This reduces the number of parameters and can achieve cross-channel information fusion, so that the convolutional layer can extract the feature differences between image pairs in more detail, thereby improving the accuracy of model prediction. The convolutional layer is a 3*3 layer.

[0086] like Figure 7 Shown is the structural diagram of the deep stitching module, which uses multiple parallel stitching branches to achieve the final multi-feature fusion.

[0087] The fifth depth splicing module layer one and the fifth depth splicing module layer two both include four parallel splicing branches, and the four parallel splicing branches are respectively a first parallel splicing branch, a second parallel splicing branch, a third parallel splicing branch and a fourth parallel splicing branch. The first parallel splicing branch includes a first 1×1 convolutional layer, the second parallel splicing branch includes a second 1×1 convolutional layer and a second 3×3 convolutional layer connected in series in sequence, the third parallel splicing branch includes a third 1×1 convolutional layer and a third 5×5 convolutional layer, and the fourth parallel splicing branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolutional layer.

[0088] The input end of the first 1×1 convolutional layer, the input end of the second 1×1 convolutional layer, the input end of the third 1×1 convolutional layer, and the input end of the fourth 3×3 pooling layer are connected in parallel as the input end of the fifth depth splicing module layer one or the fifth depth splicing module layer two; the output end of the first 1×1 convolutional layer, the output end of the second 3×3 convolutional layer, the output end of the third 5×5 convolutional layer, and the output end of the fourth 1×1 convolutional layer are connected in parallel as the output end of the fifth depth splicing module layer one or the fifth depth splicing module layer two.

[0089] The deep stitching module refers to the diagnosis of complex and non-obvious fault shapes of umbrella structures. After the last 3×3 convolutional layer of the fifth feature extraction module is completed, two layers of deep stitching modules and two layers of maximum pooling layers with a window size of 2 are introduced to enable the network to extract more detailed umbrella structure information and reduce the size of the image input fully connected layer to reduce the number of parameters of the fully connected layer and improve the calculation speed and accuracy of the model.

[0090] like Figure 6 As shown, the attention mechanism layer is set according to the following formula:

[0091] The attention mechanism layer is set according to the following formula:

[0092]

[0093] in and are the mean and variance of all neurons, is the minimum energy function, t is the target neuron of a single channel in the input feature, x i is the i-th neuron in a single channel of the input feature except the target neuron, i is the spatial dimension index, M is the number of neurons on the channel, λ is an adjustable parameter, and the present invention sets it to λ = 0.0001. H, W and C in the figure are height, width and channel respectively.

[0094] The local self-similarity of the feature map calculated by this attention mechanism layer is used to generate attention weights, focus on the key areas of umbrella fault diagnosis, suppress interference from irrelevant factors such as complex background in the image, and enhance the neural network's ability to learn the difference features of the input image.

[0095] S3. Perform a cyclic full opening test on the sun-rain umbrella to be tested, and perform image pairing processing on the real-time collected images of the sun-rain umbrella to be tested after each full opening, and input them into the trained fault diagnosis model, so as to continuously obtain fault diagnosis results, and judge in real time whether the fault diagnosis results meet the requirements for stopping detection, so as to obtain whether the number of full opening times of the sun-rain umbrella to be tested meets the quality requirements.

[0096] S31, collecting an image of the sun-rain umbrella to be tested after it is fully opened for the first time as a basic image, and initializing the number of times the sun-rain umbrella to be tested is fully opened.

[0097] S32, collecting an image of the tested umbrella after it is fully opened next time as a matching image.

[0098] S33, performing image pairing processing on the basic image and the matching image to obtain a grayscale image pair to be tested.

[0099] S34, inputting the obtained grayscale image pair to be tested into the trained fault diagnosis model obtained in step S2 to obtain a fault diagnosis result.

[0100] S35, judging whether the fault diagnosis result meets the requirement of stopping detection:

[0101] If the fault diagnosis result is good, the number of times the sun-and-rain umbrella to be tested is fully opened is increased by one, and the obtained number of times the sun-and-rain umbrella to be tested is fully opened is compared with the preset threshold of the number of times of fully opening: if the total number of times the sun-and-rain umbrella to be tested is fully opened is equal to the threshold of the number of times of fully opening, the detection is stopped, and the number of times the sun-and-rain umbrella to be tested is fully opened meets the quality requirements; if the number of times the sun-and-rain umbrella to be tested is fully opened is less than the threshold of the number of times of fully opening, the above steps S32-S34 are repeated;

[0102] If the fault diagnosis result is damage, the test is stopped, and the number of times the umbrella is fully opened does not meet the quality requirements.

[0103] The present invention inputs the traditional single image data into the traditional neural network, and the comparison results are shown in Table 1. The image pairing processing method of the present invention is used to process the data and then input it into various neural networks, and the comparison results are shown in Table 2. The number of iterations is 30.

[0104] Table 1

[0105]

[0106] Table 2

[0107]

[0108] It can be seen from the table that the image pairing processing method proposed in the present invention can greatly improve the performance of various models, and the performance of the fault diagnosis model proposed in the present invention is significantly better than the traditional conventional neural network model.

[0109] Compared with the training results of neural network models such as VGG16, the fault diagnosis model of the present invention has a small difference in accuracy and loss value between the training set and the validation set, the accuracy and loss curve of the validation set are better than those of other comparison models, and the overfitting phenomenon is significantly improved.

[0110] In the embodiment, the parameter amount of the VGG16 neural network model is 1.21×10 8 , the parameter of the present invention is 3.01×10 7 , greatly reducing energy consumption.

[0111] Table 3

[0112]

[0113] In this embodiment, an ablation experiment was also conducted on the present invention. As shown in Table 3, Model 1 is a model in which all attention layers are removed and all 1*1 convolution layers are replaced by 3*3 convolution layers compared to the present invention; Model 2 is a model in which all 1*1 convolution layers are replaced by 3*3 convolution layers compared to the present invention; and Model 3 is a model in which all attention layers are removed compared to the present invention. According to this ablation experiment, it can be clearly determined that the performance of the network model structure designed by the model of the present invention is very excellent.

[0114] The present invention applies machine vision technology to the fault diagnosis task of the umbrella, without the need for human participation in decision-making, which greatly improves the detection efficiency and accuracy. The detection method of the present invention can diagnose in real time whether the umbrella is faulty during the fatigue test, and uses the same set of algorithms for the binocular camera, which improves the efficiency and accuracy of fault diagnosis. The detection method of the present invention has certain scalability. It can collect a large amount of image data of different faults during the subsequent fatigue test of the umbrella, further enrich the defect image data set, and train and strengthen the multi-class detection model to achieve multi-fault classification detection tasks.

Claims

1. A method for diagnosing umbrella faults based on machine vision, characterized in that: The following steps are involved: S1. Use a camera to capture images of a fully opened umbrella, and perform label processing and image pairing processing on the collected images in turn to obtain a dataset of umbrella failures; S2, constructing a fault diagnosis model, inputting the obtained umbrella fault data set into the neural network diagnosis model for training, and obtaining a trained fault diagnosis model; S3. Perform a cyclic full opening test on the sun-rain umbrella to be tested, and perform image pairing processing on the real-time collected images of the sun-rain umbrella to be tested after each full opening, and input them into the trained fault diagnosis model, so as to continuously obtain fault diagnosis results, and judge in real time whether the fault diagnosis results meet the requirements for stopping detection, so as to obtain whether the number of full opening times of the sun-rain umbrella to be tested meets the quality requirements.

2. The method for diagnosing umbrella faults based on machine vision according to claim 1 is characterized in that: The step S1 is specifically as follows: S11, using a camera to capture images of a fully opened umbrella in different states as umbrella images, labeling the umbrella images to obtain good images and damaged images of the umbrella; S12, performing image pairing processing and label processing on each good image of the umbrella and other images in the collected umbrella images in sequence to obtain a plurality of grayscale image pairs; S13, combining a plurality of grayscale image pairs obtained from each good image of the umbrella to obtain a fault data set of the umbrella; S14, using the same method as step S11 to step S13 to obtain a fault data set of several umbrellas; S15. Combine the fault data sets of several umbrellas to obtain a fault data set of an umbrella.

3. The method for diagnosing umbrella faults based on machine vision according to claim 1 is characterized in that: The fault diagnosis model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module and a classification module connected in series in sequence; the first feature extraction module is mainly composed of a first cross-channel convolutional layer, a first regularization layer one, a first convolutional layer, a first regularization layer two, a first maximum pooling layer and a first attention mechanism layer connected in series in sequence; the second feature extraction module is mainly composed of a second cross-channel convolutional layer, a second regularization layer one, a second convolutional layer, a second regularization layer two and a second maximum pooling layer connected in series in sequence; the third feature extraction module is mainly composed of a third cross-channel convolutional layer, a third regularization layer one, a third convolutional layer one, a third regularization layer two, a third convolutional layer two, a third regularization layer three and a third maximum pooling layer connected in series in sequence; the fourth feature extraction module is mainly composed of a fourth cross-channel convolutional layer, a fourth regularization layer one, a fourth convolutional layer one, a fourth regularization layer two, a fourth convolutional layer two, a fourth regularization layer three and a third maximum pooling layer connected in series in sequence.

3. The fourth maximum pooling layer and the fourth attention mechanism layer are composed; the fifth feature extraction module is mainly composed of the fifth cross-channel convolutional layer, the fifth regularization layer one, the fifth convolutional layer one, the fifth regularization layer two, the fifth convolutional layer two, the fifth regularization layer three, the fifth depth splicing module layer one, the fifth maximum pooling layer one, the fifth depth splicing module layer two and the fifth maximum pooling layer two which are connected in series in sequence; the classification layer is mainly composed of a fully connected layer; the umbrella fault data set obtained in step S1 is input into the first cross-channel convolutional layer of the first feature extraction module, the output end of the first attention mechanism layer is connected to the input end of the second cross-channel convolutional layer, the output end of the second maximum pooling layer is connected to the input end of the third cross-channel convolutional layer, the output end of the third maximum pooling layer is connected to the input end of the fourth cross-channel convolutional layer, the output end of the fourth attention mechanism layer is connected to the input end of the fifth cross-channel convolutional layer, the output end of the fifth maximum pooling layer two is connected to the input end of the fully connected layer, and the output end of the fully connected layer outputs the classification result.

4. The method for diagnosing umbrella faults based on machine vision according to claim 3 is characterized in that: The first cross-channel convolutional layer, the second cross-channel convolutional layer, the third cross-channel convolutional layer, the fourth cross-channel convolutional layer and the fifth cross-channel convolutional layer are all 1×1 convolutional layers with a step size of 1 and a padding of 0; the fifth depth splicing module layer 1 and the fifth depth splicing module layer 2 each include four parallel splicing branches, the first parallel splicing branch includes a first 1×1 convolutional layer, the second parallel splicing branch includes a second 1×1 convolutional layer and a second 3×3 convolutional layer connected in series in sequence, the third parallel splicing branch includes a third 1×1 convolutional layer and a third 5×5 convolutional layer, and the fourth parallel splicing branch includes a third 1×1 convolutional layer and a third 5×5 convolutional layer. The path includes a fourth 3×3 pooling layer and a fourth 1×1 convolutional layer; the input end of the first 1×1 convolutional layer, the input end of the second 1×1 convolutional layer, the input end of the third 1×1 convolutional layer, and the input end of the fourth 3×3 pooling layer are connected in parallel as the input end of the fifth depth splicing module layer one or the fifth depth splicing module layer two; the output end of the first 1×1 convolutional layer, the output end of the second 3×3 convolutional layer, the output end of the third 5×5 convolutional layer, and the output end of the fourth 1×1 convolutional layer are connected in parallel as the output end of the fifth depth splicing module layer one or the fifth depth splicing module layer two.

5. The method for diagnosing umbrella faults based on machine vision according to claim 3 is characterized in that: The attention mechanism layer uses a convolutional neural network with neurons as calculation factors, and the attention mechanism layer is set according to the following formula: in, and are the mean and variance of all neurons, is the minimum energy function, t is the target neuron of a single channel in the input feature, x i is the i-th neuron other than the target neuron in a single channel of the input feature, i is the spatial dimension index, M is the number of neurons on the channel, and λ is an adjustable parameter.

6. The method for diagnosing umbrella faults based on machine vision according to claim 1 is characterized in that: The step S3 is specifically as follows: S31, collecting an image of the sun-rain umbrella to be tested after it is fully opened for the first time as a basic image, and initializing the number of times the sun-rain umbrella to be tested is fully opened; S32, collecting an image of the tested umbrella after it is fully opened next time as a matching image; S33, performing image pairing processing on the basic image and the matching image to obtain a grayscale image pair to be tested; S34, inputting the obtained grayscale image pair to be tested into the trained fault diagnosis model obtained in step S2 to obtain a fault diagnosis result; S35, judging whether the fault diagnosis result meets the requirement of stopping detection: If the fault diagnosis result is good, the number of times the sun-and-rain umbrella to be tested is fully opened is increased by one, and the obtained number of times the sun-and-rain umbrella to be tested is fully opened is compared with the preset threshold of the number of times of fully opening: if the total number of times the sun-and-rain umbrella to be tested is fully opened is equal to the threshold of the number of times of fully opening, the detection is stopped, and the number of times the sun-and-rain umbrella to be tested is fully opened meets the quality requirements; if the number of times the sun-and-rain umbrella to be tested is fully opened is less than the threshold of the number of times of fully opening, the above steps S32-S34 are repeated; If the fault diagnosis result is damage, the test is stopped, and the number of times the umbrella is fully opened does not meet the quality requirements.

7. The method for diagnosing umbrella faults based on machine vision according to claim 6 is characterized in that: The image pairing method in step S12 or step S33 is specifically as follows: grayscale processing is performed on the two images respectively to obtain two grayscale images, and then the two grayscale images are fused on a single channel to obtain a grayscale image pair.

8. An umbrella fault diagnosis system used in the umbrella fault diagnosis method based on machine vision as claimed in any one of claims 1 to 7, characterized in that: It includes a camera, a test device, a PLC control system and a host computer; the sunshade umbrella to be tested is clamped on the test device, the camera is arranged below the sunshade umbrella to be tested, the PLC control system is installed on the test device, the PLC control system is electrically connected to the test device and the host computer respectively, and the host computer is also electrically connected to the camera.

9. The machine vision-based umbrella fault diagnosis system according to claim 8, characterized in that: The test device includes an opening and closing mechanism and a pressure sensor, and the PLC control module includes an opening and closing control module and a pressure measuring module; the umbrella to be tested is clamped on the opening and closing mechanism, the input end and the output end of the opening and closing control module are electrically connected to the host computer and the opening and closing mechanism respectively, and the input end and the output end of the pressure measuring module are electrically connected to the pressure sensor and the host computer respectively.

10. The machine vision-based umbrella fault diagnosis system according to claim 9, characterized in that: The host computer includes an image acquisition module, a result evaluation module and a fault diagnosis module with the fault diagnosis model installed inside; the input end and the output end of the image acquisition module are respectively connected to the camera output end and the input end of the fault diagnosis module, the input end and the output end of the result evaluation module are respectively communicatively connected to the output end of the fault diagnosis module and the input end of the opening and closing control module, the input end of the result evaluation module is also communicatively connected to the output end of the pressure measurement module, and the output end of the result evaluation module is also electrically connected to the control end of the camera.

Images