A machine vision-based umbrella fault diagnosis method and system

Abstract

The application discloses a kind of sunny umbrella fault diagnosis method and system based on machine vision.The method of the application includes first using camera to shoot collection and processing to obtain sunny umbrella fault dataset, then construct fault diagnosis model, and input sunny umbrella fault dataset into fault diagnosis model to train, obtain trained fault diagnosis model, finally the image of the umbrella to be measured sunny umbrella is input to the trained fault diagnosis model, and finally whether the umbrella opening times of the umbrella to be measured sunny umbrella meets quality requirements.The system of the application includes camera, test device, PLC control system and host computer.The image pairing image method and the fault diagnosis model constructed by the application achieve good accuracy in sunny umbrella fault diagnosis, and save more energy consumption and manpower consumption.

Description

Technical Field

[0001] This invention relates to the technical field of fatigue testing for umbrellas, and more particularly to a method and system for diagnosing umbrella faults based on machine vision. Background Technology

[0002] During production and use, umbrellas undergo repeated opening and closing and are subjected to various environmental conditions. According to national standards and user needs, fatigue testing of umbrellas requires high precision, high efficiency, and the ability to simulate various real-world usage scenarios. Traditional methods for diagnosing umbrella fatigue testing rely on manual judgment, which is inefficient, prone to false positives and false negatives, and difficult to standardize. Machine vision technology, however, can provide more accurate and efficient fault detection. Currently, domestic umbrella fatigue testing systems lack automatic judgment of relevant mechanical indicators, cannot generate standard test reports and quality traceability documents, and cannot achieve intelligent real-time detection of umbrella faults. Therefore, developing a machine vision-based umbrella fatigue testing system helps to fill the gaps in national standard technical testing requirements and existing testing equipment.

[0003] Currently, fault diagnosis during umbrella fatigue testing mainly relies on traditional infrared sensor monitoring and manual diagnosis. Infrared sensor monitoring requires a sensor count matching the umbrella ribs and can only detect rib fractures. This method is not only expensive in terms of equipment and has high requirements for umbrella installation, but also has poor adaptability. Manual diagnosis struggles to detect subtle faults during fatigue testing, and since umbrellas typically open and close 400 or 600 times, it is not only labor-intensive but also prone to misjudgments and omissions due to subjective human factors.

[0004] While machine vision and deep learning technologies and methods have been widely applied in some areas of industrial inspection, existing classical neural network models and image data input methods struggle to identify fault shapes without significant umbrella shape changes in complex background testing environments. This makes it difficult to meet the demand for rapid and accurate fault diagnosis of umbrellas during fatigue testing. Therefore, providing a machine vision-based fault diagnosis method for umbrellas has become an urgent technical problem to be solved in this field. Summary of the Invention

[0005] To address the problems and deficiencies in the background technology, this invention provides a machine vision-based method and system for diagnosing umbrella faults. It uses machine learning algorithms to perform real-time fault diagnosis of umbrellas during fatigue testing, thereby solving the problems of data management and intelligent diagnosis in the current field of umbrella fatigue testing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] I. A machine vision-based method for diagnosing umbrella malfunctions, comprising the following steps:

[0008] S1. Use a camera to capture images of umbrellas fully opened. Then, perform labeling and image pairing processing on the captured images to obtain an umbrella fault dataset.

[0009] S2. Construct a fault diagnosis model. Input the obtained umbrella fault dataset into the neural network diagnosis model for training to obtain a trained fault diagnosis model.

[0010] S3. Perform cyclic full-opening detection on the umbrella under test, and input the images of the umbrella after each full opening, which are collected in real time, into the trained fault diagnosis model after image pairing processing. Continuously obtain fault diagnosis results, and judge in real time whether the fault diagnosis results meet the stop detection requirements, so as to determine whether the number of times the umbrella under test is fully opened meets the quality requirements.

[0011] S11. Use a camera to capture images of an umbrella in different states after it is fully opened as umbrella images. Label the umbrella images to obtain good and damaged images of the umbrella.

[0012] S12. For each good image of the umbrella, perform image pairing and labeling processing with other images in the acquired umbrella images in turn to obtain several grayscale image pairs.

[0013] The labeling process after image pairing is as follows: grayscale image pairs obtained by pairing good images with good images are used as positive sample image pairs, and grayscale image pairs obtained by pairing good images with damaged images are used as negative sample image pairs.

[0014] S13. Combine several grayscale image pairs obtained from each good image of the umbrella to obtain a fault dataset of the umbrella.

[0015] S14. Using the same method as steps S11-S13, we obtain a fault dataset of several umbrellas of different types.

[0016] The different types of umbrellas include manually opening and retractable umbrellas, self-opening straight umbrellas, self-opening and self-retracting umbrellas, and manually opening straight umbrellas.

[0017] S15. Combine the fault datasets of several umbrellas of different types to obtain the umbrella fault dataset.

[0018] The fault diagnosis model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, and a classification module, connected in series. The first feature extraction module mainly consists of a first cross-channel convolutional layer, a first regularization layer, a first convolutional layer, a first regularization layer, a second regularization layer, a first max pooling layer, and a first attention mechanism layer, connected in series. The second feature extraction module mainly consists of a second cross-channel convolutional layer, a second regularization layer, a second convolutional layer, a second regularization layer, and a second max pooling layer, connected in series. The third feature extraction module mainly consists of a third cross-channel convolutional layer, a third regularization layer, a third convolutional layer, a third regularization layer, a second regularization layer, and a second max pooling layer, connected in series. The first feature extraction module consists of a second regularization layer, a second convolutional layer, a third regularization layer, and a third max pooling layer, all connected in series. The second feature extraction module consists of a fourth cross-channel convolutional layer, a first regularization layer, a first convolutional layer, a second regularization layer, a second convolutional layer, a third regularization layer, a fourth max pooling layer, and a fourth attention mechanism layer, all connected in series. The third feature extraction module consists of a fifth cross-channel convolutional layer, a first regularization layer, a first convolutional layer, a second regularization layer, a second convolutional layer, a third regularization layer, a first depthwise concatenation module, a first max pooling layer, a second depthwise concatenation module, and a second max pooling layer, all connected in series. The classification layer consists of a fully connected layer.

[0019] The umbrella fault dataset obtained in step S1 is input into the first cross-channel convolutional layer of the first feature extraction module. The output of the first attention mechanism layer is connected to the input of the second cross-channel convolutional layer. The output of the second max pooling layer is connected to the input of the third cross-channel convolutional layer. The output of the third max pooling layer is connected to the input of the fourth cross-channel convolutional layer. The output of the fourth attention mechanism layer is connected to the input of the fifth cross-channel convolutional layer. The output of the fifth max pooling layer is connected to the input of the fully connected layer. The output of the fully connected layer outputs the classification result.

[0020] The first regularization layer 1, the first regularization layer 2, the second regularization layer 1, the second regularization layer 2, the third regularization layer 1, the third regularization layer 2, the third regularization layer 3, the fourth regularization layer 1, the fourth regularization layer 2, the fourth regularization layer 3, the fifth regularization layer 1, the fifth regularization layer 2, and the fifth regularization layer 3 are all regularization layers, and all of them are combinations of regularization and ReLU activation functions.

[0021] The first convolutional layer, the second convolutional layer, the third convolutional layer 1, the third convolutional layer 2, the fourth convolutional layer 2, the fourth convolutional layer 1, the fifth convolutional layer 1, and the fifth convolutional layer 2 are all 3*3 convolutional layers.

[0022] The first, second, third, fourth, and fifth cross-channel convolutional layers are all 1×1 convolutional layers with a stride of 1 and padding of 0; the convolutional layers are 3*3 layers. Both the fifth depth stitching module layer one and the fifth depth stitching module layer two include four parallel stitching branches. The first parallel stitching branch includes a first 1×1 convolutional layer, the second parallel stitching branch includes a second 1×1 convolutional layer and a second 3×3 convolutional layer connected in series, the third parallel stitching branch includes a third 1×1 convolutional layer and a third 5×5 convolutional layer, and the fourth parallel stitching branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolutional layer. The inputs of the first 1×1 convolutional layer, the second 1×1 convolutional layer, the third 1×1 convolutional layer, and the fourth 3×3 pooling layer are connected in parallel as the inputs of either the fifth depth stitching module layer one or the fifth depth stitching module layer two. The outputs of the first 1×1 convolutional layer, the second 3×3 convolutional layer, the third 5×5 convolutional layer, and the fourth 1×1 convolutional layer are connected in parallel as the outputs of either the fifth depth stitching module layer one or the fifth depth stitching module layer two.

[0023] The attention mechanism layer uses a convolutional neural network with neurons as computational factors, and the attention mechanism layer is set according to the following formula:

[0024]

[0025]

[0026] in, and Let the mean and variance of all neurons be denoted as . Let x be the minimum energy function, t be the target neuron in a single channel of the input features, and x be the minimum energy function. i For the i-th other neuron besides the target neuron in a single channel of the input feature, i is the spatial dimension index, M is the number of neurons in that channel, and λ is an adjustable parameter, which is set to λ = 0.0001 in this invention.

[0027] S31. Collect the image of the umbrella under test after it is fully opened for the first time as the base image, and initialize the number of times the umbrella under test is fully opened.

[0028] S32. Collect the image of the umbrella under test after it has been fully opened once as the matching image;

[0029] S33. Perform image pairing processing on the base image and the matching image to obtain the grayscale image pair to be tested;

[0030] S34. Input the obtained grayscale image to be tested into the trained fault diagnosis model obtained in step S2 to obtain the fault diagnosis result.

[0031] S35. Determine whether the fault diagnosis results meet the requirements for stopping the test:

[0032] If the fault diagnosis result is good, the number of times the umbrella under test is fully opened is incremented by one. The obtained number of times the umbrella under test is fully opened is compared with the preset threshold for the number of times the umbrella is fully opened: if the total number of times the umbrella under test is fully opened is equal to the threshold for the number of times the umbrella is fully opened, the test is stopped and the number of times the umbrella under test is fully opened meets the quality requirements; if the number of times the umbrella under test is fully opened is less than the threshold for the number of times the umbrella is fully opened, the above steps S32-S34 are repeated.

[0033] If the fault diagnosis result indicates damage, the testing will be stopped, and the number of times the umbrella under test can be fully opened does not meet the quality requirements.

[0034] The image pairing process in step S12 or step S33 specifically involves: performing grayscale processing on the two images respectively to obtain two grayscale images, and then performing fusion processing on the two grayscale images in a single-channel dimension to obtain a grayscale image pair. The grayscale image pair obtained by the fusion processing is a dual-channel image.

[0035] II. A Machine Vision-Based Umbrella Fault Diagnosis System

[0036] The system includes a camera, a testing device, a PLC control system, and a host computer. The umbrella to be tested is clamped on the testing device, and the camera is positioned below the umbrella to collect and detect images of the inner surface of the umbrella. The PLC control system is installed on the testing device and is electrically connected to both the testing device and the host computer. The host computer is also electrically connected to the camera.

[0037] The testing device includes an opening and closing mechanism and a pressure sensor. The PLC control module includes an opening and closing control module and a pressure measurement module. The umbrella to be tested is clamped on the opening and closing mechanism. The input and output terminals of the opening and closing control module are electrically connected to the host computer and the opening and closing mechanism, respectively. The input and output terminals of the pressure measurement module are electrically connected to the pressure sensor and the host computer, respectively. The pressure sensor is used to measure the opening and closing button force and the umbrella shaft compression force of the umbrella to be tested.

[0038] The host computer includes an image acquisition module, a result evaluation module, and a fault diagnosis module that internally carries the fault diagnosis model. The input and output terminals of the image acquisition module are respectively connected to the output terminal of the camera and the input terminal of the fault diagnosis module. The input and output terminals of the result evaluation module are respectively communicatively connected to the output terminal of the fault diagnosis module and the input terminal of the opening and closing control module. The input terminal of the result evaluation module is also communicatively connected to the output terminal of the pressure measurement module. The output terminal of the result evaluation module is also electrically connected to the control terminal of the camera.

[0039] The innovation of this invention lies in the use of image pairing processing in both the creation of the umbrella fault dataset and the fault diagnosis of the umbrella under test. This approach allows the fault diagnosis model to extract more feature association information compared to the traditional RGB three-channel single color image network input method.

[0040] The innovation of this invention lies in the first use of machine vision to diagnose the faults of umbrellas. The fault diagnosis model adopted achieves higher accuracy than traditional common network models, while still meeting the requirements of real-time performance.

[0041] The beneficial effects of this invention are:

[0042] 1. In the creation of the umbrella fault dataset, the present invention uses image pairing processing to obtain dual-channel grayscale image pairs from the collected images. Compared with the traditional RGB three-channel single color image network input method, its positive and negative sample images have better feature association information. Moreover, compared with the traditional RGB three-channel single color image network input method, its accuracy in the fault diagnosis model of the present invention is improved by 26%, reaching 99.93%.

[0043] 2. The fault diagnosis model constructed in this invention achieves high accuracy while maintaining low energy consumption, and the performance of the network model structure designed in this invention is superior to common networks such as VGG16 and ResNet18.

[0044] 3. The umbrella fault diagnosis system of the present invention is the first to use machine vision to diagnose umbrella faults, achieving a higher accuracy rate than traditional methods, meeting real-time requirements while saving more energy and manpower. Attached Figure Description

[0045] Figure 1 This is a block diagram of the overall module of the umbrella fault diagnosis system.

[0046] Figure 2 A flowchart for diagnosing malfunctions of umbrellas;

[0047] Figure 3 These are positive sample image pairs from the umbrella malfunction dataset.

[0048] Figure 4 For the negative sample image pairs in the umbrella malfunction dataset;

[0049] Figure 5 This is a network structure diagram of the fault diagnosis model;

[0050] Figure 6 This is a structural diagram of the attention mechanism layer;

[0051] Figure 7 This is a structural diagram of the deep stitching module. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] like Figure 1 As shown, the umbrella fault diagnosis system of the present invention includes a camera, a testing device, a PLC control system and a host computer.

[0054] The umbrella under test is mounted on the testing device, and a camera is positioned below it to capture and inspect images of the umbrella's inner surface. A PLC control system is installed on the testing device and electrically connected to both the testing device and a host computer, which is also electrically connected to the camera. The camera captures images of the umbrella and transmits them to the host computer. The host computer uses its built-in fault diagnosis model to sequentially detect, process, and evaluate the images. The host computer then transmits the evaluation results back to the PLC control system. Based on the evaluation results, the PLC control system controls the testing device to determine whether the umbrella should be opened and closed again. The testing device measures the pressure of the umbrella and transmits the measured pressure data to the PLC control system. The PLC control system processes the pressure data and transmits it back to the host computer, which analyzes the data to determine whether the umbrella's pressure is within acceptable limits.

[0055] The testing apparatus includes an opening and closing mechanism and a pressure sensor. The PLC control module includes an opening and closing control module and a pressure measurement module. The umbrella under test is clamped on the opening and closing mechanism. The input and output terminals of the opening and closing control module are electrically connected to the host computer and the opening and closing mechanism, respectively. The input and output terminals of the pressure measurement module are electrically connected to the pressure sensor and the host computer, respectively. The pressure sensor is used to measure the opening and closing button force and the umbrella shaft compression force of the umbrella under test.

[0056] The opening and closing control module controls the opening and closing mechanism to open and close the umbrella under test. The pressure sensor transmits the pressure results of the switch button force and umbrella shaft compression force of the umbrella under test to the pressure measurement module. The pressure measurement module processes the pressure data and transmits it to the host computer. The host computer uses its built-in fault diagnosis model to detect and evaluate the image of the umbrella under test after it is fully opened. The host computer transmits the evaluation results to the opening and closing control module. The opening and closing control module controls the opening and closing mechanism to decide whether to perform the next opening and closing process for the umbrella under test based on the evaluation results. The host computer also evaluates the received pressure data to assess whether the switch button and umbrella shaft of the umbrella under test are of acceptable quality.

[0057] The host computer includes an image acquisition module, a result evaluation module, and a fault diagnosis module with an internal fault diagnosis model. The input and output terminals of the image acquisition module are communicatively connected to the output terminal of the camera and the input terminal of the fault diagnosis module, respectively. The input and output terminals of the result evaluation module are communicatively connected to the output terminal of the fault diagnosis module and the input terminal of the opening and closing control module, respectively. The input terminal of the result evaluation module is also communicatively connected to the output terminal of the pressure measurement module, and the output terminal of the result evaluation module is also electrically connected to the control terminal of the camera.

[0058] The camera captures images of the umbrella under test and transmits them to the image acquisition module. The image acquisition module then transmits the images to the fault diagnosis module. The fault diagnosis module uses its built-in fault diagnosis model to diagnose the faults in the umbrella and obtain a fault diagnosis result. The fault diagnosis module transmits the fault diagnosis result to the result evaluation module to determine whether the test stops. The result evaluation module then transmits the result to the opening and closing control module to determine whether to proceed with the next opening and closing operation. The test stops when the fault diagnosis result indicates damage. The result evaluation module determines whether the umbrella meets the quality requirements based on whether the total number of times it has fully opened equals the threshold number of times it has fully opened. The pressure measurement module processes the pressure data and transmits it to the result evaluation module. The result evaluation module also uses the pressure data to determine whether the switch button and umbrella shaft are of acceptable quality, thus determining whether the umbrella is ultimately of acceptable quality. The result evaluation module transmits the results to the camera control terminal based on whether the umbrella meets the quality requirements for the number of times it has fully opened, and then controls the camera to take pictures.

[0059] The test apparatus also includes temperature and humidity sensors and wire-type displacement sensors; the PLC control system also includes temperature and humidity sensing modules and wire-type displacement sensing modules; the host computer also includes information management modules, data storage modules, parameter configuration modules, and real-time data display and analysis modules.

[0060] The temperature and humidity sensor measures the temperature and humidity of the test environment, and then transmits the temperature and humidity to the temperature and humidity sensing module. The temperature and humidity sensing module then transmits the processed data to the real-time data display and analysis module for analysis and display.

[0061] The pull-wire displacement sensor transmits data on whether the umbrella under test is fully open to the pull-wire displacement sensing module. The pull-wire displacement sensing module is responsible for determining whether the umbrella is fully open. If it is fully open, the pull-wire displacement sensing module transmits the fully open signal to the result evaluation module of the host computer. The result evaluation module of the host computer controls the camera to take a picture. If the umbrella is not fully open, the detection stops, and the umbrella under test does not meet the quality requirements.

[0062] The parameter configuration module is responsible for configuring the basic parameters of the umbrella and the parameters of the acquisition camera. The basic umbrella parameter configuration includes setting the sample number, inspection category, umbrella model, umbrella type, sample name, umbrella specifications, threshold for the number of times the umbrella can be fully opened, high and low values ​​for the force required to open / close the umbrella button, and high and low values ​​for the force required to compress the umbrella shaft. These basic umbrella parameters only need to be configured once and are automatically stored in the database. During testing, simply select the umbrella model, and the database will automatically filter relevant data and adjust all parameters for the current test. The acquisition camera parameter configuration includes camera exposure and gain, image scaling parameters, and video storage address.

[0063] The information management module includes a module for managing employee basic information and permissions, and a module for tracing historical test data.

[0064] The real-time data display and analysis module includes the basic parameters of the umbrella, the current number of opening and closing cycles, the force value of the switch button, the force value of compressing the umbrella shaft, the current detection speed, the test date value, the real-time test progress, the real-time temperature and humidity values, the test results, the real-time test log, the image display of the outer surface of the umbrella, and the real-time image display of the inner surface of the umbrella.

[0065] The real-time fault diagnosis of this invention runs in a separate process, unlike the traditional method of acquiring images and inputting them into the fault diagnosis model in the same thread. The method of this invention is to process data in parallel with other modules in a process manner, and to communicate by packaging data with TCP / IP protocol. This ensures that the running speed of other modules is not affected when performing fault inference. In contrast, the traditional method of acquiring images and then performing fault inference in a single thread requires waiting for the fault inference to be completed before taking a picture of the umbrella fully open. Compared with the traditional inference method, the method of this invention greatly improves the speed of model inference and meets the opening and closing speed requirements of the national standard for umbrella fatigue testing.

[0066] like Figure 2 As shown, specific embodiments of the present invention are as follows:

[0067] S1. Use a camera to capture images of umbrellas fully opened. Then, perform labeling and image pairing processing on the captured images to obtain an umbrella fault dataset.

[0068] S11. Use a camera to capture images of an umbrella in different states after it is fully opened as umbrella images. Label the umbrella images to obtain good and damaged images of the umbrella.

[0069] The experiment included capturing images of the umbrella fully open under different conditions, including different angles, different lighting levels, and the same lighting level, without any faults. Then, the umbrella angle was randomly rotated and the lighting level was adjusted to conduct continuous fatigue opening and closing tests on the umbrella until it first malfunctioned. Images of a single fault mode were captured when the umbrella was fully open under different angles, different lighting levels, and the same lighting level. Further fatigue opening and closing tests were conducted on the umbrella to cause multiple faults to occur simultaneously, and images of multiple fault modes were captured when the umbrella was fully open under different angles, different lighting levels, and the same lighting level.

[0070] S12. For each good image of the umbrella, perform image pairing and labeling processing with other images in the acquired umbrella images in turn to obtain several grayscale image pairs.

[0071] The image pairing method is specifically as follows: each of the two images is processed into grayscale to obtain two grayscale images. Then, the two grayscale images are fused in a single-channel dimension to obtain a grayscale image pair. The grayscale image pair obtained by the fusion process is a dual-channel image.

[0072] The labeling process after image pairing is as follows: grayscale image pairs obtained by pairing good images with good images are used as positive sample image pairs, and grayscale image pairs obtained by pairing good images with damaged images are used as negative sample image pairs.

[0073] In practice, the grayscale image pairs are obtained by removing duplicate images and performing image pairing processing.

[0074] S13. Combine several grayscale image pairs obtained from each good image of the umbrella to obtain a fault dataset of the umbrella.

[0075] S14. Using the same method as steps S11-S13, process to obtain fault datasets for several umbrellas of different types.

[0076] Different types of umbrellas include manually opening and retracting umbrellas, self-opening straight umbrellas, self-opening and self-retracting umbrellas, and manually opening straight umbrellas.

[0077] S15. Combine the fault datasets of several umbrellas of different types to obtain the umbrella fault dataset.

[0078] like Figure 3 and Figure 4As shown, in this embodiment, 70 umbrellas were randomly selected from 100 umbrella samples and subjected to continuous fatigue opening and closing tests. A binocular industrial camera was used as the image acquisition device to photograph the fully opened inner surface of the umbrellas, obtaining images of fault-free umbrellas and umbrellas with different fault conditions to construct a basic dataset. The original images were preprocessed. Since the original images obtained using the industrial camera are 3072×2048 pixels, their original size is too large. Directly inputting them into the model for training would result in a huge computational burden and extremely slow training speed. Therefore, the width and height of the original images were first reduced to 680×480 pixels. Using the reduced images for training not only ensures minimal loss of feature information but also significantly improves training speed and reduces detection time costs.

[0079] Because umbrella fabrics come in a variety of colors, it is necessary to convert the reduced image data into grayscale to remove color interference. This allows the neural network to pay more attention to the feature differences between image pairs, which can improve the model's accuracy.

[0080] The final umbrella fault dataset contains 20,026 grayscale image pairs, of which the training set has 12,016 grayscale image pairs, the validation set has 4,005 grayscale image pairs, and the test set has 4,005 grayscale image pairs. The proportion of faulty and non-faulty image pairs in the training set is the same.

[0081] S2. Construct a fault diagnosis model. Input the obtained umbrella fault dataset into the neural network diagnosis model for training to obtain a trained fault diagnosis model.

[0082] like Figure 5As shown, the fault diagnosis model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, and a classification module, all connected in series. The first feature extraction module mainly consists of a first cross-channel convolutional layer, a first regularization layer 1, a first convolutional layer, a first regularization layer 2, a first max pooling layer, and a first attention mechanism layer, all connected in series. The second feature extraction module mainly consists of a second cross-channel convolutional layer, a second regularization layer 1, a second convolutional layer, a second regularization layer 2, and a second max pooling layer, all connected in series. The third feature extraction module mainly consists of a third cross-channel convolutional layer, a third regularization layer 1, a third convolutional layer 1, a third regularization layer 2, a third convolutional layer 2, a third regularization layer 3, and a third max pooling layer, all connected in series. The fourth feature extraction module consists of a first cross-channel convolutional layer, a second ... regularization layer 2, a third regularization layer 2, a third regularization layer 3, and a fifth feature extraction module, all connected in series. The fifth feature extraction module consists of a first feature extraction module, a second regularization layer 1, a third regularization layer 2, a third regularization layer 3, and a fifth feature extraction module, all connected in series. The fifth feature extraction module consists of a first feature extraction module, a second regularization layer 1, a third regularization layer 2, a third regularization layer 2, a third regularization layer 3, and a fifth feature extraction module, all connected in series. The fifth feature extraction module consists of a first feature extraction module, a second regularization layer 1, a third regularization layer 2, a third regularization layer 3 The feature extraction module mainly consists of a fourth cross-channel convolutional layer, a fourth regularization layer 1, a fourth convolutional layer 1, a fourth regularization layer 2, a fourth convolutional layer 2, a fourth regularization layer 3, a fourth max pooling layer, and a fourth attention mechanism layer, all connected in series. The fifth feature extraction module mainly consists of a fifth cross-channel convolutional layer, a fifth regularization layer 1, a fifth convolutional layer 1, a fifth regularization layer 2, a fifth convolutional layer 2, a fifth regularization layer 3, a fifth depth concatenation module layer 1, a fifth max pooling layer 1, a fifth depth concatenation module layer 2, and a fifth max pooling layer 2, all connected in series. The classification layer mainly consists of a fully connected layer.

[0083] The umbrella fault dataset obtained in step S1 is input into the first cross-channel convolutional layer of the first feature extraction module. The output of the first attention mechanism layer of the first feature extraction module is connected to the input of the second cross-channel convolutional layer of the second feature extraction module. The output of the second max pooling layer of the second feature extraction module is connected to the input of the third cross-channel convolutional layer of the third feature extraction module. The output of the third max pooling layer of the third feature extraction module is connected to the input of the fourth cross-channel convolutional layer of the fourth feature extraction module. The output of the fourth attention mechanism layer of the fourth feature extraction module is connected to the input of the fifth cross-channel convolutional layer of the fifth feature extraction module. The output of the fifth max pooling layer of the fifth feature extraction module is connected to the input of the fully connected layer of the classification layer. The output of the fully connected layer of the classification layer outputs the classification result.

[0084] The first regularization layer 1, the first regularization layer 2, the second regularization layer 1, the second regularization layer 2, the third regularization layer 1, the third regularization layer 2, the third regularization layer 3, the fourth regularization layer 1, the fourth regularization layer 2, the fourth regularization layer 3, the fifth regularization layer 1, the fifth regularization layer 2, and the fifth regularization layer 3 are all regularization layers, and all of them are combinations of regularization and ReLU activation functions.

[0085] The first, second, third (first), third (second), fourth (second), fourth (first), fifth (first), and fifth (second) convolutional layers are all 3x3 convolutional layers. The first, second, third, fourth, and fifth cross-channel convolutional layers are all 1x1 convolutional layers with a stride of 1 and zero padding. This reduces the number of parameters, enables cross-channel information fusion, and allows the convolutional layers to extract feature differences between image pairs more thoroughly, improving the accuracy of model prediction. The convolutional layers are 3x3 layers.

[0086] like Figure 7 The diagram shows the structure of the deep stitching module, which utilizes multiple parallel stitching branches to achieve the final multi-feature fusion.

[0087] Both the fifth depth stitching module layer 1 and the fifth depth stitching module layer 2 include four parallel stitching branches. The four parallel stitching branches are the first parallel stitching branch, the second parallel stitching branch, the third parallel stitching branch, and the fourth parallel stitching branch. The first parallel stitching branch includes a first 1×1 convolutional layer. The second parallel stitching branch includes a second 1×1 convolutional layer and a second 3×3 convolutional layer connected in series. The third parallel stitching branch includes a third 1×1 convolutional layer and a third 5×5 convolutional layer. The fourth parallel stitching branch includes a fourth 3×3 pooling layer and a fourth 1×1 convolutional layer.

[0088] The inputs of the first 1×1 convolutional layer, the second 1×1 convolutional layer, the third 1×1 convolutional layer, and the fourth 3×3 pooling layer are connected in parallel as the inputs of either the first or second layer of the fifth depth stitching module; the outputs of the first 1×1 convolutional layer, the second 3×3 convolutional layer, the third 5×5 convolutional layer, and the fourth 1×1 convolutional layer are connected in parallel as the outputs of either the first or second layer of the fifth depth stitching module.

[0089] The depth stitching module is designed for diagnosing complex and subtle fault shapes in umbrella structures. After the last 3×3 convolutional layer of the fifth feature extraction module is completed, two depth stitching modules and two max pooling layers with a window size of 2 are introduced to enable the network to extract more detailed umbrella structure information, reduce the size of the fully connected layer of the image input, thereby reducing the number of parameters in the fully connected layer and improving the computation speed and accuracy of the model.

[0090] like Figure 6 As shown, the attention mechanism layer is set according to the following formula:

[0091] The attention mechanism layer is configured according to the following formula:

[0092]

[0093] in and Let the mean and variance of all neurons be denoted as . Let x be the minimum energy function, t be the target neuron in a single channel of the input features, and x be the minimum energy function. i The input feature is the i-th neuron other than the target neuron in a single channel, where i is the spatial dimension index, M is the number of neurons in that channel, and λ is an adjustable parameter, set to λ = 0.0001 in this invention. In the figure, H, W, and C represent height, width, and channel, respectively.

[0094] By leveraging the local self-similarity of the computational feature map in this attention mechanism layer to generate attention weights, the key areas for umbrella fault diagnosis are focused, interference from irrelevant factors such as complex backgrounds in the image is suppressed, and the neural network's ability to learn differential features from the input image is enhanced.

[0095] S3. Perform cyclic full-opening detection on the umbrella under test, and input the images of the umbrella after each full opening, which are collected in real time, into the trained fault diagnosis model after image pairing processing. Continuously obtain fault diagnosis results, and judge in real time whether the fault diagnosis results meet the stop detection requirements, so as to determine whether the number of times the umbrella under test is fully opened meets the quality requirements.

[0096] S31. Collect the image of the umbrella to be tested after it is fully opened for the first time as the base image, and initialize the number of times the umbrella to be tested is fully opened.

[0097] S32. Collect the image of the umbrella under test after it has been fully opened once as the matching image.

[0098] S33. Perform image pairing processing on the base image and the matching image to obtain the grayscale image pair to be tested.

[0099] S34. Input the obtained grayscale image to be tested into the trained fault diagnosis model obtained in step S2 to obtain the fault diagnosis result.

[0100] S35. Determine whether the fault diagnosis results meet the requirements for stopping the test:

[0101] If the fault diagnosis result is good, the number of times the umbrella under test is fully opened is incremented by one. The obtained number of times the umbrella under test is fully opened is compared with the preset threshold for the number of times the umbrella is fully opened: if the total number of times the umbrella under test is fully opened is equal to the threshold for the number of times the umbrella is fully opened, the test is stopped and the number of times the umbrella under test is fully opened meets the quality requirements; if the number of times the umbrella under test is fully opened is less than the threshold for the number of times the umbrella is fully opened, the above steps S32-S34 are repeated.

[0102] If the fault diagnosis result indicates damage, the testing will be stopped, and the number of times the umbrella under test can be fully opened does not meet the quality requirements.

[0103] This invention inputs traditional single-image data into a traditional neural network, and the comparison results are shown in Table 1. The image pairing processing method of this invention is used to process the data before inputting it into various neural networks, and the comparison results are shown in Table 2. The iteration count for each network is 30.

[0104] Table 1

[0105]

[0106] Table 2

[0107]

[0108] As can be seen from the table, the image pairing processing method proposed in this invention can greatly improve the performance of various models, and the performance of the fault diagnosis model proposed in this invention is significantly better than that of traditional conventional neural network models.

[0109] Compared with the training results of neural network models such as VGG16, the fault diagnosis model of this invention has a very small difference in accuracy and loss value between the training set and the validation set. The accuracy and loss curve of the validation set are better than other comparative models, and the overfitting phenomenon is significantly improved.

[0110] In this embodiment, the VGG16 neural network model has 1.21 × 10^6 parameters. 8 The parameter quantity of this invention is 3.01 × 10⁻⁶. 7 This greatly reduces energy consumption.

[0111] Table 3

[0112]

[0113] In this embodiment, an ablation experiment was also conducted on the present invention. As shown in Table 3, Model 1 is a model compared to the present invention with all attention layers removed and all 1*1 convolutional layers replaced with 3*3 convolutional layers; Model 2 is a model compared to the present invention with all 1*1 convolutional layers replaced with 3*3 convolutional layers; and Model 3 is a model compared to the present invention with all attention layers removed. Based on this ablation experiment, it can be clearly determined that the network model structure designed in this invention has excellent performance.

[0114] This invention applies machine vision technology to umbrella fault diagnosis, eliminating the need for human intervention in decision-making and significantly improving detection efficiency and accuracy. The detection method of this invention can diagnose whether umbrellas have malfunctioned during fatigue testing in real time. It uses the same algorithm for both binocular cameras, further improving the efficiency and accuracy of fault diagnosis. The detection method of this invention has certain scalability; it can further enrich the defect image dataset by collecting a large amount of image data of different faults during subsequent umbrella fatigue testing, and train and strengthen multi-class detection models to achieve multi-fault classification detection tasks.

Claims

1. A machine vision-based fault diagnosis method for a parasol, characterized by, It comprises the following steps: S1, using a camera to take pictures of a fully opened umbrella, and sequentially performing label processing and image pairing processing on the collected images to obtain a good umbrella fault dataset; The step S1 is specifically: S11, using a camera to take pictures of a fully opened umbrella in different states as umbrella images, and performing label processing on the umbrella images to obtain good umbrella images and damaged images; S12, sequentially performing image pairing processing and label processing on each good umbrella image and other images in the collected umbrella images to obtain a plurality of gray image pairs; S13, combining the plurality of gray image pairs obtained from each good umbrella image to obtain a fault dataset of an umbrella; S14, using the same method as steps S11-S13 to obtain a plurality of fault datasets of umbrellas; S15, combining the plurality of fault datasets of umbrellas to obtain a good umbrella fault dataset; S2, constructing a fault diagnosis model, inputting the obtained good umbrella fault dataset into a neural network diagnosis model for training, and obtaining a trained fault diagnosis model; S3, performing a complete opening detection on the to-be-tested umbrella in a loop, inputting the real-time collected images of the to-be-tested umbrella after each complete opening into the trained fault diagnosis model after image pairing processing, continuously obtaining fault diagnosis results, and real-time determining whether the fault diagnosis results meet the stop detection requirement, so as to determine whether the complete opening times of the to-be-tested umbrella meet the quality requirement; The step S3 is specifically: S31, collecting an image of the to-be-tested umbrella after the first complete opening as a base image, and initializing the complete opening times of the to-be-tested umbrella; S32, collecting an image of the to-be-tested umbrella after the next complete opening as a matching image; S33, performing image pairing processing on the base image and the matching image to obtain a to-be-tested gray image pair; S34, inputting the obtained to-be-tested gray image pair into the trained fault diagnosis model obtained in step S2 to obtain a fault diagnosis result; S35, determining whether the fault diagnosis result meets the stop detection requirement: If the fault diagnosis result is good, the complete opening times of the to-be-tested umbrella are incremented by one, and the obtained complete opening times of the to-be-tested umbrella are compared with a preset complete opening times threshold: if the total complete opening times of the to-be-tested umbrella are equal to the complete opening times threshold, the detection is stopped, and the complete opening times of the to-be-tested umbrella meet the quality requirement; if the complete opening times of the to-be-tested umbrella are less than the complete opening times threshold, the steps S32-S34 are repeated; If the fault diagnosis result is damaged, the detection is stopped, and the complete opening times of the to-be-tested umbrella do not meet the quality requirement.

2. The good umbrella fault diagnosis method based on machine vision according to claim 1, characterized in that: The fault diagnosis model comprises a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module and a classification module connected in series; the first feature extraction module mainly comprises a first cross-channel convolutional layer, a first regularization layer one, a first convolutional layer, a first regularization layer two, a first maximum pooling layer and a first attention mechanism layer connected in series; the second feature extraction module mainly comprises a second cross-channel convolutional layer, a second regularization layer one, a second convolutional layer, a second regularization layer two and a second maximum pooling layer connected in series; the third feature extraction module mainly comprises a third cross-channel convolutional layer, a third regularization layer one, a third convolutional layer one, a third regularization layer two, a third convolutional layer two, a third regularization layer three and a third maximum pooling layer connected in series; the fourth feature extraction module mainly comprises a fourth cross-channel convolutional layer, a fourth regularization layer one, a fourth convolutional layer one, a fourth regularization layer two, a fourth convolutional layer two, a fourth regularization layer three, a fourth maximum pooling layer and a fourth attention mechanism layer connected in series; the fifth feature extraction module mainly comprises a fifth cross-channel convolutional layer, a fifth regularization layer one, a fifth convolutional layer one, a fifth regularization layer two, a fifth convolutional layer two, a fifth regularization layer three, a fifth deep concatenation module layer one, a fifth maximum pooling layer one, a fifth deep concatenation module layer two and a fifth maximum pooling layer two connected in series; the classification layer mainly comprises a full connection layer; the obtained sunny and rainy umbrella fault data set in step S1 is input into the first cross-channel convolutional layer of the first feature extraction module, the output end of the first attention mechanism layer is connected with the input end of the second cross-channel convolutional layer, the output end of the second maximum pooling layer is connected with the input end of the third cross-channel convolutional layer, the output end of the third maximum pooling layer is connected with the input end of the fourth cross-channel convolutional layer, the output end of the fourth attention mechanism layer is connected with the input end of the fifth cross-channel convolutional layer, the output end of the fifth maximum pooling layer two is connected with the input end of the full connection layer, and the output end of the full connection layer outputs a classification result.

3. The sunny and rainy umbrella fault diagnosis method based on machine vision according to claim 2, characterized in that: The first cross-channel convolution layer, the second cross-channel convolution layer, the third cross-channel convolution layer, the fourth cross-channel convolution layer and the fifth cross-channel convolution layer are all 1*1 convolution layers with a step of 1 and padding of 0; the first fifth deep concatenation module layer and the second fifth deep concatenation module layer each include four parallel concatenation branches, the first parallel concatenation branch includes a first 1*1 convolution layer, the second parallel concatenation branch includes a second 1*1 convolution layer and a second 3*3 convolution layer connected in series, the third parallel concatenation branch includes a third 1*1 convolution layer and a third 5*5 convolution layer, and the fourth parallel concatenation branch includes a fourth 3*3 pooling layer and a fourth 1*1 convolution layer; the input end of the first 1*1 convolution layer, the input end of the second 1*1 convolution layer, the input end of the third 1*1 convolution layer and the input end of the fourth 3*3 pooling layer are connected in parallel as the input end of the first fifth deep concatenation module layer or the second fifth deep concatenation module layer; and the output end of the first 1*1 convolution layer, the output end of the second 3*3 convolution layer, the output end of the third 5*5 convolution layer and the output end of the fourth 1*1 convolution layer are connected in parallel as the output end of the first fifth deep concatenation module layer or the second fifth deep concatenation module layer.

4. The method according to claim 3, wherein the image pairing method in the step S12 or the step S33 is specifically as follows: two images are respectively subjected to gray processing to obtain two gray images, and then the two gray images are subjected to fusion processing on a single channel to obtain a pair of gray images. The camera, the test device, the PLC control system and the host computer; the test umbrella to be tested is clamped on the test device, the camera is arranged below the test umbrella to be tested, the PLC control system is installed on the test device, the PLC control system is electrically connected with the test device and the host computer respectively, and the host computer is also electrically connected with the camera. wherein, and are the mean and variance for all neurons, is the minimum energy function, t is the target neuron for a single channel in the input feature, x i is the i-th other neuron except the target neuron for a single channel in the input feature, i is the spatial dimension index, M is the number of neurons on this channel, and λ is a tunable parameter.

5. The machine vision-based umbrella malfunction diagnosis method according to claim 1, characterized in that: The test device includes an opening and closing mechanism and a pressure sensor, and the PLC control module includes an opening and closing control module and a pressure measurement module; the test umbrella to be tested is clamped on the opening and closing mechanism, the input end and the output end of the opening and closing control module are electrically connected with the host computer and the opening and closing mechanism respectively, and the input end and the output end of the pressure measurement module are electrically connected with the pressure sensor and the host computer respectively.

6. A system for diagnosing faults of an umbrella based on machine vision according to any one of claims 1 to 5, characterized in that: The host computer includes an image acquisition module, a result evaluation module and a fault diagnosis module in which the fault diagnosis model is embedded; the input end and the output end of the image acquisition module are connected with the output end of the camera and the input end of the fault diagnosis module respectively, the input end and the output end of the result evaluation module are communicatively connected with the output end of the fault diagnosis module and the input end of the opening and closing control module respectively, the input end of the result evaluation module is also communicatively connected with the output end of the pressure measurement module, and the output end of the result evaluation module is also electrically connected with the control end of the camera.

7. The machine vision-based umbrella fault diagnosis system according to claim 6, wherein: ​ 8.The machine vision-based umbrella fault diagnosis system according to claim 7, wherein: ​

Images