A rust spot area expansion monitoring method and system combined with a classification network

Abstract

The present application relates to the technical field of neural network model application, in particular to a rust spot area expansion monitoring method and system combined with a classification network, an image to be detected and a reference image are acquired, the reference image is an image of an initial state of a shooting area of the image to be detected; the image to be detected and the reference image are analyzed based on a neural network model, the image to be detected with changes is determined as a target image, and a changed area is marked on the target image; an image extraction is performed on the target image according to the changed area, and a target area image is obtained; the target area image is classified based on a classification model, and it is judged whether it is a rust area; the rust spot area expansion trend of an important area is monitored by a high-definition camera, the recognition accuracy is high, the monitoring can be continuously performed, the post change is not needed, the artificial negligence is avoided, and the cost is low.

Description

[0001] This case is a divisional application of the invention patent with the application date of August 14, 2023, the application number of 202311013671.7, and the name of "a rust spot area expansion monitoring method and system based on a neural network model". TECHNICAL FIELD

[0002] The present application relates to the technical field of neural network model application, in particular to a rust spot area expansion monitoring method and system combined with a classification network. BACKGROUND

[0003] In some important places, such as stations, workshops, laboratories, and key areas of airports, if the trend of rust corrosion and area expansion is not discovered in time, it may cause safety hazards. There is a wide demand for rust corrosion expansion detection in key areas.

[0004] The prior art usually collects images through a camera, and monitors and analyzes by manual work. However, manual monitoring may require a large amount of manual work, and requires continuous investment in labor costs, or relies on image comparison means, compares the original image and the change image to be detected, and finds the change area, but this method may be affected by environmental changes such as light changes, resulting in certain false positives. Therefore, how to use high-definition cameras and other automatic monitoring devices for fixed-point monitoring to realize the monitoring of rust spot area expansion in key areas and timely alarm and eliminate safety hazards is a technical problem to be solved at present. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a rust spot area expansion monitoring method and system combined with a classification network, which uses a camera to monitor the rust spot area expansion, reduces the cost and has high accuracy.

[0006] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0007] A rust spot area expansion monitoring method based on a neural network model, comprising the steps of:

[0008] S1, acquiring a to-be-detected image and a reference image, the reference image being an image of an initial state of a shooting area of the to-be-detected image;

[0009] S2, analyzing the to-be-detected image and the reference image based on a neural network model, determining the to-be-detected image with changes as a target image, and marking a change area on the target image;

[0010] S3, performing image extraction on the target image according to the change area to obtain a target area image;

[0011] S4, classifying the target region image based on the classification model to determine whether it is a rusted region.

[0012] A rust spot area expansion monitoring method combined with a classification network, comprising the steps of:

[0013] S1, obtaining a to-be-detected image and a reference image, the reference image being an image of an initial state of a shooting area of the to-be-detected image;

[0014] S2, analyzing the to-be-detected image and the reference image based on a neural network model to determine that the to-be-detected image with changes is a target image, and marking a changed region on the target image;

[0015] S3, performing image extraction on the target image according to the changed region to obtain a target region image;

[0016] S4, classifying the target region image based on a classification model to determine whether it is a rusted region;

[0017] Step S4 comprises the steps of:

[0018] S41, pre-collecting a target region image for model training, extracting a feature vector by using a pre-convolutional layer of a classification network, and performing model training by using a classification model;

[0019] S42, performing saliency detection on a to-be-detected target region image to segment a foreground region;

[0020] S43, inputting the segmented foreground region into a pre-convolutional layer of a classification network to extract a feature vector;

[0021] S44, classifying the extracted feature vector by using a trained classification model;

[0022] The classification model adopts a two-class classification model, and the classification result is a rusted region or other changed region.

[0023] S5, determining whether the area of the rusted region exceeds a set threshold value, and if so, pushing alarm information of a rusted trend change to a designated user terminal.

[0024] To solve the above technical problems, another technical solution adopted by the present application is:

[0025] The rust area expansion monitoring system based on a neural network model comprises a monitoring terminal and at least one monitoring device, wherein the monitoring terminal comprises a processor, a memory and a computer program stored in the memory; when the processor executes the computer program, the processor receives an image to be detected collected by the monitoring device and implements the steps of the rust area expansion monitoring method based on a neural network model.

[0026] The rust area expansion monitoring system combined with a classification network comprises a monitoring terminal and at least one monitoring device, wherein the monitoring terminal comprises a processor, a memory and a computer program stored in the memory; when the processor executes the computer program, the processor receives an image to be detected collected by the monitoring device and implements the following steps:

[0027] S1, obtaining an image to be detected and a reference image, wherein the reference image is an image of an initial state of a region to be photographed of the image to be detected;

[0028] S2, analyzing the image to be detected and the reference image based on a neural network model, determining that the image to be detected with changes is a target image, and marking a changed region on the target image;

[0029] S3, performing image extraction on the target image according to the changed region to obtain a target region image;

[0030] S4, classifying the target region image based on a classification model to determine whether the target region image is a rust region;

[0031] Step S4 comprises the following steps:

[0032] S41, pre-collecting a target region image to perform model training, extracting a feature vector by using a front convolutional layer of a classification network, and performing model training by using a classification model;

[0033] S42, performing saliency detection on a target region image to be detected to segment a foreground region;

[0034] S43, inputting the segmented foreground region into the front convolutional layer of the classification network to extract a feature vector;

[0035] S44, classifying the extracted feature vector by using the trained classification model;

[0036] The classification model is a two-class classification model, and the classification result is a rust region or other changed region.

[0037] S5, determining whether the area of the rust region exceeds a set threshold value, and if yes, pushing alarm information of a rust trend change to a designated user terminal.

[0038] A rust area expansion monitoring system combined with a classification network, comprising a monitoring terminal and at least one monitoring device, the monitoring terminal comprising a processor, a memory and a computer program stored in the memory, when the processor executes the computer program, receives an image to be detected collected by the monitoring device, and implements the following steps:

[0039] S1, obtaining an image to be detected and a reference image, the reference image being an image of an initial state of a shooting area of the image to be detected;

[0040] S2, analyzing the image to be detected and the reference image based on a neural network model, determining the image to be detected with changes as a target image, and marking a changed area on the target image;

[0041] S3, performing image extraction on the target image according to the changed area to obtain a target area image;

[0042] S4, classifying the target area image based on a classification model to determine whether it is a rust area;

[0043] Step S4 comprises the following steps:

[0044] S41, pre-collecting a target area image for model training, extracting a feature vector using a pre-convolutional layer of a classification network, and training a classification model;

[0045] S42, performing saliency detection on the target area image to be detected to segment a foreground area;

[0046] S43, inputting the segmented foreground area into the pre-convolutional layer of the classification network to extract a feature vector;

[0047] S44, classifying the extracted feature vector using the trained classification model;

[0048] Wherein, the classification model adopts a two-class classification model, and the classification result is a rust area or other changed area;

[0049] S5, determining whether the area of the rust area exceeds a set threshold value, and if so, pushing an alarm information of a rust trend change to a designated user terminal.

[0050] The beneficial effects of the present application are that the rust area expansion monitoring method and system based on a neural network model can obtain a to-be-detected image and a reference image, use a neural network model trained in advance to obtain the position of a change region and the category of the region, and determine whether it is a rust area, so as to realize rust area expansion detection, monitor the rust area expansion trend of an important region through a high-definition camera, has high recognition accuracy, can continuously monitor, does not need to change shifts, avoids manual negligence, and has low cost. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of a rust area expansion monitoring method based on a neural network model according to an embodiment of the present application;

[0052] Figure 2 A structural diagram of a rust area expansion monitoring system based on a neural network model according to an embodiment of the present application;

[0053] REFERENCE NUMERALS:

[0054] 1. A rust area expansion monitoring system based on a neural network model; 2. A monitoring terminal; 3. A processor; 4. A memory; 5. A monitoring device. DETAILED DESCRIPTION

[0055] To explain the technical content, purposes and effects of the present application in detail, the following will be described in combination with embodiments and the accompanying drawings.

[0056] Please refer to Figure 1 A rust area expansion monitoring method based on a neural network model, comprising the following steps:

[0057] S1. Obtain a to-be-detected image and a reference image, wherein the reference image is an image of an initial state of a shooting region of the to-be-detected image;

[0058] S2. Analyze the to-be-detected image and the reference image based on a neural network model, determine that the to-be-detected image with changes is a target image, and mark a change region on the target image;

[0059] S3. Perform image extraction on the target image according to the change region to obtain a target region image;

[0060] S4. Classify the target region image based on a classification model to determine whether it is a rust area.

[0061] From the above description, the beneficial effects of the present application are that: by acquiring the to-be-detected image and the reference image, the position and the category of the change region are acquired by using the neural network model trained in advance, whether it is a rust region is judged, so that the detection of the area expansion of rust spots is realized, the high-definition camera is used to monitor the area expansion trend of important regions, the recognition accuracy is high, uninterrupted monitoring can be carried out, the change of shifts is not needed, manual negligence is avoided, and the cost is low.

[0062] Further, the step S2 comprises the steps of:

[0063] S21, a large amount of scene knowledge is collected in advance for model training, and a scene model is obtained based on a change detection network ChangeNet;

[0064] S22, the to-be-detected image and the reference image are aligned;

[0065] S23, the to-be-detected image after the alignment operation is detected by using the scene model, and the coordinate position information of the change region is output.

[0066] From the above description, the change region of the image is detected by the scene model, the scene model is based on the change monitoring network, and the to-be-detected image and the reference image are aligned and detected, so that the change region of the to-be-detected image and the reference image can be more accurately determined.

[0067] Further, the change detection network in the step S21 is a change detection network based on a twin neural network and a full convolutional neural network;

[0068] The steps implemented by the scene model comprise:

[0069] S211, a residual network is acquired as a pre-training model by a method of transfer learning, and the composition of the residual block of the residual network comprises a convolution layer, a batch normalization BN and an activation function ReLU;

[0070] S212, features are extracted from the reference image and the to-be-detected image by a twin neural network, and different degrees of change positioning information are combined with convolution to capture coarse information and detail information in the image, and the twin neural network is based on the pre-training model;

[0071] S213, the extracted features are integrated by using a full convolutional neural network FCN, then added and classified by using the same normalization exponential function softmax, and the change region is obtained.

[0072] From the above description, when extracting the features of the same attribute, if two neural networks are used to extract the features of the pictures respectively, the extracted features are likely to not be in the same distribution domain, and the change detection network based on the twin neural network and the fully convolutional neural network can extract the features of the same distribution domain of the two input pictures, so that the similarity of the two input pictures can be better judged.

[0073] Further, step S23 further comprises the following steps:

[0074] S231, saving the scene data with image change;

[0075] The scene data is compared with the to-be-detected image and the reference image with changes;

[0076] Step S2 further comprises the following steps:

[0077] S24, updating the scene knowledge based on the saved scene data to obtain a scene model with higher precision.

[0078] From the above description, the scene data with image change is saved for each use to optimize and update the scene model, so that a scene model with higher precision is iterated.

[0079] Further, the S24 comprises the following steps:

[0080] S241, establishing a scene data accumulation module;

[0081] S242, the scene change detection module judges whether the saved scene data changes, if the scene changes, S243 and S244 are executed, otherwise, the next scene detection is waited and S242 is executed;

[0082] S243, the task model optimization module optimizes the task model using the new scene data to improve its precision in the new scene;

[0083] S244, when the scene data in the scene data accumulation module reaches the preset incremental threshold, the parameter optimization is performed by using the change detection network, the scene knowledge in different scenes is extracted by the model, and a scene model with higher precision and stronger generalization performance is obtained, so that the continuously changing scene is continuously self-adapted.

[0084] From the above description, the scene model is optimized and updated after the scene data accumulates to the preset incremental threshold, the parameter optimization is performed by using the change detection network, and the precision and generalization of the scene model in different scenes are intended to be extracted.

[0085] Further, step S4 comprises the following steps:

[0086] S41, a large number of target region images are collected in advance for model training, a feature vector is extracted by using a pre-convolutional layer of a classification network, and a classification model is used for model training;

[0087] S42, a saliency detection is performed on a target region image to be detected, and a foreground region is segmented;

[0088] S43, a feature vector is extracted by inputting the segmented foreground region into a pre-convolutional layer of a classification network;

[0089] S44, the extracted feature vector is classified by using a trained classification model.

[0090] As can be seen from the above description, the classification model is trained in advance, and when used, the foreground region is segmented by saliency detection and input into the classification network for classification.

[0091] Further, step S42 includes steps of:

[0092] S421, a saliency map is generated, a pixel-level saliency value is generated based on a color value histogram distribution of an input image, and the saliency value of each pixel point is a measure between the contrast of the pixel point and all the remaining image pixel points, and the measure formula is:

[0093]

[0094] wherein D(I k ,I i ) represents the color distance of two pixel points, I represents a pixel point set, I k and I i represent the kth pixel point and the ith pixel point in the pixel point set;

[0095] S422, a weight region merging is performed based on a Gaussian kernel function weight to obtain an improved saliency calculation formula:

[0096]

[0097] wherein σ s is a preset value, controlling the spatial weight influence, D s (r k ,r i ) represents the Euclidean distance between the centers of two regions, r k and r i represent two regions, w() represents the weight of the region, and D r () represents the color distance between the two regions;

[0098] S423, the image is binarized and segmented according to the calculated saliency map and the set threshold, and the foreground region is obtained.

[0099] From the above description, by the above steps, the saliency calculation is realized, and the foreground region is segmented.

[0100] Further, the method further comprises the steps of:

[0101] S5, determining whether the area of the rust stain region exceeds a set threshold value, and if so, pushing alarm information of a rust stain trend change to a designated user terminal.

[0102] From the above description, by setting the threshold value, the size of the preset and changed rust stain region is determined to determine whether the alarm needs to be issued.

[0103] Further, the step S5 comprises the steps of:

[0104] S51, obtaining the actual size of the image to be detected by laser ranging and camera parameters;

[0105] S52, calculating the pixel ratio of the changed region to the image to be detected;

[0106] S53, calculating the actual area of the rust stain region according to the pixel ratio and the actual size of the image to be detected;

[0107] S54, determining whether the area of the rust stain region exceeds a set threshold value, and if so, pushing alarm information of a rust stain trend change to a designated user terminal.

[0108] From the above description, based on the actual size of the reference image and the pixel ratio of the changed region to the reference image, the actual area of the changed region, i.e. the rust stain region, is determined, and then whether the alarm needs to be judged.

[0109] Please refer to Figure 2 A rust stain area expansion monitoring system based on a neural network model, comprising a monitoring terminal and at least one monitoring device, the monitoring terminal comprising a processor, a memory and a computer program stored in the memory, when the processor executes the computer program, receives the image to be detected collected by the monitoring device, and realizes the steps in the above-mentioned rust stain area expansion monitoring method based on a neural network model.

[0110] The rust stain area expansion monitoring method and system based on a neural network model are suitable for automatic rust stain area expansion monitoring.

[0111] Please refer to Figure 1 Embodiment one of the present application is:

[0112] A rust stain area expansion monitoring method based on a neural network model, comprising the steps of:

[0113] S1, acquire a to-be-detected image and a reference image, the reference image being an image of an initial state of a shooting area of the to-be-detected image;

[0114] S2, analyze the to-be-detected image and the reference image based on a neural network model, determine the to-be-detected image with changes as a target image, and mark a changed area on the target image;

[0115] The step S2 comprises the steps of:

[0116] S21, pre-acquire a large amount of scene knowledge for model training, and obtain a scene model based on a change detection network ChangeNet;

[0117] The change detection network in the step S21 is specifically a change detection network based on a twin neural network and a fully convolutional neural network;

[0118] The steps implemented by the scene model comprise:

[0119] S211, acquire a residual network as a pre-training model through a method of transfer learning, and the composition of a residual block of the residual network comprises a convolutional layer, a batch normalization BN, and an activation function ReLU.

[0120] In this embodiment, ResNet50 is used as the pre-training model.

[0121] S212, extract features from the reference image and the to-be-detected image through a twin neural network, and combine convolution to output change positioning information of different degrees, so as to capture coarse information and detailed information in the image, the twin neural network being based on the pre-training model.

[0122] In this embodiment, two twin neural networks CNN1 and CNN2 are respectively used to extract features from the reference image and the to-be-detected image, and combine convolution to output change positioning information of different degrees, so as to capture coarse information and detailed information in the image.

[0123] When it is intended to extract features of the same attribute, if two neural networks are used to extract features from the pictures respectively, the extracted features are likely to not be in the same distribution domain. At this time, a neural network can be used for feature extraction and then comparison. Therefore, the twin neural network can extract features of the same distribution domain of two input pictures, and at this time, the similarity of the two input pictures can be determined.

[0124] Two images are input from the network, and each image is extracted by a neural network. The network includes a ResNet structure, that is, a basic network including multiple modules, each module including a convolutional layer and a pooling layer to extract image features at different scales. Then, each module has a full convolutional layer with a kernel size of 1x1 after the convolutional layer and the pooling layer. The full convolutional layer is mainly used to adjust the number of channels to ensure that the feature channel numbers at different scales (under different modules) are consistent. A deconvolutional layer is connected after the full convolutional layer to make the feature dimension consistent with the input image size. The multiple modules of the basic network include features at different scales of the image. The front layers extract detailed information, and the back layers extract global information. Therefore, it is described as: combining the convolutional output of different layers to capture coarse information and detailed information in the image.

[0125] S213, using a full convolutional neural network FCN to integrate the extracted features, and then adding and classifying using the same normalization index function softmax to obtain the change region.

[0126] The image extracts features at different scales in the basic network, and their feature dimensions are different. After the full convolutional layer with a kernel size of 1x1 and the deconvolutional layer, multiple features with the same dimension are formed. At this time, they need to be normalized, so the softmax function is used for processing. The original output of the neural network is not a probability value, but a vector value obtained by complex weighting and nonlinear processing of the input value. Therefore, the softmax function is used to calculate the normalized probability of whether each pixel point (the image to be detected and the reference image) changes.

[0127] S22, aligning the image to be detected and the reference image; making the pixel points of the image to be detected and the reference image correspond to each other as much as possible.

[0128] Because the camera can shake or tilt during monitoring, image registration is needed to improve the monitoring effect. The registration method is as follows:

[0129] 1. SIFT feature point extraction is performed on the image to be detected and the reference image.

[0130] 2. Find the matching feature point pair (at least three) by similarity measurement.

[0131] 3. Obtain the spatial coordinate transformation parameters through the matching feature point pair.

[0132] 4. Image registration is performed by the coordinate transformation parameters to make the pixel point coordinates of the two images correspond to each other.

[0133] S23, using a scene model to perform change detection on the to-be-detected image after the registration alignment operation is completed, and outputting coordinate position information of a change region.

[0134] The step S23 further includes the following steps:

[0135] S231, saving scene data in which an image changes;

[0136] The scene data is compared with the to-be-detected image and the reference image in which changes exist.

[0137] S24, updating a scene knowledge based on the saved scene data to obtain a scene model with higher precision;

[0138] The S24 includes the following steps:

[0139] S241, establishing a scene data accumulation module;

[0140] S242, a scene change detection module judges whether the saved scene data changes, if the scene changes, S243 and S244 are executed, otherwise, the next scene detection is waited and S242 is executed;

[0141] S243, a task model optimization module optimizes a task model using new scene data to improve the precision of the task model in a new scene;

[0142] S244, when the scene data in the scene data accumulation module reaches a preset incremental threshold, a parameter optimization is performed using a change detection network, scene knowledge in different scenes is extracted by the model, and a scene model with higher precision and stronger generalization performance is obtained, so that continuous scene adaptation is performed on a continuously changing scene.

[0143] There are two key factors in the training of the model, one is a sample, and the other is a hyperparameter, and the optimization scheme is as follows:

[0144] 1. First, a hyperparameter table is designed to set the range of each parameter of the model, such as learning rate, iteration number, batch size, and the like;

[0145] 2. Each time new scene change data is detected, it is added to the scene data accumulation module;

[0146] 3. When the scene change data in the scene data accumulation module reaches a certain increment, the new data and the original data are combined for training, and multiple training is performed according to the hyperparameter table, and the model with the highest validation set precision is selected for updating.

[0147] S3, performing image extraction on the target image according to the change region to obtain a target region image.

[0148] S4, determining whether the scene change category of the target image is a target category according to the target region image.

[0149] In this embodiment, the target region image is classified based on the classification model to determine whether it is a rust region.

[0150] In this embodiment, the classification of the target region with changes has two categories, one is the change of the rust spot region, and the other is other changes. The features of the change region are extracted and then input to the pre-trained svm classifier to determine whether the change region belongs to the rust spot region.

[0151] Step S4 includes the following steps:

[0152] S41, a large number of target region images are collected in advance for model training, a feature vector is extracted by using the pre-convolutional layer of the classification network, and the model is trained by using the classification model.

[0153] In this embodiment, the feature vector is extracted by using the pre-convolutional layer of the classification network MobileNet-v3, and the model is trained by using the SVM classifier.

[0154] S42, performing saliency detection on the target region image to be detected to segment out the foreground region;

[0155] Step S42 includes the following steps:

[0156] S421, generating a saliency map, generating a pixel-level saliency value based on the color value histogram distribution of the input image, and the saliency value of each pixel point is a measure between the contrast of the pixel point and all the remaining image pixel points. The measure formula is:

[0157]

[0158] wherein D(I k ,I i ) represents the color distance of two pixel points, I represents a pixel point set, I k and I i represent the kth pixel point and the ith pixel point in the pixel point set.

[0159] As can be seen from the formula, the saliency of the pixel points with the same pixel value will be the same, and the pixel points with more colors will have a larger saliency value.

[0160] wherein the color distance is calculated as follows:

[0161] The color distance refers to the color distance of two pixel points, and the formula is as follows:

[0162]

[0163] wherein, C1, C2 represent color 1 and color 2, C1R represents the R channel of color 1. R, G, B represent channels.

[0164] S422, based on the weight of the Gaussian kernel function, the weight region merging is performed to obtain an improved saliency calculation formula:

[0165]

[0166] wherein, σ s is a preset value, controlling the space weight influence, D s (r k , r i ) represents the Euclidean distance between the centers of two regions, r k and r i represent two regions, w() represents the weight of the region, D r () represents the color distance between the two regions.

[0167] S423, according to the saliency map obtained by calculation and the threshold set, the image is binarized and segmented to obtain the foreground region.

[0168] S43, the extracted feature vector of the pre-convolutional layer of the classification network is inputted to the segmented foreground region.

[0169] In this embodiment, the segmented foreground region is inputted to the MobileNet-v3 pre-convolutional layer to extract the feature vector.

[0170] S44, the extracted feature vector is classified by the trained classification model SVM.

[0171] S5, determining whether the area of the rust spot region exceeds the set threshold value, if yes, the alarm information of the rust spot trend change is pushed to the specified user end;

[0172] Step S5 includes the following steps:

[0173] S51, the actual area size of the image to be detected is obtained by laser ranging and camera parameters;

[0174] S52, the pixel ratio of the change region to the image to be detected is calculated;

[0175] S53, according to the pixel ratio and the actual area size of the image to be detected, the actual area of the rust spot region is calculated;

[0176] S54, determining whether the area of the rust spot region exceeds the set threshold value, if yes, the alarm information of the rust spot trend change is pushed to the specified user end.

[0177] Referring to Figure 2 Embodiment two of the present application is:

[0178] A rust area expansion monitoring system 1 based on a neural network model, comprising a monitoring terminal 2 and at least one monitoring device 5, the monitoring terminal comprising a processor 3, a memory 4 and a computer program stored in the memory 4, when the processor 3 executes the computer program, receives the image to be detected collected by the monitoring device 5, and realizes the steps in the above-mentioned rust area expansion monitoring method based on a neural network model.

[0179] In summary, the present application provides a rust area expansion monitoring method and system based on a neural network model, by obtaining the image to be detected and the reference image, using the pre-trained neural network model to obtain the location of the change area and the category of the area, and determining whether it is a rust area, thereby realizing the detection of rust area expansion, and monitoring the rust area expansion trend of important areas through a high-definition camera, which not only has high recognition accuracy, but also can be monitored continuously without the need for shift changes, avoids human negligence, and has low cost.

[0180] The present application detects the change area of the image to be detected and the reference image according to the trained model, and can update the scene knowledge in real time, and continuously adjust the model structure using the scene; at the same time, the target area is processed in blocks, and attention weight information is used to optimize the classification effect. In actual use, complex environmental factors may cause target recognition failure, in which case the present application continuously updates the scene knowledge according to the actual scene to update the detection accuracy of the model. In classifying the target area, the present application segments the target area based on saliency, extracts the foreground area such as rust, and classifies each target image to obtain the predicted category of each sample image; wherein the saliency weight of an image block is used to indicate the influence of the image block on determining the category to which the sample image belongs.

[0181] The above-mentioned is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent transformation or direct or indirect application in related technical fields using the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A method for monitoring the expansion of rust spots combined with a classification network, characterized in that: Including steps: S1. Acquire an image to be detected and a reference image, wherein the reference image is an image of an initial state of a shooting area of ​​the image to be detected; S2. Analyzing the image to be detected and the reference image based on a neural network model, determining that the image to be detected with changes is a target image, and marking a changed area on the target image; S3, performing image extraction on the target image according to the changed area to obtain a target area image; S4, classifying the target area image based on a classification model to determine whether it is a rust spot area; Step S4 includes the steps of: S41, pre-collecting the target area image for model training, using the pre-convolutional layer of the classification network to extract feature vectors, and using the classification model to perform model training; S42, performing saliency detection on the target area image to be detected, and segmenting the foreground area; S43, inputting the segmented foreground area into the pre-convolutional layer of the classification network to extract feature vectors; S44, classifying the extracted feature vector using a trained classification model; Wherein, the classification model adopts a two-class classification model, and the classification result is rust spot area or other change area; S5. Determine whether the area of ​​the rust spot exceeds a set threshold. If so, push an alarm message of the rust spot trend change to a designated user terminal.

2. The method for monitoring the expansion of rust spots combined with a classification network according to claim 1, characterized in that: The classification network used is the MobileNet-v3 classification network, and the classification model used is the SVM classification model.

3. The method for monitoring the expansion of rust spots combined with a classification network according to claim 1, characterized in that: Step S42 includes the steps of: S421. Generate a saliency map. Generate pixel-level saliency values ​​based on the color value histogram distribution of the input image. The saliency value of each pixel is a measure of the contrast between it and all other image pixels. The measurement formula is: Among them, D(I k ,I i ) represents the color distance between two pixels, I represents the pixel set, I k and I i Represents the kth pixel and the ith pixel in the pixel set; S422. An improved saliency calculation formula is obtained by merging weighted regions based on Gaussian kernel function weight generation: Among them, σ s is the preset value, which controls the influence of spatial weight, D s (r k ,r i ) represents the Euclidean distance between two regional centers, r k and r i Represents two regions, w() represents the weight of the region, D r () indicates the color distance between two areas; S423 , performing binary segmentation on the image according to the calculated saliency map and the set threshold value to obtain the foreground area.

4. The method for monitoring the expansion of rust spots combined with a classification network according to claim 3, characterized in that: The calculation of the color distance is specifically as follows: Among them, C1 and C2 represent color 1 and color 2 respectively, and R, G, and B represent color channels.

5. The method for monitoring the expansion of rust spots combined with a classification network according to claim 1, characterized in that: Step S5 includes the steps of: S51, obtaining the actual area size of the image to be detected through laser ranging and camera parameters; S52, calculating the ratio of the number of pixels in the changed area to the number of pixels in the image to be detected; S53, calculating the actual area of ​​the rust spot region according to the pixel ratio and the actual area of ​​the image to be detected; S54: Determine whether the area of ​​the rust spot region exceeds a set threshold. If so, push an alarm message of the rust spot trend change to a designated user terminal.

6. A monitoring system for rust spot expansion combined with a classification network, characterized in that: The system includes a monitoring terminal and at least one monitoring device, wherein the monitoring terminal includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it receives an image to be detected collected by the monitoring device and implements the following steps: S1. Acquire an image to be detected and a reference image, wherein the reference image is an image of an initial state of a shooting area of ​​the image to be detected; S2. Analyzing the image to be detected and the reference image based on a neural network model, determining that the image to be detected with changes is a target image, and marking a changed area on the target image; S3, performing image extraction on the target image according to the changed area to obtain a target area image; S4, classifying the target area image based on a classification model to determine whether it is a rust spot area; Step S4 includes the steps of: S41, pre-collecting the target area image for model training, using the pre-convolutional layer of the classification network to extract feature vectors, and using the classification model to perform model training; S42, performing saliency detection on the target area image to be detected, and segmenting the foreground area; S43, inputting the segmented foreground area into the pre-convolutional layer of the classification network to extract feature vectors; S44, classifying the extracted feature vector using a trained classification model; Wherein, the classification model adopts a two-class classification model, and the classification result is rust spot area or other change area; S5. Determine whether the area of ​​the rust spot exceeds a set threshold. If so, push an alarm message of the rust spot trend change to a designated user terminal.

7. The rust spot area expansion monitoring system combined with a classification network according to claim 6, characterized in that: The classification network used is the MobileNet-v3 classification network, and the classification model used is the SVM classification model.

8. The rust spot area expansion monitoring system combined with a classification network according to claim 6, characterized in that: Step S42 includes the steps of: S421. Generate a saliency map. Generate pixel-level saliency values ​​based on the color value histogram distribution of the input image. The saliency value of each pixel is a measure of the contrast between it and all other image pixels. The measurement formula is: Among them, D(I k ,I i ) represents the color distance between two pixels, I represents the pixel set, I k and I i Represents the kth pixel and the ith pixel in the pixel set; S422. An improved saliency calculation formula is obtained by merging weighted regions based on Gaussian kernel function weight generation: Among them, σ s is the preset value, which controls the influence of spatial weight, D s (r k ,r i ) represents the Euclidean distance between two regional centers, r k and r i Represents two regions, w() represents the weight of the region, D r () indicates the color distance between two areas; S423 , performing binary segmentation on the image according to the calculated saliency map and the set threshold value to obtain the foreground area.

9. The rust spot area expansion monitoring system combined with a classification network according to claim 8, characterized in that: The calculation of the color distance is specifically as follows: Among them, C1 and C2 represent color 1 and color 2 respectively, and R, G, and B represent color channels.

10. The rust spot area expansion monitoring system combined with a classification network according to claim 6, characterized in that: Step S5 includes the steps of: S51, obtaining the actual area size of the image to be detected through laser ranging and camera parameters; S52, calculating the ratio of the number of pixels in the changed area to the number of pixels in the image to be detected; S53, calculating the actual area of ​​the rust spot region according to the pixel ratio and the actual area of ​​the image to be detected; S54: Determine whether the area of ​​the rust spot region exceeds a set threshold. If so, push an alarm message of the rust spot trend change to a designated user terminal.

Images