Trash holding net anomaly detection method, system and device and storage medium

Abstract

The invention discloses a trash holding net anomaly detection method, system and device and a storage medium, and the method comprises the steps: carrying out the analysis of a monitoring image through a semantic segmentation model, and extracting a binary segmentation mask of a trash holding net main rope; performing skeleton extraction on the segmentation mask to generate a skeleton image of the main rope; extracting a main rope curve in the skeleton image, and aligning the curve through principal component analysis; calculating a Fourier descriptor module value of the curve, and extracting morphological characteristics of the main rope; and comparing the curve feature vectors of the main rope in different time periods, calculating curve similarity, and judging whether the main rope is abnormal or not according to a similarity threshold value. The method has the advantages of high efficiency, automation, low cost, high robustness and the like, can be widely applied to monitoring the state of the trash holding net in a large-range water area, and provides reliable technical support for safe operation of water area facilities.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method, system, device and storage medium for detecting anomalies in a waste control net. Background Art

[0002] Trash nets are common protective devices used in water conservancy projects, ports, docks, and hydropower stations. They are primarily used to intercept floating debris and aquatic organisms in the water to protect the normal operation of water conservancy facilities. However, during long-term operation, trash nets may break, deform, or become loose due to factors such as water impact, external forces, and material aging. Once these abnormalities occur, they not only reduce the net's interception capacity but may also allow floating debris to enter the facility with the water flow, and in serious cases, even cause safety accidents. Therefore, real-time monitoring of the status of trash nets and abnormality detection are particularly important.

[0003] Currently, monitoring of trash nets relies primarily on manual inspections or the installation of simple sensors. However, these methods have the following drawbacks: manual inspections are labor-intensive, inefficient, and difficult to achieve real-time monitoring over large water areas. Sensor detection, however, is limited by the equipment installation environment and measurement accuracy, making it difficult to comprehensively monitor the overall status of the net. This is especially true in environments with complex currents and dense floating debris, where traditional methods struggle to quickly and accurately detect abnormalities such as main rope breaks.

[0004] In recent years, with the rapid development of computer vision and deep learning technologies, monitoring methods based on image processing have gradually emerged. By analyzing monitoring images, it is possible to accurately identify specific targets and analyze their status. This provides a new technical path for realizing intelligent monitoring of the status of wastewater control nets. Summary of the Invention

[0005] The present application provides a method, system, device and storage medium for detecting abnormalities in a waste control net to solve the above-mentioned problems.

[0006] In one aspect, the present application provides a method for detecting anomalies in a trash net, the method comprising the following steps:

[0007] Step S1: Analyze the monitoring image using the semantic segmentation model to extract the binary segmentation mask of the main rope of the pollution control net;

[0008] Step S2: Perform skeleton extraction on the segmentation mask to generate a skeleton image of the main rope;

[0009] Step S3: extract the principal rope curves in the skeleton image and align the curves through principal component analysis;

[0010] Step S4: Calculate the Fourier descriptor modulus of the curve and extract the morphological characteristics of the main rope;

[0011] Step S5: Compare the main rope curve feature vectors in different time periods, calculate the curve similarity, and determine whether there is an abnormality in the main rope based on the similarity threshold.

[0012] In one implementation of the present application, the semantic segmentation model is trained using the deep learning algorithm DeepLabv3+ model.

[0013] In one implementation of the present application, the skeleton extraction uses a morphological algorithm to reduce the binary mask layer by layer to a skeleton form with a single pixel width.

[0014] In one implementation of the present application, Fourier descriptor calculation includes the following steps:

[0015] Represent the main rope curve point set in complex form;

[0016] The discrete Fourier transform is performed on the complex point set, and the modulus values of the first 10 low-frequency Fourier descriptors are extracted as feature vectors.

[0017] In one implementation of the present application, the curve similarity is calculated using the Euclidean distance formula:

[0018]

[0019] Among them, |F1(n)| and |F2(n)| represent the Fourier descriptor modulus values of the curve at different time periods.

[0020] In one implementation of the present application, whether the main rope has an abnormality is determined based on a similarity threshold. Specifically, when the curve similarity is lower than a set threshold, it is determined that the main rope has an abnormality and an alarm is triggered.

[0021] The present application also provides a system for detecting abnormalities in a trash net, the system comprising:

[0022] The mask segmentation unit is used to analyze the monitoring image using the semantic segmentation model and extract the binary segmentation mask of the main rope of the pollution control net;

[0023] A mask extraction unit is used to perform skeleton extraction on the segmentation mask to generate a skeleton image of the main rope;

[0024] The curve alignment unit is used to extract the main rope curves in the skeleton image and align the curves through principal component analysis;

[0025] A morphological extraction unit is used to calculate the Fourier descriptor modulus of the curve and extract the morphological features of the main rope;

[0026] The abnormality judgment unit is used to compare the main rope curve feature vectors of different time periods, calculate the curve similarity, and judge whether there is an abnormality in the main rope based on the similarity threshold.

[0027] The present application also provides a device for detecting abnormalities in a waste net, the device comprising:

[0028] at least one processor; and,

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can complete the aforementioned method for detecting abnormalities in a pollution control net.

[0031] The present application also provides a non-volatile computer storage medium for detecting abnormalities in a trash net, which stores computer-executable instructions. The computer-executable instructions are executed by a processor for use in the aforementioned method for detecting abnormalities in a trash net.

[0032] The present application provides a method, system, device, and storage medium for detecting abnormalities in a waste control net, which have the following beneficial effects:

[0033] (1) The deep learning semantic segmentation technology and curve analysis method are used to automatically complete the morphological extraction and status monitoring of the main rope of the pollution control net without human supervision. Compared with manual inspection, it is more efficient and suitable for real-time monitoring of large-scale water areas.

[0034] (2) Through skeleton extraction and Fourier descriptor analysis, the present invention can perform high-precision curve similarity calculation on the main rope state and accurately capture the slight morphological changes of the main rope; at the same time, the algorithm effectively filters the noise curve and has strong environmental adaptability.

[0035] (3) It can quickly determine whether the main rope has abnormal conditions such as breakage or deformation, and trigger an alarm through the similarity threshold, providing timely and effective decision-making support for the operation and maintenance of the trash net.

[0036] (4) Intelligent monitoring of the status of the sewage net can be achieved by simply combining it with video surveillance equipment, without the need to install additional complex sensors or equipment. It has low cost and good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 A flow chart of a method for detecting abnormalities in a trash net provided in an embodiment of the present application;

[0039] Figure 2 The original graph for segmentation model reasoning provided in the embodiment of the present application;

[0040] Figure 3 The segmentation result image of the trash net provided in the embodiment of the present application;

[0041] Figure 4 The skeleton image of the trash net provided in the embodiment of the present application;

[0042] Figure 5 The skeleton image of the trash net at the second moment provided in the embodiment of the present application;

[0043] Figure 6 A diagram showing the composition of a trash net anomaly detection system provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of a device for detecting abnormalities in a waste net provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] The embodiments of the present application provide a method, system, device, and storage medium for detecting abnormalities in a trash net. The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0047] Figure 1 This is a flow chart of a method for detecting abnormalities in a waste net provided in an embodiment of the present application. Figure 1 As shown, the method mainly includes the following steps:

[0048] Step S1: Analyze the monitoring image using the semantic segmentation model to extract the binary segmentation mask of the main rope of the pollution control net;

[0049] Step S2: Perform skeleton extraction on the segmentation mask to generate a skeleton image of the main rope;

[0050] Step S3: extract the principal rope curves in the skeleton image and align the curves through principal component analysis;

[0051] Step S4: Calculate the Fourier descriptor modulus of the curve and extract the morphological characteristics of the main rope;

[0052] Step S5: Compare the main rope curve feature vectors in different time periods, calculate the curve similarity, and determine whether there is an abnormality in the main rope based on the similarity threshold.

[0053] The following is an example of Yi's water area to specifically demonstrate the embodiments of the present application. In the embodiment of the present application, first, a camera installed on a water area monitoring device is used to collect images of actual operation scenes containing the trash net, covering a variety of weather conditions and ambient lighting (such as sunny days, cloudy days, nights, etc.). The main rope area of the trash net is marked from the collected images to generate a semantic segmentation dataset containing the position of the main rope. The main categories in the dataset include "main rope" and "background". A total of 3,000 images were annotated, including 2,400 training sets and 600 verification sets.

[0054] DeepLabv3+ was used as the base network for the semantic segmentation model. The cross-entropy loss function and Adam optimizer were used during training. The initial learning rate was 0.001, and training was performed for 100 epochs. The final model achieved a mean Intersection Over Union (MIoU) of 96.2% on the validation set, accurately distinguishing the main rope of the trash net from the background area. The original image is shown in Figure 2. Figure 2 The segmentation results are shown in Figure 3 shown.

[0055] The skeleton is extracted from the segmented main rope binary image. The skeleton is defined as the center line of the main rope area that gradually shrinks inward to a single pixel width. The mathematical definition of skeleton extraction is as follows:

[0056] Suppose the input binary image is I(x,y), where (x,y) is the pixel coordinate of the image, I(x,y) = 1 represents the main rope area, and I(x,y) = 0 represents the background. Skeleton extraction can be achieved by solving the nearest distance from each pixel in the image to the boundary. The distance transformation function is defined as:

[0057]

[0058] Where d(x,y) is the Euclidean distance from the pixel to the nearest background pixel (i,j) at (x,y); (x,y) are the coordinates of the pixel point in the main rope area; (i,j) are the coordinates of the pixel point in the background area; I(i,j) = 0 indicates that the pixel (i,j) belongs to the background area.

[0059] The pixel with the largest distance value is retained as the skeleton point, and the other pixel values are 0. The skeleton image S(x,y) can be shown as:

[0060]

[0061] Where S(x,y) is the skeleton image, with a value of 1 representing a skeleton point and a value of 0 representing a non-skeleton point; d(x,y) is the distance from the pixel (x,y) to the nearest background; I(x,y) is the input binary image. Finally, the image skeleton S(x,y) is the single-pixel-wide center line of the main rope area, as shown in Figure 4 shown.

[0062] Extract all continuous curve segments from the skeleton image S(x,y), and the extracted curve segments are defined as the set of skeleton pixel points: C = {(x k ,y k )|S(x k ,y k )=1,k=1,2,...,N}

[0063] Where C is the curve point set; (x k ,y k ) is the coordinate of the point on the skeleton curve; S(x k ,y k )=1 represents the point (x k ,y k ) is the skeleton point; k is the index of the curve point; N is the total number of curve points.

[0064] To remove noise curves, set the minimum length threshold L min When the curve length L meets the conditions:

[0065]

[0066] Then the curve is retained. Where L is the actual length of the curve; (x k ,y k ) is the coordinate of the curve point; N is the total number of curve points; L min is the minimum length threshold, used to filter short curves.

[0067] For the curve segments that pass the filter, they are sorted by length and the longest curve is selected as the representative curve of the main rope.

[0068] In order to eliminate the influence of shooting angle and position on the curve shape, the main direction of the curve is aligned. Assume that the curve point set is:

[0069] C={(x k ,y k )|k=1,2,...,N}

[0070] Compute the centroid of a curve point:

[0071]

[0072] in, is the average value of the horizontal coordinates of the curve points; is the average value of the vertical coordinates of the curve points; N is the total number of curve points.

[0073] Normalize the coordinates of the curve points so that they are referenced to the centroid:

[0074]

[0075] Among them, x' k ,y' k is the normalized coordinate with the center of mass as the reference point; x k ,y k is the coordinate of the original curve point; is the centroid coordinate.

[0076] Calculate the covariance matrix of a point set

[0077]

[0078] Where Σ is the covariance matrix; σ xx ,σ yy is the variance of the normalized coordinates; σ xy is the covariance of the normalized coordinates.

[0079] Solve for the eigenvalues and eigenvectors of the covariance matrix. The eigenvector corresponding to the largest eigenvalue is the principal direction vector. Project the curve point set onto the principal direction to complete the alignment.

[0080] Furthermore, the morphological characteristics of the curve are analyzed, and the resampled curve point set is converted into a Fourier descriptor. The Fourier descriptor is a feature representation method based on the discrete Fourier transform (DFT) and is used to express the global and local shape characteristics of the curve.

[0081] First, express the resampled curve point set in complex form:

[0082] z m =x' m +iy' m ,m=1,2,...,M

[0083] Among them, z m is the plural form of the curve point; x' m is the horizontal coordinate of the resampling point; y' m ′ is the ordinate of the resampling point; i is the imaginary unit; M is the total number of resampling points.

[0084] Perform discrete Fourier transform (DFT) on the complex curve point set and calculate the Fourier coefficients:

[0085]

[0086] Among them, F(n) is the Fourier coefficient, which represents the frequency domain information; n is the frequency index; e -i2πnm / M is the basis function of Fourier transform, representing a sine wave in the form of a complex exponential; M is the total number of resampling points.

[0087] Furthermore, the modulus of the Fourier descriptor is extracted as the curve feature:

[0088]

[0089] Where |F(n)| is the modulus of the Fourier descriptor, which represents the amplitude at a specific frequency; Re(F(n)) is the real part of the Fourier coefficient; and Im(F(n)) is the imaginary part of the Fourier coefficient.

[0090] To reduce computational complexity, only the first P low-frequency Fourier descriptors (i.e., n = 0, 1, .., P-1) are selected as the main features of the curve, because the low-frequency components can better describe the overall shape characteristics of the curve:

[0091] D={|F(0)|,|F(1)|,...,|F(P-1)|}

[0092] Where D is the truncated Fourier descriptor feature vector; P is the number of selected low-frequency Fourier descriptors.

[0093] Finally, the curve similarity calculation is performed. In order to measure whether the deformation of the barrier exceeds the threshold in unit time, the images of two adjacent moments (30 minutes apart) are respectively subjected to main rope segmentation and skeleton extraction. The skeleton curve of the second moment is as follows: Figure 5 To compare the shape features of two curves at different times, the similarity between their Fourier descriptors is calculated. The similarity is measured using the Euclidean distance, as shown in the following formula:

[0094]

[0095] Where Similarity is the similarity index of the two curves. The smaller the value, the closer the shapes of the two curves are. |F1(n)| is the modulus of the n-th Fourier descriptor of curve 1. |F2(n)| is the modulus of the n-th Fourier descriptor of curve 2. P is the number of low-frequency Fourier descriptors selected.

[0096] To detect abnormalities in the main rope, the threshold of the similarity index is set to 2.0. If the similarity index of the two curves exceeds the threshold, it is determined that there may be an abnormality in the main rope of the trash net (such as a break at one end or in the middle). Figure 4 and Figure 5 The similarity index of the main rope of the blocking net in the images at two adjacent moments is 2.5, indicating that its deformation exceeds the threshold and is judged to be abnormal.

[0097] The above is a method for detecting anomalies in a trash net provided by an embodiment of the present application. Based on the same inventive concept, an embodiment of the present application also provides a system for detecting anomalies in a trash net. Figure 6 A diagram showing the composition of a trash screen anomaly detection system provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the system mainly includes: a mask segmentation unit 601, which is used to analyze the monitoring image using a semantic segmentation model and extract the binary segmentation mask of the main rope of the pollution control net; a mask extraction unit 602, which is used to perform skeleton extraction on the segmentation mask and generate a skeleton image of the main rope; a curve alignment unit 603, which is used to extract the main rope curve in the skeleton image and align the curve through principal component analysis; a morphology extraction unit 604, which is used to calculate the Fourier descriptor modulus of the curve and extract the morphological features of the main rope; and an anomaly judgment unit 605, which is used to compare the main rope curve feature vectors of different time periods, calculate the curve similarity, and judge whether the main rope has an anomaly based on the similarity threshold.

[0098] The above is a system for detecting abnormalities in a trash net provided by an embodiment of the present application. Based on the same inventive concept, an embodiment of the present application also provides a device for detecting abnormalities in a trash net. Figure 7 A schematic diagram of a device for detecting abnormalities in a waste net provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device mainly includes: at least one processor 701; and a memory 702 in communication with the at least one processor; wherein the memory 702 stores instructions that can be executed by the at least one processor 701, and the instructions are executed by the at least one processor 701 so that the at least one processor 701 can complete the aforementioned method for detecting abnormalities in a waste control net.

[0099] In addition, an embodiment of the present application further provides a non-volatile computer storage medium for detecting abnormalities in a trash net, which stores computer-executable instructions. The computer-executable instructions are executed by a processor to implement the aforementioned method for detecting abnormalities in a trash net.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0103] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0104] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments.

[0105] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0106] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for detecting abnormalities in a trash net, characterized in that: The method comprises the following steps: Step S1: Analyze the monitoring image using the semantic segmentation model to extract the binary segmentation mask of the main rope of the pollution control net; Step S2: Perform skeleton extraction on the segmentation mask to generate a skeleton image of the main rope; Step S3: extract the principal rope curves in the skeleton image and align the curves through principal component analysis; Step S4: Calculate the Fourier descriptor modulus of the curve and extract the morphological characteristics of the main rope; Step S5: Compare the main rope curve feature vectors in different time periods, calculate the curve similarity, and determine whether there is an abnormality in the main rope based on the similarity threshold.

2. The method for detecting abnormalities in a trash net according to claim 1, wherein: The semantic segmentation model is trained using the deep learning algorithm DeepLabv3+ model.

3. The method for detecting abnormalities in a trash net according to claim 1, wherein: The skeleton extraction adopts a morphological algorithm to reduce the binary mask layer by layer to a skeleton form with a single pixel width.

4. The method for detecting abnormalities in a trash net according to claim 1, wherein: The Fourier descriptor calculation includes the following steps: Represent the main rope curve point set in complex form; The discrete Fourier transform is performed on the complex point set, and the modulus values of the first 10 low-frequency Fourier descriptors are extracted as feature vectors.

5. The method for detecting abnormalities in a trash net according to claim 1, wherein: The calculation of curve similarity uses the Euclidean distance formula: Among them, |F1(n)| and |F2(n)| represent the Fourier descriptor modulus values of the curve at different time periods.

6. The method for detecting abnormalities in a trash net according to claim 1, characterized in that: The main rope is judged to have abnormality according to the similarity threshold. Specifically, when the curve similarity is lower than the set threshold, the main rope is judged to have abnormality and an alarm is triggered.

7. A trash net abnormality detection system, characterized in that: The system comprises: The mask segmentation unit is used to analyze the monitoring image using the semantic segmentation model and extract the binary segmentation mask of the main rope of the pollution control net; A mask extraction unit is used to perform skeleton extraction on the segmentation mask to generate a skeleton image of the main rope; The curve alignment unit is used to extract the main rope curves in the skeleton image and align the curves through principal component analysis; A morphological extraction unit is used to calculate the Fourier descriptor modulus of the curve and extract the morphological features of the main rope; The abnormality judgment unit is used to compare the main rope curve feature vectors of different time periods, calculate the curve similarity, and judge whether there is an abnormality in the main rope based on the similarity threshold.

8. A device for detecting abnormalities in a trash net, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can complete the method for detecting anomalies in a pollution control net according to any one of claims 1 to 7.

9. A non-volatile computer storage medium for detecting abnormalities in a trash net, storing computer-executable instructions, characterized in that: The computer executable instructions are executed by a processor to implement the method for detecting anomalies in a trash net according to any one of claims 1 to 7.

Images