Drainage network diagnosis method, device, electronic equipment and computer storage medium

Abstract

The present invention provides a drainage pipe network diagnosis method, device, electronic equipment and computer storage medium, and relates to the technical field of municipal pipe network. The method comprises: obtaining an image to be detected of a water supply pipe network, and preprocessing the image to be detected; extracting features of the preprocessed image to be detected based on a feature extraction unit to obtain an N-dimensional scale feature map; fusing features of the N-dimensional scale map to obtain a fused N-dimensional scale feature map; enhancing features of the fused N-dimensional scale feature map based on a feature enhancement unit to obtain a final feature map; inputting the final feature map into a detector, and outputting defect classification and defect position of the image to be detected; fusing semantic features of different levels by fully connected mapping, designing individual channel branches to mine dependencies between scenes, and designing common channel branches to establish common spatial information associations to enhance feature expression, thereby reducing manual workload and improving the accuracy of defect diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of municipal pipe networks, and in particular to a drainage pipe network diagnosis method, device, electronic equipment and computer storage medium. Background Art

[0002] The drainage network system is an important part of urban infrastructure, and undertakes key tasks such as ensuring residents' daily life, flood prevention and drainage, and sewage treatment. With the acceleration of urbanization, the maintenance and management of drainage networks have become increasingly important. However, the aging and damage of pipes will have a serious impact on the normal operation of the city, so regular inspection and maintenance of the drainage network is very necessary. At present, the solutions for diagnosing drainage network defects mainly include: manual diagnosis methods and traditional target detection diagnosis methods that introduce computer vision technology.

[0003] However, manual diagnosis methods rely on manual frame-by-frame diagnosis and analysis of all video images, which often consumes a lot of manpower, has a large workload and low efficiency. Traditional target detection and diagnosis methods introduce computer vision technology and use detection algorithms to analyze images recorded by robots. Feature information is gradually diluted in the long network cascade transmission, which makes it difficult to cope with the diverse complexities of pipeline defect diagnosis scenarios, resulting in low accuracy in actual defect diagnosis. Summary of the invention

[0004] The purpose of the present invention is to provide a drainage network diagnosis method, device, electronic device and computer storage medium, which utilize fully connected mapping to fuse semantic features of different levels during image recognition, design individual channel branches to mine the dependencies between scenes, design common channel branches to establish common spatial information associations and enhance feature expression, thereby reducing manual workload and improving the accuracy of defect diagnosis.

[0005] In a first aspect, the present invention provides a drainage network diagnosis method, comprising:

[0006] Acquire the image to be detected of the water supply network, and preprocess the image to be detected; wherein the preprocessing includes: resolution normalization processing and / or pixel value normalization processing;

[0007] Based on the feature extraction unit, feature extraction is performed on the preprocessed image to be detected to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2;

[0008] The N-dimensional scale map features are subjected to feature fusion to obtain a fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear interpolation upsampling, convolution calculation downsampling or identity mapping;

[0009] Based on the feature enhancement unit, the fused N-dimensional scale feature map is feature enhanced to obtain a final feature map; wherein the feature enhancement unit includes: an individual channel branch and a common channel branch;

[0010] The final feature map is input into the detector, which outputs the defect classification and defect location of the image to be detected.

[0011] In some preferred embodiments of the present invention, the feature scales of the N levels of feature spaces in the feature extraction unit decrease successively; and the channel dimensions of the N levels of feature spaces in the feature extraction unit are all the same.

[0012] In some preferred embodiments of the present invention, the N-dimensional scale feature map includes: N feature maps;

[0013] The step of fusing the N-dimensional scale feature map to obtain a fused N-dimensional scale feature map includes:

[0014] Each feature map is segmented, and each feature map corresponds to M segmentation feature maps after segmentation; among them, ; M is a positive integer greater than 1;

[0015] The segmentation feature map is sampled according to a preset feature scale to generate a generated feature map corresponding to M feature scales except for the feature scale corresponding to the own feature image; if the feature scale of the generated feature map is larger than the feature scale of the segmentation feature map, the sampling operation is bilinear interpolation upsampling; if the feature scale of the generated feature map is smaller than the feature scale of the segmentation feature map, the sampling operation is convolution calculation downsampling;

[0016] The generated feature map is fully connected and fused with the feature map with identity mapping to obtain a fused N-dimensional scale feature map.

[0017] In some preferred embodiments of the present invention, the step of performing feature enhancement on the fused N-dimensional scale feature map based on the feature enhancement unit to obtain the final feature map includes:

[0018] The fused feature map is divided into two branches with the same channels, and vector calculations are performed separately; one branch performs individual vector calculations, and the other branch performs common vector calculations;

[0019] The results of the two vector calculations are concatenated and added point by point to the fused feature map;

[0020] The result of point-by-point addition is convolved to obtain the final feature map.

[0021] In some preferred embodiments of the present invention, the step of calculating the personality vector includes:

[0022] The image of the input personality channel branch is subjected to maximum pooling in the channel direction to obtain the personality matrix and maximum pooling in the width and height direction to obtain the personality vector;

[0023] Perform matrix multiplication on the personality matrix and the image of the input personality channel branch to obtain the first personality feature map;

[0024] Perform matrix multiplication of the personality vector and the image of the input personality channel branch to obtain a second personality feature map;

[0025] Add the first sex characteristic graph and the second sex characteristic graph point by point to obtain a sex characteristic graph.

[0026] In some preferred embodiments of the present invention, the step of calculating the commonality vector includes:

[0027] The image of the input common channel branch is mean pooled in the channel direction to obtain the commonality matrix and mean pooled in the width and height direction to obtain the commonality vector;

[0028] Perform matrix multiplication on the commonality matrix and the image of the input individual channel branch to obtain a first commonality feature map;

[0029] Perform matrix multiplication on the commonality vector and the image of the input individuality channel branch to obtain a second commonality feature map;

[0030] The first common feature map and the second common feature map are added point by point to obtain a common feature map.

[0031] In some preferred embodiments of the present invention, the detector includes: a classification subnetwork and a regression subnetwork arranged in parallel;

[0032] The classification subnetwork includes multiple convolutional layers, and the multiple convolutional layers of the classification subnetwork share the remaining parameters except the final output dimension; the regression subnetwork includes multiple convolutional layers, and the multiple convolutional layers of the regression subnetwork share the remaining parameters except the final output dimension.

[0033] In a second aspect, the present invention provides a drainage network diagnostic device, comprising:

[0034] A preprocessing module is used to obtain the image to be detected of the water supply network and preprocess the image to be detected; wherein the preprocessing includes: resolution standardization processing and / or pixel value normalization processing;

[0035] A feature extraction module is used to extract features from the preprocessed image to be detected based on the feature extraction unit to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2;

[0036] A feature fusion module is used to fuse the features of the N-dimensional scale map to obtain a fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear interpolation upsampling, convolution calculation downsampling or identity mapping;

[0037] A feature enhancement module is used to perform feature enhancement on the fused N-dimensional scale feature map based on the feature enhancement unit to obtain a final feature map; wherein the feature enhancement unit includes: an individual channel branch and a common channel branch;

[0038] The diagnosis module is used to input the final feature map into the detector and output the defect classification and defect location of the image to be detected.

[0039] In a third aspect, the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the drainage network diagnostic method of the first aspect.

[0040] In a fourth aspect, the present invention provides a computer storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the drainage network diagnostic method of the first aspect.

[0041] The present invention brings the following beneficial effects:

[0042] The present invention provides a drainage network diagnosis method, device, electronic device and computer storage medium, including: obtaining an image to be detected of a water supply network, and preprocessing the image to be detected; wherein the preprocessing includes: resolution standardization processing and / or pixel value normalization processing; based on a feature extraction unit, feature extraction is performed on the preprocessed image to be detected to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2; feature fusion is performed on the features of the N-dimensional scale map to obtain a fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear Interpolation upsampling, convolution downsampling or identity mapping; based on the feature enhancement unit, the fused N-dimensional scale feature map is enhanced to obtain the final feature map; wherein the feature enhancement unit includes: individual channel branches and common channel branches; the final feature map is input into the detector, and the defect classification and defect position of the image to be detected are output; in the image recognition process, the semantic features of different levels are fused by fully connected mapping, and the individual channel branches are designed to mine the dependency relationship between scenes, and the common channel branches are designed to establish a common spatial information association to enhance the feature expression, thereby reducing the manual workload and improving the accuracy of defect diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of a drainage network diagnosis method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a fully connected fusion process of an image feature pyramid provided by an embodiment of the present invention;

[0046] Figure 3 A network structure diagram of an image feature enhancement unit provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a personalized channel branch network structure provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a common channel branch network structure provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of a detector network structure provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the structure of a drainage network diagnostic device provided by an embodiment of the present invention;

[0051] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0052] Icon: 310 - preprocessing module; 320 - feature extraction module; 330 - feature fusion module; 340 - feature enhancement module; 350 - diagnosis module; 400 - memory; 401 - processor; 402 - bus; 403 - communication interface. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0056] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0057] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0058] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] The drainage network system is an important part of urban infrastructure, and undertakes key tasks such as ensuring the daily life of residents, flood prevention and drainage, and sewage treatment. With the acceleration of urbanization, the maintenance and management of the drainage network has become increasingly important. However, the aging and damage of the pipes will have a serious impact on the normal operation of the city, so regular inspection and maintenance of the drainage network is very necessary. At present, there are mainly two solutions for diagnosing drainage network defects:

[0060] Manual diagnosis method: relies on manual analysis of the images recorded by the robot inside the pipeline, and professionals judge the damage and health of the pipeline. Since all video images need to be diagnosed and analyzed frame by frame, this process is detailed but often consumes a lot of manpower, has a large workload, and is inefficient.

[0061] Traditional target detection and diagnosis methods: computer vision technology is introduced to analyze images recorded by the robot using detection algorithms. Although this process can replace manual review and improve the efficiency of defect diagnosis, the traditional general target detection framework mostly uses deeper neural networks and fixed anchor point ratios, which can easily lead to the gradual dilution of feature information in the long network cascade transmission, making it difficult to cope with the diverse complexity of pipeline network defect diagnosis scenarios, resulting in low accuracy in actual defect diagnosis.

[0062] Based on this, the purpose of the present invention is to provide a drainage network diagnosis method, device, electronic device and computer storage medium, which utilizes fully connected mapping to fuse semantic features of different levels during image recognition, designs individual channel branches to mine the dependencies between scenes, designs common channel branches to establish common spatial information associations and thus enhance feature expression, thereby reducing manual workload and improving the accuracy of defect diagnosis.

[0063] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0064] Embodiment 1

[0065] The present invention provides a drainage network diagnosis method. Figure 1 The flowchart of a drainage network diagnosis method provided by an embodiment of the present invention is shown, and the method includes:

[0066] Step S102, obtaining the image to be detected of the water supply network, and preprocessing the image to be detected; wherein the preprocessing includes: resolution normalization processing and / or pixel value normalization processing.

[0067] Specifically, we first need to collect image data of the water supply network. These images can be obtained through closed-circuit television (CCTV) cameras or other types of pipeline inspection equipment. The acquired images may come from different times and environmental conditions, so their quality and resolution may vary. In order to ensure the accuracy of subsequent processing, these images need to be preprocessed.

[0068] Resolution normalization: Adjust all images to a uniform resolution so that subsequent processing can be performed under the same standard. This usually involves scaling operations so that each image has the same size. In some preferred embodiments of the present invention, the size of the image after resolution normalization is fixed at 600×600.

[0069] Pixel value normalization: Normalize the pixel value range of the image to a standard range (such as 0 to 1 or 0 to 255) to eliminate the impact of illumination changes and improve the robustness of the algorithm. In some preferred embodiments of the present invention, the pixel value size will be compressed to a closed interval [0,1].

[0070] In some preferred embodiments of the present invention, pixel value normalization is performed using the following formula:

[0071] ;

[0072] in, is the original pixel value, is the minimum pixel value, is the maximum pixel value, is the normalized pixel value.

[0073] Step S104, based on the feature extraction unit, feature extraction is performed on the preprocessed image to be detected to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2.

[0074] Specifically, feature extraction is a core step in image processing, and its purpose is to extract useful information from the preprocessed image for subsequent analysis. In this embodiment, feature extraction is performed through a feature space containing N levels. Each level corresponds to a different scale and can capture different features of the image from coarse to fine. For example, a lower level may capture larger structural features, while a higher level can capture finer details. This multi-scale approach helps to improve the accuracy of defect detection.

[0075] Step S106, performing feature fusion on the N-dimensional scale map features to obtain a fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear interpolation upsampling, convolution calculation downsampling or identity mapping.

[0076] Specifically, feature fusion combines feature maps of different scales to form a more comprehensive and rich feature representation. In this process, a variety of sampling operations can be used to ensure that features of different scales can be effectively combined.

[0077] Bilinear interpolation upsampling: This is a commonly used image scaling method that can change the size of an image while maintaining its quality. Here, it can be used to scale up the feature maps at a coarser scale so that they are aligned with the feature maps at a finer scale.

[0078] Convolution downsampling: The size of the feature map is reduced through convolution operations, which can shrink the feature maps of finer scales so that they can be combined with feature maps of other scales.

[0079] Identity mapping: In some cases, it may be necessary to keep feature maps of certain scales unchanged. In this case, the identity mapping can be used to directly pass these feature maps to the next step.

[0080] Step S108, based on the feature enhancement unit, feature enhancement is performed on the fused N-dimensional scale feature map to obtain a final feature map; wherein the feature enhancement unit includes: an individual channel branch and a common channel branch.

[0081] Specifically, feature enhancement is to further process the fused feature map to improve the accuracy and robustness of defect detection. In this embodiment, the feature enhancement unit includes two branches: an individual channel branch and a common channel branch.

[0082] Individual channel branches: Focus on enhancing features that are particularly important for a specific defect type. For example, if a defect often appears as a specific texture or shape, this branch will enhance those features. Common channel branches: Aim to enhance features that are generally important for multiple defect types. This helps improve the model's generalization ability for different types of defects.

[0083] Step S110, input the final feature map into the detector, and output the defect classification and defect position of the image to be detected.

[0084] Specifically, the last step is to input the enhanced feature map into a detector, which is responsible for identifying and classifying defects in the image. The output of the detector includes the type of each defect and their location in the image. This information is very useful for maintenance personnel because it can directly guide the focus and priority of maintenance work.

[0085] Exemplarily, common defect categories include: cracks, misalignments, deformations, etc.; the defect categories are usually marked on defective output images to achieve defect identification and positioning.

[0086] The present invention provides a drainage network diagnosis method, comprising: obtaining an image to be detected of a water supply network, and preprocessing the image to be detected; wherein the preprocessing comprises: resolution standardization processing and / or pixel value normalization processing; based on a feature extraction unit, feature extraction is performed on the preprocessed image to be detected to obtain an N-dimensional scale feature map; wherein the feature extraction unit comprises: a feature space of N levels; N is a positive integer greater than 2; feature fusion is performed on the features of the N-dimensional scale map to obtain a fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process comprises at least one of the following: bilinear interpolation upsampling, convolution Calculate downsampling or identity mapping; based on the feature enhancement unit, perform feature enhancement on the fused N-dimensional scale feature map to obtain the final feature map; wherein the feature enhancement unit includes: individual channel branches and common channel branches; input the final feature map into the detector, and output the defect classification and defect position of the image to be detected; in the image recognition process, use the fully connected mapping to fuse the semantic features of different levels, design the individual channel branches to mine the dependency between scenes, design the common channel branches to establish the common spatial information association and then enhance the feature expression, thereby reducing the manual workload and improving the accuracy of defect diagnosis.

[0087] A drainage network diagnosis method provided by an embodiment of the present invention takes the CNN cascade feature pyramid as the entry point, uses fully connected mapping to fuse semantic features at different levels, and then designs a multi-scale feature enhancement channel suitable for the network defect diagnosis scenario: the individuality channel branch (ICB) can mine the dependencies between scenes and highlight the individual description of the feature map, and the generality channel branch (GCB) can encode the features of each level of the image, establish a common spatial information association, and enhance the feature expression. The combination of the two can largely solve the drawbacks of directly using traditional general target detection methods for defect diagnosis, and has better diagnostic and recognition effects when dealing with complex phenomena such as overlap, truncation, and occlusion between defects.

[0088] Embodiment 2

[0089] On the basis of the above-mentioned embodiments, the embodiments of the present invention provide another drainage network diagnosis method. A defect detector specifically for the pipeline network defect diagnosis scenario is designed. Considering that the high-level semantics in the image feature space are helpful for learning category information, but the large receptive field and small resolution are not conducive to positioning, while the bottom-level semantics are helpful for learning location information, but the semantics are too weak to be conducive to model classification, this embodiment uses a fully connected mapping to fuse the features generated by the CNN cascade, realizes information exchange between different directions and levels, and makes up for the macro and micro information that is continuously diluted in the fusion path. On this basis, this embodiment also introduces a multi-scale feature enhancement channel to strengthen the individual characteristics between defects of different categories and the common characteristics between defects of the same category, realizes the final labeling of the pipeline network defect category and location information, and achieves better accuracy in complex and diverse detection scenarios such as overlap, truncation, and occlusion between defects.

[0090] In some preferred embodiments of the present invention, the feature scales of the N levels of feature spaces in the feature extraction unit decrease successively; and the channel dimensions of the N levels of feature spaces in the feature extraction unit are all the same.

[0091] Specifically, the normalized image is passed to CNN (Convolutional Neural Networks) to construct a multi-level feature space. Assume that there are 5 levels of feature maps in the feature space. For example, the feature scales corresponding to each level feature map in this space are , and the channel dimension of each feature map is 256.

[0092] Further, in some preferred embodiments of the present invention, the N-dimensional scale feature map includes: N feature maps; the step of fusing the N-dimensional scale feature map to obtain the fused N-dimensional scale feature map includes: segmenting each feature map, and each feature map after segmentation corresponds to M segmentation feature maps; wherein, ; M is a positive integer greater than 1; the segmentation feature map is sampled according to a preset feature scale to generate a generated feature map corresponding to M feature scales except the feature scale corresponding to its own feature image; if the feature scale of the generated feature map is larger than the feature scale of the segmentation feature map, the sampling operation is bilinear interpolation upsampling; if the feature scale of the generated feature map is smaller than the feature scale of the segmentation feature map, the sampling operation is convolution calculation downsampling; the generated feature map is fully connected and fused with the feature map subjected to identity mapping to obtain a fused N-dimensional scale feature map.

[0093] Specifically, for the above multi-scale features Perform fully connected feature fusion to obtain fused multi-scale features This process strictly follows the channel segmentation and splicing fusion method, which mainly involves three calculations: bilinear interpolation upsampling, downsampling with a convolution kernel size of 3 and a convolution step of 2, and the network identity mapping of f(x)=x.

[0094] See also Figure 2 The schematic diagram of a fully connected fusion process of an image feature pyramid provided by an embodiment of the present invention is shown in FIG. Each scale feature map in the set is split by channel to obtain four scales: The feature map of the segmentation calculation formula is as follows:

[0095] ;

[0096] in, They are all segmentation feature maps. For split operation.

[0097] The four segmented feature maps are upsampled by bilinear interpolation at different scales and downsampled with a convolution kernel size of 3 and a convolution step size of 2 to generate four feature level scales other than their own scale.

[0098] By splicing the feature maps according to the channel dimension, the four feature maps after up and down sampling are combined with the original feature maps obtained by identity mapping. Perform fully connected fusion to construct fusion features rich in semantics at all levels , and the feature map scales are , where W and H belong to the set , for example in and In, W and H are both 75; in and In the example, W and H are both 38. The splicing calculation formula is as follows:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, For conduct The result of upsampling by bilinear interpolation is For conduct The convolution is used to calculate the down-sampling result; For connection operation.

[0105] Furthermore, in some preferred embodiments of the present invention, the step of performing feature enhancement on the fused N-dimensional scale feature map based on the feature enhancement unit to obtain the final feature map includes: dividing the fused feature map into two branches with the same channels, and performing vector calculations on each branch; wherein one branch performs individual vector calculations and the other branch performs common vector calculations; concatenating the results of the two vector calculations and adding them point by point to the fused feature map; and convolving the result of the point-by-point addition to obtain the final feature map.

[0106] For details, see Figure 3 The network structure diagram of an image feature enhancement unit provided by an embodiment of the present invention is shown in FIG. The multi-scale feature enhancement channel is input, which designs two parallel network branches, namely the individuality channel branch (ICB) and the generality channel branch (GCB). The former can mine the dependencies between scenes and highlight the individual description of the pipeline network defects, while the latter can encode the multi-scale information of the image and establish the common spatial information association between the defects. The feature map finally output by the multi-scale feature enhancement channel is recorded as , its scale is still , where W and H belong to the set .

[0107] Furthermore, in some preferred embodiments of the present invention, the step of calculating the personality vector includes: performing maximum pooling in the channel direction on the image of the input personality channel branch to obtain a personality matrix and performing maximum pooling in the width and height directions to obtain a personality vector; performing matrix multiplication on the personality matrix and the image of the input personality channel branch to obtain a first personality feature map; performing matrix multiplication on the personality vector and the image of the input personality channel branch to obtain a second personality feature map; and adding the first personality feature map and the second personality feature map point by point to obtain a personality feature map.

[0108] For details, see Figure 4The schematic diagram of a personalized channel branch network structure provided by an embodiment of the present invention is shown. The input of ICB comes from half of the feature map of each level after full connection fusion, which is divided by channel. Its scale is W×H×256, followed by two parallel branches for calculating the personality matrix of the original feature in the channel direction and the personality vector in the width and height direction, using the maximum pooling method. One is the maximum pooling in the channel direction, and then the sigmoid function is activated to obtain the personality matrix; one channel is the maximum pooling in the width and height direction, and then two full connection operations and two sigmoid function activations are performed in succession to obtain the personality vector; finally, the calculated personality matrix and personality vector are multiplied with the original feature respectively, and then added point by point to obtain the final personality feature map with a scale of W×H×256.

[0109] Furthermore, in some preferred embodiments of the present invention, the step of calculating the commonality vector includes: performing mean pooling on the image of the input common channel branch in the channel direction to obtain a commonality matrix and performing mean pooling on the image in the width and height directions to obtain a commonality vector; performing matrix multiplication on the commonality matrix and the image of the input individual channel branch to obtain a first commonality feature map; performing matrix multiplication on the commonality vector and the image of the input individual channel branch to obtain a second commonality feature map; and adding the first commonality feature map and the second commonality feature map point by point to obtain a commonality feature map.

[0110] For details, see Figure 5 As shown in the figure, a schematic diagram of a common channel branch network structure provided by an embodiment of the present invention, the network structure of GCB is similar to that of ICB, except that mean pooling technology is used instead of maximum pooling technology. One is mean pooling in the channel direction, and then sigmoid function activation is performed to obtain a commonality matrix; one channel is mean pooling in the width and height directions, and then two full connection operations and two sigmoid function activations are performed continuously to obtain a commonality vector; finally, the commonality matrix in the channel direction and the commonality vector in the width and height directions are multiplied by the original features and then added point by point, to obtain a commonality feature map with a scale of W×H×256.

[0111] Further, in some preferred embodiments of the present invention, the detector includes: a classification subnetwork and a regression subnetwork arranged in parallel;

[0112] The classification subnetwork includes multiple convolutional layers, and the multiple convolutional layers of the classification subnetwork share the remaining parameters except the final output dimension; the regression subnetwork includes multiple convolutional layers, and the multiple convolutional layers of the regression subnetwork share the remaining parameters except the final output dimension.

[0113] For details, see Figure 6 The detector network structure diagram of the embodiment of the present invention is shown in FIG. 1 , wherein the detector adopts a one-stage direct regression detection method, and the output of the multi-scale feature enhancement unit of the CNN network is Add after (Classification subnetwork) and (Regression subnetwork) Two parallel subnetworks perform final defect classification and location. The head parameters of the two subnetworks are shared. Each layer consists of 4 convolution kernels with a size of 3 and The convolutional layers are composed of nonlinear activation functions. In each sub-network, the structures are exactly the same except for the output dimension of the last layer. In the network structure of the detector, A represents the number of anchor boxes, and K represents the total number of defect categories to be detected.

[0114] Furthermore, in some preferred embodiments of the present invention, the classification subnetwork and the regression subnetwork share the remaining parameters except the final output dimension.

[0115] Since the traditional drainage network defect diagnosis and identification scenes often involve complex and diverse situations such as truncation, occlusion, and overlap, it is inappropriate to directly copy the traditional general target detection algorithm that relies on CNN to this scene. A drainage network diagnosis method provided by an embodiment of the present invention optimizes the traditional detection framework, and uses a fully connected feature fusion method to compensate for the image semantics that are gradually lost during the CNN cascade process, enriching the feature expression. Secondly, a multi-scale feature enhancement channel is designed to enhance the individual features and common features between network defects, so as to achieve the purpose of being more sensitive to the category location information of potential targets, thereby making the detector more robust when diagnosing defects in complex scenes.

[0116] Embodiment 3

[0117] Based on the above embodiments, the present invention provides a drainage network diagnosis device. Figure 7 The structural diagram of a drainage network diagnostic device provided by an embodiment of the present invention is shown, and the device includes:

[0118] The preprocessing module 310 is used to obtain the image to be detected of the water supply network and perform preprocessing on the image to be detected; wherein the preprocessing includes: resolution standardization processing and / or pixel value normalization processing.

[0119] The feature extraction module 320 is used to extract features from the preprocessed image to be detected based on the feature extraction unit to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2.

[0120] The feature fusion module 330 is used to fuse the N-dimensional scale map features to obtain a fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear interpolation upsampling, convolution calculation downsampling or identity mapping.

[0121] The feature enhancement module 340 is used to perform feature enhancement on the fused N-dimensional scale feature map based on the feature enhancement unit to obtain a final feature map; wherein the feature enhancement unit includes: an individual channel branch and a common channel branch.

[0122] The diagnosis module 350 is used to input the final feature map into the detector and output the defect classification and defect location of the image to be detected.

[0123] Furthermore, in some preferred embodiments of the present invention, the feature scales of the N levels of feature spaces in the feature extraction unit decrease successively; and the channel dimensions of the N levels of feature spaces in the feature extraction unit are all the same.

[0124] Further, in some preferred embodiments of the present invention, the N-dimensional scale feature map includes: N feature maps; a feature fusion module 330, which is used to segment each feature map, and each feature map corresponds to M segmentation feature maps after segmentation; wherein, ; M is a positive integer greater than 1; the segmentation feature map is sampled according to a preset feature scale to generate a generated feature map corresponding to M feature scales except the feature scale corresponding to its own feature image; if the feature scale of the generated feature map is larger than the feature scale of the segmentation feature map, the sampling operation is bilinear interpolation upsampling; if the feature scale of the generated feature map is smaller than the feature scale of the segmentation feature map, the sampling operation is convolution calculation downsampling; the generated feature map is fully connected and fused with the feature map subjected to identity mapping to obtain a fused N-dimensional scale feature map.

[0125] Furthermore, in some preferred embodiments of the present invention, the feature enhancement module 340 is used to divide the fused feature map into two branches with the same channels, and perform vector calculations separately; wherein, one branch performs individual vector calculations, and the other branch performs common vector calculations; the results of the two vector calculations are concatenated and added point by point to the fused feature map; the result of the point-by-point addition is convolved to obtain the final feature map.

[0126] Furthermore, in some preferred embodiments of the present invention, the feature enhancement module 340 is used to perform maximum pooling in the channel direction on the image of the input personality channel branch to obtain a personality matrix and to perform maximum pooling in the width and height directions to obtain a personality vector; perform matrix multiplication on the personality matrix and the image of the input personality channel branch to obtain a first personality feature map; perform matrix multiplication on the personality vector and the image of the input personality channel branch to obtain a second personality feature map; and add the first personality feature map and the second personality feature map point by point to obtain a personality feature map.

[0127] Furthermore, in some preferred embodiments of the present invention, the feature enhancement module 340 is used to perform mean pooling on the image of the input common channel branch in the channel direction to obtain a common matrix and to perform mean pooling on the image of the width and height direction to obtain a common vector; perform matrix multiplication on the common matrix and the image of the input individual channel branch to obtain a first common feature map; perform matrix multiplication on the common vector and the image of the input individual channel branch to obtain a second common feature map; and add the first common feature map and the second common feature map point by point to obtain a common feature map.

[0128] Furthermore, in some preferred embodiments of the present invention, the detector includes: a classification subnetwork and a regression subnetwork arranged in parallel; the classification subnetwork includes multiple layers of convolutional layers, and the multiple layers of convolutional layers of the classification subnetwork share the remaining parameters except the final output dimension; the regression subnetwork includes multiple layers of convolutional layers, and the multiple layers of convolutional layers of the regression subnetwork share the remaining parameters except the final output dimension.

[0129] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the drainage network diagnostic device described above can refer to the corresponding process in the aforementioned embodiment of the drainage network diagnostic method, and will not be repeated here.

[0130] Embodiment 4

[0131] The embodiment of the present invention also provides an electronic device for running a drainage network diagnosis method; see Figure 8 The structural schematic diagram of an electronic device provided by an embodiment of the present invention is shown, and the electronic device includes a memory 400 and a processor 401, wherein the memory 400 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 401 to implement the above-mentioned drainage network diagnosis method.

[0132] Further, Figure 8 The electronic device shown further includes a bus 402 and a communication interface 403 , and the processor 401 , the communication interface 403 and the memory 400 are connected via the bus 402 .

[0133] The memory 400 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0134] The processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 401. The above processor 401 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 400, and the processor 401 reads the information in the memory 400 and completes the steps of the method of the above embodiment in combination with its hardware.

[0135] An embodiment of the present invention further provides a computer storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the above-mentioned business recommendation method. The specific implementation can be found in the method embodiment and will not be repeated here.

[0136] The computer program products of the drainage network diagnostic method, device and electronic device provided in the embodiments of the present invention include a computer storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0138] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0139] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A drainage network diagnosis method, characterized in that: include: Acquire the image to be detected of the water supply network, and preprocess the image to be detected; The preprocessing includes: resolution standardization processing and / or pixel value normalization processing; Based on the feature extraction unit, the preprocessed image to be detected is subjected to feature extraction to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2; the N-dimensional scale feature map includes: N feature maps; The N-dimensional scale map features are subjected to feature fusion to obtain the fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear interpolation upsampling, convolution calculation downsampling or identity mapping; Based on the feature enhancement unit, the fused N-dimensional scale feature map is feature enhanced to obtain a final feature map; wherein the feature enhancement unit includes: an individual channel branch and a common channel branch; Inputting the final feature map into a detector, and outputting the defect classification and defect location of the image to be detected; The step of performing feature enhancement on the fused N-dimensional scale feature map based on the feature enhancement unit to obtain a final feature map includes: The fused feature map is divided into two branches with the same channels, and vector calculations are performed respectively; one branch performs individual vector calculations, and the other branch performs common vector calculations; The results of the two vector calculations are concatenated and added point by point to the fused feature map; Convolving the result of point-by-point addition to obtain the final feature map; The step of calculating the personality vector includes: The image input to the personality channel branch is subjected to maximum pooling in the channel direction to obtain a personality matrix and maximum pooling in the width and height direction to obtain a personality vector; Performing matrix multiplication on the personality matrix and the image input into the personality channel branch to obtain a first personality feature map; Performing matrix multiplication on the personality vector and the image input into the personality channel branch to obtain a second personality feature map; Adding the first personality characteristic graph and the second personality characteristic graph point by point to obtain a personality characteristic graph; The step of calculating the commonality vector comprises: The image input into the common channel branch is subjected to mean pooling in the channel direction to obtain a commonality matrix and mean pooling in the width and height directions to obtain a commonality vector; Perform matrix multiplication on the commonality matrix and the image input into the individual channel branch to obtain a first commonality feature map; Performing matrix multiplication on the commonality vector and the image input into the individual channel branch to obtain a second commonality feature map; The first common feature map and the second common feature map are added point by point to obtain a common feature map.

2. The drainage network diagnosis method according to claim 1, characterized in that: The feature scales of the N levels of feature spaces in the feature extraction unit decrease in sequence; and the channel dimensions of the N levels of feature spaces in the feature extraction unit are all the same.

3. The drainage network diagnosis method according to claim 2, characterized in that: The N-dimensional scale feature map includes: N feature maps; The step of fusing the N-dimensional scale feature map to obtain the fused N-dimensional scale feature map comprises: Each of the feature maps is segmented, and after segmentation, each of the feature maps corresponds to M segmentation feature maps; wherein, ; M is a positive integer greater than 1; The segmentation feature map is sampled according to a preset feature scale to generate a generated feature map corresponding to M feature scales except the feature scale corresponding to the own feature image; wherein, if the feature scale of the generated feature map is larger than the feature scale of the segmentation feature map, the sampling operation is the bilinear interpolation upsampling; if the feature scale of the generated feature map is smaller than the feature scale of the segmentation feature map, the sampling operation is the convolution calculation downsampling; The generated feature map is fully connected and fused with the feature map after the identity mapping to obtain the fused N-dimensional scale feature map.

4. The drainage network diagnosis method according to claim 1, characterized in that: The detector comprises: a classification subnetwork and a regression subnetwork arranged in parallel; The classification subnetwork includes multiple convolutional layers, and the multiple convolutional layers of the classification subnetwork share the remaining parameters except the final output dimension; The regression sub-network includes multiple layers of convolutional layers, and the multiple layers of convolutional layers of the regression sub-network share the remaining parameters except the final output dimension.

5. A drainage network diagnostic device, characterized in that: include: A preprocessing module, used to obtain the image to be detected of the water supply network and preprocess the image to be detected; The preprocessing includes: resolution standardization processing and / or pixel value normalization processing; A feature extraction module, used for extracting features from the preprocessed image to be detected based on a feature extraction unit to obtain an N-dimensional scale feature map; wherein the feature extraction unit includes: a feature space of N levels; N is a positive integer greater than 2; the N-dimensional scale feature map includes: N feature maps; A feature fusion module is used to perform feature fusion on the N-dimensional scale map features to obtain the fused N-dimensional scale feature map; wherein the sampling operation in the feature fusion process includes at least one of the following: bilinear interpolation upsampling, convolution calculation downsampling or identity mapping; A feature enhancement module, used for performing feature enhancement on the fused N-dimensional scale feature map based on a feature enhancement unit to obtain a final feature map; wherein the feature enhancement unit includes: an individual channel branch and a common channel branch; A diagnosis module, used for inputting the final feature map into a detector and outputting the defect classification and defect location of the image to be detected; A feature enhancement module is used to divide the fused feature map into two branches with the same channels, and perform vector calculations on each branch; one branch performs individual vector calculations, and the other branch performs common vector calculations; the results of the two vector calculations are concatenated and added point by point to the fused feature map; the result of the point-by-point addition is convolved to obtain the final feature map; The feature enhancement module is used to calculate the personality vector, and the steps of calculating the personality vector include: performing maximum pooling in the channel direction on the image input to the personality channel branch to obtain a personality matrix and a maximum pooling in the width and height direction to obtain a personality vector; performing matrix multiplication on the personality matrix and the image input to the personality channel branch to obtain a first personality feature map; performing matrix multiplication on the personality vector and the image input to the personality channel branch to obtain a second personality feature map; adding the first personality feature map and the second personality feature map point by point to obtain a personality feature map; The feature enhancement module is used to calculate the common vector, and the steps of calculating the common vector include: performing mean pooling in the channel direction on the image input to the common channel branch to obtain a common matrix and performing mean pooling in the width and height directions to obtain a common vector; performing matrix multiplication on the common matrix and the image input to the individual channel branch to obtain a first common feature map; performing matrix multiplication on the common vector and the image input to the individual channel branch to obtain a second common feature map; and adding the first common feature map and the second common feature map point by point to obtain a common feature map.

6. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the drainage network diagnosis method described in any one of claims 1 to 4 above.

7. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions. When the computer executable instructions are called and executed by the processor, the computer executable instructions prompt the processor to implement the drainage network diagnosis method according to any one of claims 1 to 4.

Images