Method and device for detecting dike danger based on thermal infrared image segmentation

Abstract

The embodiment of the present invention discloses a method and device for detecting dike danger based on thermal infrared image segmentation. The method includes: obtaining an image to be tested that includes the dike facilities to be tested, and the image to be tested is a thermal infrared image; inputting the image to be tested into an image segmentation network to determine the location of the dangerous area in the image to be tested, and the dangerous area is the area where the dangerous situation exists in the dike facilities to be tested; the image segmentation network includes a feature extraction module and an area determination module; in the feature extraction module, at each target network node, feature values ​​of at least one level of the image to be tested are received by jump connection to perform feature fusion to obtain at least one original feature map, and the feature values ​​of at least one level include: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level and adjacent levels as the current network node. The present invention can avoid safety risk issues.

Description

Technical Field

[0001] The present invention relates to the technical field of levee engineering, and more specifically to a levee danger detection method and device based on thermal infrared image segmentation. Background Art

[0002] Levee projects are used to control water flow, prevent floods, and regulate and distribute water. Once high water levels occur during the flood season, dangerous situations will occur, which often catches people off guard and results in a passive situation in flood control. Therefore, it is necessary to monitor, inspect, and troubleshoot levees. At present, due to the lack of effective, reliable, flexible, and non-contact levee project monitoring technology and related suspected dangerous situation detection algorithms, manual network inspections are still required when the flood control situation is severe. The environment around levee projects is complex, and it is difficult for levee inspectors to detect potential dangerous situations such as small cracks and water seepage in time, resulting in low inspection efficiency and difficulty in detecting dangerous situations in a timely and accurate manner. Especially in the event of floods or special circumstances, it is necessary to increase the intensity and frequency of levee inspections, which greatly increases the safety risks of levee inspectors. Summary of the invention

[0003] The present invention is proposed in consideration of the above problems. According to one aspect of the present invention, a method for detecting dike danger based on thermal infrared image segmentation is provided, comprising: obtaining an image to be tested containing dike facilities to be tested, the image to be tested being a thermal infrared image; inputting the image to be tested into an image segmentation network to determine the location of the danger area in the image to be tested, the danger area being the area where danger exists in the dike facilities to be tested; wherein the image segmentation network comprises a feature extraction module and an area determination module, the feature extraction module is used to extract at least one original feature map of the image to be tested, and the area determination module is used to determine the danger area based on at least one original feature map; in the feature extraction module, at each target network node in at least one target network node, feature values ​​of at least one level of the image to be tested are received by means of a jump connection for feature fusion to obtain at least one original feature map, the feature values ​​of at least one level comprising: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level as the current network node and at an adjacent level.

[0004] Furthermore, the feature extraction module includes at least one layer of sub-network, the network nodes of each layer of sub-network include a head node and a non-head node, and at least one target network node is a non-head node in at least one layer of sub-network; the image to be tested is input into the image segmentation network to determine the location of the dangerous area in the image to be tested, including: in the feature extraction module, for each layer of sub-network, the feature value extracted at the previous node adjacent to the head node is obtained through the head node of the sub-network of the layer; through the non-head node of the sub-network of the layer, a first feature value is obtained from at least part of the nodes in the same sub-network as the non-head node and located before the non-head node, a second feature value is obtained from at least one node of at least one layer of sub-network adjacent to the sub-network where the non-head node is located, and the first feature value is obtained from the first node of the sub-network of the layer. The first feature value and the second feature value are fused to obtain at least one original feature map; in the area determination module, at least one original feature map is fused to obtain a global feature map; the global feature map is pixel clustered to obtain a dangerous area; wherein, the previous node adjacent to the first node includes the first node of the previous layer of sub-network before the sub-network where the first node is located, and at least one node of at least one layer of sub-network adjacent to the sub-network where the non-first node is located includes: a node in the next adjacent sub-network of the sub-network where the non-first node is located and in the j-1th convolutional layer, and / or a node in the previous adjacent sub-network of the sub-network where the non-first node is located and in the jth convolutional layer, wherein j represents the sequence number of the convolutional layer where the non-first node is located in the corresponding sub-network.

[0005] Furthermore, the first eigenvalue and the second eigenvalue are fused, including: assigning weights of channel attention and spatial attention to the first eigenvalue and the second eigenvalue, and fusing them based on the assigned weights.

[0006] Furthermore, pixel clustering is performed on the global feature map to obtain the dangerous area, including: for every two pixels in the global feature map, whether the two pixels are adjacent pixels is determined based on whether there is an overlapping area in the neighborhood areas corresponding to the two pixels, wherein if there is an overlapping area, the two pixels are determined to be adjacent pixels, and the neighborhood area corresponding to any pixel is a region including eight neighboring pixels of the pixel; if the two pixels are adjacent pixels, whether the two pixels belong to the same connected domain is determined based on the grayscale values ​​of the two pixels; and the connected domain determined based on all the pixels in the global feature map is used as the dangerous area.

[0007] Furthermore, an image to be tested including the levee facilities to be tested is obtained, including: obtaining an initial thermal infrared image including the levee and water conservancy facilities to be tested acquired by an infrared camera and a point cloud image including the levee and water conservancy facilities to be tested acquired by a lidar; performing image fusion on the initial thermal infrared image and the point cloud image to obtain the image to be tested; wherein the channel of the image to be tested includes a color value channel and an intensity channel, the color value channel includes a color value corresponding to each pixel point of the image to be tested determined based on the initial thermal infrared image, and the intensity channel includes an echo intensity value corresponding to each pixel point of the image to be tested determined based on the point cloud image.

[0008] Furthermore, before fusing the initial thermal infrared image and the point cloud image to obtain the image to be tested, the method further includes: contrast enhancement of the initial thermal infrared image.

[0009] Furthermore, the initial thermal infrared image is contrast enhanced by the following formula:

[0010]

[0011] Where I(x,y) stretched represents the contrast-enhanced pixel value of the pixel with coordinates (x, y) in the initial thermal infrared image, I(x, y) represents the original pixel value of the pixel with coordinates (x, y) in the initial thermal infrared image, and I max Represents the maximum pixel value in the initial thermal infrared image, I min Indicates the minimum pixel value in the initial thermal infrared image.

[0012] Furthermore, the image to be tested includes an intensity channel, which includes an echo intensity value corresponding to each pixel point of the image to be tested, determined based on the point cloud image of the levee facility to be tested; the image to be tested is input into an image segmentation network to determine the position of the dangerous area in the image to be tested, including: inputting the image to be tested into the image segmentation network to obtain a dangerous area mask, the dangerous area mask is used to indicate the position of the dangerous area in the image to be tested; or, the method also includes: obtaining the dangerous area mask according to the position of the dangerous area; wherein each at least one original feature map corresponds to at least one dangerous area mask; the method also includes: calculating the danger weight of the dangerous area according to the area of ​​the dangerous area mask, the position information of the dangerous area mask and the corresponding echo intensity value in the dangerous area mask, the danger weight is used to indicate the degree of danger of the dangerous area.

[0013] Furthermore, the location information includes the distance between the area corresponding to the dangerous area mask and the historical dangerous area, and the danger weight is calculated by the following formula:

[0014]

[0015] Where W is the danger weight, S is the area of ​​the dangerous area mask, d is the distance between the area corresponding to the dangerous area mask and the historical dangerous area, and Avgi is the average value of the echo intensity value corresponding to each pixel in the dangerous area mask.

[0016] Furthermore, the image segmentation network is trained in the following manner: obtaining a sample image containing sample embankment facilities and corresponding annotation data, the annotation data being used to indicate the annotation position of the sample dangerous area in the sample image; inputting the sample image into the image segmentation network to determine the predicted position of the sample dangerous area in the sample image; calculating the loss value of the image segmentation network based on the annotation position and the predicted position to train the image segmentation network; wherein the method for obtaining the annotation data includes: using morphological gradient edge detection on the sample image, and filling the contour obtained by the edge detection to determine the annotation position of the sample dangerous area in the sample image.

[0017] According to another aspect of the present invention, there is also provided a levee danger detection device based on thermal infrared image segmentation, comprising: an acquisition module, which is used to acquire an image to be tested containing a levee facility to be tested; an input module, which is used to input the image to be tested into an image segmentation network to determine the location of a danger area in the image to be tested, the danger area being an area where danger exists in the levee facility to be tested; wherein the image segmentation network comprises a feature extraction module and an area determination module, the feature extraction module is used to extract at least one original feature map of the image to be tested, and the area determination module is used to determine the danger area based on at least one original feature map; in the feature extraction module, at each target network node in at least one target network node, feature values ​​of at least one level of the image to be tested are received by means of a jump connection for feature fusion to obtain at least one original feature map, and the feature values ​​of at least one level include: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level and at an adjacent level as the current network node.

[0018] According to another aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are run by the processor, they are used to execute the above-mentioned embankment hazard detection method based on thermal infrared image segmentation.

[0019] According to another aspect of the present invention, there is also provided a storage medium on which program instructions are stored, and the program instructions are used to execute the above-mentioned embankment danger detection method based on thermal infrared image segmentation when running.

[0020] According to another aspect of the present invention, there is also provided a computer program product, comprising computer program instructions, which are used to execute the above-mentioned embankment hazard detection method based on thermal infrared image segmentation when running.

[0021] According to the embankment hazard detection method based on thermal infrared image segmentation of an embodiment of the present invention, a embankment hazard detection method based on thermal infrared image segmentation can detect the image to be tested containing the embankment facilities to be tested based on an image segmentation network. This method is an intelligent and automated detection method that does not require manual inspections, has high inspection efficiency and accuracy, can detect dangerous situations in a timely and accurate manner, and can avoid safety risks in manual inspections.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 A schematic flow chart of a method for detecting dike danger based on thermal infrared image segmentation according to an embodiment of the present invention is shown;

[0025] Figure 2 A schematic diagram of a U-shaped sub-network according to an embodiment of the present invention is shown;

[0026] Figure 3 A schematic diagram showing a densely nested structure of a feature extraction module according to an embodiment of the present invention is shown;

[0027] Figure 4 A schematic diagram showing a feature extraction module according to an embodiment of the present invention;

[0028] Figure 5 A schematic diagram showing a feature extraction module according to an embodiment of the present invention;

[0029] Figure 6 A schematic diagram showing a dike danger detection device based on thermal infrared image segmentation according to an embodiment of the present invention;

[0030] Figure 7A schematic diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0032] The danger of dike engineering is usually caused by the pressure difference and permeable layer between the water levels on both sides of the dike. Generally speaking, the danger of dike engineering occurs on the back slope. However, the back slope has a complex environment and is usually covered with vegetation, trees, country roads, etc. Under severe weather conditions, such as rain, high-resolution visible light image detection and segmentation algorithms usually fail. Long-wave infrared rays can penetrate clouds and rain, are not affected by lighting conditions, and can work all day. Therefore, thermal infrared images containing dike facilities acquired based on long-wave infrared can still provide reliable information under severe weather conditions. The characteristics of thermal infrared images can reflect the hazards and urgency of dike engineering dangers. However, thermal infrared images usually do not have regularized features such as shape, size, position, texture, etc. Therefore, when performing intelligent detection of dangerous areas on thermal infrared images containing dike facilities, traditional target detection networks or image segmentation networks are difficult to achieve satisfactory results due to the lack of standardized data sets for training networks.

[0033] In the prior art, the dangerous conditions of levee projects are mainly detected through artificial networking, which has low inspection efficiency and is difficult to detect dangerous conditions in a timely and accurate manner. At the same time, there are safety risks for patrol personnel during the inspection process. The embodiment of the present invention provides a method for detecting dangerous conditions of levee projects based on an image segmentation network to detect images to be tested that include levee facilities to be tested. This method is an intelligent and automated detection method that does not require manual inspections, has high inspection efficiency and accuracy, can detect dangerous conditions in a timely and accurate manner, and can avoid safety risks in manual inspections.

[0034] See also Figure 1 As shown, it is a schematic flow chart of a method for detecting dike danger based on thermal infrared image segmentation according to an embodiment of the present invention. Figure 1 As shown, the method for detecting dike danger based on thermal infrared image segmentation includes step S110 and step S120.

[0035] Step S110, obtaining an image to be tested that includes the levee facilities to be tested, where the image to be tested is a thermal infrared image.

[0036] Exemplarily, the levee facilities to be tested may be facilities of levee projects, including but not limited to river levees, river levees, lake levees, sea levees, and levees in low-lying areas of reservoirs and flood storage areas. Optionally, the image to be tested may simply contain the levee facilities to be tested. For example, after collecting an initial image for the physical space where the levee facilities to be tested are located, the levee facilities to be tested may be detected from the initial image, and the image area where the levee facilities to be tested are located may be extracted from the initial image to form the image to be tested. For another example, after collecting an initial image for the physical space where the levee facilities to be tested are located, the pixel values ​​of the image area other than the levee facilities to be tested in the initial image may be set to a preset pixel value (e.g., 0), and the initial image after the setting may be used as the image to be tested. Optionally, the image to be tested may include other ground objects while containing the levee facilities to be tested, that is, various tangible objects on the ground other than the levee facilities, such as vegetation, trees, country roads, etc. The image to be tested can be a thermal infrared image, which can display the thermal infrared radiation information that is invisible to the human eye and is radiated by the ground objects and collected and recorded by the thermal infrared detection device (also called infrared camera). This thermal infrared radiation information can be used to identify the ground objects and reflect the surface parameters (such as temperature, emissivity, humidity, thermal inertia, etc.).

[0037] Step S120, input the image to be tested into the image segmentation network to determine the location of the dangerous area in the image to be tested, and the dangerous area is the area where there is danger in the levee facility to be tested. The image segmentation network includes a feature extraction module and a region determination module, the feature extraction module is used to extract at least one original feature map of the image to be tested, and the region determination module is used to determine the dangerous area based on at least one original feature map; in the feature extraction module, at each target network node in at least one target network node, feature values ​​of at least one level of the image to be tested are received by skip connection to perform feature fusion to obtain at least one original feature map, and the feature values ​​of at least one level include: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level and adjacent levels as the current network node.

[0038] Exemplarily, the feature extraction module in the image segmentation network can be implemented using a residual network (ResNet). The ResNet network constructs a network by introducing residual blocks, and adds the input directly to the layer output through a jump connection. Specifically, a standard residual block usually includes at least two convolutional layers and an identity mapping, which allows the input data to pass directly without going through all the convolutional layers. In a specific embodiment, a U-type paradigm ResNet network can be used to form a feature extraction module in the image segmentation network by stacking at least one U-type subnetwork (i.e., residual block). The image to be tested is input into the image segmentation network to perform regional segmentation on the image to be tested, and a dangerous area can be obtained. The dangerous situation may be, for example, a dangerous situation of a dike project, including water seepage, pipe bursts, leaks, etc.

[0039] For example, the feature extraction module can be a U-shaped sub-network (U-Net network), or can be formed by stacking multiple U-shaped sub-networks. Figure 2 As shown in FIG. 1 , it is a schematic diagram of a U-shaped subnetwork according to an embodiment of the present invention. The U-shaped subnetwork is composed of an encoder, a decoder, and a jump connection layer, wherein the jump connection layer is located between the downsampling path (encoder) and the upsampling path (decoder) of the network. In the downsampling path, the network extracts the features of the image by downsampling, and reduces the spatial dimension of the input to obtain the feature map of the input image, and then connects the feature to the corresponding layer in the upsampling path through the jump connection layer. Each layer in the upsampling path gradually enlarges the spatial dimension of the feature map and restores it to the size of the image to be tested. Please refer to Figure 3 As shown, it is a schematic diagram of a dense nested structure of a feature extraction module according to an embodiment of the present invention. The feature extraction module can be obtained by stacking multiple U-shaped sub-networks to construct a dense nested structure. Figure 3 In the feature extraction module, the dense nested structure is constructed by 4 layers of U-shaped sub-networks of different depths. The top-down U-shaped sub-networks increase in the feature extraction module. Each U-shaped sub-network can extract the features of objects of different sizes. The higher the level of the U-shaped sub-network, the higher the level of the U-shaped sub-network, which can obtain higher-level information and extract the features of larger objects. Figure 4 As shown, it is a schematic diagram of a feature extraction module according to an embodiment of the present invention.

[0040] Figure 4 By setting interconnected network nodes in the middle path of the stacked U-shaped sub-network, a 5-layer sub-network (each sub-network is Figure 4The feature extraction module is composed of a sub-network (arranged horizontally in the middle), each layer of the sub-network includes one or more network nodes, and the connection mode of each network node is a three-way interactive jump connection. In the feature extraction module, each network node can obtain a feature value obtained from at least one network node at the same level and adjacent levels of the network node, and the network node fuses the obtained feature values ​​to generate an original feature map.

[0041] The above technical solution can use the image segmentation network to detect dangerous areas in the image to be tested. The image segmentation network has network nodes that transmit features in a jump connection manner, which can alleviate the gradient vanishing problem caused by increasing the network depth in the neural network, reduce the probability of the image segmentation network losing smaller target objects in the image to be tested, and make small-area dangerous areas of embankment projects less likely to be lost in the deep network of the image segmentation network, thereby improving the accuracy of dangerous area detection; determining the dangerous area in the image to be tested based on the feature map obtained by the image segmentation network can achieve dangerous detection in a larger range, saving manpower and time.

[0042] According to the levee hazard detection method based on thermal infrared image segmentation according to an embodiment of the present invention, the image to be tested containing the levee facilities to be tested can be detected based on the image segmentation network. This method is an intelligent and automated detection method that does not require manual inspections. It has high inspection efficiency and accuracy, can detect dangerous situations in a timely and accurate manner, and can avoid safety risks in manual inspections.

[0043] Optionally, the feature extraction module may include at least one layer of sub-network, the network nodes of each layer of sub-network include a head node and a non-head node, and at least one target network node is a non-head node in at least one layer of sub-network.

[0044] See also Figure 5 , which is a schematic diagram of a feature extraction module according to an embodiment of the present invention. The feature extraction module may be referred to as a dense nested interactive module (DNIM). Figure 5 The feature extraction module in the Figure 5 The network consists of a network of L(i,j), i=0,1,2,3,4, j=0,1,2,3,4, where j represents the serial number of the convolutional layer in the corresponding subnetwork, and i represents the serial number of the downsampling layer in the corresponding subnetwork. The first node may include a network node with j=0, such as L(1,0), and the non-first node may include a network node with j>0, such as L(1,1).

[0045] Optionally, in step S120, the image to be tested is input into an image segmentation network to determine the location of the dangerous area in the image to be tested, including:

[0046] Step S121, in the feature extraction module, for each layer of sub-network, the feature value extracted at the previous node adjacent to the first node is obtained through the first node of the layer of sub-network.

[0047] Please continue reading Figure 5 As shown in the figure, the first node of each layer of the U-shaped sub-network only receives the feature transfer of the common skip connection layer, that is, the feature value extracted at the previous node L(a-1,0) adjacent to the first node L(a,0). For example, the first node L(2,0) obtains the feature value extracted at the first node L(1,0). The feature value obtained at the first node L(i,0) can be expressed by the following formula (1):

[0048] L(i,0)=P max (F(L i-1,0 )) (1)

[0049] Where P max (·) denotes the maximum pooling with a stride of 2, and F(·) denotes the cascaded convolutional layers built using the same convolutional blocks, i ≥ 1.

[0050] Optionally, in step S120, the image to be tested is input into an image segmentation network to determine the location of the dangerous area in the image to be tested, and further includes:

[0051] Step S122, in the feature extraction module, for each layer of sub-network, through the non-first node of the layer of sub-network, obtain a first eigenvalue from at least some of the nodes that are in the same sub-network as the non-first node and are located before the non-first node, obtain a second eigenvalue from at least one node of at least one layer of sub-network adjacent to the sub-network where the non-first node is located, and fuse the first eigenvalue and the second eigenvalue to obtain at least one original feature graph.

[0052] Please continue reading Figure 5As shown, the non-first node L(0,j) adjacent to i=0 and j≥1 obtains the feature values ​​output by the nodes L(0,0),...,L(0,j-1) in the same subnetwork and before the non-first node L(0,j), and the node L(1,j-1) in at least one layer of subnetwork adjacent to the subnetwork where the non-first node L(0,j) is located. For example, the non-first node L(0,2) obtains the feature values ​​output by the nodes L(0,0) and L(0,1) in the same subnetwork and before the non-first node L(0,2), and the feature value output by the node L(1,1) in the next layer of subnetwork adjacent to the subnetwork where the non-first node L(0,2) is located. For a non-first node L(i,j) where i≠0 and j>1, it obtains the feature values ​​extracted from L(i,0),...,L(i,j-1) which is in the same subnetwork as the non-first node L(i,j) and is located before the non-first node L(i,j), and the feature values ​​output by the node L(i-1,j) and / or the node L(i+1,j-1) of at least one layer (previous layer and / or next layer) of the subnetwork adjacent to the subnetwork where the non-first node L(i,j) is located. For example, the non-first node L(2,2) obtains the feature values ​​output by the first nodes L(2,0) and L(2,1) which are in the same subnetwork as the non-first node and are located before the non-first node, and the feature values ​​output by the non-first node L(3,1) and the non-first node L(1,2) of at least one layer of the subnetwork adjacent to the subnetwork where the non-first node is located. More specifically, the feature value obtained at the non-first node L(i,j) can be expressed by the following formula (2):

[0053]

[0054] Where P max (·) denotes max pooling with a stride of 2, F(·) denotes a cascaded convolutional layer built using the same convolutional block, U(·) denotes an upsampling layer, [·,·] denotes a concatenated layer, and j>0.

[0055] Please note that the above Figure 5 The described embodiment is only an example. For a non-first node, when obtaining the first characteristic value, it can be obtained from all nodes in the same sub-network as the non-first node and located before the non-first node, or from some nodes in the same sub-network as the non-first node and located before the non-first node. Figure 5 For the non-first node L(1,2), the first eigenvalue obtained can come from L(1,0) alone, or from L(1,1) alone, or from both L(1,0) and L(1,1).

[0056] Optionally, in step S120, the image to be tested is input into an image segmentation network to determine the location of the dangerous area in the image to be tested, and further includes:

[0057] Exemplarily, fusing the first eigenvalue and the second eigenvalue may include: assigning weights of channel attention and spatial attention to the first eigenvalue and the second eigenvalue, and fusing based on the assigned weights. For the feature map obtained by merging the first eigenvalue and the second eigenvalue (which may be referred to as the feature map to be fused), weights may be assigned in the channel dimension and the spatial dimension, the corresponding channel attention weights and spatial attention weights may be determined, and the assigned channel attention weights and spatial attention weights may be used to perform channel attention operations and spatial attention operations on the feature map to be fused. When fusing the feature map to be fused, the size of the feature map to be fused may be first enlarged to The generated fused feature map can be expressed as the following formula (3):

[0058]

[0059] Step S123, in the region determination module, at least one original feature map is fused to obtain a global feature map. Optionally, at least one original feature map can be fused by a feature pyramid fusion module to obtain a global feature map. The scale of the global feature map (which can be called resolution, including width and height) can be consistent with the scale of the image to be tested.

[0060] Optionally, in step S120, the image to be tested is input into an image segmentation network to determine the location of the dangerous area in the image to be tested, and further includes:

[0061] Step S124: In the area determination module, the global feature map is subjected to pixel clustering to obtain a dangerous area.

[0062] Exemplarily, pixel clustering can divide the pixels in an image into several groups so that the pixels in the same group have similar features. Optional traditional clustering algorithms include K-means, hierarchical clustering, DBSCAN, etc. These algorithms can be directly applied to the clustering of pixels, that is, the image is regarded as a data set in a high-dimensional space, the pixels are regarded as data points, and then the pixels are grouped using a clustering algorithm. In the clustering process, the color, texture and other features of the pixels can be used as the basis for clustering; you can also choose a clustering algorithm based on image segmentation, which first divides the image into several regions, and then clusters the pixels in the same region as a group.

[0063] Optionally, the previous node adjacent to the first node includes the first node of the previous layer of subnetwork before the subnetwork where the first node is located, and at least one node of at least one layer of subnetwork adjacent to the subnetwork where the non-first node is located includes: a node in the next adjacent subnetwork of the subnetwork where the non-first node is located and in the j-1th convolutional layer, and / or a node in the previous adjacent subnetwork of the subnetwork where the non-first node is located and in the jth convolutional layer, where j represents the sequence number of the convolutional layer where the non-first node is located in the corresponding subnetwork.

[0064] Please continue reading Figure 5 As shown, for the first node L(0,0), there is no previous node adjacent to the first node L(0,0). For other first nodes, there is a previous node adjacent to the first node. For example, for the first node L(1,0), the previous node adjacent to the first node L(1,0) includes the first node L(0,0) of the previous layer of subnetwork before the subnetwork where the first node L(1,0) is located. For a non-first node, at least one node of at least one layer of subnetwork adjacent to the subnetwork where the non-first node is located includes: a node in the previous adjacent subnetwork and / or the next adjacent subnetwork of the subnetwork where the non-first node is located and in the jth convolutional layer. Taking the non-first node L(2,1) as an example, at least one node of at least one layer of subnetwork adjacent to the subnetwork where the non-first node L(2,1) is located includes: the node L(1,1) that is in the previous adjacent subnetwork before the subnetwork where the non-first node L(2,1) is located and is in the 1st convolutional layer, and the node L(3,0) that is in the next adjacent subnetwork after the subnetwork where the non-first node L(2,1) is located and is in the 0th convolutional layer.

[0065] The above technical solution adds a densely nested interactive feature fusion strategy in the image segmentation network, fully utilizes the interactive features at different levels and the complex relationship between the features, and enhances the ability of the image segmentation network to effectively detect and segment small targets; the above technical solution uses the attention mechanism to assign weights to the feature map to achieve feature enhancement, so that the features of the image to be tested can be adaptively adjusted in both the channel dimension and the spatial dimension, so that the dangerous area finally obtained is more accurate.

[0066] Exemplarily, fusing the first eigenvalue and the second eigenvalue may include: assigning weights of channel attention and spatial attention to the first eigenvalue and the second eigenvalue, and fusing based on the assigned weights. Optionally, assigning weights of channel attention and spatial attention to the first eigenvalue and the second eigenvalue, and fusing based on the assigned weights, includes:

[0067] Merge the first eigenvalue and the second eigenvalue to obtain a feature map to be fused;

[0068] Perform a first pooling operation along the spatial dimension on the feature map to be fused to obtain a first feature map;

[0069] Normalizing the feature values ​​in the first feature map to obtain a normalized second feature map, wherein the feature values ​​of the second feature map under each channel are channel attention weights corresponding to each channel of the feature map to be fused;

[0070] Perform a second pooling operation along the channel dimension on the feature map to be fused to obtain a third feature map;

[0071] Convolving the third feature map, and normalizing the feature values ​​in the feature map obtained by the convolution to obtain a normalized fourth feature map, wherein the feature value corresponding to each of the multiple first feature map regions in the fourth feature map represents the spatial attention weight of the corresponding second feature map region in the feature map to be fused, and the position of the first feature map region in the fourth feature map is consistent with the position of the corresponding second feature map region in the feature map to be fused;

[0072] The channel attention weight corresponding to each channel in the feature map to be fused is multiplied by the corresponding channel of the feature map to be fused, and the spatial attention weight corresponding to each second feature map area in the feature map to be fused is multiplied by the corresponding second feature map area of ​​the feature map to be fused to obtain the fused feature map (i.e., the feature map output by the network node currently performing the fusion).

[0073] Exemplarily, the first pooling operation can be any pooling operation along the spatial dimension (width and height). The first pooling operation can include global average pooling and / or global maximum pooling along the spatial dimension of the feature map to be fused to obtain the first feature map. When the first feature map is input into a multi-layer perceptron (MLP), the multi-layer perceptron learns the common features between the channels corresponding to each first feature map, calculates the attention weights between the channels corresponding to each first feature map, measures the degree of association between the channels corresponding to each first feature map, and maps the first feature map to a channel weight vector with different strengths to form a second feature map. The number of channels of the first feature map is equal to the number of channels of the feature map to be fused, but the width and height of the first feature map are compressed to 1, that is, if the feature map to be fused is of size H*W*C, then the first feature map obtained after the first pooling operation is of size 1*1*C, where C represents the number of channels, H represents the height, and W represents the width. The number of channels of the second feature map is equal to the number of channels of the feature map to be fused. The eigenvalues ​​of the second feature map are normalized to generate a third feature map. The number of channels of the third feature map is equal to the number of channels of the feature map to be fused, and if any channel c1 is selected in the third feature map, there is a channel c2 in the feature map to be fused, corresponding to channel c1. The second pooling operation can be any pooling operation along the channel dimension. The second pooling operation may include global maximum pooling and / or global average pooling. The height and width of the third feature map are equal to those of the feature map to be fused, and the number of channels of the third feature map is 1. The fourth feature map obtained after convolution is H*W*1, where C represents the number of channels of the feature map to be fused, H represents the height, and W represents the width. Normalization can be implemented using any normalization function, such as a Softmax function or a Sigmoid function. More specifically, in an embodiment of the present invention, corresponding to feature L, its channel attention weight can be calculated by the following formula (4), and the spatial attention weight can be calculated by the following formula (5):

[0074]

[0075]

[0076] In the formula, represents element-by-element multiplication, σ represents the sigmoid function, P max (·) represents the maximum pooling with a stride of 2, P avg (·) represents average pooling, f 7×7 represents the convolution operation of the 7×7 filter. The channel attention weight corresponding to each channel in the feature map to be fused is multiplied by the corresponding channel of the feature map to be fused. The spatial attention weight corresponding to each second feature map area in the feature map to be fused is multiplied by the corresponding second feature map area of ​​the feature map to be fused. The enhanced feature can be expressed by the following formula (6):

[0077]

[0078] Through the above scheme, the first pooling operation can extract key channel information and map the attention weight distribution from all channels to a few key channels containing important information. By learning the importance of each channel, the influence of channels with smaller weights can be further reduced to prevent overfitting. The second pooling operation can focus on the local area of ​​the image to be tested, reduce the calculation of a large number of pixels by the image segmentation network, reduce the complexity of calculation, and combine the channel attention weight with the spatial attention weight, so that the image segmentation network can pay attention to both spatial and channel information during the learning process, so that the feature information of the dangerous area obtained is richer.

[0079] Optionally, in step S123, pixel clustering is performed on the global feature map to obtain a dangerous area, including:

[0080] For every two pixels in the global feature map, whether the two pixels are adjacent pixels is determined based on whether there is an overlapping area in the neighborhood areas corresponding to the two pixels. If there is an overlapping area, the two pixels are determined to be adjacent pixels, and the neighborhood area corresponding to any pixel is the area including the eight neighborhood pixels of the pixel. If the two pixels are adjacent pixels, whether the two pixels belong to the same connected domain is determined based on the grayscale values ​​of the two pixels. The connected domain determined based on all the pixels in the global feature map is used as the dangerous area.

[0081] Exemplarily, an eight-connected neighborhood clustering algorithm can be used to cluster pixels belonging to the same target (herein, the dangerous area) together and calculate the centroid of each target. If in the global feature map, for any pixel A and any pixel B, if the area where the eight neighboring pixels of pixel A are located overlaps with the area where the eight neighboring pixels of pixel B are located, then pixel A and pixel B are judged to be adjacent pixels. When pixel A and pixel B are adjacent pixels, if the grayscale values ​​of pixel A and pixel B are the same or belong to the same preset grayscale interval, then pixel A and pixel B are judged to belong to the same connected domain, that is, the same target. After determining all the targets of the global feature map, the centroid of each target can be optionally calculated, and the position of the centroid is used to represent the position of the dangerous area.

[0082] The above technical solution can better understand the local structure and global features in the image through pixel clustering, and divide the global feature map into different areas at the pixel level, which is helpful to determine the location and scope of the dangerous area.

[0083] Optionally, in step S110, obtaining an image to be tested that includes the levee facilities to be tested includes:

[0084] An initial thermal infrared image including the levee facilities to be measured collected by an infrared camera and a point cloud image including the levee facilities to be measured collected by a lidar are obtained; the initial thermal infrared image and the point cloud image are fused to obtain the image to be measured.

[0085] Exemplarily, an infrared camera can be used to collect an initial thermal infrared image containing the levee facilities to be measured, and a laser radar can be used to collect a point cloud image containing the levee facilities to be measured. Exemplarily, an infrared camera uses a photosensitive element to detect infrared rays radiated outward from an object, and converts these thermal radiation information into electrical signals through technologies such as photoelectric conversion and signal processing. Through further processing and calculation, the electrical signals can be converted into a visualized grayscale image, i.e., an initial thermal infrared image. Any object with a temperature higher than absolute zero will emit electromagnetic waves, and long-wave infrared has a strong penetrating ability, and can observe ground objects through clouds, haze, rain and snow and other weather conditions. In addition, the higher the temperature of an object, the stronger the electromagnetic wave signal it radiates. If there is a temperature difference between the detection target and the background, the energy difference of the radiation is manifested as the edge contour of the detection target on the infrared image. In a thermal infrared image, the lower the temperature of an object, the lower the grayscale value of the pixel corresponding to the object, and the closer it is to black. In an embodiment of the present invention, the pixels of the initial thermal infrared image may be optionally transformed inversely first, that is, each pixel in the initial thermal infrared image is subtracted from its corresponding grayscale value by 255, so that the grayscale value of the corresponding pixel of the object with lower temperature in the initial thermal infrared image is higher and closer to white. It should be understood by those skilled in the art that the inverse transformation operation of the pixels of the initial thermal infrared image is not a necessary step, and it is optional according to the specific parameters of the image segmentation network. The laser radar measures the distance from the target to the laser radar receiver by emitting a laser beam to obtain accurate three-dimensional point cloud data and corresponding point cloud images, wherein the three-dimensional point cloud data includes the actual spatial information of the pixels in the point cloud image. The field of view of the device for acquiring the point cloud image (such as the above-mentioned laser radar) is at least partially the same as the field of view of the device for acquiring the thermal infrared image (such as the above-mentioned infrared camera), that is, the objects observed by the laser radar and the objects observed by the infrared camera are at least partially the same. For example, the fields of view of the laser radar and the infrared camera may overlap at least at the dike facility to be measured.

[0086] Exemplarily, the point cloud image may be composed of a set of points in a three-dimensional space, each point usually having three coordinate values ​​of x, y, and z, and the initial thermal infrared image contains the temperature distribution information of the scene in the image. Exemplarily, the point cloud image and the initial thermal infrared image may be fused at the pixel level and / or at the feature level through spatial registration (spatial alignment of the point cloud image and the initial thermal infrared image) and attribute registration (registering the geometric position information corresponding to the point cloud image and the temperature information corresponding to the initial thermal infrared image), and the fused data may be three-dimensionally reconstructed to obtain the image to be measured, wherein the feature-level fusion is performed by extracting features and merging features of the initial thermal infrared image and the point cloud image to form a fused feature vector. Those skilled in the art should understand that multi-source images can generate new images in a prescribed common spatial coordinate system according to a certain algorithm, and the aforementioned image fusion process is only an optional implementation method. The present invention does not specifically limit the method of image fusion, and it is sufficient to meet the matching standard of the spatial position and spatial coordinates of the objects corresponding to the initial thermal infrared image and the point cloud image in the image to be measured.

[0087] Optionally, the channel of the image to be tested includes a color value channel and an intensity channel, the color value channel includes a color value corresponding to each pixel point of the image to be tested determined based on the initial thermal infrared image, and the intensity channel includes an echo intensity value corresponding to each pixel point of the image to be tested determined based on the point cloud image.

[0088] Exemplarily, the image to be tested obtained by image fusion may include a color value channel, and the color value of the color value channel is determined based on the color value of the initial thermal infrared image (i.e., the grayscale value mentioned above). Exemplarily, the image to be tested obtained by image fusion may also include an intensity channel, and the echo intensity value of the intensity channel is determined based on the echo intensity value of the point cloud image, and the echo intensity value serves as an additional channel of the image to be tested.

[0089] The above technical scheme can effectively integrate the pixel information of the initial thermal infrared image of the embankment facility to be tested and the point cloud data information of the embankment facility to be tested into the same image, so that the spatial information and point cloud data information of the embankment facility to be tested correspond to each other and are fully combined, making the features of each pixel point in the image to be tested richer and more intuitive.

[0090] Optionally, before fusing the initial thermal infrared image and the point cloud image to obtain the image to be tested, the embankment danger detection method based on thermal infrared image segmentation further includes:

[0091] Perform contrast enhancement on the initial thermal infrared image.

[0092] Exemplarily, the contrast enhancement method of the initial thermal infrared image may include one or more of histogram equalization, histogram piecewise linear transformation, histogram regulation, etc. This embodiment does not limit the specific method of enhancing the contrast.

[0093] Due to the characteristics of the detection elements of the infrared camera and the differences in the temperature distribution of the ground objects, the initial thermal infrared image will only show a more obvious grayscale change when the temperature of the ground objects changes drastically. In addition, the rapid movement of the drone will aggravate the blurriness of the initial thermal infrared image. The initial thermal infrared image usually has a low contrast. The above technical solution can effectively alleviate the problem of low contrast of thermal infrared images and unclear edges of the targets to be measured in the image (the levee facilities to be measured or the dangerous areas on the levee facilities to be measured). Through image enhancement, the contrast between the background and the target to be measured in the image to be measured can be improved, which is beneficial to the feature extraction of the image to be measured by the image segmentation network and helps to accurately extract the dangerous areas in the image to be measured.

[0094] Optionally, the initial thermal infrared image is contrast enhanced by the following formula (7):

[0095]

[0096] Where I(x,y) stretched represents the contrast-enhanced pixel value of the pixel with coordinates (x, y) in the initial thermal infrared image, I(x, y) represents the original pixel value of the pixel with coordinates (x, y) in the initial thermal infrared image, and I max Represents the maximum pixel value in the initial thermal infrared image, I min Indicates the minimum pixel value in the initial thermal infrared image.

[0097] For example, if the pixel value of a pixel point in the initial thermal infrared image is 55, the maximum pixel value in the initial thermal infrared image is 200, and the minimum pixel value is 20, then after contrast enhancement of the pixel point, the pixel value of the pixel point obtained is I=(55-20)*255 / (200-20)=49.5.

[0098] The above technical solution stretches the grayscale value of the initial thermal infrared image to the range of 0-255 by contrast enhancement, which can make the edge of the target to be measured in the initial thermal infrared image clearer and help distinguish the background of the initial thermal infrared image and the outline of the target to be measured.

[0099] Optionally, the image to be measured includes an intensity channel, and the intensity channel includes an echo intensity value corresponding to each pixel point of the image to be measured, which is determined based on the point cloud image of the levee facility to be measured;

[0100] Inputting the image to be tested into the image segmentation network to determine the position of the dangerous area in the image to be tested, including: inputting the image to be tested into the image segmentation network to obtain a dangerous area mask, the dangerous area mask is used to indicate the position of the dangerous area in the image to be tested; or, the levee dangerous condition detection method based on thermal infrared image segmentation also includes: obtaining the dangerous area mask according to the position of the dangerous area;

[0101] Wherein, each at least one original feature map corresponds to at least one dangerous area mask; the embankment dangerous condition detection method based on thermal infrared image segmentation also includes:

[0102] The danger weight of the dangerous area is calculated according to the area of ​​the dangerous area mask, the location information of the dangerous area mask and the corresponding echo intensity value in the dangerous area mask. The danger weight is used to indicate the degree of danger of the dangerous area.

[0103] Exemplarily, by inputting the image to be tested into the image segmentation network, at least one dangerous area mask can be obtained. After obtaining the dangerous area mask, its area and / or location information can be determined. Optionally, the location information of the dangerous area mask can be represented by the image coordinates of the centroid of the dangerous area mask. In the image segmentation network, each layer of subnetwork outputs an original feature map respectively. Since the depth of the subnetwork in the image segmentation network is different, the scale of the features extracted by each layer of subnetwork is also different. Therefore, each original feature map can correspond to the dangerous area mask detected at different scales. The global feature map obtained by fusing the original feature map may include the dangerous area mask corresponding to the original feature map. Since the higher the water content of the region and the lower the temperature of the region, the lower the echo intensity value of the pixel point corresponding to the region, wherein the echo intensity value of the pixel point corresponding to the pure water body is close to 0, the embodiment of the present invention can distinguish the water body area from other land feature areas except the water body by the echo intensity value of the dangerous area mask. Generally speaking, suspended particles, dissolved substances, aquatic plants or underwater structures in the water body may affect the propagation and reflection of the laser, thereby generating some echo signals in the point cloud, so that the dangerous area mask has a non-zero intensity value. The echo intensity value of the dangerous area mask can be set to be negatively correlated with the water content of the actual area corresponding to the dangerous area mask, that is, the lower the echo intensity value of the dangerous area mask, the higher the water content of the corresponding actual area.

[0104] The above technical solution uses the geometric area size, location information of the dangerous area mask finally output by the image segmentation network and the correlation between the corresponding echo intensity value in the dangerous area mask and the dangerousness of the dangerous area to determine the danger weight. The size of the danger weight can be used to determine the degree of danger of the dangerous area, which helps to locate and screen out the specific geographical locations of dangerous areas with higher risks, facilitating rapid investigation and emergency repair.

[0105] Optionally, the location information includes the distance between the area corresponding to the dangerous area mask and the historical dangerous area, and the danger weight is calculated by the following formula (8):

[0106]

[0107] Where W is the danger weight, S is the area of ​​the dangerous area mask, d is the distance between the area corresponding to the dangerous area mask and the historical dangerous area, and Avgi is the average value of the echo intensity value corresponding to each pixel in the dangerous area mask.

[0108] Exemplarily, the distance between the area corresponding to the dangerous area mask and the historical dangerous area can be the straight-line distance between the centroid of the area corresponding to the dangerous area mask and the centroid of the historical dangerous area. The historical dangerous area can be a previously detected dangerous area on the embankment facility to be tested. The average value of the echo intensity value corresponding to each pixel point in the dangerous area mask can be obtained by calculating the ratio of the sum of the echo intensity values ​​corresponding to each pixel point in the dangerous area mask to the number of pixels in the dangerous area mask. Specifically, the larger the calculated danger weight is, the greater the degree of danger of the dangerous area corresponding to the danger weight is. In a specific embodiment, the basic unit of the area S of the dangerous area mask is set to 10,000 square meters, the basic unit of the distance d between the area corresponding to the dangerous area mask and the historical dangerous area is set to kilometers, and the basic unit of the echo intensity value Avgi corresponding to each pixel point in the dangerous area mask is set to dBZ. There is a dangerous area mask Mask1, whose area is 120,000 square meters, the distance between it and the historical dangerous area is 10 kilometers, and the average echo intensity value corresponding to each pixel point in the dangerous area mask is 10 dBZ, then the danger weight

[0109] The above technical scheme utilizes the relationship that the geometric area size of the dangerous area is positively correlated with the danger level of the dangerous area, and the distance between the dangerous area and the historical dangerous area, and the echo intensity value within the dangerous area are negatively correlated with the danger level of the dangerous area, to quantify the danger level of the dangerous area and make the danger of each dangerous area more intuitive.

[0110] Optionally, the image segmentation network is trained by:

[0111] Acquire a sample image containing a sample dike facility and corresponding annotation data, where the annotation data is used to indicate an annotation position of a sample dangerous area in the sample image;

[0112] The sample image is input into the image segmentation network to determine the predicted position of the sample dangerous area in the sample image;

[0113] The loss value of the image segmentation network is calculated according to the marked position and the predicted position to train the image segmentation network;

[0114] The method of obtaining the annotation data includes: using morphological gradient edge detection on the sample image, and filling the contour obtained by the edge detection to determine the annotation position of the sample dangerous area in the sample image.

[0115] Exemplarily, the sample image may simply include sample levee facilities, or may include sample levee facilities and other features except the sample levee facilities, such as vegetation, trees, country roads, etc. Sample levee facilities may be, for example, river levees, river levees, lake levees, sea levees, and levees in low-lying areas of reservoirs and flood storage areas. The sample levee facilities may be of the same type as or different from the levee facilities to be tested. The sample levee facilities and the levee facilities to be tested may be the same levee facilities or different levee facilities. Preferably, the sample levee facilities may be levee facilities of the same type as the levee facilities to be tested. The annotated data corresponding to the sample image may be the image coordinates corresponding to the centroid of the sample dangerous area in the sample image. Those skilled in the art should understand that the loss value of the image segmentation network calculated based on the annotated position and the predicted position is used to adjust the parameters of the image segmentation network so that the loss value of the obtained image segmentation network meets the preset numerical requirements, and the specific training process will not be described in detail here. Those skilled in the art should understand that morphological gradient edge detection is an image edge detection algorithm based on grayscale morphology, and the contour line of the sample dangerous area in the sample image can be obtained through morphological gradient edge detection. The edge detection process of morphological gradient edge detection is not described here. After the contour line of the sample dangerous area in the sample image is obtained by the morphological gradient edge detection method, the contour line can be manually checked, modified and supplemented. The above-mentioned method of obtaining the annotation data is only an example, and the annotation data can also be annotated in other suitable ways, such as manual annotation.

[0116] The above technical solution is based on morphological gradient edge detection, which filters out areas with significant gradient changes in thermal infrared images, and can quickly and accurately determine the actual spatial information of the sample dangerous area of ​​the sample image. The image segmentation network is trained according to the predicted position and marked position of the sample dangerous area of ​​the sample image, so that the obtained image segmentation network can efficiently and accurately complete the regional segmentation of the image to be tested in the application stage, and then accurately extract the dangerous area.

[0117] See also Figure 6 As shown, it is a schematic diagram of a dike danger detection device 600 based on thermal infrared image segmentation according to an embodiment of the present invention. According to another aspect of the embodiment of the present application, a dike danger detection device based on thermal infrared image segmentation is also provided. Figure 6As shown, the levee danger detection device 600 based on thermal infrared image segmentation includes:

[0118] An acquisition module 610 is used to acquire an image to be tested that includes the levee facilities to be tested;

[0119] An input module 620, which is used to input the image to be tested into the image segmentation network to determine the location of the dangerous area in the image to be tested, where the dangerous area is the area where there is a dangerous situation in the embankment facility to be tested;

[0120] Among them, the image segmentation network includes a feature extraction module and a region determination module, the feature extraction module is used to extract at least one original feature map of the image to be tested, and the region determination module is used to determine the dangerous area based on the at least one original feature map; in the feature extraction module, at each target network node in at least one target network node, feature values ​​of at least one level of the image to be tested are received by jump connection to perform feature fusion to obtain at least one original feature map, and the feature values ​​of at least one level include: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level and adjacent levels as the current network node.

[0121] See also Figure 7 As shown, it is a schematic diagram of an electronic device 700 according to an embodiment of the present invention. According to another aspect of the present invention, there is also provided an electronic device 700, comprising: a processor 710 and a memory 720, wherein the memory 720 stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned levee danger detection method based on image segmentation when the processor is running.

[0122] According to another aspect of the embodiment of the present application, a computer-readable storage medium is also provided. The storage medium stores a computer program / instruction, and when the computer program / instruction is executed by the processor, the above-mentioned levee danger detection method based on thermal infrared image segmentation is implemented. The storage medium may include, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0123] According to another aspect of the present invention, there is also provided a computer program product, comprising computer program instructions, which are used to execute the above-mentioned embankment hazard detection method based on thermal infrared image segmentation when running.

[0124] A person skilled in the art can easily understand the implementation structure, working principle and beneficial effects of the levee danger detection device based on thermal infrared image segmentation, electronic device, computer readable storage medium and computer program product by reading the above-mentioned levee danger detection method based on thermal infrared image segmentation. For the sake of brevity, it will not be described here.

[0125] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0127] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0128] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0129] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0130] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0131] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0132] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules of the levee hazard detection device based on thermal infrared image segmentation according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0133] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0134] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting dike danger based on thermal infrared image segmentation, characterized in that: include: Acquire an image to be tested that includes the levee facilities to be tested, wherein the image to be tested is a thermal infrared image; Inputting the image to be tested into an image segmentation network to determine the location of a dangerous area in the image to be tested, wherein the dangerous area is an area in the embankment facility to be tested where a dangerous situation exists; The image segmentation network includes a feature extraction module and a region determination module, wherein the feature extraction module is used to extract at least one original feature map of the image to be tested, and the region determination module is used to determine the dangerous area based on the at least one original feature map; in the feature extraction module, at each target network node in at least one target network node, feature values ​​of at least one level of the image to be tested are received by jump connection to perform feature fusion to obtain the at least one original feature map, and the feature values ​​of at least one level include: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level and adjacent levels as the current network node; The feature extraction module includes at least one layer of sub-network, the network nodes of each layer of sub-network include the head node and the non-head node, and the at least one target network node is the non-head node in the at least one layer of sub-network; The step of inputting the image to be tested into an image segmentation network to determine the location of the dangerous area in the image to be tested includes: In the feature extraction module, for each layer of sub-network, Acquire, through the first node of the sub-network of the layer, a feature value extracted at a previous node adjacent to the first node; Obtain a first eigenvalue from at least some nodes in the same subnetwork as the non-first node and before the non-first node through the non-first node of the layer of subnetwork, obtain a second eigenvalue from at least one node of at least one layer of subnetwork adjacent to the subnetwork where the non-first node is located, and fuse the first eigenvalue and the second eigenvalue to obtain the at least one original feature graph; In the area determination module, fusing the at least one original feature map to obtain a global feature map; Perform pixel clustering on the global feature map to obtain the dangerous area; Among them, the previous node adjacent to the first node includes the first node of the previous layer of sub-network before the sub-network where the first node is located, and at least one node of at least one layer of sub-network adjacent to the sub-network where the non-first node is located includes: a node in the next adjacent sub-network of the sub-network where the non-first node is located and in the j-1th convolutional layer, and / or a node in the previous adjacent sub-network of the sub-network where the non-first node is located and in the jth convolutional layer, wherein j represents the sequence number of the convolutional layer where the non-first node is located in the corresponding sub-network.

2. The method according to claim 1, characterized in that The fusing the first eigenvalue and the second eigenvalue comprises: The first eigenvalue and the second eigenvalue are weighted for channel attention and spatial attention, and are fused based on the weights assigned.

3. The method according to claim 1, characterized in that The step of clustering pixels of the global feature map to obtain the dangerous area includes: For every two pixels in the global feature map, Determining whether the two pixel points are adjacent pixel points according to whether there is an overlapping area in the neighborhood areas corresponding to the two pixel points, wherein if there is an overlapping area, the two pixel points are determined to be adjacent pixel points, and the neighborhood area corresponding to any pixel point is an area including eight neighboring pixel points of the pixel point; If the two pixels are adjacent pixels, determining whether the two pixels belong to the same connected domain according to the grayscale values ​​of the two pixels; A connected domain determined based on all pixels in the global feature map is used as the dangerous area.

4. The method according to any one of claims 1 to 3, characterized in that: The step of acquiring the image to be tested including the levee facilities to be tested comprises: Acquire an initial thermal infrared image containing the levee facilities to be measured collected by an infrared camera and a point cloud image containing the levee facilities to be measured collected by a laser radar; Performing image fusion on the initial thermal infrared image and the point cloud image to obtain the image to be tested; Among them, the channel of the image to be tested includes a color value channel and an intensity channel, the color value channel includes a color value corresponding to each pixel point of the image to be tested determined based on the initial thermal infrared image, and the intensity channel includes an echo intensity value corresponding to each pixel point of the image to be tested determined based on the point cloud image.

5. The method according to claim 4, characterized in that Before fusing the initial thermal infrared image and the point cloud image to obtain the image to be measured, the method further includes: The initial thermal infrared image is contrast enhanced.

6. The method according to claim 5, characterized in that The initial thermal infrared image is contrast enhanced by the following formula: Where I(x,y) stretched represents the contrast-enhanced pixel value of the pixel point with coordinates (x, y) in the initial thermal infrared image, I(x, y) represents the original pixel value of the pixel point with coordinates (x, y) in the initial thermal infrared image, and I max represents the maximum pixel value in the initial thermal infrared image, I min Indicates the minimum pixel value in the initial thermal infrared image.

7. The method according to any one of claims 1 to 3, characterized in that: The image to be measured includes an intensity channel, and the intensity channel includes an echo intensity value corresponding to each pixel point of the image to be measured, which is determined based on the point cloud image of the levee facility to be measured; The step of inputting the image to be tested into an image segmentation network to determine the position of the dangerous area in the image to be tested includes: inputting the image to be tested into an image segmentation network to obtain a dangerous area mask, wherein the dangerous area mask is used to indicate the position of the dangerous area in the image to be tested; or the method further includes: obtaining the dangerous area mask according to the position of the dangerous area; Wherein, each of the at least one original feature map corresponds to at least one dangerous area mask; the method further comprises: The danger weight of the dangerous area is calculated according to the area of ​​the dangerous area mask, the position information of the dangerous area mask and the corresponding echo intensity value in the dangerous area mask. The danger weight is used to represent the danger degree of the dangerous area.

8. The method according to claim 7, characterized in that The location information includes the distance between the area corresponding to the dangerous area mask and the historical dangerous area, and the danger weight is calculated by the following formula: Wherein, W is the danger weight, S is the area of ​​the dangerous area mask, d is the distance between the area corresponding to the dangerous area mask and the historical dangerous area, and Avgi is the average value of the echo intensity value corresponding to each pixel point in the dangerous area mask.

9. The method according to any one of claims 1 to 3, characterized in that: The image segmentation network is trained in the following way: Acquire a sample image containing a sample dike facility and corresponding annotation data, wherein the annotation data is used to indicate an annotation position of a sample dangerous area in the sample image; Inputting the sample image into the image segmentation network to determine the predicted position of the sample dangerous area in the sample image; Calculating a loss value of the image segmentation network according to the marked position and the predicted position to train the image segmentation network; The method for acquiring the annotation data includes: applying morphological gradient edge detection to the sample image, and filling the contour obtained by the edge detection to determine the annotation position of the sample dangerous area in the sample image.

10. A device for detecting dike danger based on thermal infrared image segmentation, characterized in that: include: An acquisition module, which is used to acquire an image to be tested that includes the levee facilities to be tested; An input module, which is used to input the image to be tested into an image segmentation network to determine the location of a dangerous area in the image to be tested, wherein the dangerous area is an area in the levee facility to be tested where a dangerous situation exists; The image segmentation network includes a feature extraction module and a region determination module, wherein the feature extraction module is used to extract at least one original feature map of the image to be tested, and the region determination module is used to determine the dangerous area based on the at least one original feature map; in the feature extraction module, at each target network node in at least one target network node, feature values ​​of at least one level of the image to be tested are received by jump connection to perform feature fusion to obtain the at least one original feature map, and the feature values ​​of at least one level include: feature values ​​obtained from at least one network node at the same level as the current network node, or feature values ​​obtained from at least one network node at the same level and adjacent levels as the current network node; The feature extraction module includes at least one layer of sub-network, the network nodes of each layer of sub-network include the head node and the non-head node, and the at least one target network node is the non-head node in the at least one layer of sub-network; The feature extraction module comprises: A first acquisition submodule, for each layer of sub-network, is used to acquire, through the first node of the layer of sub-network, a feature value extracted at a previous node adjacent to the first node; A second acquisition submodule, for each layer of subnetwork, is used to obtain a first eigenvalue from at least some nodes in the same subnetwork as the non-first node and before the non-first node through a non-first node of the layer of subnetwork, obtain a second eigenvalue from at least one node of at least one layer of subnetwork adjacent to the subnetwork where the non-first node is located, and fuse the first eigenvalue and the second eigenvalue to obtain the at least one original feature graph; The area determination module comprises: A fusion submodule, configured to fuse the at least one original feature map to obtain a global feature map; A clustering submodule, which is used to cluster pixels of the global feature map to obtain the dangerous area; Among them, the previous node adjacent to the first node includes the first node of the previous layer of sub-network before the sub-network where the first node is located, and at least one node of at least one layer of sub-network adjacent to the sub-network where the non-first node is located includes: a node in the next adjacent sub-network of the sub-network where the non-first node is located and in the j-1th convolutional layer, and / or a node in the previous adjacent sub-network of the sub-network where the non-first node is located and in the jth convolutional layer, wherein j represents the sequence number of the convolutional layer where the non-first node is located in the corresponding sub-network.

11. An electronic device, comprising: A processor and a memory, wherein the memory stores computer program instructions, and wherein the computer program instructions are used by the processor to execute the embankment hazard detection method based on thermal infrared image segmentation as described in any one of claims 1 to 9 when the processor is running.

12. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the embankment hazard detection method based on thermal infrared image segmentation as described in any one of claims 1 to 9 when running.

13. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the embankment hazard detection method based on thermal infrared image segmentation as described in any one of claims 1 to 9 when running.

Images