River ice distribution intelligent extraction method, device and equipment and storage medium

Abstract

The application discloses a kind of river ice distribution intelligent extraction method, device, equipment and storage medium.The method includes obtaining the current river ice image set corresponding to target river ice monitoring area by unmanned aerial vehicle remote sensing;River ice is identified to current river ice image set by pre-set river ice identification network model and obtains corresponding river ice identification image set, U-Net branch network in pre-set river ice identification network model is used to obtain the river ice feature map marked river ice distribution, ResNet branch network is used to obtain the feature distinguishing map distinguishing river bank and river channel, feature distinguishing map and river ice feature map are fused;Finally, according to river ice identification image set, the corresponding river ice distribution data is extracted.The current river ice image set obtained by the pre-set river ice identification network model established by the application can be identified for river ice, and river ice distribution data can be obtained in real time, which not only improves the efficiency of judging ice disaster, but also improves the grasping ability of ice flow process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a river ice distribution intelligent extraction method, device, equipment and storage medium. BACKGROUND

[0002] Due to the different geographical positions of the upper and lower reaches of the Yellow River, the river flows from southwest to northeast, the lower reach freezes early, warms up late, and the freezing time is long, the upper reach freezes late, warms up early, and the freezing time is short; after entering winter, the ice water of the upper reaches flows down, easily congests in the downstream, forms ice dam and ice bridge, blocks the water section, raises the water level, and forms ice flood. If the ice congestion is serious, the water level is lifted too high, the beach is flooded, and the beach houses are often flooded, farmland is flooded, embankments are waterlogged, and engineering facilities are damaged by ice, and in serious cases, ice flood is poured and ice flood overflows the embankment, which seriously threatens the safety of people's life and property in the beach area.

[0003] Therefore, ice condition monitoring is a key link for anti-ice decision, water resource optimization scheduling, and water engineering operation management. The traditional ice condition monitoring method is based on visual observation, which is completely dependent on the eyes and experience of the observation personnel according to the ice condition observation specification. The visual observation method can observe a wide range of content, but due to the difference between individual observation, the observation data often has differences, and the obtained river ice distribution result is inaccurate, and the operation mode is low in efficiency.

[0004] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a river ice distribution intelligent extraction method, device, equipment and storage medium, which aims to solve the technical problem of low efficiency of the operation mode of traditional artificial patrol and judgment of ice flood disaster.

[0006] In order to achieve the above purpose, the present application provides a river ice distribution intelligent extraction method, which comprises the following steps:

[0007] Obtain a current river ice image set corresponding to a target river ice monitoring area, the current river ice image set is obtained in an orthographic shooting mode based on unmanned aerial vehicle remote sensing technology;

[0008] The preset river ice recognition network model is used for river ice recognition on the current river ice image set, so as to obtain a river ice recognition image set corresponding to the current river ice image set. The preset river ice recognition network model comprises a U-Net branch network and a ResNet branch network. The U-Net branch network is used for extracting image features of an input image and obtaining a river ice feature map marking a river ice distribution based on the image features. The ResNet branch network is used for obtaining multi-scale semantic information of the input image and obtaining a feature distinguishing map distinguishing a river bank from a river channel based on the multi-scale semantic information. The preset river ice recognition network model performs feature fusion on the feature distinguishing map and the river ice feature map at a connection position of the U-Net branch network and the ResNet branch network.

[0009] River ice distribution data corresponding to the target river ice monitoring area is extracted according to the river ice recognition image set.

[0010] Optionally, the preset river ice recognition network model further comprises a feature fusion module. A first input end of the feature fusion module is connected with the U-Net branch network, and a second input end of the feature fusion module is connected with the ResNet branch network.

[0011] The step of performing river ice recognition on the current river ice image set by using the preset river ice recognition network model to obtain a river ice recognition image set corresponding to the current river ice image set comprises the following steps.

[0012] The current river ice image set is input into the U-Net branch network to extract image features, so as to obtain a river ice feature image.

[0013] The current river ice image set is input into the ResNet branch network to perform image residual error processing, so as to obtain a river bank segmentation image.

[0014] The river ice feature image and the river bank segmentation image are subjected to weight processing by using the feature fusion module, so as to obtain a river ice recognition image set corresponding to the current river ice image set.

[0015] Optionally, the ResNet branch network comprises a Res residual unit and an up-sampling unit. An output end of the Res residual unit is connected with an input end of the up-sampling unit, and an output end of the up-sampling unit is connected with the second input end of the feature fusion module.

[0016] The step of inputting the current river ice image set into the ResNet branch network to perform image residual error processing to obtain a river bank segmentation image comprises the following steps.

[0017] inputting the current river ice image set into the Res residual unit for hierarchical image residual processing to obtain residual images of each level, wherein the Res residual unit comprises Res residual blocks of a preset number of levels;

[0018] performing significant analysis on the residual images of each level according to a channel attention mechanism, and inputting the analysis result into the up-sampling unit for up-sampling processing to obtain a river bank segmentation image.

[0019] Optionally, the U-Net branch network comprises a down-sampling layer, an up-sampling layer and a network connection layer.

[0020] The step of inputting the current river ice image set into the U-Net branch network for image feature extraction to obtain a river ice feature image comprises:

[0021] inputting the current river ice image set into the down-sampling layer for hierarchical down-sampling to obtain shallow feature images of each level;

[0022] inputting the shallow feature images of each level of the down-sampling layer into each level of the up-sampling layer through the network connection layer for hierarchical up-sampling to extract deep features, and obtaining a river ice feature image according to the deep features.

[0023] Optionally, the feature fusion module comprises a scale balancing unit, a global weight unit and a fusion unit.

[0024] The step of processing the river ice feature image and the river bank segmentation image through the feature fusion module to obtain a river ice recognition image set corresponding to the current river ice image set comprises:

[0025] inputting the river ice feature image and the river bank segmentation image into the scale balancing unit for feature balancing to obtain an initial fusion image;

[0026] inputting the initial fusion image into the global weight unit for global weight processing to obtain a global fusion image;

[0027] performing product processing on the initial fusion image to obtain a product fusion image corresponding to the initial fusion image;

[0028] inputting the initial fusion image, the global fusion image and the product fusion image into the fusion unit for weighted processing to obtain a river ice recognition image set corresponding to the current river ice image set.

[0029] Optionally, the scale balancing unit has a structure of 1x1Conv+BN+ReLU, the global weight unit has a structure of Global pooling+1x1Conv+BN+Sigmiod, and the feature fusion module is formed by the product branch of the global weight unit and the scale balancing unit and the fusion unit spliced with the scale balancing unit.

[0030] wherein, BN is a normalization module, ReLU is an activation function, 1x1Conv is 1x1 convolution, Sigmiod is a Sigmiod activation function, and Global pooling is global pooling.

[0031] Optionally, before the step of obtaining a current river ice image set corresponding to a target river ice monitoring area, the current river ice image set is obtained in an orthographic shooting mode based on unmanned aerial vehicle remote sensing technology, the method further comprises the following steps of:

[0032] obtaining a preset number of image training sets;

[0033] performing background marking on the image training sets by using an image marking tool to obtain a marked image training set;

[0034] iteratively training an initial river ice recognition network model to be trained according to the marked image training set to obtain a preset river ice recognition network model.

[0035] In addition, to achieve the above object, the application further provides a device for intelligently extracting river ice distribution, which comprises:

[0036] an image acquisition module configured to acquire a current river ice image set corresponding to a target river ice monitoring area, the current river ice image set being acquired in an orthographic shooting mode based on unmanned aerial vehicle remote sensing technology;

[0037] a river ice recognition module configured to perform river ice recognition on the current river ice image set by using a preset river ice recognition network model to obtain a river ice recognition image set corresponding to the current river ice image set, the preset river ice recognition network model comprising a U-Net branch network and a ResNet branch network, the U-Net branch network being configured to extract image features of an input image and obtain a river ice feature map marking river ice distribution based on the image features, the ResNet branch network being configured to acquire multi-scale semantic information of the input image and obtain a feature distinguishing map distinguishing river banks from river courses based on the multi-scale semantic information, and the preset river ice recognition network model being configured to perform feature fusion on the feature distinguishing map and the river ice feature map at a connection position of the U-Net branch network and the ResNet branch network;

[0038] An ice distribution module is configured to extract ice distribution data corresponding to the target ice monitoring area according to the ice recognition image set.

[0039] In addition, to achieve the above object, the present application further provides an ice distribution intelligent extraction device, which comprises a memory, a processor and an ice distribution intelligent extraction program stored in the memory and executable on the processor, and the ice distribution intelligent extraction program is configured to implement the steps of the ice distribution intelligent extraction method as described above.

[0040] In addition, to achieve the above object, the present application further provides a storage medium, which stores an ice distribution intelligent extraction program, and the ice distribution intelligent extraction program implements the steps of the ice distribution intelligent extraction method as described above when executed by a processor.

[0041] The present application first acquires a current ice image set corresponding to a target ice monitoring area, which is acquired in an orthographic shooting mode based on unmanned aerial vehicle remote sensing technology; then performs ice recognition on the current ice image set through a preset ice recognition network model to obtain an ice recognition image set corresponding to the current ice image set, the preset ice recognition network model comprising a U-Net branch network and a ResNet branch network, the U-Net branch network being configured to extract image features of an input image and obtain an ice feature map marking ice distribution based on the image features, the ResNet branch network being configured to acquire multi-scale semantic information of the input image and obtain a feature distinguishing map distinguishing between riverbanks and river courses based on the multi-scale semantic information, and the preset ice recognition network model performing feature fusion of the feature distinguishing map and the ice feature map at the connection between the U-Net branch network and the ResNet branch network; finally, ice distribution data corresponding to the target ice monitoring area is extracted according to the ice recognition image set. Compared with the existing ice condition monitoring by manual visual inspection, the present application can acquire river ice distribution data in real time by using the preset ice recognition network model to perform ice recognition on the current ice image set obtained by unmanned aerial vehicle visible light remote sensing, which not only improves the efficiency of judging ice flood disaster, but also enhances the ability to grasp the process of ice flood, and provides scientific and accurate data support for ice flood prevention and control. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a structural schematic diagram of an ice distribution intelligent extraction device of a hardware running environment related to an embodiment scheme of the present application;

[0043] Figure 2 is a flowchart of a first embodiment of an ice distribution intelligent extraction method of the present application;

[0044] Figure 3A flowchart of the ResNet branch network in the second embodiment of the river ice distribution intelligent extraction method of the present application is shown in Figure 6;

[0045] Figure 4 A flowchart of the channel attention mechanism in the second embodiment of the river ice distribution intelligent extraction method of the present application is shown in Figure 7;

[0046] Figure 5 A flowchart of the feature fusion module in the second embodiment of the river ice distribution intelligent extraction method of the present application is shown in Figure 8;

[0047] Figure 6 A structural diagram of the preset river ice recognition network model in the second embodiment of the river ice distribution intelligent extraction method of the present application is shown in Figure 9;

[0048] Figure 7 A flowchart of the spatial attention mechanism in the second embodiment of the river ice distribution intelligent extraction method of the present application is shown in Figure 10;

[0049] Figure 8 A flowchart of the third embodiment of the river ice distribution intelligent extraction method of the present application is shown in Figure 11;

[0050] Figure 9 A structural block diagram of the first embodiment of the river ice distribution intelligent extraction device of the present application is shown in Figure 12.

[0051] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are intended to explain the present application, but not to limit the present application.

[0053] Reference Figure 1 , Figure 1 A structural diagram of the river ice distribution intelligent extraction device related to the hardware running environment of the embodiment scheme of the present application is shown in Figure 13.

[0054] As Figure 1As shown, the river ice distribution intelligent extraction device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 can be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the river ice distribution intelligent extraction device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0056] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a river ice distribution intelligent extraction program.

[0057] In Figure 1 As shown in the river ice distribution intelligent extraction device, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the river ice distribution intelligent extraction device of the present application can be arranged in the river ice distribution intelligent extraction device, and the river ice distribution intelligent extraction device calls the river ice distribution intelligent extraction program stored in the memory 1005 through the processor 1001, and executes the river ice distribution intelligent extraction method provided by the embodiment of the present application.

[0058] The embodiment of the present application provides a river ice distribution intelligent extraction method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the river ice distribution intelligent extraction method of the present application.

[0059] In this embodiment, the river ice distribution intelligent extraction method includes the following steps:

[0060] Step S10: acquiring a current river ice image set corresponding to the target river ice monitoring area, the current river ice image set being acquired in an orthographic shooting mode based on a UAV remote sensing technology.

[0061] It should be noted that the execution subject of the method of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a mobile phone, a computer, a small server, etc., and can also be other electronic devices capable of achieving the same or similar functions, such as the above-mentioned river ice distribution intelligent extraction device, and the present embodiment does not limit this. Hereinafter, the river ice distribution intelligent extraction method provided by the present embodiment and each of the following embodiments will be described in detail with reference to the above-mentioned river ice distribution intelligent extraction device (referred to as extraction device).

[0062] It should be understood that the target river ice monitoring area is a river channel area that needs to be monitored in real time due to the need to prevent ice dam and ice bridge disasters after entering winter. For example, the Yellow River, due to the different geographical locations of the upper and lower reaches of the Yellow River, after entering winter, the ice water is in the lower reaches, which is prone to congestion, forming ice dams and ice bridges, blocking the water section, causing the water level to be high, forming ice floods, and seriously threatening the safety of people's lives and property in the beach area. Therefore, river ice monitoring can be carried out in the area where ice flood blocking is prone to occur in the Yellow River to judge the ice flood disaster in advance.

[0063] It should be noted that the current river ice image set is a set of images about the river channel obtained by using a UAV to shoot when monitoring the target river ice monitoring area. The current river ice image set is obtained in an orthographic shooting mode, which can fully reflect the current river ice distribution of the river section. According to the size of the river section and the clarity of the image, the number of images in the current river ice image set is not limited.

[0064] In actual operation, the above-mentioned extraction device can be carried on a UAV to realize the operation of intelligent extraction of river ice distribution. The UAV uses remote sensing technology to acquire the current river ice image set in an orthographic shooting mode, and transmits the current river ice image set to the extraction device for intelligent extraction of river ice distribution to acquire river ice distribution data in real time. With the continuous improvement of UAV and sensor technology, small UAVs can be used for complex river environment monitoring by carrying multiple sensors in low-altitude remote sensing measurement. The UAV platform has the characteristics of high efficiency, low cost and flexible operation. It not only overcomes the difficulty of obtaining image data manually, but also avoids image distortion caused by side shooting from the shore, improves the accuracy of raw data, and provides a new technical means for monitoring large-scale ice flood change information, which can greatly improve the tracking and prevention and control capability of ice flood.

[0065] Step S20: river ice recognition is performed on the current river ice image set by a preset river ice recognition network model to obtain a river ice recognition image set corresponding to the current river ice image set. The preset river ice recognition network model includes a U-Net branch network and a ResNet branch network. The U-Net branch network is used to extract image features of an input image and obtain a river ice feature map marking river ice distribution based on the image features. The ResNet branch network is used to obtain multi-scale semantic information of the input image and obtain a feature distinguishing map distinguishing river banks from river courses based on the multi-scale semantic information. The preset river ice recognition network model performs feature fusion of the feature distinguishing map and the river ice feature map at a connection position of the U-Net branch network and the ResNet branch network.

[0066] It should be noted that the preset river ice recognition network model is a network model that can recognize river ice and is pre-built. The part of the preset river ice recognition network model that realizes the function of recognizing river ice is mainly the U-Net branch network and the ResNet branch network. The U-Net branch network can be used to recognize the river ice region in the input image. It can extract image features of the input image and mark the river ice region according to the image features to obtain a river ice feature map marking the river ice. Or other network models that can achieve the same or similar function, which are not limited in the embodiment.

[0067] Because the turbidity of the Yellow River is different at different times and in different river sections, the pure U-Net branch network is easy to misclassify the land on the river bank and the turbid water body as the same feature, which affects the accuracy of river ice recognition. Therefore, in order to improve the accuracy of river ice recognition, the ResNet branch network can be used to segment the river bank and the river course as foreground and background of the input image, mark the river bank part, and obtain a feature distinguishing map distinguishing the river bank from the river course. Or other network models that can achieve the same or similar function, which are not limited in the embodiment. The ResNet branch network can sample the input image multiple times to obtain semantic information (i.e., multi-scale semantic information), and according to the obtained semantic information, the river bank and the river course can be segmented. Finally, at the connection position of the U-Net branch network and the ResNet branch network, the feature distinguishing map and the river ice feature map are vector weighted and added to themselves to generate fusion features to obtain a river ice recognition image set marking the river ice distribution.

[0068] Step S30: river ice distribution data corresponding to the target river ice monitoring area is extracted according to the river ice recognition image set.

[0069] It should be noted that the preset river ice recognition network model can recognize the current river ice image set obtained by the unmanned aerial vehicle shooting to obtain a river ice recognition image set marked with a river ice region. According to the river ice recognition image set, the river ice distribution data of the river section can be further extracted, and the real-time obtained river ice distribution data can be used to judge the flow ice growth and consumption process, grasp the ice disaster situation, and provide scientific and accurate data support for ice flood prevention and control.

[0070] Further, in order to improve the accuracy of feature fusion of the feature distinction map and the river ice feature map, and obtain more accurate river ice recognition image set, the preset river ice recognition network model further comprises a feature fusion module, a first input end of the feature fusion module is connected with the U-Net branch network, and a second input end of the feature fusion module is connected with the ResNet branch network. The embodiment further comprises the following steps in step S20:

[0071] Step S21: inputting the current river ice image set into the U-Net branch network for image feature extraction to obtain a river ice feature image.

[0072] It should be noted that the U-Net branch network can add a channel attention mechanism and a spatial attention mechanism when performing down-sampling on the current river ice image set to extract image features, so as to improve the ability of the network to recognize detailed features. The present embodiment does not limit this.

[0073] Step S22: inputting the current river ice image set into the ResNet branch network for image residual processing to obtain a river bank segmentation image.

[0074] It can be understood that the ResNet branch network can contain a certain number of Res residual blocks to prevent the degradation of deep learning and improve the model convergence speed. Meanwhile, batch normalization can be added to the operation of the Res residual block to perform batch processing on the data obtained by the residual block and improve the accuracy of the ResNet branch network. The present embodiment does not limit this.

[0075] Step S23: performing weight processing on the river ice feature image and the river bank segmentation image through the feature fusion module to obtain a river ice recognition image set corresponding to the current river ice image set.

[0076] It should be noted that a channel attention mechanism can be added before the weight processing of the river ice feature image and the river bank segmentation image to improve the fusion of important information of the river ice feature image and the river bank segmentation image, thereby improving the accuracy of the preset river ice recognition network model in recognizing river ice. The present embodiment does not limit this.

[0077] In a specific implementation, the current river ice image set is first input into the U-Net branch network to perform image feature extraction, and a river ice feature image is obtained; then the current river ice image set is input into the ResNet branch network to perform image residual processing, and a river bank segmentation image is obtained; finally, the river ice feature image and the river bank segmentation image are processed by the feature fusion module to obtain the river ice recognition image set corresponding to the current river ice image set, so as to improve the accuracy of feature fusion of the feature distinguishing image and the river ice feature image through the feature fusion module, and obtain a more accurate river ice recognition image set.

[0078] The embodiment first acquires a current river ice image set corresponding to a target river ice monitoring area, and the current river ice image set is acquired in an orthographic shooting mode based on an unmanned aerial vehicle remote sensing technology; then a river ice recognition network model is used to perform river ice recognition on the current river ice image set, and a river ice recognition image set corresponding to the current river ice image set is obtained, the river ice recognition network model includes a U-Net branch network and a ResNet branch network, the U-Net branch network is used to extract image features of an input image and obtain a river ice feature image marking river ice distribution based on the image features, the ResNet branch network is used to acquire multi-scale semantic information of the input image and obtain a feature distinguishing image distinguishing a river bank from a river channel based on the multi-scale semantic information, and the feature distinguishing image and the river ice feature image are fused at a connection between the U-Net branch network and the ResNet branch network; finally, river ice distribution data corresponding to the target river ice monitoring area is extracted according to the river ice recognition image set. Compared with the existing ice condition monitoring by manual visual inspection, since the current river ice image set obtained by the unmanned aerial vehicle visible light remote sensing is recognized by the river ice recognition network model, the river ice distribution data of the river channel can be acquired in real time, the efficiency of judging the ice flood disaster is improved, the grasping ability of the flow ice formation and disappearance process is improved, and scientific and accurate data support is provided for the ice flood prevention and control.

[0079] Based on the first embodiment, considering the construction of the preset river ice recognition network model, a second embodiment is provided.

[0080] In the embodiment, the river ice recognition network model is used to perform river ice recognition on the current river ice image set, and a river ice recognition image set corresponding to the current river ice image set is obtained. Figure 3 , Figure 3 FIG. 2 is a flowchart of the ResNet branch network in the second embodiment of the river ice distribution intelligent extraction method of the present application. The ResNet branch network includes a Res residual unit and an up-sampling unit, an output end of the Res residual unit is connected to an input end of the up-sampling unit, and an output end of the up-sampling unit is connected to a second input end of the feature fusion module; the step S22 includes:

[0081] Step S221: inputting the current river ice image set into the Res residual unit for hierarchical image residual processing to obtain residual images of each level, wherein the Res residual unit comprises Res residual blocks of a preset number of levels.

[0082] Step S222: performing significant analysis on the residual images of each level according to the channel attention mechanism, and inputting the analysis result into the up-sampling unit for up-sampling processing to obtain a river bank segmentation image.

[0083] It should be noted that the Res residual unit is a residual edge mechanism in the ResNet branch network, and adding Res residual blocks of a certain preset number of levels (for example, Res1, Res2, Res3, Res4, etc.) can improve the learning depth of the network. The new network layer at least will not make the effect worse than the original one, and can relatively stably improve the effect of the model by deepening the number of layers. In addition, another advantage of using the Res residual unit is that it can avoid the problem of gradient disappearance. Because of this feature, it can train a network of several hundred or even thousands of layers.

[0084] It can be understood that the up-sampling unit can up-sample the output features of the Res residual unit multiple times to obtain multi-scale semantic information, so that the river bank and river channel can be accurately segmented according to the obtained semantic information.

[0085] It should be understood that the channel attention mechanism can enhance the channels of feature maps useful for the current task and suppress the feature channels not useful for the current task. According to the channel attention mechanism, the attention function for distinguishing the river bank and the river channel can be improved. In the actual process, reference Figure 4 , Figure 4 is a flowchart of the channel attention mechanism in the second embodiment of the intelligent river ice distribution extraction method of the present application. The channel attention mechanism comprises a global weight unit, the structure of the global weight unit is Global pooling+1x1Conv+BN+Sigmiod, and the channel attention mechanism is composed of the product branch (Mul) of the output features provided by the global weight unit and the Res residual unit and the output feature weighted splicing (Sum) provided by the Res residual unit, wherein BN is a normalization module, 1x1Conv is a 1x1 convolution, Sigmiod is a Sigmiod activation function, and Global pooling is a global pooling. According to the channel attention mechanism, the attention of the output features provided by the Res residual unit can be improved to improve the function of distinguishing the river bank and the river channel.

[0086] In a specific implementation, the current river ice image set is input into the Res residual unit for hierarchical image residual processing to obtain residual images of each level; then the channel attention mechanism is used to perform significant analysis on the residual images of each level to improve the attention of the output features provided by the Res residual unit to improve the function of distinguishing river banks and river courses; finally, the analysis results are input into the up-sampling unit for up-sampling processing to obtain a river bank segmentation image.

[0087] Further, in order to obtain global context information, a global context vector weighting can be performed after the channel attention mechanism after the last residual block (for example, the Res4 residual block) in the Res residual unit, and the vector is obtained by global pooling of the output feature map of Res4.

[0088] Further, in order to improve the function of the U-Net branch network for identifying river ice, the U-Net branch network includes a down-sampling layer, an up-sampling layer, and a network connection layer; and the step S21 includes: inputting the current river ice image set into the down-sampling layer for hierarchical down-sampling to obtain shallow feature images of each level; inputting the shallow feature images of each level of the down-sampling layer into the up-sampling layer of each level through the network connection layer to extract deep features by hierarchical up-sampling, and obtaining a river ice feature image according to the deep features.

[0089] It should be noted that the down-sampling layer belongs to the feature extraction part, and it is a contraction network that reduces the picture size layer by layer through a certain number of down-sampling (for example, four down-sampling), and in this process of continuous down-sampling, the feature extraction finally obtains shallow feature images of each level.

[0090] Further, in order to improve the ability of the network to identify detailed features, a channel attention mechanism and a spatial attention mechanism can also be added in the process of down-sampling in the down-sampling layer, which is not limited in this embodiment.

[0091] It can be understood that the up-sampling layer is also called an expansion network, which makes the picture size larger, and the number of layers of the up-sampling layer corresponds to the number of layers of the down-sampling layer, four up-samplings are used to extract deep features, and in the process of up-sampling, the number of channels of the picture is halved, which is contrary to the change of the number of channels of the feature extraction of the down-sampling layer. The process of up-sampling fuses the shallow information of the down-sampling layer, that is, splices the shallow feature images of the down-sampling layer.

[0092] It should be understood that the role of the network connection layer is to perform copy and crop splicing, and there are a corresponding number of splicing operations in the U-Net branch network, for example, there are four downsampling operations in the downsampling layer, and there are a corresponding four splicing operations in the network connection layer. The purpose of the network connection layer is to fuse feature information, and the shallow feature information of each layer of the downsampling layer is spliced and fused with the deep information of each layer of the upsampling layer, so that the deep and shallow information is fused. When splicing, not only the picture size should be consistent, but also the feature dimensions should be the same, so that splicing can be performed, that is, the feature fusion of each layer of the downsampling layer and each layer of the upsampling layer is performed one by one.

[0093] Further, considering that the U-Net branch network and the ResNet branch network are connected, the preset river ice recognition network model performs feature fusion on the river ice feature image and the river bank segmentation image, and the feature fusion module includes a scale balancing unit, a global weight unit and a fusion unit; the step S23 includes: inputting the river ice feature image and the river bank segmentation image into the scale balancing unit to perform feature balancing, and obtaining an initial fusion image; inputting the initial fusion image into the global weight unit to perform global weight processing, and obtaining a global fusion image; performing product processing on the initial fusion image to obtain a product fusion image corresponding to the initial fusion image; inputting the initial fusion image, the global fusion image and the product fusion image into the fusion unit to perform weighted processing, and obtaining a river ice recognition image set corresponding to the current river ice image set.

[0094] It should be noted that the scale balancing unit is used to balance the size between the river ice feature image transmitted by the U-Net branch network and the river bank segmentation image transmitted by the ResNet branch network, so as to improve the accuracy of image fusion.

[0095] The structure of the feature fusion module can be referred to Figure 5 , Figure 5 is a flowchart of the feature fusion module in the second embodiment of the intelligent river ice distribution extraction method of the present application; the structure of the scale balancing unit is 1x1Conv+BN+ReLU, the structure of the global weight unit is Global pooling+1x1Conv+BN+Sigmiod, and the feature fusion module is formed by the product branch of the global weight unit and the scale balancing unit and the fusion unit spliced with the scale balancing unit and the fusion unit; wherein, BN is a normalization module, ReLU is an activation function, 1x1Conv is a 1x1 convolution, Sigmiod is a Sigmiod activation function, and Global pooling is a global pooling.

[0096] In a specific implementation, when the river ice feature image and the river bank segmentation image are fused at the connection of the two branches of the U-Net branch network and the ResNet branch network, first, the river ice feature image and the river bank segmentation image are merged (Concat), and then the merged image is subjected to 1x1 convolution (1x1Conv), batch normalization (BN) and ReLU activation function for linear correction to balance the scale of the feature map, to obtain an initial fusion image; then, a weight vector is obtained like a channel attention module, that is, the initial fusion image is sequentially subjected to global pooling, convolution (1x1Conv kernel), batch normalization (BN) and Sigmoid operation to introduce a global weight vector to obtain a global fusion image; the initial fusion image is subjected to product processing (Mul) to obtain a product fusion image corresponding to the initial fusion image; finally, the initial fusion image, the global fusion image and the product fusion image are input into the fusion unit for weighting (Sum) to generate fusion features, to obtain a river ice recognition image set.

[0097] In actual consideration, the implementation process of the preset river ice recognition network model is as shown in Figure 6 Figure 6 FIG. 1 is a structural schematic diagram of a preset river ice recognition network model in a second embodiment of the river ice distribution intelligent extraction method of the present application; the network has two branches, the branch in the upper part of the figure is a U-Net branch network, which takes the U-Net network structure as a prototype and is responsible for recognizing river ice; the branch in the lower part of the figure is a ResNet branch network, which takes the ResNet network structure with added channel attention mechanism as a prototype and is responsible for distinguishing river banks and the Yellow River; a feature fusion module (Fusion) weights the feature maps processed by connection and processing by vectors and adds them to themselves to generate fusion features, and the specific method is as follows:

[0098] (1) U-Net branch network: after the current river ice image set is input, 3x3 convolution kernels are mainly used in the U-Net branch network for processing including downsampling, pooling, upsampling (Up conv 2x2) and horizontal connection (i.e. network connection layer).

[0099] ​In the U-Net branch network, a left-right symmetric encoder-decoder structure is adopted, and a down-sampling layer and an up-sampling layer are added. The down-sampling layer can compress the input image features and retain key information, acting as an encoder. The up-sampling layer can recover the feature map, thereby outputting a segmentation image of the same size as the original image. The network uses a skip connection (Copy and crop) to fuse the pixel-level features and semantic-level features of the image, thereby realizing pixel-level semantic segmentation and being suitable for identifying and extracting high-resolution small targets. When using U-Net to construct the river ice extraction branch network, 3x3 convolution, ReLU activation function (Conv 3x3, ReLU), and 2x2 maximum pooling (Maxpool 2x2) are used for down-sampling to extract river ice feature information. After pooling, the number of feature map channels remains unchanged, and the size is reduced to 1 / 2 of the original size. Down-sampling is performed 4 times, and at the same time, a channel attention mechanism (Position attention) and a spatial attention mechanism are added during down-sampling to improve the network's ability to identify detailed features. Finally, a 1x1 convolution kernel (conv 1x1) is used for channel adjustment to convert the number of feature layers to be fused to the number of classes, and finally a river ice feature image marking the distribution of river ice is obtained.

[0100] The spatial attention mechanism is as shown in Figure 7 Figure 7 is a flowchart of the spatial attention mechanism in the second embodiment of the intelligent river ice distribution extraction method of the present application. The spatial attention mechanism includes a pooling unit, and the structure of the pooling unit is Maxpool+Avgpool+1x1Conv+BN+Sigmiod. The spatial attention mechanism is composed of the pooling unit, a multiplication branch (Mul) of output features provided by down-sampling, and a weighted splicing (Sum) of output features provided by down-sampling. Among them, BN is a normalization module, 1x1Conv is a 1x1 convolution, Sigmiod is a Sigmiod activation function, Maxpool is maximum pooling, and Avgpool is average pooling. According to the spatial attention mechanism, the spatial relationship between the output features provided by the down-sampling can be improved.

[0101] ​(2) The ResNet branch network adds a residual learning mechanism, effectively preventing the degradation of the deep learning network, and improving the model convergence speed. When using ResNet to construct the river ice extraction network branch, a 7x7 convolution kernel (Conv7x7) is used, a maximum pooling operation (Max pool 2x2) is performed to obtain the corresponding pool, and the branch backbone network includes 4 residual blocks, which are marked as Res1, Res2, Res3 and Res4. Among them, Res1 is the original residual block, the convolution step of Res2, Res3 and Res4 is set to 1, and a batch normalization layer (Mul) is added based on Res4 Relu, and a channel attention module (Channel attention) is added after each residual block in Res2, Res3 and Res4. In addition, in order to obtain the global context information, the feature map output after Res4 is subjected to global average pooling, and is input into the channel attention module after Res4 as a global context vector for weighting operation; finally, the outputs of the two channel attention modules after Res2 and Res3 and the channel attention module after Res4 are concatenated and upsampled 4 times (Concat 16Upsample) to obtain multi-scale semantic information, distinguish the river bank and the river channel for the input image, and obtain the river bank segmentation image.

[0102] (3) When the two branches are fused, first, 1x1 convolution, batch normalization and ReLU unit are performed to balance the scale of the feature map, and then a global weight vector is introduced to weight the feature map to generate fused features. The river ice recognition image set corresponding to the current river ice image set, the fusion mechanism process is as shown in Figure 5

[0103] The embodiment uses unmanned aerial vehicle remote sensing and deep learning semantic segmentation technology for river ice recognition and extraction, proposes a double-branch semantic segmentation construction preset river ice recognition network model, which can simultaneously consider the spatial correlation information between pixels and perform pixel-level classification. The ResNet branch network in the preset river ice recognition network model is used to segment the Yellow River and the river bank as foreground and background, and the Yellow River part is extracted first, which can greatly avoid the interference of background elements such as grass and buildings on the river bank during river ice extraction; then the U-Net branch network is combined to perform small-scale river ice segmentation and recognition, and the river ice is further identified; thereby the river ice can be quickly recognized and extracted, and the traditional river ice monitoring method is more flexible.

[0104] Reference Figure 8 , Figure 8 is a flowchart of the third embodiment of the river ice distribution intelligent extraction method of the present application.

[0105] ​Based on the above embodiments, in this embodiment, considering the training of the preset river ice recognition network model, the accuracy of the preset river ice recognition network model in performing river ice recognition on the input image is improved, and before the step S10, the method further comprises:

[0106] Step S01: Obtain a preset number of image training sets.

[0107] It should be noted that the image training set is an image set for training the preset river ice recognition network model. A certain preset number (for example, 200) of unmanned aerial orthographic photograph sets of different times and different river sections during the ice flood period can be obtained, and the photograph size can be 5472x3648 specifications. The number of photographs and the size specifications are not limited in this embodiment.

[0108] Step S02: Use an image labeling tool to mark the background of the image training set to obtain a labeled image training set.

[0109] In a specific implementation, the image training set needs to be background labeled to make data labels. The image labeling tool (such as Adobe Photoshop) can be used to label the river bank and river ice area of the image training set. When the preset river ice recognition network model is trained, the non-river bank and river ice area will be automatically treated as a water body background.

[0110] Step S03: Iteratively train the initial river ice recognition network model to be trained according to the labeled image training set to obtain a preset river ice recognition network model.

[0111] In actual consideration, using the completed labeled image training set, the labeled image training set can be randomly divided into a training data set and a test data set according to a 7:3 ratio, and the training data set can be input into the initial river ice recognition network model constructed according to the above double-branch semantic segmentation for training. In actual algorithm operation, the photograph can be completed as a black background along the height direction, and the converted photograph size can be 5472x5472 size specifications. The model output result is re-cropped back to the normal photograph size 5472x3648 size specifications. This embodiment does not limit this. Finally, using the trained model, the current river ice image set is predicted, and the river ice distribution is extracted.

[0112] This embodiment obtains a preset number of image training sets, uses an image labeling tool to mark the background of the image training set to obtain a labeled image training set, and finally iteratively trains the initial river ice recognition network model to be trained according to the labeled image training set to obtain a preset river ice recognition network model, thereby improving the accuracy of the preset river ice recognition network model in performing river ice recognition on the input image, and improving the efficiency of river ice recognition.

[0113] In addition, the present application also provides a storage medium, wherein the storage medium stores a river ice distribution intelligent extraction program, and the river ice distribution intelligent extraction program is executed by a processor to implement the steps of the river ice distribution intelligent extraction method.

[0114] Reference Figure 9 , Figure 9 The present application is a river ice distribution intelligent extraction device.

[0115] As Figure 9 shown, the river ice distribution intelligent extraction device provided by the present application comprises:

[0116] An image acquisition module 901 is configured to acquire a current river ice image set corresponding to a target river ice monitoring area, wherein the current river ice image set is acquired in an orthographic shooting mode based on a UAV remote sensing technology.

[0117] A river ice recognition module 902 is configured to perform river ice recognition on the current river ice image set by using a preset river ice recognition network model, and obtain a river ice recognition image set corresponding to the current river ice image set, wherein the preset river ice recognition network model comprises a U-Net branch network and a ResNet branch network, the U-Net branch network is configured to extract image features of an input image and obtain a river ice feature map marking river ice distribution based on the image features, the ResNet branch network is configured to obtain multi-scale semantic information of the input image and obtain a feature distinguishing map distinguishing river banks and river channels based on the multi-scale semantic information, and the preset river ice recognition network model performs feature fusion on the feature distinguishing map and the river ice feature map at a connection position of the U-Net branch network and the ResNet branch network.

[0118] A river ice distribution module 903 is configured to extract river ice distribution data corresponding to the target river ice monitoring area according to the river ice recognition image set.

[0119] The embodiment first acquires a current river ice image set corresponding to a target river ice monitoring area, the current river ice image set being acquired in an orthographic shooting mode based on a UAV remote sensing technology; then river ice recognition is performed on the current river ice image set through a preset river ice recognition network model to obtain a river ice recognition image set corresponding to the current river ice image set, the preset river ice recognition network model comprising a U-Net branch network and a ResNet branch network, the U-Net branch network being used to extract image features of an input image and obtain a river ice feature map marking river ice distribution based on the image features, the ResNet branch network being used to acquire multi-scale semantic information of the input image and obtain a feature distinguishing map distinguishing river banks from river courses based on the multi-scale semantic information, the preset river ice recognition network model performing feature fusion on the feature distinguishing map and the river ice feature map at a connection between the U-Net branch network and the ResNet branch network; finally, river ice distribution data corresponding to the target river ice monitoring area is extracted according to the river ice recognition image set. Compared with existing ice condition monitoring by manual visual inspection, since the preset river ice recognition network model is used to perform river ice recognition on a current river ice image set obtained by a UAV visible light remote sensing, river course river ice distribution data can be acquired in real time, which not only improves the efficiency of judging ice flood disaster conditions, but also improves the grasping ability of the flow ice formation and disappearance process, thereby providing scientific and accurate data support for ice flood prevention and control.

[0120] Other embodiments or specific implementations of the river ice distribution intelligent extraction device can refer to the above method embodiments, which will not be described here.

[0121] It should be noted that in this document, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0122] The above embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0123] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0124] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for intelligent extraction of river ice distribution, characterized in that, The intelligent extraction method for river ice distribution includes: Acquire the current river ice image set corresponding to the target river ice monitoring area. The current river ice image set is acquired based on UAV remote sensing technology in orthophoto mode. The river ice recognition network model is used to identify river ice in the current river ice image set to obtain a river ice recognition image set corresponding to the current river ice image set. The preset river ice recognition network model includes a U-Net branch network and a ResNet branch network. The U-Net branch network is used to extract image features of the input image and obtain a river ice feature map that marks the distribution of river ice based on the image features. The ResNet branch network is used to obtain multi-scale semantic information of the input image and obtain a feature distinction map that distinguishes riverbanks from river channels based on the multi-scale semantic information. At the connection between the U-Net branch network and the ResNet branch network, the preset river ice recognition network model performs feature fusion between the feature distinction map and the river ice feature map. Extract river ice distribution data corresponding to the target river ice monitoring area based on the river ice identification image set; The ResNet branch network includes a Res residual unit and an upsampling unit. The output of the Res residual unit is connected to the input of the upsampling unit, and the output of the upsampling unit is connected to the second input of the feature fusion module. The current river ice image set is input into the Res residual unit for hierarchical image residual processing to obtain residual images at each level. The Res residual unit contains a preset number of Res residual blocks. The residual images at each level are significantly analyzed based on the channel attention mechanism, and the analysis results are input into the upsampling unit for upsampling processing to obtain the riverbank segmentation image; The U-Net branch network includes a downsampling layer, an upsampling layer, and a network connection layer. The current river ice image set is input into the downsampling layer for hierarchical downsampling to obtain shallow feature images of each layer; The shallow feature images of each level of the downsampling layer are input to each level of the upsampling layer through the network connection layer to extract deep features, and the river ice feature image is obtained based on the deep features.

2. The intelligent extraction method for river ice distribution as described in claim 1, characterized in that, The preset river ice recognition network model further includes a feature fusion module. The first input of the feature fusion module is connected to the U-Net branch network, and the second input of the feature fusion module is connected to the ResNet branch network. The step of performing river ice identification on the current river ice image set using a preset river ice identification network model to obtain the river ice identification image set corresponding to the current river ice image set includes: The current set of river ice images is input into the U-Net branch network for image feature extraction to obtain river ice feature images; The current river ice image set is input into the ResNet branch network for image residual processing to obtain riverbank segmentation images; The feature fusion module performs weighted processing on the river ice feature image and the riverbank segmentation image to obtain the river ice recognition image set corresponding to the current river ice image set.

3. The intelligent extraction method for river ice distribution as described in claim 2, characterized in that, The feature fusion module includes a scale balancing unit, a global weighting unit, and a fusion unit; The step of performing weighted processing on the river ice feature image and the riverbank segmentation image through the feature fusion module to obtain the river ice recognition image set corresponding to the current river ice image set includes: The river ice feature image and the riverbank segmentation image are input into the scale balancing unit for feature balancing to obtain an initial fused image; The initial fused image is input into the global weight unit for global weight processing to obtain the global fused image; Perform product processing on the initial fused image to obtain the product fused image corresponding to the initial fused image; The initial fused image, the global fused image, and the product fused image are input into the fusion unit for weighted processing to obtain the river ice recognition image set corresponding to the current river ice image set.

4. The intelligent extraction method for river ice distribution as described in claim 3, characterized in that, The scale balancing unit has a 1x1 Conv+BN+ReLU structure, the global weight unit has a Global pooling+1x1 Conv+BN+Sigmoid structure, and the feature fusion module is formed by the product branch of the global weight unit and the scale balancing unit, and by splicing the scale balancing unit and the fusion unit. Wherein, BN is the normalization module, ReLU is the activation function, 1x1Conv is the 1x1 convolution, Sigmoid is the Sigmoid activation function, and Global pooling is the global pooling.

5. The intelligent extraction method for river ice distribution as described in any one of claims 1 to 4, characterized in that, Before the step of acquiring the current river ice image set corresponding to the target river ice monitoring area, wherein the current river ice image set is acquired based on UAV remote sensing technology in orthophoto mode, the method further includes: Obtain a preset number of image training sets; The image training set is labeled with a background using an image labeling tool to obtain a labeled image training set. The initial river ice recognition network model to be trained is iteratively trained based on the labeled image training set to obtain the preset river ice recognition network model.

6. A smart extraction device for river ice distribution, characterized in that, The device performs the intelligent extraction method for river ice distribution as described in claim 1, and the device includes: The image acquisition module is used to acquire the current river ice image set corresponding to the target river ice monitoring area. The current river ice image set is acquired based on UAV remote sensing technology in orthophoto mode. A river ice recognition module is used to perform river ice recognition on the current river ice image set through a preset river ice recognition network model, and obtain a river ice recognition image set corresponding to the current river ice image set. The preset river ice recognition network model includes a U-Net branch network and a ResNet branch network. The U-Net branch network is used to extract image features of the input image and obtain a river ice feature map that marks the distribution of river ice based on the image features. The ResNet branch network is used to obtain multi-scale semantic information of the input image and obtain a feature distinction map that distinguishes riverbanks and river channels based on the multi-scale semantic information. At the connection between the U-Net branch network and the ResNet branch network, the preset river ice recognition network model performs feature fusion between the feature distinction map and the river ice feature map. The river ice distribution module is used to extract river ice distribution data corresponding to the target river ice monitoring area based on the river ice identification image set.

7. A smart extraction device for river ice distribution, characterized in that, The device includes: a memory, a processor, and a smart ice distribution extraction program stored in the memory and executable on the processor, the smart ice distribution extraction program being configured to implement the steps of the smart ice distribution extraction method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a smart extraction program for river ice distribution, which, when executed by a processor, implements the steps of the smart extraction method for river ice distribution as described in any one of claims 1 to 5.

Images