A training method and device of a semantic segmentation neural network based on water surface data

Abstract

The application provides a training method and device of a semantic segmentation neural network based on water surface data. The method comprises the following steps: obtaining open-pit mine flood control dam water surface data, performing binaryzation processing on the water surface data, and obtaining binaryzation image data; a first candidate semantic segmentation neural network is constructed, wherein the first candidate semantic segmentation neural network is composed of an encoder and a decoder; public water surface data is obtained, the first candidate semantic segmentation neural network is coarsely trained according to the public water surface data, and after the training is completed, a second candidate semantic segmentation neural network is obtained; the open-pit mine flood control dam water surface data is input into the second candidate semantic segmentation neural network, segmentation image data is obtained, a loss value is calculated based on the segmentation image and the binaryzation image data, the second candidate semantic segmentation neural network is finely trained according to the loss value, and the training is completed until a semantic segmentation neural network is obtained. The two-step training method can improve the accuracy of the water surface data segmentation task.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a training method and apparatus for a semantic segmentation neural network based on water surface data. Background Technology

[0002] Semantic segmentation is a fundamental research area in computer vision (CV). It can assign a specific category to each pixel in an image. For example, it can analyze objects in an image or video stream and label their category pixel by pixel. Semantic segmentation has been widely used in many fields such as autonomous driving, smart cities, and medical image processing.

[0003] Currently, there are several detection schemes for the scenario of water level detection in open-pit mine flood control dams: setting up fixed water gauges in the water and detecting the fixed water gauges based on target detection to detect the water level; setting up multiple different colored marker lines from top to bottom on the riverbank, inferring the position of the water surface based on deep learning methods, and comparing it with the marker lines to obtain different water level levels; and directly detecting the water level based on sensors.

[0004] However, setting up a fixed water gauge in the water requires a relatively flat bottom and the bottom should not change frequently. Setting up multiple different colored marker lines from top to bottom on the riverbank requires a relatively flat and undulating bottom and surface, and the riverbank should be relatively flat and perpendicular to the horizontal plane. These requirements are too demanding. Using sensors is too expensive and easily affected by the environment. Summary of the Invention

[0005] To address the aforementioned issues, a training method and apparatus for a semantic segmentation neural network based on water surface data is proposed to separate the water surface from the dam body, thereby facilitating the observation of whether the water level exceeds the limit.

[0006] The first aspect of this application proposes a training method for a semantic segmentation neural network based on water surface data, comprising:

[0007] Obtain water surface data of the open-pit mine flood control dam, and perform binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data;

[0008] Construct a first candidate semantic segmentation neural network, wherein the first candidate semantic segmentation neural network consists of an encoder and a decoder, the encoder is used to reduce the dimensionality of the input image and encode it, and the decoder is used to extract the semantic label of each pixel in the encoded image;

[0009] Obtain publicly available water surface data, and perform coarse training on the first candidate semantic segmentation neural network based on the publicly available water surface data. After training, a second candidate semantic segmentation neural network is obtained.

[0010] The water surface data of the open-pit mine flood control dam is input into the second candidate semantic segmentation neural network to obtain segmented image data. Based on the segmented image and the binarized image data, the loss value is calculated. The second candidate semantic segmentation neural network is fine-trained according to the loss value until the training is completed, and a trained semantic segmentation neural network is obtained.

[0011] Optionally, the step of acquiring water surface data of the open-pit mine flood control dam and performing binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data includes:

[0012] Multiple surveillance cameras were deployed along the waterfront to obtain water surface data of the open-pit mine flood control dam;

[0013] For the first image in the open-pit mine flood control dam water surface data, a closed polygon composed of a preset set of points is used to surround the water surface area in the first image;

[0014] The first image is binarized, with pixels in the enclosed area assigned red and pixels in the unenclosed area assigned black.

[0015] The remaining images in the open-pit mine flood control dam water surface data are subjected to the binarization process described above. After all images are labeled, the binarized image data is obtained.

[0016] Optionally, the encoder includes five convolutional networks, which are used for:

[0017] Maxim pooling, interspersed across five convolutional networks, is used to reduce the dimensionality of the input image;

[0018] In the fourth and fifth convolutional networks, dilated convolutions are used to increase the receptive field of the encoder.

[0019] Optionally, the decoder includes:

[0020] The first attention refinement module is used for deep dimensionality reduction using global average pooling and normalization, fusing input image features and IMU features, and reweighting them;

[0021] The first void space pyramid pooling module is used to refine the segmentation of small water structures based on the input image features and IMU features;

[0022] The first feature fusion module is used to fuse the output features of the first attention refinement module and the first void space pyramid pooling module, and generate 1024 feature channels.

[0023] The second attention refinement module is used to fuse features from the third convolutional network, features from IMU channel encoding, and features transmitted from the first feature fusion module.

[0024] The second feature fusion module is used to sample the features transmitted by the second attention refinement module, fuse the low-dimensional features transmitted by the second attention refinement module with the high-dimensional features transmitted by the second convolutional network, and reweight the fused features.

[0025] Optionally, the decoder further includes:

[0026] The second hollow spatial pyramid pooling module is used to segment the water structure of the input image and extract the semantic label of each pixel. The second hollow spatial pyramid pooling module has a built-in softmax function.

[0027] Optionally, when training the first candidate semantic segmentation neural network and the second candidate semantic segmentation neural network, the method further includes:

[0028] The high-resolution features encoded by the second convolutional network in the encoder and the IMU-encoded features are input into the second feature fusion module in the decoder;

[0029] The medium-resolution features and IMU-encoded features encoded by the second convolutional network in the encoder are input into the second attention refinement module of the decoder;

[0030] The low-resolution features and IMU-encoded features encoded by the fifth convolutional network in the encoder are input into the first attention refinement module of the decoder, and the encoded information is decoded using stacked deconvolutional layers, mapping the extracted information onto the two-dimensional plane of the original input space.

[0031] Optionally, the method further includes:

[0032] Add a residual module between the encoder and the decoder;

[0033] Zero-filling sparse operation is performed on the convolution kernels of the pre-numbered convolutional networks in the encoder.

[0034] Optionally, the second candidate semantic segmentation neural network adopts the cross-entropy function as the loss function, and the expression of the loss function is:

[0035]

[0036] Among them, y ic p represents the label of the i-th pixel. ic It is the predicted probability that pixel i belongs to category c.

[0037] The second aspect of this application proposes a training device for a semantic segmentation neural network based on water surface data, comprising:

[0038] The acquisition module is used to acquire water surface data of the open-pit mine flood control dam, and to perform binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data;

[0039] A construction module is used to construct a first candidate semantic segmentation neural network, wherein the first candidate semantic segmentation neural network consists of an encoder and a decoder, the encoder is used to reduce the dimensionality of the input image and encode it, and the decoder is used to extract the semantic label of each pixel in the encoded image;

[0040] The coarse training module is used to acquire public water surface data and perform coarse training on the first candidate semantic segmentation neural network based on the public water surface data. After the training is completed, the second candidate semantic segmentation neural network is obtained.

[0041] The fine training module is used to input the water surface data of the open-pit mine flood control dam into the second candidate semantic segmentation neural network to obtain segmented image data. Based on the segmented image and the binarized image data, a loss value is calculated. The second candidate semantic segmentation neural network is fine-trained according to the loss value until the training is completed, and a trained semantic segmentation neural network is obtained.

[0042] A third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the methods described in the first aspect above.

[0043] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0044] By constructing a semantic segmentation neural network, the parts concerning the water surface and the dam body in the water surface data can be separated, which is beneficial for subsequent water level detection. By using coarse training and fine training methods to train the candidate semantic segmentation neural network, the robustness of the semantic segmentation neural network can be improved, thereby improving the accuracy of water surface data segmentation tasks.

[0045] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0046] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart illustrating a training method for a semantic segmentation neural network based on water surface data, according to an embodiment of this application.

[0048] Figure 2 This is a schematic diagram of the encoder and decoder structure according to an embodiment of this application;

[0049] Figure 3 This is a block diagram illustrating a training device for a semantic segmentation neural network based on water surface data, according to an embodiment of this application.

[0050] Figure 4 It is a block diagram of an electronic device. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0052] Figure 1 This is a flowchart illustrating a training method for a semantic segmentation neural network based on water surface data according to an embodiment of this application, including the following steps:

[0053] Step 101: Obtain water surface data of the open-pit mine flood control dam, and perform binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data.

[0054] In this embodiment, a monitoring camera with multiple IMU sensors is deployed at the water's edge. The shooting angle and position of the monitoring camera are adjusted until the monitoring area can completely cover the entire flood control dam body. Water surface images under different environmental and water level conditions are collected by the monitoring camera.

[0055] It should be noted that the IMU sensor is an inertial measurement unit, which is a sensor used to detect and measure acceleration and rotational motion. The IMU sensor can record the inertial data when the camera is collecting image data of the water surface of the open-pit mine flood control dam.

[0056] In addition, in this embodiment of the application, the image in the water surface data of the open-pit mine flood control dam is binarized, specifically:

[0057] For the first image in the open-pit mine flood control dam water surface data, a closed polygon composed of a preset set of points is used to enclose the water surface area in the first image;

[0058] The first image is binarized, with pixels in the enclosed area assigned red and pixels in the unenclosed area assigned black.

[0059] The remaining images in the open-pit mine flood control dam water surface data are subjected to the binarization process described above. After all images are labeled, the binarized image data is obtained.

[0060] Step 102: Construct a first candidate semantic segmentation neural network, which consists of an encoder and a decoder. The encoder is used to reduce the dimensionality of the input image and encode it, and the decoder is used to extract the semantic label of each pixel in the encoded image.

[0061] In this embodiment, the candidate semantic segmentation neural network consists of an encoder and a decoder. The encoder uses a convolutional network with residual and dilated convolutions. Five convolutional networks and interspersed max pooling are used to reduce the dimensionality of image information. In the latter two convolutional networks, dilated convolutions are used to increase the receptive field of the encoder, so that it can extract more deep image features when encoding the input image.

[0062] The decoder needs to fuse the feature information encoded by the image encoder and the feature information encoded by the IMU encoder, and then decode and refine them into the final segmentation output. It selects features with different resolutions from the outputs of the second, third, and fifth convolutional networks to participate in the fusion, and uses an attention refinement module, a holed spatial pyramid pooling module, and a feature fusion module to fuse the image encoded information and the IMU encoded information, respectively.

[0063] Specifically, such as Figure 2 As shown, the relationship between the encoder and decoder is as follows:

[0064] (1) First, the feature E5 encoded by the last convolutional network in the image encoder is simultaneously input into the first hollow spatial pyramid pooling module ASPP1 and the first attention refinement module ARM1, which have three pyramid layers. The ASPP1 module is used to prune the coarse-resolution E5 features to refine the segmentation of small water structures, while the ARM1 module uses global average pooling and normalized depth dimensionality reduction to process the fusion and reweighting of image features and IMU features.

[0065] (2) The first feature fusion module FFM1 of the adaptive fusion method fuses the trimmed features of ASPP1 module and the output features of ARM1 module to generate 1024 feature channels.

[0066] (3) The features fused by the first feature fusion module FFM1 are further fused with the E3 features and IMU channel features encoded by the third convolutional network in the image encoder by the second attention refinement module ARM2. Specifically:

[0067] The ARM2 module first fuses the IMU channel features and the E3 features from the encoder. Next, it uses a set of 1×1 convolutions to double the number of feature channels. Each channel is added to the features from the FFM1 from the decoder. Finally, the refined and reweighted features from the ARM2 module, along with the E2 features encoded by the second convolutional network of the image encoder and the IMU feature channels, are input into the second feature fusion module FFM2.

[0068] (4) The second feature fusion module FFM2 implements a more complex fusion path than the second attention refinement module ARM2. By upsampling the output features of the ARM2 module and connecting them with the E2 features and IMU feature channels, the FFM2 module combines the low-dimensional features from the ARM2 module and the high-dimensional features from the E2 of the image encoder. The depth of the output feature channel is halved by a 3×3 convolution block and normalized by a batch normalization module.

[0069] It should be noted that the calculation of the weight vector of the second feature fusion module FFM2 is similar to that of ARM1, and it uses reweighted features to complete the selection and fusion of image coding features and IMU coding features.

[0070] (5) The decoder output finally extracts the semantic label of each pixel. Adding the second hole spatial pyramid pooling module ASPP2 can significantly improve the segmentation of water structure while only increasing the computational overhead by a small amount.

[0071] Therefore, a softmax ASPP2 module is introduced at the end of the decoder to perform tag classification while decoding.

[0072] It should be noted that the resolution of the output image after fusion and decoding by the decoder is one-quarter of the input resolution, so it is upsampled four times to match the resolution of the input image.

[0073] In addition, zero-filling sparsity is performed on the convolutional kernels of the convolutional layers after the encoder, which extracts more contextual information by using the sparse convolutional layers. A residual module is added between the encoder and decoder, and the residual connection is used to help extract more multi-scale information and make the training of the neural network easier to converge.

[0074] Step 103: Obtain public water surface data, perform coarse training on the first candidate semantic segmentation neural network based on the public water surface data, and obtain the second candidate semantic segmentation neural network after training.

[0075] In this embodiment of the application, as described in step 102, the data processing during training, and the linkage between the encoder and decoder, are as follows:

[0076] The high-resolution features encoded by the second convolutional network in the encoder and the IMU-encoded features are input into the second feature fusion module FFM2 in the decoder;

[0077] The medium-resolution features and IMU-encoded features encoded by the second convolutional network in the encoder are input into the second attention refinement module ARM2 of the decoder;

[0078] The low-resolution features encoded by the fifth convolutional network in the encoder and the IMU-encoded features are input into the first attention refinement module ARM1 of the decoder, and the encoded information is decoded using stacked deconvolutional layers, mapping the extracted information onto the two-dimensional plane of the original input space.

[0079] In this embodiment of the application, the first step is to use the disclosed water surface data to coarsely train the first candidate semantic segmentation neural network, so that the candidate network learns the general features of the water surface.

[0080] In one possible embodiment, the publicly available water surface data is the HydroSHEDS dataset.

[0081] Additionally, it is understandable that model training is an iterative process. The model is trained by continuously adjusting its network parameters until the overall loss function value is less than the preset value, or the overall loss function value no longer changes or changes slowly, at which point the model converges and a well-trained model is obtained.

[0082] Alternatively, training can be considered complete once a preset number of training sessions has been completed.

[0083] Alternatively, training can be considered complete once a preset training time has been reached.

[0084] Step 104: Input the water surface data of the open-pit mine flood control dam into the second candidate semantic segmentation neural network to obtain segmented image data. Calculate the loss value based on the segmented image and the binarized image data. Perform fine training on the second candidate semantic segmentation neural network according to the loss value until the training is completed, and obtain the trained semantic segmentation neural network.

[0085] In this embodiment of the application, the second candidate semantic segmentation neural network obtained after coarse training is fine-trained.

[0086] Specifically, during training, the original image is used as the input to the neural network. The loss is calculated by comparing the output segmented image with the binarized image data after binarization using labeled points. The network parameters are updated by propagating the loss value from the back end of the network forward.

[0087] It should be noted that when the dataset is too large, the loss is calculated for 16 or 32 samples in the dataset first, and the weights are updated until all data has been traversed, completing one training cycle. This process is repeated multiple times to complete the training of the network.

[0088] Furthermore, since image semantic segmentation can also be understood as pixel classification, this application uses the cross-entropy function, which performs well in classification problems, as expressed in:

[0089]

[0090] Among them, y ic p represents the label of the i-th pixel. ic It is the predicted probability that pixel i belongs to category c, where c = 0 or 1.

[0091] Therefore, a segmented image can be obtained through a semantic segmentation neural network, which facilitates subsequent water level detection.

[0092] This application embodiment constructs a semantic segmentation neural network to separate the water surface and dam body portions in the water surface data, which is beneficial for subsequent water level detection. By using coarse and fine training methods to train the candidate semantic segmentation neural network, the robustness of the semantic segmentation neural network can be improved, thereby improving the accuracy of the water surface data segmentation task.

[0093] Figure 3 This is a block diagram of a training device 300 for a semantic segmentation neural network based on water surface data, according to an embodiment of this application, comprising:

[0094] The acquisition module 310 is used to acquire water surface data of the open-pit mine flood control dam, and to perform binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data;

[0095] The construction module 320 is used to construct the first candidate semantic segmentation neural network, wherein the first candidate semantic segmentation neural network consists of an encoder and a decoder. The encoder is used to reduce the dimensionality of the input image and encode it, and the decoder is used to extract the semantic label of each pixel in the encoded image.

[0096] The coarse training module 330 is used to acquire public water surface data and perform coarse training on the first candidate semantic segmentation neural network based on the public water surface data. After the training is completed, the second candidate semantic segmentation neural network is obtained.

[0097] The fine training module 340 is used to input the water surface data of the open-pit mine flood control dam into the second candidate semantic segmentation neural network to obtain segmented image data. Based on the segmented image and the binarized image data, the loss value is calculated. The second candidate semantic segmentation neural network is fine trained according to the loss value until the training is completed, and the trained semantic segmentation neural network is obtained.

[0098] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0099] Figure 4 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0100] like Figure 4 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0101] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the voice command response method. For example, in some embodiments, the voice command response method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the voice command response method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the voice command response method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0108] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a semantic segmentation neural network based on water surface data, characterized in that, include: Obtain water surface data of the open-pit mine flood control dam, and perform binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data; Construct a first candidate semantic segmentation neural network, wherein the first candidate semantic segmentation neural network consists of an encoder and a decoder, the encoder is used to reduce the dimensionality of the input image and encode it, and the decoder is used to extract the semantic label of each pixel in the encoded image; The encoder includes five convolutional networks, which are used to: perform dimensionality reduction on the input image using maximum pooling interspersed in the five convolutional networks; and increase the receptive field of the encoder by using dilated convolution in the fourth and fifth convolutional networks. The decoder includes: a first attention refinement module for using global average pooling and normalized depth dimensionality reduction to fuse input image features and IMU features, and then reweighting them; a first dilated spatial pyramid pooling module for refining the segmentation of small water structures based on the input image features and IMU features; a first feature fusion module for fusing the output features of the first attention refinement module and the first dilated spatial pyramid pooling module to generate 1024 feature channels; a second attention refinement module for fusing features from a third convolutional network, features from IMU channel encoding, and features transmitted from the first feature fusion module; and a second feature fusion module for upsampling the features transmitted by the second attention refinement module, fusing the low-dimensional features transmitted by the second attention refinement module with the high-dimensional features transmitted by the second convolutional network, and then reweighting the fused features. Obtain publicly available water surface data, and perform coarse training on the first candidate semantic segmentation neural network based on the publicly available water surface data. After training, a second candidate semantic segmentation neural network is obtained. The water surface data of the open-pit mine flood control dam is input into the second candidate semantic segmentation neural network to obtain segmented image data. Based on the segmented image and the binarized image data, the loss value is calculated. The second candidate semantic segmentation neural network is fine-trained according to the loss value until the training is completed, and a trained semantic segmentation neural network is obtained.

2. The method according to claim 1, characterized in that, The process of acquiring water surface data of the open-pit mine flood control dam and binarizing the water surface data to obtain binarized image data includes: Multiple surveillance cameras were deployed along the waterfront to obtain water surface data of the open-pit mine flood control dam; For the first image in the open-pit mine flood control dam water surface data, a closed polygon composed of a preset set of points is used to surround the water surface area in the first image; The first image is binarized, with pixels in the enclosed area assigned red and pixels in the unenclosed area assigned black. The remaining images in the open-pit mine flood control dam water surface data are subjected to the binarization process described above. After all images are labeled, the binarized image data is obtained.

3. The method according to claim 1, characterized in that, The decoder also includes: The second hollow spatial pyramid pooling module is used to segment the water structure of the input image and extract the semantic label of each pixel. The second hollow spatial pyramid pooling module has a built-in softmax function.

4. The method according to claim 3, characterized in that, Training the first candidate semantic segmentation neural network and the second candidate semantic segmentation neural network also includes: The high-resolution features encoded by the second convolutional network in the encoder and the IMU-encoded features are input into the second feature fusion module in the decoder; The medium-resolution features and IMU-encoded features encoded by the second convolutional network in the encoder are input into the second attention refinement module of the decoder; The low-resolution features and IMU-encoded features encoded by the fifth convolutional network in the encoder are input into the first attention refinement module of the decoder, and the encoded information is decoded using stacked deconvolutional layers, mapping the extracted information onto the two-dimensional plane of the original input space.

5. The method according to claim 1, characterized in that, Also includes: Add a residual module between the encoder and the decoder; Zero-filling sparse operation is performed on the convolution kernels of the pre-numbered convolutional networks in the encoder.

6. The method according to claim 1, characterized in that, The second candidate semantic segmentation neural network uses the cross-entropy function as its loss function, and the expression for the loss function is as follows: in, This represents the label of the i-th pixel. It is the predicted probability that pixel i belongs to category c.

7. A training device for a semantic segmentation neural network based on water surface data, characterized in that, include: The acquisition module is used to acquire water surface data of the open-pit mine flood control dam, and to perform binarization processing on the water surface data of the open-pit mine flood control dam to obtain binarized image data; A construction module is used to construct a first candidate semantic segmentation neural network, wherein the first candidate semantic segmentation neural network consists of an encoder and a decoder, the encoder is used to reduce the dimensionality of the input image and encode it, and the decoder is used to extract the semantic label of each pixel in the encoded image; The encoder includes five convolutional networks, which are used to: perform dimensionality reduction on the input image using maximum pooling interspersed in the five convolutional networks; and increase the receptive field of the encoder by using dilated convolution in the fourth and fifth convolutional networks. The decoder includes: a first attention refinement module for using global average pooling and normalized depth dimensionality reduction to fuse input image features and IMU features, and then reweighting them; a first dilated spatial pyramid pooling module for refining the segmentation of small water structures based on the input image features and IMU features; a first feature fusion module for fusing the output features of the first attention refinement module and the first dilated spatial pyramid pooling module to generate 1024 feature channels; a second attention refinement module for fusing features from a third convolutional network, features from IMU channel encoding, and features transmitted from the first feature fusion module; and a second feature fusion module for upsampling the features transmitted by the second attention refinement module, fusing the low-dimensional features transmitted by the second attention refinement module with the high-dimensional features transmitted by the second convolutional network, and then reweighting the fused features. The coarse training module is used to acquire public water surface data and perform coarse training on the first candidate semantic segmentation neural network based on the public water surface data. After the training is completed, the second candidate semantic segmentation neural network is obtained. The fine training module is used to input the water surface data of the open-pit mine flood control dam into the second candidate semantic segmentation neural network to obtain segmented image data. Based on the segmented image and the binarized image data, a loss value is calculated. The second candidate semantic segmentation neural network is fine-trained according to the loss value until the training is completed, and a trained semantic segmentation neural network is obtained.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-6.

Images