Urban water depth estimation method and system based on multi-reference system
By combining the multi-reference system with the water accumulation relationship segmentation model and the depth estimation model, the urban water accumulation depth is calculated, and the error problems caused by high equipment costs, small coverage and reference differences in the existing technology are solved, achieving more efficient and accurate estimation of water accumulation depth.
Patent Information
- Application Number
- CN202411156508.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing urban water depth estimation method has problems caused by high equipment costs, limited coverage, and differences in reference types and specifications. The method based on visual algorithm requires a lot of manual calibration work, which is a large amount of workload.
The depth of the urban water accumulation area is calculated by using a multi-reference system. By obtaining the image to be estimated after water accumulation and the image before water accumulation, combined with the relationship segmentation model and depth estimation model of the multi-reference object and water accumulation relationship, the depth estimation model is used to calculate the depth of the water accumulation area. Eigenvectors, output accurate water accumulation depth value and mask position, avoiding error effects caused by reference type and specifications.
It improves the accuracy of estimation of water accumulation depth, reduces errors caused by differences in reference types and specifications, reduces manual calibration workload, and improves the efficiency and accuracy of estimation.
Smart Images

Figure CN118691838B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for estimating waterlogging depth, and more specifically to a method and system for estimating urban waterlogging depth based on a multi-reference system. Background Art
[0002] At present, one of the key goals of urban waterlogging prevention and control is to effectively control ground waterlogging during rainfall, among which the depth of waterlogging has become one of the evaluation criteria. Traditional monitoring methods such as water level sensors, water gauge identification or manual monitoring have many problems, including high equipment and maintenance costs, insufficient monitoring range and untimely response.
[0003] Among the existing methods, the use of water level sensors has high accuracy, but is expensive and has limited coverage; water gauge recognition based on visual algorithms faces the problems of low water gauge coverage and inconsistent standards, especially when it cannot be accurately identified in the middle of the road; in the current single reference object depth estimation method, due to the different perspectives, ranges and sizes of the reference object and the accumulated water, only the relative height can be calculated after flooding, which cannot accurately reflect the actual situation of the flooded area. For example, the height difference of the sidewalk will lead to deviations in the position information of the lamp pole, and the difference in the size of the curb will cause inaccurate information.
[0004] In addition, Chinese patent CN201811403004.9 mentions a road waterlogging image detection and early warning method based on a hybrid model. Its logic is to first manually calibrate waterlogging pictures of different depths, then use the camera to identify the waterlogging in the actual scene, and use the encoding and decoding technology to determine the similarity between the actual waterlogging depth and the calibrated picture. However, this method requires a lot of manual calibration work, and it is necessary to draw multiple water level calibration gears in advance, which is a lot of work.
[0005] Therefore, it is necessary to design a new method to avoid the errors caused by different reference types and specifications when estimating the depth of water accumulation, so as to improve the accuracy of estimation. Summary of the invention
[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a method and system for estimating the depth of urban waterlogging based on a multi-reference system.
[0007] To achieve the above object, the present invention adopts the following technical solution: a method for estimating urban waterlogging depth based on a multi-reference system, comprising:
[0008] Obtain the image to be estimated after water accumulation;
[0009] Input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information;
[0010] Obtaining an image before water accumulation corresponding to the image to be estimated, so as to obtain an image before water accumulation;
[0011] Inputting the image to be estimated and the image before waterlogging into a depth estimation model, and calculating the depth difference between the flooded area and the unflooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding;
[0012] Calculating a final water depth value according to the first eigenvector and the second eigenvector;
[0013] Output the final water depth value and the corresponding waterlogged area mask position.
[0014] Its further technical solution is: the multi-reference object and water accumulation relationship segmentation model is obtained by using a teacher network trained with a labeled data set to assign pseudo labels to an unlabeled data set to form a pseudo-labeled data set, and the labeled data set is used as a sample set to train a student network created based on a Transformer architecture that is consistent with the teacher network.
[0015] A further technical solution is: the image to be estimated is input into a segmentation model of multiple reference objects and water accumulation relationship to perform water accumulation area segmentation and depth estimation to obtain a first feature vector, including:
[0016] Performing data enhancement on the image to be estimated and inputting it into a segmentation model of the relationship between a reference object and water accumulation;
[0017] The multi-reference object and water accumulation relationship segmentation model generates a feature vector for the image to be estimated after data enhancement, and sequentially passes through a normalization layer, a self-attention mechanism, and a multi-layer perceptron to obtain water accumulation segmentation information and corresponding information of reference objects above the water accumulation;
[0018] Calculate the height of the submerged part according to the proportional relationship between the reference object information above the waterlogged area and the actual target object, so as to obtain the water depth value corresponding to the waterlogged area;
[0019] The water depth value corresponding to the waterlogged area is added to the feature vector corresponding to the waterlogged segmentation information to form a first feature vector.
[0020] A further technical solution is: the height of the submerged part is calculated according to the proportional relationship between the reference object information above the submerged water and the actual target object to obtain the water depth value corresponding to the water accumulation area, including:
[0021] Determine the width and height information of the mask pixels of the reference object submerged by the accumulated water according to the information of the reference object submerged by the accumulated water;
[0022] Obtain the width-height ratio of the stored actual reference object from the segmentation model of the relationship between multiple reference objects and water accumulation;
[0023] The actual height of the reference object above the waterlogged object is calculated based on the mask pixel width and height information of the reference object above the waterlogged object and the width and height ratio of the actual reference object;
[0024] Calculate the difference between the actual height of the reference object and the actual height of the reference object above the waterlogged area to obtain the water depth value corresponding to the waterlogged area.
[0025] A further technical solution is: the image to be estimated and the image before waterlogging are input into a depth estimation model, and the difference between the depth of the waterlogged area and the depth of the unwaterlogged area is calculated in combination with the segmented feature vector to obtain a second feature vector, including:
[0026] Inputting the image to be estimated and the image before water accumulation into a depth estimation model for feature encoding, obtaining feature vectors before and after water accumulation, and forming two sets of depth feature vectors;
[0027] The segmentation feature vector is combined with two groups of depth feature vectors to perform feature fusion to obtain fused depth feature vectors before and after water accumulation;
[0028] The difference between the fused depth feature vectors before and after the water accumulation is calculated to obtain the second feature vector.
[0029] A further technical solution is: the segmentation feature vector is combined with two sets of depth feature vectors to perform feature fusion to obtain the fused depth feature vectors before and after water accumulation, including:
[0030] Performing feature alignment using the segmentation feature vector and two groups of depth feature vectors;
[0031] The cosine similarity between the aligned deep feature vectors is calculated to determine the fused deep feature vectors before and after water accumulation.
[0032] A further technical solution is: the difference between the depth feature vectors before and after the fusion of the water accumulation is calculated to obtain a second feature vector, including:
[0033] The difference between the fused depth feature vectors before and after water accumulation is calculated using the Euclidean distance to obtain the second feature vector.
[0034] A further technical solution is: calculating the final water depth value according to the first eigenvector and the second eigenvector includes:
[0035] Performing up and down sampling on the first eigenvector and the second eigenvector to obtain sampling results corresponding to the first eigenvector and the second eigenvector;
[0036] Multiplying the sampling results corresponding to the first eigenvector and the second eigenvector to obtain a multiplication result;
[0037] The multiplication result is input into a decoder for decoding, and the final water depth value is determined in combination with the mask position of the waterlogged area.
[0038] A further technical solution is: the multiplication result is input into a decoder for decoding, and the final water depth value is determined in combination with the mask position of the waterlogged area, including:
[0039] The multiplication result is sequentially passed through a 1*1 convolution layer, two layers of inverse convolution, multiple 3*3 convolution layers and a ReLU activation function in the decoder to obtain the actual depth value represented by each pixel. The actual depth value represented by the corresponding pixel is extracted in combination with the mask position of the water accumulation area, and the maximum depth value among the extracted actual depth values is determined to obtain the final water accumulation depth value.
[0040] The present invention also provides a device for estimating the depth of urban waterlogging based on a multi-reference system, comprising:
[0041] A post-water accumulation image acquisition unit, used for acquiring an image to be estimated after water accumulation;
[0042] A first estimation unit is used to input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information;
[0043] A pre-waterlogging image acquisition unit, used to acquire an image before waterlogging occurs corresponding to the image to be estimated, so as to obtain the pre-waterlogging image;
[0044] A second estimation unit is used to input the image to be estimated and the image before waterlogging into a depth estimation model, and calculate the depth difference between the flooded area and the unflooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding;
[0045] A calculation unit, configured to calculate a final water depth value according to the first eigenvector and the second eigenvector;
[0046] The output unit is used to output the final water depth value and the corresponding waterlogged area mask position.
[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention obtains the waterlogging area and depth information from the image after waterlogging through a multi-reference object and waterlogging relationship segmentation model, and then calculates the depth change of the submerged area by comparing the images before and after waterlogging and combining semantic information to obtain a depth feature vector; based on the waterlogging area and depth information and the depth feature vector, the accurate waterlogging depth value and the mask position of the waterlogging area are calculated and output, thereby avoiding the error influence caused by different reference object types, specifications, etc. when estimating the waterlogging depth, and improving the accuracy of the estimation.
[0048] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0050] Figure 1 A schematic diagram of a flow chart of a method for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0051] Figure 2 A schematic diagram of a sub-process of a method for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0052] Figure 3 A schematic diagram of a sub-process of a method for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0053] Figure 4 A schematic diagram of a sub-process of a method for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0054] Figure 5 A schematic diagram of a sub-process of a method for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0055] Figure 6 A schematic diagram of a sub-process of a method for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0056] Figure 7A schematic diagram of a reference street lamp being submerged provided by an embodiment of the present invention;
[0057] Figure 8 A schematic diagram of a reference street lamp provided in an embodiment of the present invention after being submerged and the water recedes;
[0058] Fig. 9 A schematic diagram of a water accumulation area provided by an embodiment of the present invention;
[0059] Fig.10 A schematic diagram of a waterlogged area after the depth value provided by an embodiment of the present invention is visualized;
[0060] Fig.11 A schematic block diagram of a device for estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention;
[0061] Fig.12 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0064] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0065] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0066] See also Figure 1 , Figure 1A schematic flow chart of a method for estimating the depth of urban waterlogging based on a multi-reference system provided in an embodiment of the present invention. The method for estimating the depth of urban waterlogging based on a multi-reference system is applied to a server. The server exchanges data with a terminal and a camera, and uses the camera to capture an image after waterlogging. The server performs depth calculations on multiple reference objects on the image to form a first eigenvector, and then compares the images before and after waterlogging to obtain a second eigenvector of difference. Finally, the first eigenvector is multiplied by the second eigenvector to obtain the most accurate distribution map of waterlogging depth, which takes into account the waterlogging depths of multiple reference objects and avoids the influence of errors caused by different reference object types and specifications. By comparing the image depth values with and without waterlogging, the real situation information of the flooded area can be supplemented, thereby eliminating information bias caused by different factors. Thus, when estimating the depth of waterlogging, the influence of errors caused by different reference object types and specifications is avoided, and the accuracy of the estimation is improved.
[0067] Figure 1 Schematic diagram of the process of estimating urban waterlogging depth based on a multi-reference system provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S160.
[0068] S110, obtaining an image to be estimated after water accumulation.
[0069] In this embodiment, the image to be estimated refers to an image of a certain designated area where water accumulates.
[0070] S120. Input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; and the segmentation feature vector includes image feature semantic information.
[0071] In this embodiment, the corresponding water depth value refers to the depth at which the water submerges each reference object.
[0072] The above-mentioned multi-reference object and water accumulation relationship segmentation model is obtained by using the teacher network trained by the labeled dataset to assign pseudo labels to the unlabeled dataset to form a pseudo-labeled dataset and the labeled dataset as a sample set to train a student network created based on the Transformer architecture that is consistent with the teacher network.
[0073] Specifically, collect pictures of water accumulation and water accumulation scenes, and establish a data set containing these pictures; use the water accumulation recognition model to recognize the pictures in the data set, identify the water accumulation areas, and crop these areas to form a training data set; preprocess the established training data set, including image enhancement, size unification and other operations, and then extract the feature vector of the image; use the network model to train the preprocessed data set, with the goal of learning the relationship between multiple reference objects (such as vehicles, pedestrians, curbs, lamp posts, etc.) and water accumulation. The output of the model is to be able to accurately segment these reference objects and water accumulation areas for subsequent depth estimation and other analysis, thereby forming a segmentation model for the relationship between multiple reference objects and water accumulation.
[0074] Among them, the water accumulation recognition model is formed by training a neural network with water accumulation pictures with water accumulation labels and water accumulation scene pictures. Of course, it can also be formed by training other classifiers.
[0075] The training process of the multi-reference object and water accumulation relationship segmentation model uses two identical networks built based on ViT (Vision Transformer) and multi-layer MLP (Multi-layer Perceptron) in the Transformer architecture, one for the teacher network and the other for the student network. The knowledge distillation technology is used to improve the generalization ability and performance of the model.
[0076] The specific training process of the multi-reference object and water accumulation relationship segmentation model includes: using the labeled data set to train the teacher network, the teacher network can extract effective feature representations when processing images; using the trained teacher network to predict the unlabeled data set and generate pseudo labels. These pseudo labels can be regarded as a form of labeling of the unlabeled data set for subsequent student network training; the student network training will use the labeled data set and the pseudo-label data set generated by the teacher network. In this way, the student network can be jointly trained from labeled and unlabeled data, and learn a wider range of image representations through pseudo-labels; perform various data enhancements on the input image, including cropping (local and global perspectives), color perturbation, Gaussian blur, exposure enhancement, etc. These enhancement operations help to increase the model's robustness and generalization ability to different variant inputs, especially Specifically, the student network receives all preprocessed cropped images, while the teacher network only receives cropped images from a global perspective, which helps the multi-reference and water accumulation relationship segmentation model to respond from local to global, and trains the student network to infer a wider range of contextual information; for images after data augmentation, feature vectors are extracted by the student network and the teacher network respectively, and these feature vectors are scaled to between 0 and 1 by the softmax function, and their cross entropy loss is calculated. This process encourages the student network to imitate the output of the teacher network, thereby learning better feature representations; SGD (stochastic gradient descent) is used to backpropagate and update the parameters of the student network to minimize the loss function. At the same time, the parameters of the student network are updated to the teacher network through the exponential moving average (EMA) method to maintain the stability and effect of the teacher network parameters.
[0077] By utilizing pseudo labels generated from unlabeled data, the student network can learn more data distribution characteristics, thereby improving its generalization ability for new data; using pseudo labels generated by the teacher network for training can effectively expand the training data set, thereby reducing the risk of overfitting of the student network on labeled data; using a variety of data augmentation techniques can help the model learn more robust and diverse feature representations to combat changes and noise in the input data.
[0078] Combining the training process of the teacher model and the student model, and using data augmentation and knowledge distillation techniques, the performance and generalization ability of the image segmentation model are effectively improved.
[0079] The trained multi-reference object and water accumulation relationship segmentation model can perform water accumulation area segmentation and depth estimation on the image to be estimated.
[0080] In one embodiment, see Figure 2 The above-mentioned step S120 may include steps S121~S124.
[0081] S121, performing data enhancement on the image to be estimated, and inputting the image into a segmentation model of the relationship between multiple reference objects and water accumulation.
[0082] In this embodiment, data enhancement is first performed on the image to be estimated, such as cropping, rotation, color enhancement, etc., to increase the robustness and generalization ability of the model; data enhancement helps the model extract more diverse and broader image features.
[0083] S122, generating a feature vector for the image to be estimated after data enhancement by the multi-reference object and water accumulation relationship segmentation model, and sequentially passing through a normalization layer, a self-attention mechanism and a multi-layer perceptron to obtain water accumulation segmentation information and corresponding information of reference objects submerged by water accumulation.
[0084] In this embodiment, the water accumulation segmentation information includes the mask position of the water accumulation area, and the corresponding water accumulation submerged upper reference object information includes the position information of the water accumulation submerged upper reference object of the water accumulation area.
[0085] Specifically, first, the multi-reference object and water accumulation relationship segmentation model normalizes the feature vector and then inputs it into the self-attention mechanism, which can effectively capture the global dependencies in the input features. Next, the feature vector output by the attention mechanism is input into a multi-layer perceptron, which contains two linear layers and a ReLU activation function. This step aims to introduce nonlinear transformations to integrate the features extracted by the attention mechanism and combine the water accumulation segmentation information and the reference object information above the water accumulation.
[0086] S123, calculating the height of the submerged part according to the proportional relationship between the reference object information above the waterlogging and the actual target object, so as to obtain the water depth value corresponding to the waterlogging area.
[0087] In this embodiment, the water depth value corresponding to the waterlogged area refers to the water depth estimated by the multi-reference object and waterlogging relationship segmentation model.
[0088] In one embodiment, see Figure 3 The above-mentioned step S123 may include steps S1231~S1234.
[0089] S1231. Determine the width and height information of the mask pixels of the reference object submerged by the accumulated water according to the information of the reference object submerged by the accumulated water.
[0090] In this embodiment, the mask pixel refers to the pixel value or the number of pixels in a specific area in the image. The mask pixel width and height information of the reference object submerged by the accumulated water refers to the width and height of the pixel range covered by the reference object submerged by the accumulated water in the image.
[0091] S1232. Obtain the stored width-to-height ratio of the actual reference object from the multi-reference object and water accumulation relationship segmentation model.
[0092] In this embodiment, the stored width-to-height ratios of the actual reference objects are obtained from the multi-reference object and water accumulation relationship segmentation model. These ratios are obtained in advance by measuring and analyzing the actual scene or standard reference objects. This information is usually obtained through the training and learning process of the model so that it can be used subsequently to calculate the height of the actual reference object.
[0093] S1233, calculating the actual height of the reference object submerged by the accumulated water based on the width and height information of the mask pixels of the reference object submerged by the accumulated water and the width and height ratio of the actual reference object.
[0094] In this embodiment, the actual height of the reference object above which the accumulated water submerges refers to the height of the reference object above which the accumulated water submerges in the actual physical world.
[0095] For example: The pixel width of the mask where the water submerges the reference object above; The pixel height of the mask where the water floods the upper reference object. These values can be measured from the pixel values of the reference object information flooded by the water. is the width-to-height ratio of the actual reference object; Calculate the actual height of the reference object above the water level The mask pixel information and the actual proportional relationship are used to convert the pixel height observed in the image into the actual physical height, thereby achieving an accurate estimation of the reference object height.
[0096] S1234. Calculate the difference between the actual height of the reference object and the actual height of the reference object above the waterlogged area to obtain the water depth value corresponding to the waterlogged area.
[0097] In this embodiment, the water depth value corresponding to the waterlogged area refers to the difference between the actual height of the reference object and the actual height of the reference object above which the waterlogged area submerges.
[0098] S124. Add the water depth value corresponding to the waterlogged area to the feature vector corresponding to the waterlogged segmentation information to form a first feature vector.
[0099] In this embodiment, the water depth value corresponding to the water accumulation area is added to the mask position of the corresponding water accumulation area to form a first feature vector.
[0100] A multi-reference object and water accumulation relationship segmentation model is used to estimate the depth of the water accumulation area in the estimated image. The water accumulation depths of multiple reference objects are taken into account, avoiding the error influence caused by different reference object types and specifications.
[0101] However, when estimating the depth of waterlogging using the segmentation model of the relationship between multiple reference objects and waterlogging, it was found that due to the differences in the viewing angle, range, and size of the reference objects, only the relative height of the reference objects could be calculated, and the actual situation of the flooded area could not be accurately confirmed. Figures 7 and 8 As shown in the figure, it can be seen that the submerged reference objects (such as lamp poles) have information bias due to the different heights of the sidewalks; similarly, the submerged reference objects (such as curbs) have information bias due to their inconsistent sizes, which leads to the fact that the use of multiple reference objects and water accumulation relationship segmentation model to estimate the water accumulation depth may not necessarily have a high accuracy rate.
[0102] In order to solve the problem of low accuracy due to information deviation introduced by different reference objects, in this embodiment, by comparing the depth values of the images with and without water accumulation, the real situation information of the flooded area is supplemented, thereby reducing the information deviation introduced by different factors. In this way, the actual depth of the flooded area can be more accurately reflected, helping to effectively eliminate errors caused by factors such as inconsistent viewing angles and reference object sizes.
[0103] The depth estimated by the depth estimation model combined with the images before and after waterlogging can not only improve the accuracy of waterlogging depth estimation, but also more accurately evaluate and respond to the potential risks and impacts brought by waterlogging in practical applications.
[0104] S130 , obtaining an image before waterlogging that corresponds to the image to be estimated, so as to obtain the image before waterlogging.
[0105] In this embodiment, the image before waterlogging refers to an image of the designated area before waterlogging occurs.
[0106] S140, inputting the image to be estimated and the image before waterlogging into a depth estimation model, and calculating the depth difference between the flooded area and the non-flooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding.
[0107] In this embodiment, by inputting pictures before and after water accumulation into the depth estimation model, the depth feature vectors of these pictures can be obtained. By using the existing mask position of the water accumulation area, the depth feature vector corresponding to the water accumulation area can be extracted. Then, the depth feature vectors before and after the water accumulation are compared and calculated to obtain a feature vector difference to form a second feature vector, which contains the encoding of the depth value. The depth of the water accumulation can be calculated by multiplying the second feature vector with the first feature vector generated by the student model. This method can reduce the depth information deviation caused by different reference objects, and by integrating the outputs of different models, it can more effectively utilize the depth information in the picture and improve the stability and accuracy of the results.
[0108] Among them, the depth estimation model is a model used to calculate the depth or distance of each pixel in the image. In this embodiment, this depth estimation model is used to analyze the images before and after the water accumulation, so as to calculate the depth difference between the flooded area and the unflooded area. This model is usually based on computer vision and deep learning technology, and can predict the distance or depth information of each pixel to the camera based on the input image.
[0109] Specifically, this depth estimation model can be a neural network based on deep learning, such as a convolutional neural network (CNN) or other models designed to regress depth information. Such models are usually trained on a large number of labeled data sets to learn to extract depth features from images and predict depth information.
[0110] In one embodiment, see Figure 4 The above-mentioned step S140 may include steps S141~S143.
[0111] S141 , inputting the image to be estimated and the image before water accumulation into a depth estimation model for feature encoding, obtaining feature vectors before and after water accumulation, and forming two sets of depth feature vectors.
[0112] In this embodiment, the depth feature vector refers to a vector that expresses spatial information and depth information in an image; through the depth estimation model, more accurate water depth information can be obtained, rather than just the visual surface of the image.
[0113] S142: Use the segmentation feature vector in combination with two groups of depth feature vectors to perform feature fusion to obtain fused depth feature vectors before and after water accumulation.
[0114] In this embodiment, the fused depth feature vectors before and after water accumulation can take into account the semantic segmentation information and depth estimation information of the image at the same time, thereby improving the accuracy and robustness of water accumulation depth estimation; fusing features from different sources can help reduce the error caused by a single feature source, thereby more accurately estimating the water accumulation depth.
[0115] In one embodiment, see Figure 5 , the above-mentioned step S142 may include steps S1421~S1422.
[0116] S1421. Perform feature alignment using the segmentation feature vector and two groups of depth feature vectors.
[0117] In this embodiment, feature alignment refers to mapping data represented by different features to the same feature space or the same scale to facilitate subsequent feature fusion and processing.
[0118] Feature alignment can ensure that each feature vector is aligned before and after fusion, avoiding errors caused by feature mismatch or inconsistency.
[0119] S1422: Calculate the cosine similarity between the aligned depth feature vectors to determine the fused depth feature vectors before and after the water accumulation.
[0120] Specifically, cosine similarity is a commonly used similarity measure, which is suitable for comparing the direction and similarity between vectors; in this way, the image feature semantic information of the segmentation feature vector can be used to guide the depth estimation, and finally two complete feature vectors are obtained, which are represented as the depth features before and after the water accumulation.
[0121] S143. Calculate the difference between the fused depth feature vectors before and after the water accumulation to obtain a second feature vector.
[0122] In this embodiment, considering the difference between the two feature vectors before and after waterlogging occurs, the Euclidean distance is used to calculate the difference between the fused depth feature vectors before and after waterlogging to obtain the second feature vector.
[0123] S150: Calculate a final water depth value according to the first eigenvector and the second eigenvector.
[0124] In this embodiment, the final water depth value refers to the depth value calculated by integrating the first eigenvector and the second eigenvector after processing. The first eigenvector is generated by the student model; the second eigenvector is obtained by comparing and calculating the difference between the depth eigenvectors before and after water accumulation, and contains depth information encoding. These two eigenvectors can be multiplied or other appropriate operations to obtain a more stable and accurate water depth value.
[0125] In one embodiment, see Figure 6 , the above-mentioned step S150 may include steps S151~S153.
[0126] S151. Perform up-sampling and down-sampling on the first feature vector and the second feature vector to obtain sampling results corresponding to the first feature vector and the second feature vector.
[0127] In this embodiment, both the first feature vector and the second feature vector include depth information calculated by different models, and the features are upsampled and downsampled to unify the dimensions of the first feature vector and the second feature vector.
[0128] S152: Multiply the sampling results corresponding to the first eigenvector and the second eigenvector to obtain a multiplication result.
[0129] In this embodiment, the multiplication result refers to the result obtained by multiplying the sampling results corresponding to the first eigenvector and the second eigenvector.
[0130] The purpose of multiplication is to strengthen the depth information in the vector while suppressing the influence of irrelevant features to a certain extent.
[0131] S153: Input the multiplication result into a decoder for decoding, and determine a final water depth value in combination with the mask position of the waterlogged area.
[0132] In this embodiment, the multiplication result is sequentially passed through a 1*1 convolution layer, two layers of inverse convolution, multiple 3*3 convolution layers and a ReLU activation function in the decoder to obtain the actual depth value represented by each pixel. The actual depth value represented by the corresponding pixel is extracted in combination with the mask position of the water accumulation area, and the maximum depth value among the extracted actual depth values is determined to obtain the final water accumulation depth value.
[0133] Specifically, the decoder consists of multiple convolutional layers. First, the multiplication result is reduced in dimension through a 1×1 convolutional layer, which helps to reduce the complexity of subsequent calculations. Next, two layers of inverse convolution are used for upsampling to gradually restore the spatial dimensions of the feature map so that it matches the size of the original input image, that is, the image before water accumulation. Finally, through multiple 3×3 convolutional layers and ReLU activation function processing, the value at each position of the output tensor represents the actual depth value of the corresponding pixel. The depth value of the corresponding area is extracted through the mask position of the waterlogged area, and the maximum depth value in the area is calculated as the final waterlogged depth value.
[0134] S160: Output the final water accumulation depth value and the corresponding water accumulation area mask position.
[0135] After adding the final water depth value to the corresponding waterlogging area mask position, the visualization output can be performed. The waterlogging area mask position is as follows: Fig. 9 As shown, the final water depth value is added and visualized to output, such as Fig.10 shown.
[0136] The method of this embodiment utilizes a system of multiple reference objects instead of relying on a single reference object. Reference objects of different types and specifications can provide diverse depth information, thereby significantly reducing the errors and uncertainties that may be caused by a single reference object; by comparing the depth information of multiple reference objects, the depth of water accumulation on urban roads can be estimated more accurately; in addition, the method uses a deep learning model, especially a teacher network and a student network based on the Transformer architecture. These networks can deeply understand the laws of water accumulation depth in rich urban scenes by learning annotated data sets and pseudo-label data sets, and improve the ability and accuracy of the model to handle complex scenes; at the same time, data enhancement technology is used in the process of processing the image to be estimated, which helps to increase the diversity of data and improve the generalization ability of the model. Using feature fusion technology, features from different reference objects and depth information are integrated, further improving the accuracy and precision of water accumulation depth in complex urban road scenes; in addition, the difference or similarity between the calculated depth feature vectors is also considered, and the change of water accumulation depth is evaluated through these quantitative indicators, rather than relying solely on a single pixel difference or a simple before-after comparison. This refined calculation method can more accurately reflect the change of water accumulation depth on actual urban roads. The comprehensive use of multiple reference objects, deep learning models, data enhancement and feature fusion techniques, as well as refined deep feature calculation and alignment methods, effectively improves the ability to accurately estimate the depth of water in complex urban scenes and achieves higher accuracy targets than traditional methods.
[0137] The above-mentioned urban waterlogging depth estimation method based on a multi-reference object system obtains the waterlogging area and depth information from the image after waterlogging through a multi-reference object and waterlogging relationship segmentation model, and then calculates the depth change of the flooded area by comparing the images before and after waterlogging and combining semantic information to obtain a depth feature vector; based on the waterlogging area and depth information and the depth feature vector, it calculates and outputs the accurate waterlogging depth value and the mask position of the waterlogging area, thereby avoiding the error influence caused by different reference object types and specifications when estimating the waterlogging depth, thereby improving the accuracy of the estimation.
[0138] Fig.11 is a schematic block diagram of an urban flood depth estimation device 300 based on a multi-reference system provided by an embodiment of the present invention. Fig.11 As shown, corresponding to the above urban water depth estimation method based on a multi-reference system, the present invention also provides an urban water depth estimation device 300 based on a multi-reference system. The urban water depth estimation device 300 based on a multi-reference system includes a unit for executing the above urban water depth estimation method based on a multi-reference system, and the device can be configured in a server. Fig.11The urban waterlogging depth estimation device 300 based on a multi-reference system includes a post-waterlogging image acquisition unit 301, a first estimation unit 302, a pre-waterlogging image acquisition unit 303, a second estimation unit 304, a calculation unit 305 and an output unit 306.
[0139] The post-water accumulation image acquisition unit 301 is used to acquire the image to be estimated after water accumulation; the first estimation unit 302 is used to input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information; the pre-water accumulation image acquisition unit 303 is used to acquire the image before water accumulation corresponding to the image to be estimated to obtain the image before water accumulation; the second estimation unit 304 is used to input the image to be estimated and the pre-water accumulation image into the depth estimation model, and calculate the depth difference between the flooded area and the non-flooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information coding; the calculation unit 305 is used to calculate the final water accumulation depth value according to the first feature vector and the second feature vector; the output unit 306 is used to output the final water accumulation depth value and the corresponding mask position of the water accumulation area.
[0140] In one embodiment, the first estimation unit 302 includes an enhancement subunit, a segmentation subunit, a height calculation subunit, and an addition subunit.
[0141] An enhancement subunit is used to perform data enhancement on the image to be estimated and input it into a multi-reference object and water accumulation relationship segmentation model; a segmentation subunit is used to generate a feature vector for the image to be estimated after data enhancement by the multi-reference object and water accumulation relationship segmentation model, and sequentially pass through a normalization layer, a self-attention mechanism and a multi-layer perceptron to obtain water accumulation segmentation information and corresponding reference object information above the water accumulation; a height calculation subunit is used to calculate the height of the submerged part according to the proportional relationship between the reference object information above the water accumulation and the actual target object, so as to obtain the water accumulation depth value corresponding to the water accumulation area; an adding subunit is used to add the water accumulation depth value corresponding to the water accumulation area to the feature vector corresponding to the water accumulation segmentation information to form a first feature vector.
[0142] In one embodiment, the height calculation subunit includes a width and height information determination module, a ratio acquisition module, an actual height calculation module, and a water depth calculation module.
[0143] A width and height information determination module is used to determine the mask pixel width and height information of the reference object above the waterlogging based on the information of the reference object above the waterlogging; a ratio acquisition module is used to obtain the stored width and height ratio of the actual reference object from the multi-reference object and waterlogging relationship segmentation model; an actual height calculation module is used to calculate the actual height of the reference object above the waterlogging based on the mask pixel width and height information of the reference object above the waterlogging and the width and height ratio of the actual reference object; a waterlogging depth calculation module is used to calculate the difference between the actual height of the reference object and the actual height of the reference object above the waterlogging, so as to obtain the waterlogging depth value corresponding to the waterlogging area.
[0144] In one embodiment, the second estimation unit 304 includes an encoding subunit, a fusion subunit, and a difference calculation subunit.
[0145] The encoding subunit is used to input the image to be estimated and the image before water accumulation into the depth estimation model for feature encoding, so as to obtain feature vectors before and after water accumulation, and form two groups of depth feature vectors; the fusion subunit is used to use the segmentation feature vector in combination with the two groups of depth feature vectors to perform feature fusion, so as to obtain the fused depth feature vectors before and after water accumulation; the difference calculation subunit is used to calculate the difference between the fused depth feature vectors before and after water accumulation, so as to obtain a second feature vector.
[0146] In one embodiment, the fusion subunit includes an alignment module and a similarity calculation module.
[0147] The alignment module is used to perform feature alignment with the segmentation feature vector and the two groups of depth feature vectors; the similarity calculation module is used to calculate the cosine similarity between the aligned depth feature vectors to determine the fused depth feature vectors before and after the water accumulation.
[0148] In one embodiment, the difference calculation subunit is used to calculate the difference between the fused depth feature vectors before and after the water accumulation by using the Euclidean distance to obtain the second feature vector.
[0149] In one embodiment, the calculation unit 305 includes an up- and down-sampling subunit, a multiplication subunit, and a decoding subunit.
[0150] An up and down sampling subunit is used to perform up and down sampling on the first eigenvector and the second eigenvector to obtain sampling results corresponding to the first eigenvector and the second eigenvector; a multiplication subunit is used to multiply the sampling results corresponding to the first eigenvector and the second eigenvector to obtain a multiplication result; a decoding subunit is used to input the multiplication result into a decoder for decoding, and determine the final water depth value in combination with the mask position of the water accumulation area.
[0151] In one embodiment, the decoding subunit is used to sequentially pass the multiplication result through a 1*1 convolution layer, two layers of inverse convolution, multiple 3*3 convolution layers and a ReLU activation function in the decoder to obtain an actual depth value represented by each pixel, extract the actual depth value represented by the corresponding pixel in combination with the mask position of the water accumulation area, and determine the maximum depth value among the extracted actual depth values to obtain the final water accumulation depth value.
[0152] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned urban waterlogging depth estimation device based on a multi-reference system and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.
[0153] The above-mentioned urban flood depth estimation device 300 based on the multi-reference system can be implemented in the form of a computer program. The computer program can be used in Fig.12 Runs on the computer device shown.
[0154] See also Fig.12 , Fig.12 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.
[0155] See also Fig.12 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .
[0156] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a method for estimating urban water depth based on a multi-reference system.
[0157] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0158] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for estimating the depth of urban waterlogging based on a multi-reference system.
[0159] The network interface 505 is used to communicate with other devices over the network. Fig.12The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0160] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:
[0161] Acquire an image to be estimated after water accumulation; input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information; acquire an image before water accumulation corresponding to the image to be estimated to obtain an image before water accumulation; input the image to be estimated and the image before water accumulation into a depth estimation model, and calculate the depth difference between the flooded area and the non-flooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding; calculate a final water accumulation depth value based on the first feature vector and the second feature vector; and output a final water accumulation depth value and the corresponding mask position of the water accumulation area.
[0162] Among them, the multi-reference object and water accumulation relationship segmentation model is obtained by using the teacher network trained by the labeled data set to assign pseudo labels to the unlabeled data set to form a pseudo-labeled data set, and the labeled data set is used as a sample set to train a student network created based on the Transformer architecture and consistent with the teacher network.
[0163] In one embodiment, when the processor 502 implements the step of inputting the image to be estimated into the multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain the first feature vector, the processor 502 specifically implements the following steps:
[0164] Data enhancement is performed on the image to be estimated, and the image is input into a multi-reference object and water accumulation relationship segmentation model; a feature vector is generated by the multi-reference object and water accumulation relationship segmentation model for the image to be estimated after data enhancement, and the image is sequentially passed through a normalization layer, a self-attention mechanism, and a multi-layer perceptron to obtain water accumulation segmentation information and corresponding reference object information above the water accumulation; the height of the submerged part is calculated according to the proportional relationship between the reference object information above the water accumulation and the actual target object to obtain the water accumulation depth value corresponding to the water accumulation area; the water accumulation depth value corresponding to the water accumulation area is added to the feature vector corresponding to the water accumulation segmentation information to form a first feature vector.
[0165] In one embodiment, when the processor 502 implements the step of calculating the height of the submerged part according to the proportional relationship between the reference object information above the submerged water and the actual target object to obtain the water depth value corresponding to the water accumulation area, the processor 502 specifically implements the following steps:
[0166] Determine the mask pixel width and height information of the reference object above the waterlogging based on the information of the reference object above the waterlogging; obtain the stored width and height ratio of the actual reference object from the multi-reference object and waterlogging relationship segmentation model; calculate the actual height of the reference object above the waterlogging based on the mask pixel width and height information of the reference object above the waterlogging and the width and height ratio of the actual reference object; calculate the difference between the actual height of the reference object and the actual height of the reference object above the waterlogging to obtain the water depth value corresponding to the waterlogged area.
[0167] In one embodiment, when the processor 502 implements the step of inputting the image to be estimated and the image before water accumulation into the depth estimation model, and calculating the difference between the depth of the waterlogged area and the depth of the area not flooded by water in combination with the segmentation feature vector to obtain the second feature vector, the processor 502 specifically implements the following steps:
[0168] The image to be estimated and the image before water accumulation are input into a depth estimation model for feature encoding to obtain feature vectors before and after water accumulation to form two groups of depth feature vectors; the segmentation feature vector is combined with the two groups of depth feature vectors to perform feature fusion to obtain the fused depth feature vectors before and after water accumulation; the difference between the fused depth feature vectors before and after water accumulation is calculated to obtain a second feature vector.
[0169] In one embodiment, when the processor 502 implements the step of using the segmentation feature vector to combine with two groups of depth feature vectors to perform feature fusion to obtain the fused depth feature vectors before and after water accumulation, the processor 502 specifically implements the following steps:
[0170] The segmentation feature vector is used to align features with two groups of depth feature vectors; and the cosine similarity between the aligned depth feature vectors is calculated to determine the fused depth feature vectors before and after the water accumulation.
[0171] In one embodiment, when the processor 502 implements the step of calculating the difference between the depth feature vectors before and after the fusion of the water accumulation to obtain the second feature vector, the processor 502 specifically implements the following steps:
[0172] The difference between the fused depth feature vectors before and after water accumulation is calculated using the Euclidean distance to obtain the second feature vector.
[0173] In one embodiment, when the processor 502 implements the step of calculating the final water depth value according to the first feature vector and the second feature vector, the processor 502 specifically implements the following steps:
[0174] The first eigenvector and the second eigenvector are sampled up and down to obtain sampling results corresponding to the first eigenvector and the second eigenvector; the sampling results corresponding to the first eigenvector and the second eigenvector are multiplied to obtain a multiplication result; the multiplication result is input into a decoder for decoding, and the final water depth value is determined in combination with the mask position of the water accumulation area.
[0175] In one embodiment, when the processor 502 implements the step of inputting the multiplication result into the decoder for decoding and determining the final water depth value in combination with the mask position of the water accumulation area, the processor 502 specifically implements the following steps:
[0176] The multiplication result is sequentially passed through a 1*1 convolution layer, two layers of inverse convolution, multiple 3*3 convolution layers and a ReLU activation function in the decoder to obtain the actual depth value represented by each pixel. The actual depth value represented by the corresponding pixel is extracted in combination with the mask position of the water accumulation area, and the maximum depth value among the extracted actual depth values is determined to obtain the final water accumulation depth value.
[0177] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0178] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.
[0179] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:
[0180] Acquire an image to be estimated after water accumulation; input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information; acquire an image before water accumulation corresponding to the image to be estimated to obtain an image before water accumulation; input the image to be estimated and the image before water accumulation into a depth estimation model, and calculate the depth difference between the flooded area and the non-flooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding; calculate a final water accumulation depth value based on the first feature vector and the second feature vector; and output a final water accumulation depth value and the corresponding mask position of the water accumulation area.
[0181] Among them, the multi-reference object and water accumulation relationship segmentation model is obtained by using the teacher network trained by the labeled data set to assign pseudo labels to the unlabeled data set to form a pseudo-labeled data set, and the labeled data set is used as a sample set to train a student network created based on the Transformer architecture and consistent with the teacher network.
[0182] In one embodiment, when the processor executes the computer program to implement the step of inputting the image to be estimated into multiple reference objects and water accumulation relationship segmentation models to perform water accumulation area segmentation and depth estimation to obtain a first feature vector, the processor specifically implements the following steps:
[0183] Data enhancement is performed on the image to be estimated, and the image is input into a multi-reference object and water accumulation relationship segmentation model; a feature vector is generated by the multi-reference object and water accumulation relationship segmentation model for the image to be estimated after data enhancement, and the image is sequentially passed through a normalization layer, a self-attention mechanism, and a multi-layer perceptron to obtain water accumulation segmentation information and corresponding reference object information above the water accumulation; the height of the submerged part is calculated according to the proportional relationship between the reference object information above the water accumulation and the actual target object to obtain the water accumulation depth value corresponding to the water accumulation area; the water accumulation depth value corresponding to the water accumulation area is added to the feature vector corresponding to the water accumulation segmentation information to form a first feature vector.
[0184] In one embodiment, when the processor executes the computer program to implement the step of calculating the height of the submerged part according to the proportional relationship between the reference object information above the submerged water and the actual target object to obtain the water depth value corresponding to the water accumulation area, the processor specifically implements the following steps:
[0185] Determine the mask pixel width and height information of the reference object above the waterlogging based on the information of the reference object above the waterlogging; obtain the stored width and height ratio of the actual reference object from the multi-reference object and waterlogging relationship segmentation model; calculate the actual height of the reference object above the waterlogging based on the mask pixel width and height information of the reference object above the waterlogging and the width and height ratio of the actual reference object; calculate the difference between the actual height of the reference object and the actual height of the reference object above the waterlogging to obtain the water depth value corresponding to the waterlogged area.
[0186] In one embodiment, when the processor executes the computer program to implement the step of inputting the image to be estimated and the image before water accumulation into the depth estimation model, and calculating the difference between the depth of the waterlogged area and the depth of the area not flooded by water in combination with the segmentation feature vector to obtain the second feature vector, the processor specifically implements the following steps:
[0187] The image to be estimated and the image before water accumulation are input into a depth estimation model for feature encoding to obtain feature vectors before and after water accumulation to form two groups of depth feature vectors; the segmentation feature vector is combined with the two groups of depth feature vectors to perform feature fusion to obtain the fused depth feature vectors before and after water accumulation; the difference between the fused depth feature vectors before and after water accumulation is calculated to obtain a second feature vector.
[0188] In one embodiment, when the processor executes the computer program to implement the step of using the segmentation feature vector to combine with two sets of depth feature vectors to perform feature fusion to obtain the fused depth feature vectors before and after water accumulation, the processor specifically implements the following steps:
[0189] The segmentation feature vector is used to align features with two groups of depth feature vectors; and the cosine similarity between the aligned depth feature vectors is calculated to determine the fused depth feature vectors before and after the water accumulation.
[0190] In one embodiment, when the processor executes the computer program to implement the step of calculating the difference between the depth feature vectors before and after the fusion of the water accumulation to obtain the second feature vector, the processor specifically implements the following steps:
[0191] The difference between the fused depth feature vectors before and after water accumulation is calculated using the Euclidean distance to obtain the second feature vector.
[0192] In one embodiment, when the processor executes the computer program to implement the step of calculating the final water depth value according to the first eigenvector and the second eigenvector, the processor specifically implements the following steps:
[0193] The first eigenvector and the second eigenvector are sampled up and down to obtain sampling results corresponding to the first eigenvector and the second eigenvector; the sampling results corresponding to the first eigenvector and the second eigenvector are multiplied to obtain a multiplication result; the multiplication result is input into a decoder for decoding, and the final water depth value is determined in combination with the mask position of the water accumulation area.
[0194] In one embodiment, when the processor executes the computer program to implement the step of inputting the multiplication result into a decoder for decoding and determining the final water depth value in combination with the mask position of the waterlogged area, the processor specifically implements the following steps:
[0195] The multiplication result is sequentially passed through a 1*1 convolution layer, two layers of inverse convolution, multiple 3*3 convolution layers and a ReLU activation function in the decoder to obtain the actual depth value represented by each pixel. The actual depth value represented by the corresponding pixel is extracted in combination with the mask position of the water accumulation area, and the maximum depth value among the extracted actual depth values is determined to obtain the final water accumulation depth value.
[0196] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.
[0197] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0198] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0199] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.
[0201] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for estimating urban water depth based on a multi-reference system, characterized in that: include: Obtain the image to be estimated after water accumulation; Input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information; Obtaining an image before water accumulation corresponding to the image to be estimated, so as to obtain an image before water accumulation; Inputting the image to be estimated and the image before waterlogging into a depth estimation model, and calculating the depth difference between the flooded area and the unflooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding; Calculating a final water depth value according to the first eigenvector and the second eigenvector; Output the final water depth value and the corresponding mask position of the water area; The multi-reference object and water accumulation relationship segmentation model is obtained by using a teacher network trained with a labeled data set to assign pseudo labels to an unlabeled data set to form a pseudo-labeled data set, and the labeled data set is used as a sample set to train a student network created based on a Transformer architecture and consistent with the teacher network; The calculating a final water depth value according to the first eigenvector and the second eigenvector includes: Performing up and down sampling on the first eigenvector and the second eigenvector to obtain sampling results corresponding to the first eigenvector and the second eigenvector; Multiplying the sampling results corresponding to the first eigenvector and the second eigenvector to obtain a multiplication result; The multiplication result is input into a decoder for decoding, and the final water depth value is determined in combination with the mask position of the waterlogged area.
2. The urban water depth estimation method based on a multi-reference system according to claim 1 is characterized in that: The step of inputting the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector includes: Performing data enhancement on the image to be estimated and inputting it into a segmentation model of the relationship between a reference object and water accumulation; The multi-reference object and water accumulation relationship segmentation model generates a feature vector for the image to be estimated after data enhancement, and sequentially passes through a normalization layer, a self-attention mechanism, and a multi-layer perceptron to obtain water accumulation segmentation information and corresponding information of reference objects above the water accumulation; Calculate the height of the submerged part according to the proportional relationship between the reference object information above the waterlogged area and the actual target object, so as to obtain the water depth value corresponding to the waterlogged area; The water depth value corresponding to the waterlogged area is added to the feature vector corresponding to the waterlogged segmentation information to form a first feature vector.
3. The urban water depth estimation method based on a multi-reference system according to claim 2 is characterized in that: The calculating the height of the submerged part according to the proportional relationship between the reference object information above the submerged water and the actual target object to obtain the water depth value corresponding to the water accumulation area includes: Determine the width and height information of the mask pixels of the reference object submerged by the accumulated water according to the information of the reference object submerged by the accumulated water; Obtain the width-height ratio of the stored actual reference object from the segmentation model of the relationship between multiple reference objects and water accumulation; The actual height of the reference object above the waterlogged object is calculated based on the mask pixel width and height information of the reference object above the waterlogged object and the width and height ratio of the actual reference object; Calculate the difference between the actual height of the reference object and the actual height of the reference object above the waterlogged area to obtain the water depth value corresponding to the waterlogged area.
4. The urban water depth estimation method based on a multi-reference system according to claim 1 is characterized in that: The step of inputting the image to be estimated and the image before waterlogging into a depth estimation model, and calculating the difference between the depth of the waterlogged area and the depth of the area not flooded by waterlogging in combination with the segmented feature vector to obtain a second feature vector includes: Inputting the image to be estimated and the image before water accumulation into a depth estimation model for feature encoding, obtaining feature vectors before and after water accumulation, and forming two sets of depth feature vectors; The segmentation feature vector is combined with two groups of depth feature vectors to perform feature fusion to obtain fused depth feature vectors before and after water accumulation; The difference between the fused depth feature vectors before and after the water accumulation is calculated to obtain the second feature vector.
5. The urban water depth estimation method based on a multi-reference system according to claim 4 is characterized in that: The step of combining the segmentation feature vector with two sets of depth feature vectors to perform feature fusion to obtain the fused depth feature vectors before and after water accumulation includes: Performing feature alignment using the segmentation feature vector and two groups of depth feature vectors; The cosine similarity between the aligned deep feature vectors is calculated to determine the fused deep feature vectors before and after water accumulation.
6. The urban water depth estimation method based on a multi-reference system according to claim 4 is characterized in that: The calculating the difference between the depth feature vectors before and after the fusion of the water accumulation to obtain the second feature vector includes: The difference between the fused depth feature vectors before and after water accumulation is calculated using the Euclidean distance to obtain the second feature vector.
7. The urban water depth estimation method based on a multi-reference system according to claim 1 is characterized in that: The step of inputting the multiplication result into a decoder for decoding, and determining a final water depth value in combination with the mask position of the waterlogged area, comprises: The multiplication result is sequentially passed through a 1*1 convolution layer, two layers of inverse convolution, multiple 3*3 convolution layers and a ReLU activation function in the decoder to obtain the actual depth value represented by each pixel. The actual depth value represented by the corresponding pixel is extracted in combination with the mask position of the water accumulation area, and the maximum depth value among the extracted actual depth values is determined to obtain the final water accumulation depth value.
8. An urban water depth estimation device based on a multi-reference system, characterized in that: include: A post-water accumulation image acquisition unit, used for acquiring an image to be estimated after water accumulation; A first estimation unit is used to input the image to be estimated into a multi-reference object and water accumulation relationship segmentation model to perform water accumulation area segmentation and depth estimation to obtain a first feature vector and a segmentation feature vector, wherein the first feature vector includes a mask position of the water accumulation area and a corresponding water accumulation depth value; the segmentation feature vector includes image feature semantic information; A pre-waterlogging image acquisition unit, used to acquire an image before waterlogging occurs corresponding to the image to be estimated, so as to obtain the pre-waterlogging image; A second estimation unit is used to input the image to be estimated and the image before waterlogging into a depth estimation model, and calculate the depth difference between the flooded area and the unflooded area in combination with the segmentation feature vector to obtain a second feature vector, wherein the second feature vector includes depth information encoding; A calculation unit, configured to calculate a final water depth value according to the first eigenvector and the second eigenvector; An output unit, used to output the final water depth value and the corresponding water area mask position; The multi-reference object and water accumulation relationship segmentation model is obtained by using a teacher network trained with a labeled data set to assign pseudo labels to an unlabeled data set to form a pseudo-labeled data set, and the labeled data set is used as a sample set to train a student network created based on a Transformer architecture and consistent with the teacher network; The calculation unit includes an up and down sampling subunit, a multiplication subunit and a decoding subunit; the up and down sampling subunit is used to perform up and down sampling on the first feature vector and the second feature vector to obtain sampling results corresponding to the first feature vector and the second feature vector; the multiplication subunit is used to multiply the sampling results corresponding to the first feature vector and the second feature vector to obtain a multiplication result; the decoding subunit is used to input the multiplication result into a decoder for decoding, and determine the final water depth value in combination with the mask position of the water accumulation area.
Citation Information
Cited By
Tree lodging detection method and system combined with reference object image analysis
CN119495029A
Tree fall detection method and system combined with reference image analysis
CN119495029B
