Road image segmentation method and device based on pixel-level self-supervised learning

Abstract

The application provides a road image segmentation method and device based on pixel-level self-supervised learning, which comprises the following steps: collecting an initial road image according to a patrol route; performing preprocessing on the initial road image based on bilinear interpolation to obtain a multi-scale road image set; inputting the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively to obtain corresponding feature maps and heat maps; inputting the obtained heat maps into a region comparison module to determine a comparison learning result through pixel-by-pixel comparison; inputting the feature maps and the comparison learning result into a multi-scale attention module to obtain a classification result, and obtaining a target segmentation result according to the overall classification result. According to the technical scheme of the embodiment, the pixel-level similarity can be obtained through comparison learning, and the road classification can be realized through the multi-scale attention mechanism, so that the image segmentation of the road image can be automatically realized under the condition that the labeled data is small, and the image recognition precision and the automation degree are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a road image segmentation method and device based on pixel-level self-supervised learning. BACKGROUND

[0002] At present, with the development and wide application of unmanned aerial vehicle technology, unmanned aerial vehicle remote sensing is very important in fine measurement of road surface attached objects and three-dimensional reconstruction of urban buildings, and road information is widely used in many fields. Different roads have different ground cover, road surface radiation differences, imaging blur and other problems, and the road image presents non-uniformity of edge information and irregularity of feature deformation distribution, which increases the uncertainty of road feature expression and the difficulty of road extraction model construction. The commonly used method is to increase the labeled features in the model training process, but this will consume a lot of manpower, the image processing efficiency is low, and the selected features may not be representative, which limits the image recognition accuracy. SUMMARY

[0003] The following is a summary of the subject matter described in detail in this document. This summary is not intended to limit the scope of protection of the claims.

[0004] The embodiment of the present application provides a road image segmentation method and device based on pixel-level self-supervised learning, which can automatically realize image segmentation of road images, improve image recognition accuracy and automation degree.

[0005] In a first aspect, the embodiment of the present application provides a road image segmentation method based on pixel-level self-supervised learning, applied to an unmanned aerial vehicle, comprising:

[0006] acquiring a preset inspection route, and collecting an initial road image according to the inspection route;

[0007] performing preprocessing on the initial road image based on bilinear interpolation to obtain a multi-scale road image set;

[0008] inputting the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively, performing a heat map generation operation based on feature extraction through the main segmentation network to obtain a first feature map and a first heat map, and performing a heat map generation operation based on feature extraction through the auxiliary segmentation network to obtain a second feature map and a second heat map, the network structure of the main segmentation network and the auxiliary segmentation network being the same, and the network weight of the main segmentation network being greater than that of the auxiliary segmentation network;

[0009] inputting the first heat map, the second heat map and the second feature map into a region comparison module, and determining a comparison learning result according to the first heat map and the second heat map through pixel-by-pixel comparison;

[0010] inputting the first feature map and the contrast learning result into the multi-scale attention module to obtain a classification result of each pixel of the first feature map, and obtaining a target segmentation result according to all the classification results.

[0011] In some embodiments, the initial road image is preprocessed based on bilinear interpolation to obtain a multi-scale road image set, including:

[0012] A preset scale factor is obtained, and an interpolation distance value is determined according to the scale factor;

[0013] An initial coordinate corresponding to each pixel of the initial road image in a three-dimensional coordinate is determined;

[0014] According to each initial coordinate and the interpolation distance value, two interpolations are performed in the longitudinal axis direction and one interpolation is performed in the transverse axis direction to obtain four neighbor point sets;

[0015] A multi-scale road image set is obtained according to the pixel points corresponding to each neighbor point set.

[0016] In some embodiments, the heat map generation operation based on feature extraction includes:

[0017] A multi-scale target image is input into a target segmentation network for feature extraction to obtain a multi-scale feature vector set, the multi-scale feature vector set including a plurality of multi-scale feature vectors;

[0018] Convolution calculation is performed on the multi-scale feature vector set and a preset convolution module to obtain a mapping feature map, the mapping feature map including a plurality of grids, each grid uniquely corresponding to a multi-scale feature vector;

[0019] Each grid of the mapping feature map is determined as a target heat map.

[0020] In some embodiments, the network weight of the auxiliary segmentation network is determined by the following formula:

[0021] θ G ←αθ G +(1-α)θ F , wherein a is a preset momentum value, θ F is the network weight of the main segmentation network, and θ G is the network weight of the auxiliary segmentation network.

[0022] In some embodiments, the contrast learning result is determined according to the first heat map and the second heat map by pixel-by-pixel contrast, including:

[0023] A plurality of category region masks are generated according to the second heat map;

[0024] determine a category prototype according to each of the category region masks and the second feature map;

[0025] determine a similarity between each pixel of the first heat map and the category prototype according to the first feature map, and determine the similarity as the contrast learning result;

[0026] wherein an expression of the category region mask is:

[0027] wherein P is a pixel position, is a kth category region mask of the pixel position P, k is a natural number greater than 1, is the second heat map, threshold is a preset threshold, C fg is a preset road category;

[0028] wherein an expression of the category prototype is:

[0029] wherein Pt k is a kth category prototype, is the second feature map;

[0030] wherein the similarity is calculated by the following formula:

[0031] wherein, is the first feature map.

[0032] In some embodiments, before the first heat map, the second heat map and the second feature map are input into the region contrast module, the method further comprises:

[0033] determining a contrast loss function of the region contrast module, wherein an expression of the contrast loss function is:

[0034] wherein Tc is a preset temperature coefficient, and ω is a number of category prototypes.

[0035] In some embodiments, before the first feature map and the contrast learning result are input into the multi-scale attention module, the method further comprises:

[0036] determining a cosine distance between each pixel according to the first heat map and the second heat map;

[0037] determining a loss function of the multi-scale attention module according to the cosine distance.

[0038] In a second aspect, an embodiment of the present application provides a road image segmentation device based on pixel-level self-supervised learning, comprising:

[0039] An image acquisition unit is configured to acquire a preset inspection route and collect an initial road image according to the inspection route.

[0040] A preprocessing unit is configured to perform a preprocessing based on bilinear interpolation on the initial road image to obtain a multi-scale road image set.

[0041] An image injection unit is configured to input the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively, perform a heat map generation operation based on feature extraction through the main segmentation network to obtain a first feature map and a first heat map, perform a heat map generation operation based on feature extraction through the auxiliary segmentation network to obtain a second feature map and a second heat map, the network structure of the main segmentation network is the same as that of the auxiliary segmentation network, and the network weight of the main segmentation network is greater than that of the auxiliary segmentation network.

[0042] A contrast learning unit is configured to input the first heat map, the second heat map and the second feature map into a region contrast module, and determine a contrast learning result according to the first heat map and the second heat map through pixel-by-pixel contrast.

[0043] A classification unit is configured to input the first feature map and the contrast learning result into the multi-scale attention module to obtain a classification result of each pixel of the first feature map, and obtain a target segmentation result according to all the classification results.

[0044] In a third aspect, an embodiment of the present application provides a UAV, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the road image segmentation method based on pixel-level self-supervised learning as described in the first aspect when executing the computer program.

[0045] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program for executing the road image segmentation method based on pixel-level self-supervised learning as described in the first aspect.

[0046] The embodiment of the present application comprises: acquiring a preset inspection route, collecting an initial road image according to the inspection route; performing preprocessing on the initial road image based on bilinear interpolation to obtain a multi-scale road image set; inputting the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively, performing feature extraction-based heat map generation operation through the main segmentation network to obtain a first feature map and a first heat map, performing feature extraction-based heat map generation operation through the auxiliary segmentation network to obtain a second feature map and a second heat map, the network structure of the main segmentation network and the auxiliary segmentation network is the same, and the network weight of the main segmentation network is greater than that of the auxiliary segmentation network; inputting the first heat map, the second heat map and the second feature map into a region comparison module, and determining a comparison learning result according to the first heat map and the second heat map through pixel-by-pixel comparison; inputting the first feature map and the comparison learning result into the multi-scale attention module to obtain the classification result of each pixel of the first feature map, and obtaining a target segmentation result according to all the classification results. According to the technical scheme of the embodiment, the heat maps obtained by the two segmentation networks with the same structure are compared and learned to obtain the pixel-level similarity, and then the multi-scale attention mechanism is used for road classification, so that the image segmentation of the road image is automatically realized under the condition of less labeled data, and the image recognition precision and the automation degree are improved.

[0047] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and achieved by the structure particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings are included to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0049] Figure 1 is a flowchart of a road image segmentation method based on pixel-level self-supervised learning provided by an embodiment of the present application;

[0050] Figure 2 is a flowchart for obtaining a multi-scale road image set provided by another embodiment of the present application;

[0051] Figure 3 is a calculation schematic diagram of bilinear interpolation method provided by another embodiment of the present application;

[0052] Figure 4 is a flowchart for determining a target heat map provided by another embodiment of the present application;

[0053] Figure 5 is a flowchart of determining a contrast learning result provided by another embodiment of the present application;

[0054] Figure 6 is a flowchart of determining a contrast loss function provided by another embodiment of the present application;

[0055] Figure 7 is a flowchart of determining a loss function of a multi-scale attention module provided by another embodiment of the present application;

[0056] Figure 8 is a structural diagram of a road image segmentation device based on pixel-level self-supervised learning provided by another embodiment of the present application;

[0057] Figure 9 is a device diagram of a UAV provided by another embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0059] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "target" and the like in the specification, claims or above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0060] The application provides a road image segmentation method and device based on pixel-level self-supervised learning, and the method comprises the following steps: acquiring a preset inspection route, collecting an initial road image according to the inspection route, performing preprocessing on the initial road image based on bilinear interpolation to obtain a multi-scale road image set, inputting the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively, performing a heat map generation operation based on feature extraction through the main segmentation network to obtain a first feature map and a first heat map, performing a heat map generation operation based on feature extraction through the auxiliary segmentation network to obtain a second feature map and a second heat map, the network structure of the main segmentation network is the same as that of the auxiliary segmentation network, and the network weight of the main segmentation network is greater than that of the auxiliary segmentation network, inputting the first heat map, the second heat map and the second feature map into a region comparison module, determining a comparison learning result according to the first heat map and the second heat map through pixel-by-pixel comparison, inputting the first feature map and the comparison learning result into a multi-scale attention module to obtain a classification result of each pixel of the first feature map, and obtaining a target segmentation result according to all the classification results. According to the technical scheme of the embodiment, the heat maps obtained through two segmentation networks with the same structure are compared to obtain a pixel-level similarity, and then a multi-scale attention mechanism is used for road classification, so that the image segmentation of the road image can be automatically realized under the condition that the labeled data is small, and the image recognition precision and the automation degree are improved.

[0061] As Figure 1 shown, Figure 1 is a flowchart of a road image segmentation method based on pixel-level self-supervised learning provided by an embodiment of the application. The road image segmentation method based on pixel-level self-supervised learning is applied to a UAV and comprises the following steps:

[0062] In step S110, a preset inspection route is acquired, and an initial road image is collected according to the inspection route.

[0063] In step S120, preprocessing is performed on the initial road image based on bilinear interpolation to obtain a multi-scale road image set.

[0064] In step S130, the multi-scale road image set is input into a main segmentation network and an auxiliary segmentation network respectively, a first feature map and a first heat map are obtained by performing a heat map generation operation based on feature extraction through the main segmentation network, a second feature map and a second heat map are obtained by performing a heat map generation operation based on feature extraction through the auxiliary segmentation network, the network structure of the main segmentation network is the same as that of the auxiliary segmentation network, and the network weight of the main segmentation network is greater than that of the auxiliary segmentation network.

[0065] Step S140, input the first heat map, the second heat map and the second feature map into a region contrast module, and determine a contrast learning result according to the first heat map and the second heat map through pixel-by-pixel contrast;

[0066] Step S150, input the first feature map and the contrast learning result into a multi-scale attention module to obtain a classification result of each pixel of the first feature map, and obtain a target segmentation result according to all the classification results.

[0067] It should be noted that the inspection route in the embodiment can be determined according to a common path planning algorithm, and the path planning is not limited too much in the embodiment, and the inspection route of the unmanned aerial vehicle can be determined.

[0068] It should be noted that the initial road image can be obtained by a camera carried on the unmanned aerial vehicle, for example, in the process of flight of the unmanned aerial vehicle according to the inspection route, a video image is captured by the camera, and each frame of the video image is determined as the initial road image; for another example, the image is captured at a preset period during the flight of the unmanned aerial vehicle, and the specific mode can be selected according to the time requirement.

[0069] It should be noted that the preprocessing of bilinear interpolation can effectively increase the image scale, and provide more image samples for model training and network segmentation.

[0070] It should be noted that the network structures of the main segmentation network and the auxiliary segmentation network can be the same, while in the actual segmentation process, the main segmentation network can be optimized by gradient descent, and the auxiliary segmentation network supports parameter updating by exponential momentum update.

[0071] It should be noted that the feature extraction to obtain the feature map is a technology familiar to those skilled in the art, and will not be described in detail here. After obtaining the first feature map and the second feature map, a plurality of grids can be obtained by convolution of the feature map through a convolution window, and each grid obtained after calculation is determined as a heat map by adding data between different channels.

[0072] It should be noted that in order to stably generate self-supervision for training the main network, the embodiment also provides a region contrast module and a multi-scale attention module. The region contrast module can realize similarity determination through contrast learning, solving the problem of automatic annotation of road image segmentation. The multi-scale attention module is used to adapt to the problem that narrow roads are difficult to finely segment in the road segmentation process. Through the technical solution of the embodiment, fine segmentation of the road image can be realized under the condition that the obtained image is less. Through contrast learning and multi-scale attention mechanism, the road and the obstacle are distinguished, and the accuracy of road recognition is improved.

[0073] In addition, in an embodiment, with reference to Figure 2 ,Figure 1 Step S120 of the illustrated embodiment also includes, but is not limited to, the following steps:

[0074] Step S210: Obtain the preset scale factor and determine the interpolation distance value based on the scale factor;

[0075] Step S220: Determine the initial coordinates of each pixel in the initial road image in three-dimensional coordinates;

[0076] Step S230: Based on each initial coordinate and interpolation distance value, perform two interpolations in the vertical axis direction and one interpolation in the horizontal axis direction to obtain four neighbor point sets;

[0077] Step S240: Obtain a multi-scale road image set based on the pixels corresponding to each neighbor set.

[0078] It should be noted that the scale factor can be selected according to actual needs. In order to better illustrate the technical solution of this embodiment, the following uses three scale factors of 0.5, 1 and 2 as examples.

[0079] The drone flies along a designated route, circling points, and preprocesses the collected aerial images. The data pairs are represented as (I,t), where I represents the input image, and t = {c1, c2} represents two forms of classification labels for a binary-hot encoded vector, indicating roads and non-roads respectively. To better illustrate the principle, images I with scale factors of 0.5, 1.0, and 2.0 are respectively denoted by the symbol I. s I m and I l It means. I s and I l Through image I m It is obtained by bilinear interpolation, where I m The original image I is represented by I = {I}. s ,I m ,I l} represents the set of these multi-scale images.

[0080] like Figure 3 As shown, taking a scale factor of 1 as an example, we take the four neighboring points around point I = (W, H), interpolate twice in the y-direction (or x-direction), and then interpolate once in the x-direction (or y-direction) to obtain the value f(W, H) of point (W, H). Let the four neighboring points be (i, j), (i, j+1), (i+1, j), and (i+1, j+1). Draw a straight line through (W, H) parallel to the x-axis, intersecting the edge formed by the four neighboring points at points (i, H) and (i+1, H). First, interpolate in the y-direction and calculate the values ​​f(i, H) and f(i+1, H) at the intersection points. f(i, H) is then calculated by interpolating f(i, j+1) and f(i, j).

[0081] According to the above manner, the value of f(W, H) is:

[0082] f(W, H) = a [βf(i+1, j+1) + (1-β)f(i+1, j)] + (1-a) [βf(i, j+1) + (1-β)f(i, j)], where (W, H) is any point of the image, i is the interpolation distance value in the horizontal axis direction, j is the interpolation distance value in the vertical axis direction, a = x-i, β = y-j. By the above method, the image is processed to obtain a multi-scale road image set with scale factors of 0.5, 1.0, and 2.0. Multiplying i and j by the scale factor can achieve the effect of averaging low-pass filtering, the edges are smoothed, a relatively coherent output image is generated, and the image processing efficiency is improved.

[0083] It should be noted that one interpolation by the above method can obtain four adjacent points, and the pixels corresponding to the coordinates of the adjacent points are pixels of a scale road image. After interpolation for each pixel of the original image, four adjacent point sets are obtained, each adjacent point set is an image set, and four scale road image sets are obtained.

[0084] In addition, in an embodiment, the step S130 of the embodiment shown in Figure 4 , Figure 1 may further include but not limited to the following steps:

[0085] Step S410, inputting the multi-scale target image into the target segmentation network to extract features and obtain a multi-scale feature vector set, the multi-scale feature vector set including a plurality of multi-scale feature vectors;

[0086] Step S420, performing convolution calculation on the multi-scale feature vector set and the preset convolution module to obtain a mapping feature map, the mapping feature map including a plurality of grids, each grid uniquely corresponding to a multi-scale feature vector;

[0087] Step S430, determining each grid of the mapping feature map as a target heat map.

[0088] It should be noted that the multi-scale target image I is first input into the main segmentation network N and the auxiliary segmentation network Z to obtain a multi-scale feature vector set of two branches, denoted as and Then, a 1x1 convolution module is used to calculate the mapping feature map E (E e R D×H×W), where D is the number of channels of the feature space vector, H is the height, and W is the width. Since the convolution window is 1x1, the data between different channels of the feature space vector are added, for example, the size of the input data body is I1=(W1, H1, D1), and the size of the output data body is I2=(W2, H2, D2), which satisfies the following expressions: W2=(W1-F+2P) / S+1; H2=(H1-F+2P) / S+1; D2=K, where K is the number of filters, F is the spatial size of the filter, S is the step size, and P is the number of zero padding. The corresponding set of target heat maps of each network is obtained after processing by the above formula, and is respectively denoted as and Then, all the sub-vectors of E N and T N are interpolated to the size of I m . The heat map of the kth road class is further scaled to T K ← T K / mαx(T K ).

[0089] In addition, in an embodiment, the network weight of the auxiliary segmentation network is determined by the following formula:

[0090] θ G ← αθ G +(1-α)θ F , where α is a preset momentum value, θ F is the network weight of the main segmentation network, and θ G is the network weight of the auxiliary segmentation network.

[0091] It should be noted that in the actual training process of the segmentation network, θ G is updated by θ F , and the value of α can be 0.99, so that the auxiliary segmentation network can provide stable self-supervised pixel-level contrastive learning for the main segmentation network. Of course, the value of α can also be adjusted according to the learning accuracy requirement, which is not limited here.

[0092] In addition, in an embodiment, the step S140 of the embodiment shown in Figure 5 , Figure 1 further includes but is not limited to the following steps:

[0093] Step S510, generating a plurality of class region masks according to the second heat map;

[0094] Step S520, determining a class prototype according to each class region mask and the second feature map;

[0095] Step S530, determining the similarity between each pixel of the first heat map and the class prototype according to the first feature map, and determining the similarity as the contrastive learning result.

[0096] wherein the expression of the class region mask is:

[0097] wherein P is a pixel position, is the kth class region mask of the pixel position P, k is a natural number greater than 1, is the second heat map, threshold is a preset threshold, C fg is a preset road class;

[0098] wherein the expression of the class prototype is:

[0099] wherein Pt k is the kth class prototype, is the second feature map;

[0100] wherein the similarity is calculated by the following formula:

[0101] wherein, is the first feature map.

[0102] It should be noted that the multi-scale map obtained in the embodiment shown in Figure 4 is input into the region contrast module, and the class region mask is generated by thresholding the heat map from the auxiliary segmentation network. These masks are used for pixel-level self-supervision to guide the classification of each pixel. According to the class region mask, the representative features are fused. In order to increase the representativeness, the present application simultaneously eliminates the erroneous nature of self-supervision, aggregates the representative features of each class in all images in the batch dimension, and defines the generated vector as the prototype of the class classification. Then, according to the class region mask, each pixel-level feature of the main segmentation network is trained to be close to the corresponding class prototype, while being far away from the prototypes of other classes. Due to the lack of region information, only image-level supervision can lead to inaccurate results. On the other hand, through pixel-level supervision based on class region mask and class prototype, the proposed region contrast module effectively solves the inaccuracy of the heat map by guiding the classification of each pixel through contrast.

[0103] It should be noted that the heat map can provide the positioning region of each class during the training stage, and the heat map is classified as a pixel-by-pixel score map of a certain class. For each pixel, a class region mask is generated. In order to further correct the rough positioning result of the heat map, the pixels with an activation value greater than a preset threshold can be generated in the process of generating the class region mask. Therefore, the acquisition of the kth mask can be expressed as: wherein P is a pixel position, is the kth class region mask of the pixel position P, k is a natural number greater than 1, For the second heat map, the threshold is a preset threshold, C fg The preset road class.

[0104] Since the semi-supervised learning of deep learning usually uses a class region mask to do pseudo-labeling, however, in view of the characteristics of the unmanned aerial vehicle affecting the road segmentation application, the segmentation of the road needs a refined boundary, and the quality and stability of the self-supervision of the embodiment are both advantageous, the model of the embodiment does not use a saliency module or a pre-trained classifier to correct the region information score, but directly obtains the class region mask from the heat map of the road segmentation support network.

[0105] It should be noted that in the region contrast module, the class region mask is not only used for self-supervised learning of the unmanned aerial vehicle image pixels, but also used for class prototype generation to obtain a prototype of each class. Even though pixel-level self-supervision is an effective way to impose spatial constraints through image-level supervision, self-supervision still has the possibility of inaccuracy, therefore, in order to avoid the derailment phenomenon caused by self-supervision, the embodiment uses feature-level contrastive learning instead of directly calculating the cross-entropy loss between the heat map set and the class region mask. According to the class region mask of the kth class, the support network feature E Z On average, the prototype of the kth class can be determined as: Wherein, Pt k is the kth class prototype, is the second feature map.

[0106] It should be noted that the pixel-level feature obtained by the main segmentation network can be obtained in the form of contrastive learning. First, map the feature and the prototype to a D-dimensional space using the L2 norm, and then define the similarity between them using the dot product and the exponential function: Wherein, is the first feature map, p is the pixel position, N is the main segmentation network, is the kth class prototype.

[0107] In addition, in an embodiment, with reference to Figure 6 Before performing step S140 of the embodiment shown in FIG. 1, the following steps are further included but are not limited to: Figure 1

[0108] Step S610, determine the contrastive loss function of the region contrast module;

[0109] Wherein, the expression of the contrastive loss function is:

[0110] Wherein, Tc is a pre-set temperature coefficient, and ω is the number of class prototypes.

[0111] ​It should be noted that the goal of the region contrast module is to ensure that the pixels of roads and non-roads in the road images acquired by the UAV are close to their corresponding class prototypes in terms of pixel-level features, while being far away from other class prototypes. Based on this, when determining the contrast loss function, class intelligent prototypes can be constructed from multiple images, and these features can be trained to be close to the corresponding prototypes. Based on maximizing mutual information, the following contrast loss function is obtained: in, This is the first feature map, where p is the pixel position and Tc is the temperature coefficient. This represents the prototype of the k-th class, where ω is the number of classes, which is set to 2 here. The contrastive losses of all classes are then summed to calculate the result.

[0112] Additionally, in one embodiment, reference is made to Figure 7 In execution Figure 1 Before step S140 in the illustrated embodiment, the following steps may also be included, but are not limited to:

[0113] Step S710: Determine the cosine distance between each pixel based on the first heatmap and the second heatmap;

[0114] Step S720: Determine the loss function of the multi-scale attention module based on the cosine distance.

[0115] It should be noted that the sparse activation problem can be solved by using a multi-scale attention module, which leverages the high localization capability of multi-scale inference. Since the network focuses on different regions based on the resolution of the input image, an attention matrix can be defined based on the differences between the heatmaps of the main segmentation network and the auxiliary segmentation network. The corresponding attention scores are used as weighting parameters for generating multi-scale inference heatmaps, which are then used for self-supervised training of the main network. Furthermore, the multi-scale inference heatmap (i.e., msinf-CAM) is an aggregation of heatmaps obtained at multiple image scales, thereby addressing the sparse activation problem.

[0116] It should be noted that multi-scale inference methods can also eliminate large variations in heatmaps from multi-scale images to generate pseudo-labels. Heatmaps obtained from images at a specific resolution contain meaningful information that is difficult to obtain at other resolutions. Therefore, through multi-scale inference technology, the inference performance of existing segmentation networks is greatly improved. The multi-scale attention module is based on this principle, fully utilizing the high localization capability of heatmaps while guiding each scale of heatmap to learn more information from different heatmaps at different scales. Based on this, the cosine distance (ξ) in this embodiment can be calculated for categories present in image-level monitoring, as shown below: Where i, j∈{s, m, l}, represent the scaling factor, and k·k2 represents the predefined L2 norm. For T N and TZ Vectorization is performed, and the dimension of the cosine distance matrix is RNx3x3. The cosine distance matrix is used to create a multi-scale self-supervised heat map obtained from the main network. The target heat map is a weighted sum of the heat map at different scales according to the cosine distance, that is, the target heat map is: wherein, is the second heat map of the Kth class; meanwhile, the loss of the multi-scale attention module can be defined as: wherein, is the first heat map of the Kth class, and k·k1 represents a preset L1 norm.

[0117] It should be noted that, in order to train the segmentation network model, the total loss function of the model can be determined according to the contrast loss function and the loss function of the multi-scale attention module, and the expression is as follows: L our =λ1L cls +λ2L RCM +λ3L man , wherein L our is the total loss function, L cls represents the binary cross-entropy loss between the image-level class prediction of the main segmentation network and the image-level classification label, L RCM is the contrast loss function, L man is the loss function of the multi-scale attention module, and λ1, λ2 and λ3 are balance parameters between loss terms, and the specific values are determined according to actual requirements.

[0118] In addition, with reference to Figure 8 , the embodiment of the present application provides a road image segmentation device based on pixel-level self-supervised learning, which comprises

[0119] An image acquisition unit 810 is configured to acquire a preset inspection route and collect an initial road image according to the inspection route;

[0120] A preprocessing unit 820 is configured to perform preprocessing on the initial road image based on bilinear interpolation to obtain a multi-scale road image set;

[0121] An image injection unit 830 is configured to input the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively, perform a heat map generation operation based on feature extraction through the main segmentation network to obtain a first feature map and a first heat map, and perform a heat map generation operation based on feature extraction through the auxiliary segmentation network to obtain a second feature map and a second heat map, wherein the network structure of the main segmentation network and the auxiliary segmentation network is the same, and the network weight of the main segmentation network is greater than that of the auxiliary segmentation network;

[0122] The contrast learning unit 840 is configured to input the first heat map, the second heat map and the second feature map into a region contrast module, and determine a contrast learning result according to the first heat map and the second heat map through pixel-by-pixel contrast.

[0123] The classification unit 850 is configured to input the first feature map and the contrast learning result into a multi-scale attention module, obtain a classification result of each pixel of the first feature map, and obtain a target segmentation result according to all the classification results.

[0124] In addition, referring to Figure 9 One embodiment of the present application further provides a UAV, which comprises a memory 910, a processor 920 and a computer program stored in the memory 910 and executable on the processor 920.

[0125] The processor 920 and the memory 910 can be connected through a bus or other means.

[0126] The non-transitory software program and instructions required for implementing the pixel-level self-supervised learning-based road image segmentation method of the above embodiment are stored in the memory 910, and when executed by the processor 920, the pixel-level self-supervised learning-based road image segmentation method in the above embodiment is executed, for example, the method steps S110 to S150 in the above description Figure 1 , the method steps S210 to S240 in the above description Figure 2 , the method steps S410 to S430 in the above description Figure 4 , the method steps S510 to S530 in the above description Figure 5 , the method step S610 in the above description Figure 6 , and the method steps S710 to S720 in the above description Figure 7 .

[0127] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0128] In addition, one embodiment of the present application further provides a computer readable storage medium storing a computer program, which is executed by a processor or a controller, for example, by a processor in the above UAV embodiment, so that the above processor executes the pixel-level self-supervised learning-based road image segmentation method in the above embodiment, for example, executes the method steps S110 to S150 in the above description Figure 1 , the method steps S210 to S240 in the above description Figure 2the method steps S210 to S240 in the method of Figure 4 the method steps S410 to S430 in the method of Figure 5 the method steps S510 to S530 in the method of Figure 6 the method step S610 in the method of Figure 7 the method steps S710 to S720 in the method of. Those of ordinary skill in the art will appreciate that all or some of the steps, means, or components in the above-disclosed methods can be implemented as software, firmware, hardware, or any suitable combination thereof. Some or all of the components or means can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable storage media, which can comprise any medium or means that embodies or stores the software and from which a computer can read. Computer readable storage media can include computer storage media (or non-transitory storage media) and communication media (or transitory storage media). Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other storage medium which can be used to store the desired information and which can be accessed by a computer. Further, it should be appreciated that a computer program product can include a computer readable medium having executable instructions embodied therein, and that such a computer program product can be distributed over network(s) coupled with computer(s). For example, the computer readable medium can comprise a transmission medium such as a carrier wave carrying the program segments, and a computer program product can comprise executable instructions embodied in a carrier wave.

[0129] Other embodiments of the application will be apparent to those of ordinary skill in the art from a consideration of the specification and practice of the application disclosed herein. It is intended that the application encompass any and all variations, uses, or adaptations of the application which are reasonably

[0130] It is to be understood that the application is not limited to the precise construction described in the specification above and shown in the drawings, and that various modifications and changes can be made by those of ordinary skill in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

[0131] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A road image segmentation method based on pixel-level self-supervised learning, characterized in that, Applied to drones, including: Obtain a preset inspection route and collect initial road images based on the inspection route; The initial road images are preprocessed based on bilinear interpolation to obtain a multi-scale road image set; The multi-scale road image set is input into a main segmentation network and an auxiliary segmentation network, respectively. The main segmentation network performs a feature extraction-based heatmap generation operation to obtain a first feature map and a first heatmap. The auxiliary segmentation network performs a feature extraction-based heatmap generation operation to obtain a second feature map and a second heatmap. The main segmentation network and the auxiliary segmentation network have the same network structure, and the network weights of the main segmentation network are greater than those of the auxiliary segmentation network. The first heatmap, the second heatmap, and the second feature map are input into the region comparison module, and the comparison learning result is determined based on the first heatmap and the second heatmap by comparing them pixel by pixel. The first feature map and the contrastive learning result are input into the multi-scale attention module to obtain the classification result of each pixel of the first feature map, and the target segmentation result is obtained based on all the classification results; The step of determining the contrast learning result by comparing the first heatmap and the second heatmap pixel by pixel includes: Multiple category region masks are generated based on the second heatmap; The category prototype is determined based on the category region mask for each category and the second feature map; Based on the first feature map, the similarity between each pixel of the first heatmap and the category prototype is determined, and the similarity is determined as the contrastive learning result; The expression for the category region mask is as follows: Where p is the pixel position. Let p be the mask for the k-th category region at pixel location p, where k is a natural number greater than 1. This is the second heatmap. For the preset threshold, For the preset road type, For the c-th element of the second heatmap, This is the k-th element of the second heatmap; The expression for the category prototype is: ,in, For the k-th category prototype, This is the second feature map; The similarity is calculated using the following formula: ,in, This is the first feature map. It is the prototype of the kth category.

2. The road image segmentation method based on pixel-level self-supervised learning according to claim 1, characterized in that, The preprocessing of the initial road image based on bilinear interpolation yields a multi-scale road image set, including: Obtain a preset scale factor, and determine the interpolation distance value based on the scale factor; Determine the initial coordinates of each pixel in the initial road image in three-dimensional coordinates; Based on each initial coordinate and the interpolation distance value, two interpolations are performed in the vertical axis direction and one interpolation is performed in the horizontal axis direction to obtain four neighbor point sets; A multi-scale road image set is obtained based on the pixels corresponding to each of the neighboring point sets.

3. The road image segmentation method based on pixel-level self-supervised learning according to claim 1, characterized in that, The heatmap generation operation based on feature extraction includes: A multi-scale target image is input into a target segmentation network for feature extraction to obtain a multi-scale feature vector set, which includes multiple multi-scale feature vectors. Convolution calculation is performed using the multi-scale feature vector set and a preset convolution module to obtain a mapping feature map. The mapping feature map includes multiple grids, and each grid uniquely corresponds to the multi-scale feature vector. Each grid of the mapped feature map is determined as the target heatmap.

4. The road image segmentation method based on pixel-level self-supervised learning according to claim 1, characterized in that, The network weights of the auxiliary segmentation network are determined by the following formula: Where α is a preset momentum value, These are the network weights of the main segmentation network. These are the network weights that assist in segmenting the network.

5. The road image segmentation method based on pixel-level self-supervised learning according to claim 1, characterized in that, Before inputting the first heatmap, the second heatmap, and the second feature map into the region comparison module, the method further includes: Determine the contrast loss function for the region contrast module, wherein the expression for the contrast loss function is: Where Tc is a pre-set temperature coefficient. N represents the number of category prototypes, and N is the main segmentation network.

6. The road image segmentation method based on pixel-level self-supervised learning according to claim 5, characterized in that, Before inputting the first feature map and the contrastive learning result into the multi-scale attention module, the method further includes: The cosine distance between each pixel is determined based on the first heatmap and the second heatmap; The loss function of the multi-scale attention module is determined based on the cosine distance.

7. A road image segmentation device based on pixel-level self-supervised learning, characterized in that, include: The image acquisition unit is used to acquire a preset inspection route and collect initial road images based on the inspection route; The preprocessing unit is used to perform bilinear interpolation-based preprocessing on the initial road image to obtain a multi-scale road image set. An image injection unit is used to input the multi-scale road image set into a main segmentation network and an auxiliary segmentation network respectively. The main segmentation network performs a feature extraction-based heatmap generation operation to obtain a first feature map and a first heatmap. The auxiliary segmentation network performs a feature extraction-based heatmap generation operation to obtain a second feature map and a second heatmap. The main segmentation network and the auxiliary segmentation network have the same network structure, and the network weights of the main segmentation network are greater than those of the auxiliary segmentation network. The contrast learning unit is used to input the first heatmap, the second heatmap, and the second feature map into the region contrast module, and determine the contrast learning result based on the first heatmap and the second heatmap by comparing them pixel by pixel; A classification unit is used to input the first feature map and the contrastive learning result into a multi-scale attention module to obtain the classification result of each pixel of the first feature map, and to obtain the target segmentation result based on all the classification results; The step of determining the contrast learning result by comparing the first heatmap and the second heatmap pixel by pixel includes: Multiple category region masks are generated based on the second heatmap; The category prototype is determined based on the category region mask for each category and the second feature map; Based on the first feature map, the similarity between each pixel of the first heatmap and the category prototype is determined, and the similarity is determined as the contrastive learning result; The expression for the category region mask is as follows: Where p is the pixel position. Let p be the mask for the k-th category region at pixel location p, where k is a natural number greater than 1. This is the second heatmap. For the preset threshold, For the preset road type, For the c-th element of the second heatmap, This is the k-th element of the second heatmap; The expression for the category prototype is: ,in, For the k-th category prototype, This is the second feature map; The similarity is calculated using the following formula: ,in, This is the first feature map. It is the prototype of the kth category.

8. An unmanned aerial vehicle (UAV), comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the road image segmentation method based on pixel-level self-supervised learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to execute the road image segmentation method based on pixel-level self-supervised learning as described in any one of claims 1 to 6.

Images