Sky-ground segmentation line detection method, device and equipment based on image context information

Abstract

This application relates to a method, apparatus, and device for detecting top-to-bottom lines based on image context information in the field of image semantic segmentation technology. The method includes: labeling an acquired preprocessed outdoor scene natural image dataset to obtain training samples; constructing a top-to-bottom line detection model, which includes an encoder, a decoder, and a semantic supervision network; training the model using the training samples to obtain a trained top-to-bottom line detection model; and using the trained model to detect top-to-bottom lines in the image to be detected and labeling them. This method utilizes the context module in the encoder to output multi-scale context information to overcome the influence of weather changes, the semantic embedding module in the decoder to narrow the gap between semantic information and spatial information and improve prediction accuracy, and the semantic boundary output by the semantic supervision network to enhance the recognition ability of small, difficult-to-distinguish samples.

Description

Technical Field

[0001] This application relates to the field of image semantic segmentation technology, and in particular to a method, apparatus and device for detecting top and bottom dividing lines based on image context information. Background Technology

[0002] With the development of miniature inertial sensors, new avionics technologies, and high-speed chip technology, the functions of drones have been significantly improved. In most jobs requiring monitoring or detection technologies, such as traffic monitoring, border patrol, search and rescue, and wildfire fighting, drones are widely used as low-cost tools. However, in some special environments, such as deep mountains, offshore waters, or deserts where stable signals are lacking, absent, or even blocked, it is impossible to use base station or satellite signals to provide navigation for safe drone flight. Therefore, offline maps supplemented by robust sky-ground divider detection algorithms are necessary to assist drones in obstacle detection, autonomous landing, and navigation. As mentioned above, robust sky-ground divider detection algorithms are crucial for drones to complete tasks in areas with poor or blocked signals.

[0003] With the application of deep learning in image semantic segmentation, sky-ground line detection technology has developed rapidly. However, due to factors such as weather changes, terrain, and light intensity, classic deep learning algorithms are not directly applicable to sky-ground line detection tasks. Furthermore, some non-sky regions with sky features, such as specular reflections of the sky, can affect the deep learning algorithm's ability to identify sky regions. Therefore, this invention addresses these difficult-to-segment samples, overcoming the influence of weather changes to a certain extent, and implementing a robust sky-ground line detection algorithm, which is of great significance for the safe navigation of unmanned aerial vehicles (UAVs). Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and device for detecting top and bottom lines based on image context information to address the aforementioned technical problems.

[0005] A method for detecting top and bottom dividing lines based on image context information, the method comprising:

[0006] The preprocessed outdoor scene natural image dataset was labeled to obtain training samples.

[0007] A model for detecting top-to-bottom lines is constructed, comprising an encoder, a decoder, and a semantic supervision network. The encoder includes a main network consisting of the first four convolutional modules of a VGG16 network, and a context module. The main network downsamples the training samples using the four convolutional modules, outputting four feature maps of different sizes. The context module processes the feature maps of different sizes output after the fourth convolutional module operation using dilated convolution, global average pooling, and channel attention mechanisms to obtain multi-scale context information feature maps. The semantic supervision network fuses all feature maps of different sizes and multi-scale context information feature maps using bilinear interpolation and a residual correction module to obtain semantic boundary feature maps. The decoder fuses the multi-scale context information feature maps and feature maps of different sizes at various levels using deconvolution and a semantic embedding module to obtain the sample prediction result.

[0008] Based on the semantic boundary feature map, the annotation of the training samples, and the sample prediction results obtained by inputting the training set into the sky-ground dividing line detection model, the sky-ground dividing line detection model is trained to obtain a trained sky-ground dividing line detection model.

[0009] The acquired image to be detected is input into the trained sky-ground dividing line detection model to obtain the prediction result of the image to be detected. If the prediction result of the image to be detected contains a sky-ground dividing line, the sky-ground dividing line is marked.

[0010] A device for detecting top and bottom dividing lines based on image context information, the device comprising:

[0011] The training sample determination module is used to annotate the acquired preprocessed outdoor scene natural image dataset to obtain training samples.

[0012] A module for constructing a top-to-bottom line detection model is included. This model comprises an encoder, a decoder, and a semantic supervision network. The encoder consists of a main network composed of the first four convolutional modules of a VGG16 network, and a context module. The main network downsamples the training samples using the four convolutional modules, outputting four feature maps of different sizes. The context module processes the feature maps of different sizes output after the fourth convolutional module operation using dilated convolution, global average pooling, and channel attention mechanisms to obtain multi-scale context information feature maps. The semantic supervision network fuses all feature maps of different sizes and the multi-scale context information feature maps using bilinear interpolation and a residual correction module to obtain a semantic boundary feature map. The decoder fuses the multi-scale context information feature maps and feature maps of different sizes at each level using deconvolution and a semantic embedding module to obtain the sample prediction result.

[0013] The sky-ground dividing line detection model training module is used to train the sky-ground dividing line detection model based on the semantic boundary feature map, the annotation of the training samples, and the sample prediction results obtained by inputting the training set into the sky-ground dividing line detection model, so as to obtain a trained sky-ground dividing line detection model.

[0014] The top-to-bottom line detection and labeling module is used to input the acquired image to be detected into the trained top-to-bottom line detection model to obtain the prediction result of the image to be detected. If the prediction result of the image to be detected contains a top-to-bottom line, the top-to-bottom line is labeled.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0016] The aforementioned method, apparatus, and device for detecting top-to-bottom lines based on image context information include: labeling a preprocessed dataset of outdoor scene natural images to obtain training samples; constructing a top-to-bottom line detection model, which includes an encoder, a decoder, and a semantic supervision network; the encoder uses the first four layers of a VGG16 network as the main network, and adds a context module at the bottom layer of the main network to output multi-scale context information; to narrow the gap between semantic and spatial information and improve the model's prediction accuracy, a semantic embedding module is added to the decoder; to increase the model's ability to recognize small, difficult-to-distinguish samples, a semantic supervision network is designed based on high-level semantic feature maps, and the model's prediction is supervised by the proportion of the loss function; the top-to-bottom line detection model is trained using the training samples to obtain a trained top-to-bottom line detection model; the trained top-to-bottom line detection model is used to detect the image to be detected and the top-to-bottom lines are labeled. This method overcomes the influence of weather changes by using multi-scale context information, and at the same time utilizes semantic boundaries to enhance the recognition ability of difficult-to-distinguish samples. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for detecting top and bottom lines based on image context information in one embodiment;

[0018] Figure 2 Here is a structural diagram of the skyline detection model in another embodiment;

[0019] Figure 3 This is a schematic diagram of the training process for the top-to-bottom line detection model in another embodiment;

[0020] Figure 4 Here is a structural diagram of the context module in another embodiment;

[0021] Figure 5 Here is a structural diagram of the semantic embedding module in another embodiment;

[0022] Figure 6 This is a structural diagram of the RRB module in another embodiment;

[0023] Figure 7 This is a structural block diagram of a top-to-bottom line detection device based on image context information in one embodiment;

[0024] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] Sky region detection network (SRDN) is a model for detecting the sky region.

[0027] In one embodiment, such as Figure 1 As shown, a method for detecting top and bottom dividing lines based on image context information is provided. This method includes the following steps:

[0028] Step 100: Label the preprocessed outdoor scene natural image dataset to obtain training samples.

[0029] Specifically, in order to handle input images of different resolutions, the images in the image dataset are preprocessed by centering and uniformly sizing to 352*480, and the preprocessed images are labeled and used as training samples.

[0030] Step 102: Construct a model for detecting the top and bottom dividing lines.

[0031] The top-to-bottom line detection model includes an encoder, a decoder, and a semantic supervision network.

[0032] The encoder consists of a main network composed of the first four convolutional modules of the VGG16 network, and a context module. The main network is used to downsample the training samples using the four convolutional modules to output four feature maps of different sizes. The context module is used to process the feature maps of different sizes output after the operation of the fourth convolutional module using dilated convolution, global average pooling, and channel attention mechanisms to obtain multi-scale context information feature maps.

[0033] The semantic supervision network is used to fuse all feature maps of different sizes and multi-scale contextual information feature maps using bilinear interpolation and a modified residual module to obtain semantic boundary feature maps.

[0034] The decoder is used to fuse multi-scale contextual information feature maps and feature maps of different sizes at various levels using deconvolution and semantic embedding modules to obtain sample prediction results.

[0035] The structure diagram of the skyline detection model is as follows: Figure 2 As shown.

[0036] Specifically, the top-to-bottom line detection model is based on an encoding and decoding structure. It uses the first four convolutional modules of the classic image classification network VGG16 as the main encoder network, and adds a context module at the bottom layer of the main network to output multi-scale context information.

[0037] The context module outputs context information at different scales in parallel using a spatial pyramid structure through four dilated convolutions with different dilation rates and a global average pooling. Feature maps are fused via spatial channel concatenation, followed by a channel attention mechanism to enhance the model's attention to the sky region. The context module primarily outputs multi-scale context information. Although fully convolutional networks use an end-to-end, pixel-to-pixel learning approach, individual pixels have connections with their neighboring pixels (of the same or different categories). Therefore, rich context information can improve the model's ability to recognize individual pixels, especially under adverse weather conditions such as fog or rain. In such conditions, rain or fog pixels cover the entire image. If end-to-end, pixel-to-pixel learning is still used, ignoring the connections between individual pixels and other pixels, the model's output accuracy will be affected, and it may even fail to recognize pixels.

[0038] The decoder performs conventional feature fusion, used to fuse multi-scale contextual information feature maps F. pThe algorithm begins by reconstructing feature maps layer by layer using deconvolution until they match the original image size. During this reconstruction, feature maps of the same size are fused using concat. The Semantic Embedding Branch (SEB) then bridges the gap between lower-level feature maps rich in spatial and color information and higher-level feature maps rich in semantic information, improving the model's accuracy in skyline detection. The SEB incorporates more resolution information into higher-level, semantically rich but spatially lacking feature maps, then fuses these with lower-level, spatially rich but semantically lacking feature maps. This reduces the gap between semantic and spatial information in the high- and low-level feature maps, improving the model's segmentation accuracy. Higher-level feature maps refer to those output by the third and fourth convolutional modules of the main network, while lower-level feature maps refer to those output by the first and second convolutional modules. Lower-level feature maps contain rich spatial information (resolution information) but relatively little semantic information, while higher-level feature maps contain rich semantic information but relatively little spatial information.

[0039] To enhance the model's ability to recognize small, indistinguishable samples, a semantic supervision network was designed based on high-level semantic feature maps, and the model's predictions were supervised by the proportion of the loss function. This semantic supervision network is used to fuse the output feature maps of each convolutional module of the main network and the context information feature maps to output semantic boundary feature maps. Starting with the multi-scale context information feature map Fp, the feature maps are reconstructed layer by layer using bilinear interpolation until they are the same size as the original image. During the reconstruction process, feature maps of the same size are fused using a concat method. To prevent gradient vanishing during training, a residual module was added to the supervision network, ultimately outputting the semantic boundary feature map.

[0040] Step 104: Based on the semantic boundary feature map, the annotation of the training samples, and the sample prediction results obtained by inputting the training set into the sky-ground dividing line detection model, train the sky-ground dividing line detection model to obtain the trained sky-ground dividing line detection model.

[0041] Step 106: Input the acquired image to be detected into the trained sky-ground dividing line detection model to obtain the prediction result of the image to be detected. If the prediction result of the image to be detected contains sky-ground dividing lines, then mark the sky-ground dividing lines.

[0042] The above-mentioned method for detecting top-to-bottom lines based on image context information includes: labeling a preprocessed dataset of natural outdoor scene images to obtain training samples; constructing a top-to-bottom line detection model, which includes an encoder, a decoder, and a semantic supervision network; the encoder uses the first four layers of a VGG16 network as the main network, and adds a context module at the bottom layer of the main network to output multi-scale context information; to narrow the gap between semantic and spatial information and improve the model's prediction accuracy, a semantic embedding module is added to the decoder; to increase the model's ability to recognize small, difficult-to-distinguish samples, a semantic supervision network is designed based on high-level semantic feature maps, and the model's prediction is supervised by the proportion of the loss function; the top-to-bottom line detection model is trained using the training samples to obtain a trained top-to-bottom line detection model. The trained top-to-bottom line detection model is then used to detect the image to be detected and the top-to-bottom lines are labeled. This method overcomes the influence of weather changes by using multi-scale context information, and at the same time utilizes semantic boundaries to enhance the ability to recognize difficult-to-distinguish samples.

[0043] In one embodiment, such as Figure 3 Step 104 includes the following steps:

[0044] Step 300: Input the training samples into the main network of the encoder to obtain four feature maps of different sizes.

[0045] Step 302: Input the feature maps of different sizes output by the fourth convolutional module into the context module of the encoder to obtain multi-scale context information feature maps.

[0046] Step 304: Input all feature maps of different sizes and multi-scale contextual information feature maps into the semantic supervision network to obtain semantic boundary feature maps.

[0047] Step 306: Input all feature maps of different sizes and multi-scale contextual information feature maps into the decoder to obtain the sample prediction results.

[0048] Step 308: Perform reverse training on the sky-ground dividing line detection model based on the semantic boundary feature map, sample prediction results, and the annotation of the training samples to obtain the trained sky-ground dividing line detection model.

[0049] In one embodiment, each of the four convolutional modules of the main network includes a convolutional layer, batch normalization, and a ReLU activation function. Step 300 includes: inputting training samples into the first convolutional module of the main network for convolution and batch normalization, and then activating them with the ReLU function to obtain a first feature map of different sizes; inputting the first feature map of different sizes into the second convolutional module of the main network for convolution and batch normalization, and then activating them with the ReLU function to obtain a second feature map of different sizes; inputting the second feature map of different sizes into the third convolutional module of the main network for convolution and batch normalization, and then activating them with the ReLU function to obtain a third feature map of different sizes; and inputting the third feature map of different sizes into the fourth convolutional module of the main network for convolution and batch normalization, and then activating them with the ReLU function to obtain a fourth feature map of different sizes.

[0050] Specifically, the training samples are input into the encoder's main network and undergo four convolutions with a stride of 2. Each convolution is followed by batch normalization and ReLU activation to prevent gradient vanishing during training, resulting in four feature maps of different sizes: X1, X2, X3, and X4. The fourth feature map, X4, is then input into the context module, outputting a multi-scale context information feature map F. p .

[0051] In one embodiment, the structure of the context module is as follows: Figure 4 As shown, the context module includes: four dilated convolutional layers, one global average pooling layer, and one channel attention mechanism. The four dilated convolutional layers and the global average pooling layer are designed as parallel structures using a spatial pyramid structure, followed by the channel attention mechanism. The dilation rates of the four dilated convolutional layers are 1, 6, 12, and 18, respectively. The dilated convolutional layers are responsible for outputting context information at different scales, and the global average pooling layer outputs global context information. Step 302 includes: inputting the feature maps of different sizes output by the fourth convolutional module into the four dilated convolutional layers and the global average pooling layer, respectively, to obtain context information at five different scales; and then concatenating and fusing all context information at different scales on the channel before inputting it into the channel attention mechanism to obtain a multi-scale context information feature map. Figure 4 In this context, mul represents multiplication, and sum represents addition.

[0052] Specifically, such as Figure 2As shown, the encoder consists of the first four layers of the classic VGG16 classification algorithm and a context module. Considering that the model is mainly used for small mobile devices, such as drones, the last downsampling operation and fully connected layers of VGG16 were removed to reduce model parameters, thereby reducing the algorithm's computational requirements and energy consumption. Simultaneously, a context module was added at the lower end of the model to output multi-scale contextual information feature maps, reducing the impact of weather and light intensity changes. The specific operation flow is as follows:

[0053] Step S11: Input the original image (size: 352*480), perform a convolution operation with a stride of 2 and a kernel size of 3*3, and then perform a Batch Norm and ReLU activation operation to prevent gradient vanishing during training. Output the first feature map X1 of different sizes (size: 176*240).

[0054] Step S12: Input the first feature map X1 of different sizes (size: 176*240), perform a convolution operation with a stride of 2 and a kernel size of 3*3, and then perform a Batch Norm and ReLU activation operation to prevent the training gradient from vanishing, and output the second feature map X2 of different sizes (size: 88*120).

[0055] Step S13: Input the second feature map X2 of different size (size: 88*120), perform a convolution operation with a stride of 2 and a kernel size of 3*3, and then perform a Batch Norm and ReLU activation operation to prevent the training gradient from vanishing, and output the third feature map X3 of different size (size: 44*60).

[0056] Step S14: Input the third feature map X3 of different size (size: 44*60), perform a convolution operation with a stride of 2 and a kernel size of 3*3, and then perform a Batch Norm and ReLU activation operation to prevent the training gradient from vanishing, and output the fourth feature map X4 of different size (size: 22*30).

[0057] Step S15: Input the fourth feature map of different sizes X4 (size: 22*30) into the context module, such as... Figure 3 As shown, after five convolutional operations with a stride of 1, a kernel size of 3*3, and dilation rates of 1, 6, 12, and 18 respectively, and a global average pooling operation, five contextual information feature maps of different scales are obtained. These feature maps are then fused using a concat method, and finally processed by a channel attention mechanism to output a multi-scale contextual information feature map F. p (Dimensions: 22*30).

[0058] In one embodiment, the decoder includes three semantic embedding modules and four deconvolution layers; step 306 includes: spatially superimposing the multi-scale context information feature map with a fourth feature map of different sizes and inputting it into the first deconvolution layer to obtain a fourth feature map; inputting the fourth feature map of different sizes and a third feature map of different sizes into the first semantic embedding module, performing a convolution operation on the fourth feature map of different sizes, and performing bilinear interpolation on the obtained convolution features to obtain a restored feature map, and performing element-wise multiplication and fusion of the restored feature map with the third feature map of different sizes to obtain a third semantic feature map; and then... The first feature map is obtained by spatially superimposing the second feature map with the third feature map and then inputting the second feature map into the second deconvolution layer. The third feature map and the second feature map at different scales are then input into the second semantic embedding module to obtain the second semantic feature map. The second semantic feature map and the third feature map are then spatially superimposed with the third feature map and then input into the third deconvolution layer to obtain the second feature map. The second feature map at different scales and the first feature map at different scales are then input into the third semantic embedding module to obtain the first semantic feature map. The first semantic feature map and the second feature map are then spatially superimposed with the second feature map and then input into the fourth deconvolution layer to obtain the sample prediction result.

[0059] Specifically, such as Figure 2 The decoder consists of four deconvolutional layers with a stride of 2 and a kernel size of 4*4, and three semantic embedding modules (SEBs). The deconvolutional layers are responsible for layer-by-layer image reconstruction. The SEBs are responsible for incorporating spatial information into high-level feature maps rich in semantic information but lacking spatial information, thus bridging the gap between semantic and spatial information and achieving good feature map fusion results. The semantic embedding modules use bilinear interpolation to restore the high-level semantically rich feature maps to the same size as the low-level spatially rich feature maps. Then, they perform convolution operations on the two feature maps to incorporate more spatial information into the semantic feature maps, reducing the gap between semantic and spatial information and improving the model's segmentation accuracy. A schematic diagram of the semantic embedding module is shown below. Figure 5 As shown.

[0060] The specific operation steps of the decoder are as follows.

[0061] Step S21: As Figure 2 As shown in the Feature fusion path, the concat operation is used to combine the feature maps F output by the context module. p Feature fusion is performed with the fourth feature map X4 of different sizes, followed by a deconvolution operation with a stride of 2 and a kernel size of 4*4, outputting the fourth feature map Y4. Simultaneously, the fourth feature map X4 of different sizes and the third feature map X3 of different sizes are input into the semantic embedding module (SEB), such as... Figure 5As shown, the fourth feature map X4, which is rich in semantic information but lacks spatial information, undergoes a convolution operation with a stride of 1 and a kernel size of 3*3, followed by a bilinear interpolation operation to restore the feature map size to the same as the third feature map X3. Finally, the feature map output by bilinear interpolation is fused with the third feature map X3 by element-wise multiplication to output the third semantic feature map O3.

[0062] Step S22: Use the concat operation to fuse the third semantic feature map O3 and the fourth feature map Y4. After one deconvolution operation with a stride of 2 and a kernel size of 4*4, output the third feature map Y3. At the same time, input the third feature map X3 and the second feature map X2 into the semantic embedding module (SEB) to execute the function of the semantic embedding module and output the second semantic feature map O2.

[0063] Step S23: Use the "concat" operation to fuse the second semantic feature map O2 and the third feature map Y3. After one deconvolution operation with a stride of 2 and a kernel size of 4*4, output the second feature map Y2. At the same time, input feature maps X2 and X1 into the semantic embedding module (SEB) to execute the function of the semantic embedding module and output the first semantic feature map O1.

[0064] Step S24: Use the "concat" operation to fuse the first semantic feature map O1 and the second feature map Y2. After one deconvolution operation with a stride of 2 and a kernel size of 4*4, output the sample prediction result Y1.

[0065] In one embodiment, the semantic supervision network includes nine correction residual modules; step 304 includes: inputting a multi-scale context information feature map and a fourth different-size feature map into the first correction residual module and the second correction residual module respectively, and then fusion the output features of the two correction residual modules in a spatial channel to obtain a fourth fused feature; inputting the fourth fused feature and the third different-size feature map into the third correction residual module and the fourth correction residual module respectively, and then fusion the output features of the two correction residual modules in a spatial channel to obtain a third fused feature; inputting the third fused feature and the second different-size feature map into the fifth correction residual module and the sixth correction residual module respectively, and then fusion the output features of the two correction residual modules in a spatial channel to obtain a second fused feature; inputting the second fused feature and the first different-size feature map into the seventh correction residual module and the eighth correction residual module respectively, and then fusion the output features of the two correction residual modules in a spatial channel to obtain a first fused feature; and inputting the first fused feature into the ninth correction residual module to obtain a semantic boundary feature map.

[0066] Specifically, the principle of semantic supervision networks is relatively simple, consisting of a multi-scale contextual information feature map F output by the context module. p Initially, the process can be performed layer by layer upwards. Bilinear interpolation is used for upsampling. During training, to prevent gradient vanishing, a Corrected Residual Module (RRB) is added to the semantic supervision network. The structure of the RRB module is as follows: Figure 6 As shown. The specific steps are as follows;

[0067] Step S31: From the multi-scale contextual information feature map F p Initially, after passing through the Residual Correction Module (RRB) once, the fourth feature map X4 of different sizes also passes through the Residual Correction Module (RRB) once. The outputs of the two Residual Correction Modules are fused by concat to obtain the fourth fused feature map S4.

[0068] Step S32: The fourth feature map S4 passes through the correction residual module RRB once, and the third feature map X3 of different sizes also passes through the correction residual module RRB once. The outputs of the two residual modules are fused by concat to obtain the third fused feature map S3.

[0069] Step S33: Feature map S3 passes through a correction residual module RRB once, and feature map X2 also passes through a correction residual module RRB once. The outputs of the two residual modules are fused by concat to obtain the second fused feature map S2.

[0070] Step S34: The second fused feature map S2 passes through the correction residual module RRB once, and the first feature map X1 of different sizes also passes through the correction residual module RRB once. The output results of the two residual modules are fused by concat to obtain the first fused feature map S1.

[0071] Step S35: The first fused feature map S1 outputs the semantic boundary feature map S0 after passing through the corrected residual module RRB once.

[0072] In one embodiment, the residual correction module includes: a point convolutional layer, two 3*3 convolutional layers, and one batch normalization layer; the steps are as follows: inputting the first fused feature into the ninth residual correction module to obtain a semantic boundary feature map, including: inputting the first fused feature into the point convolutional layer of the ninth residual correction module to obtain point convolutional features; inputting the point convolutional features into the first 3*3 convolutional layer, and inputting the obtained convolution result into the batch normalization layer for normalization processing; activating the normalization result with the ReLU function and then inputting it into the second 3*3 convolutional layer to obtain residual branch features; summing the point convolutional features and the residual branch features and then activating them with the ReLU function to obtain a semantic boundary feature map.

[0073] In one embodiment, step 308 includes: constructing a total model loss function; the total model loss function is a weighted sum of the semantic supervision network loss function and the decoder loss function; wherein the semantic supervision network loss function is the focus loss function and the decoder loss function is the cross-entropy loss function; and the sky-ground dividing line detection model is back-trained based on the semantic boundary feature map, sample prediction results, training sample annotations, and the total model loss function to obtain a trained sky-ground dividing line detection model.

[0074] In a validation embodiment, the selected image datasets include: the Camvid dataset, which contains a total of 701 road scene images with a resolution of 360*480, of which the training set has 367 images, the validation set has 101 images, and the test set has 101 images; and the vKITTI dataset, which contains road scenes under different weather conditions such as rain, fog, sunshine, and overcast skies, totaling 2126 images with a size of 1242*375, and also includes scenes with left and right deviations of 15° and 30°.

[0075] Table 1. Comparison of SRDN with other classic semantic segmentation networks (Camvid dataset)

[0076]

[0077] To illustrate the specific performance of the proposed Skyline Detection Network (SRDN), a comparative experiment was conducted with existing classical semantic segmentation networks. The dataset used in the experiment was the Camvid dataset, and the results are shown in Table 1. Comparing the second and third columns of Table 1, it can be seen that the semantic segmentation network using VGG16 as the main network has a smaller model parameter size than the semantic segmentation networks using ResNet and Xception as the main networks. This is because the network depth of VGG16 is smaller than that of ResNet and Xception. Considering that the network proposed in this invention is mainly used for functions such as assisted positioning in drone or car navigation, and that the battery capacity and storage capacity of mobile devices are limited, they cannot support the long-term operation of networks with relatively deep network depth and large model parameter size. Therefore, the shallower VGG16 was chosen as the main network. Compared with existing classical semantic segmentation networks, the model parameter size of SRDN is relatively smaller.

[0078] Analysis of model inference time, as shown in the fourth column of Table 1, reveals that SRDN requires relatively less computation, only exceeding that of the DFN network compared to the classic semantic segmentation networks listed in the table. Analysis of model prediction accuracy, as shown in the training and testing columns of Table 1, demonstrates that SRDN achieves relatively high prediction accuracy in both training and testing for the sky-ground line labeling task, compared to the classic semantic segmentation networks listed in the table.

[0079] While using classic segmentation networks directly for top-to-bottom line detection can achieve high prediction accuracy, accurate labeling of these lines requires relatively good edge segmentation capabilities. Apart from SegNet, the edge segmentation performance of the listed classic semantic segmentation networks was relatively poor. SegNet utilizes pooling indices to retain sufficient edge resolution information in the encoder, and then uses skip connections to fuse this information into the feature map of the decoder, achieving a better edge segmentation effect. However, this series of operations, such as pooling indices and skip connections, increases the model's parameter size and computational cost, resulting in longer inference times. As shown in the fourth column of Table 1, the SegNet model has the longest inference time and also has a larger parameter size. The increased computational cost also increases energy consumption, all of which hinder the model's application on small mobile devices. The SRDN network uses a semantic embedding module (SED) to narrow the gap between semantic and spatial information, enabling a better fusion between low-level feature maps rich in spatial information but lacking semantic information and high-level feature maps rich in semantic information but lacking spatial information. This ultimately improves the model's prediction accuracy and achieves a better edge segmentation effect.

[0080] Table 2 Comparative experiments under different weather conditions (vKITTI dataset)

[0081]

[0082]

[0083] Table 2 lists the experimental results of classic semantic segmentation networks and SRDN under different weather conditions, based on the virtual dataset vKITTI. In Table 2, the first column lists the names of the classic semantic segmentation networks, and the second column lists the average interleaving ratio (mIoU) of each semantic segmentation network under the same scene in five weather conditions: morning, cloudy, sunny, rainy, and foggy. The data in Table 2 shows that SRDN and Deeplabv3+ outperform other classic semantic segmentation networks under these five common weather conditions. This is because SRDN and Deeplabv3+ output contextual information at different scales during the encoding process and integrate this information into the feature maps output by the decoder. Other classic semantic segmentation networks still use point-to-point, pixel-to-pixel recognition methods, ignoring the relationships between pixels when defining pixel semantics. This is detrimental to the model's resistance to weather changes, because raindrops or moisture from clouds and fog cover the entire image, affecting the pixel-by-pixel recognition method that ignores the relationships between pixels. Utilizing contextual information, allowing the network to focus more on the relationships between pixels, is more conducive to the model's resistance to weather changes.

[0084] Most classic semantic segmentation networks classify small, non-sky regions with similar features to the sky as sky regions, such as mirrors reflecting the sky or white walls. However, it's noteworthy that FCNs (Full-Cut Networks) do not classify these small, sky-similar non-sky regions as sky regions. This is because the semantic information contained in high-level feature maps, after being scaled down to 1 / 16, 1 / 32, or even 1 / 64 of the original image, is not sensitive to classifying these small, sky-similar non-sky regions as sky regions, indiscriminately classifying them as having the same semantics as nearby pixels. In contrast, the FCN decoder directly upsamples and restores the high-level feature maps at 1 / 8, 1 / 16, and 1 / 32 of the original image, thus correctly classifying these non-sky regions. Inspired by this, this invention designs a semantic supervision network that uses high-level feature maps to output semantic boundaries, supervising the model's output to enhance its ability to identify small, sky-similar non-sky regions. However, the vanishing gradient problem was discovered during training. Therefore, a residual structure, namely the modified residual module (RRB), was added to the supervised network to prevent the vanishing gradient and enable the supervised network to train and output semantic boundaries normally.

[0085] It should be understood that, although Figure 1 , Figure 3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0086] In one embodiment, such as Figure 7 As shown, a device for detecting top and bottom lines based on image context information is provided, including: a training sample determination module, a top and bottom line detection model construction module, a top and bottom line detection model training module, and a top and bottom line detection and labeling module, wherein:

[0087] The training sample determination module is used to annotate the acquired preprocessed outdoor scene natural image dataset to obtain training samples.

[0088] The top-to-bottom line detection model construction module is used to build a top-to-bottom line detection model, which includes an encoder, a decoder, and a semantic supervision network. The encoder consists of a main network composed of the first four convolutional modules of a VGG16 network, and a context module. The main network is used to downsample the training samples using the four convolutional modules to output four feature maps of different sizes. The context module is used to process the feature maps of different sizes output after the operation of the fourth convolutional module using dilated convolution, global average pooling, and channel attention mechanisms to obtain multi-scale context information feature maps. The semantic supervision network is used to fuse all feature maps of different sizes and multi-scale context information feature maps using bilinear interpolation and a residual correction module to obtain semantic boundary feature maps. The decoder is used to fuse multi-scale context information feature maps and feature maps of different sizes at each level using deconvolution and a semantic embedding module to obtain the sample prediction results.

[0089] The top-to-bottom line detection model training module is used to train the top-to-bottom line detection model based on the semantic boundary feature map, the annotation of training samples, and the sample prediction results obtained by inputting the training set into the top-to-bottom line detection model, so as to obtain the trained top-to-bottom line detection model.

[0090] The top-to-bottom line detection and labeling module is used to input the acquired image to be detected into the trained top-to-bottom line detection model to obtain the prediction result of the image to be detected. If the prediction result of the image to be detected contains a top-to-bottom line, then the top-to-bottom line is labeled.

[0091] In one embodiment, the sky-ground dividing line detection model training module is further used to input training samples into the main network of the encoder to obtain four feature maps of different sizes; input the feature maps of different sizes output by the fourth convolution module into the context module of the encoder to obtain multi-scale context information feature maps; input all feature maps of different sizes and multi-scale context information feature maps into the semantic supervision network to obtain semantic boundary feature maps; input all feature maps of different sizes and multi-scale context information feature maps into the decoder to obtain sample prediction results; and perform reverse training on the sky-ground dividing line detection model based on the semantic boundary feature maps, sample prediction results, and annotations of the training samples to obtain a trained sky-ground dividing line detection model.

[0092] In one embodiment, each of the four convolutional modules of the main network includes a convolutional layer, batch normalization, and a ReLU activation function. The training module for the top-to-bottom line detection model is further configured to input training samples into the first convolutional module of the main network for convolution and batch normalization, and then activate them using the ReLU function to obtain a first feature map of different sizes; input the first feature map of different sizes into the second convolutional module of the main network for convolution and batch normalization, and then activate it using the ReLU function to obtain a second feature map of different sizes; input the second feature map of different sizes into the third convolutional module of the main network for convolution and batch normalization, and then activate it using the ReLU function to obtain a third feature map of different sizes; and input the third feature map of different sizes into the fourth convolutional module of the main network for convolution and batch normalization, and then activate it using the ReLU function to obtain a fourth feature map of different sizes.

[0093] In one embodiment, the context module includes: four dilated convolutional layers, one global average pooling layer, and one channel attention mechanism. The four dilated convolutional layers and the global average pooling layer are designed as parallel structures using a spatial pyramid structure, followed by the channel attention mechanism. The dilation rates of the four dilated convolutional layers are 1, 6, 12, and 18, respectively. The training module for the top-to-bottom line detection model is also used to input the feature maps of different sizes output by the fourth convolutional module into the four dilated convolutional layers and the global average pooling layer, respectively, to obtain context information at five different scales. All context information at different scales is concatenated and fused on the channel before being input into the channel attention mechanism to obtain a multi-scale context information feature map.

[0094] In one embodiment, the decoder includes three semantic embedding modules and four deconvolutional layers; the top-to-bottom line detection model training module is further used to spatially superimpose multi-scale context information feature maps and fourth different-size feature maps into the first deconvolutional layer to obtain a fourth feature map; the fourth different-size feature map and the third different-scale feature map are input into the first semantic embedding module, and a convolution operation is performed on the fourth different-size feature map, and the obtained convolutional features are bilinearly interpolated to obtain a restored feature map, and the restored feature map is fused with the third different-scale feature map element-wise to obtain a third semantic feature map; the fourth different-size feature map is then used to train the top-to-bottom line detection model training module, which is also used to spatially superimpose multiplication of the multi-scale context information feature map and input the fourth different-size feature map into the first deconvolutional layer to obtain a fourth feature map; the fourth different-size feature map and the third different-scale feature map are input into the first semantic embedding module, and a convolution operation is performed on the fourth different-size feature map, and the obtained convolutional features are bilinearly interpolated to obtain a restored feature map, and the restored feature map is fused with the third different-scale feature map element-wise to obtain a third semantic feature map; the fourth different-size feature map is then used to train the top-to-bottom line detection model training module, which is used to train the top-to-bottom line detection model training module, and input the fourth different-size feature map into the first semantic embedding module, which is used to train the fourth different-size feature map into the first semantic embedding ... The third semantic feature map is obtained by spatially stacking the third and fourth feature maps and then inputting them into the second deconvolution layer. The third feature map and the second feature map at different scales are then input into the second semantic embedding module to obtain the second semantic feature map. The second semantic feature map and the third feature map are then spatially stacked and input into the third deconvolution layer to obtain the second feature map. The second feature map at different scales and the first feature map at different scales are then input into the third semantic embedding module to obtain the first semantic feature map. The first semantic feature map and the second feature map are then spatially stacked and input into the fourth deconvolution layer to obtain the sample prediction result.

[0095] In one embodiment, the semantic supervision network includes nine corrected residual modules; the sky-ground dividing line detection model training module is further configured to input multi-scale context information feature maps and fourth different-size feature maps into the first and second corrected residual modules respectively, and then fuse the output features of the two corrected residual modules in a spatial channel to obtain a fourth fused feature; input the fourth fused feature and the third different-size feature map into the third and fourth corrected residual modules respectively, and then fuse the output features of the two corrected residual modules in a spatial channel to obtain a third fused feature; input the third fused feature and the second different-size feature map into the fifth and sixth corrected residual modules respectively, and then fuse the output features of the two corrected residual modules in a spatial channel to obtain a second fused feature; input the second fused feature and the first different-size feature map into the seventh and eighth corrected residual modules respectively, and then fuse the output features of the two corrected residual modules in a spatial channel to obtain a first fused feature; and input the first fused feature into the ninth corrected residual module to obtain a semantic boundary feature map.

[0096] In one embodiment, the residual correction module includes: a point convolutional layer, two 3*3 convolutional layers, and one batch normalization layer; the top and bottom line segmentation detection model training module is further used to input the first fused feature into the point convolutional layer of the ninth residual correction module to obtain point convolutional features; input the point convolutional features into the first 3*3 convolutional layer, and input the obtained convolution result into the batch normalization layer for normalization processing; after activating the normalization result with the ReLU function, input it into the second 3*3 convolutional layer to obtain residual branch features; and after summing the point convolutional features and the residual branch features, activate them with the ReLU function to obtain a semantic boundary feature map.

[0097] In one embodiment, the sky-ground dividing line detection model training module is further used to construct the model's total loss function; the model's total loss function is a weighted sum of the semantic supervision network loss function and the decoder loss function; wherein, the semantic supervision network loss function is the focus loss function, and the decoder loss function is the cross-entropy loss function; based on the semantic boundary feature map, sample prediction results, training sample annotations, and the model's total loss function, the sky-ground dividing line detection model is back-trained to obtain the trained sky-ground dividing line detection model.

[0098] Specific limitations regarding the image context-based top-to-bottom line detection device can be found in the limitations of the image context-based top-to-bottom line detection method described above, and will not be repeated here. Each module in the aforementioned image context-based top-to-bottom line detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0099] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for detecting top and bottom lines based on image context information. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0100] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image context information-based sky-ground segmentation line detection method, characterized in that, The method includes: The preprocessed outdoor scene natural image dataset is labeled to obtain training samples; A model for detecting top-to-bottom lines is constructed, comprising an encoder, a decoder, and a semantic supervision network. The encoder includes a main network consisting of the first four convolutional modules of a VGG16 network, and a context module. The main network operates on the training samples using the four convolutional modules to output four feature maps of different sizes. The context module processes the feature maps of different sizes output after the fourth convolutional module operation using dilated convolution, global average pooling, and channel attention mechanisms to obtain multi-scale context information feature maps. The semantic supervision network fuses all feature maps of different sizes and multi-scale context information feature maps using bilinear interpolation and a residual correction module to obtain semantic boundary feature maps. The decoder fuses the multi-scale context information feature maps and feature maps of different sizes at each level using deconvolution and a semantic embedding module to obtain the sample prediction result. Based on the semantic boundary feature map, the annotation of the training samples, and the sample prediction results obtained by inputting the training samples into the sky-ground dividing line detection model, the sky-ground dividing line detection model is trained to obtain a trained sky-ground dividing line detection model. The acquired image to be detected is input into the trained sky-ground dividing line detection model to obtain the prediction result of the image to be detected. If the prediction result of the image to be detected contains a sky-ground dividing line, the sky-ground dividing line is marked. Specifically, the sky-ground dividing line detection model is trained based on the semantic boundary feature map, the annotation of the training samples, and the sample prediction results obtained by inputting the training samples into the sky-ground dividing line detection model, to obtain a trained sky-ground dividing line detection model, including: The training samples are input into the main network of the encoder to obtain four feature maps of different sizes; The feature maps of different sizes output by the fourth convolutional module are input into the context module of the encoder to obtain multi-scale context information feature maps; All the feature maps of different sizes and the multi-scale context information feature maps are input into the semantic supervision network to obtain the semantic boundary feature map; All the feature maps of different sizes and the multi-scale context information feature maps are input into the decoder to obtain the sample prediction results; The sky-ground dividing line detection model is back-trained based on the semantic boundary feature map, the sample prediction results, and the annotations of the training samples to obtain a trained sky-ground dividing line detection model. The top-to-bottom line detection model is back-trained based on the semantic boundary feature map, sample prediction results, and the annotations of the training samples to obtain a trained top-to-bottom line detection model, including: Construct the total model loss function; the total model loss function is a weighted sum of the semantic supervision network loss function and the decoder loss function; wherein, the semantic supervision network loss function is the focus loss function, and the decoder loss function is the cross-entropy loss function; Based on the semantic boundary feature map, the sample prediction results, the annotation of the training samples, and the model's total loss function, the sky-ground dividing line detection model is back-trained to obtain a trained sky-ground dividing line detection model.

2. The method of claim 1, wherein, The main network's four convolutional modules each include: a convolutional layer, batch normalization, and a ReLU activation function; The training samples are input into the main network of the encoder to obtain four feature maps of different sizes, including: The training samples are input into the first convolutional module of the main network for convolution and batch normalization, and then activated using the ReLU function to obtain the first feature maps of different sizes. The first feature map of different sizes is input into the second convolutional module of the main network for convolution and batch normalization, and then activated by the ReLU function to obtain the second feature map of different sizes. The second feature map of different sizes is input into the third convolutional module of the main network for convolution and batch normalization, and then activated by the ReLU function to obtain the third feature map of different sizes. The third feature map of different sizes is input into the fourth convolutional module of the main network for convolution and batch normalization, and then activated by the ReLU function to obtain the fourth feature map of different sizes.

3. The method according to claim 1, characterized in that, The context module includes: four dilated convolutional layers, one global average pooling layer, and one channel attention mechanism. The four dilated convolutional layers and the global average pooling layer are designed as parallel structures using a spatial pyramid structure, followed by the channel attention mechanism. The dilation rates of the four dilated convolutional layers are 1, 6, 12, and 18, respectively. The feature maps of different sizes output by the fourth convolutional module are input into the context module of the encoder to obtain multi-scale context information feature maps, including: The feature maps of different sizes output by the fourth convolutional module are input into four dilated convolutional layers and a global average pooling layer, respectively, to obtain contextual information at five different scales. All contextual information at different scales is spliced ​​and fused on the channel and then input into the channel attention mechanism to obtain a multi-scale contextual information feature map.

4. The method according to claim 2, characterized in that, The decoder includes three semantic embedding modules and four deconvolutional layers; All the feature maps of different sizes and the multi-scale context information feature maps are input into the decoder to obtain sample prediction results, including: The multi-scale context information feature map and the fourth feature map of different sizes are spatially superimposed and then input into the first deconvolution layer to obtain the fourth feature map; The fourth feature map of different sizes and the third feature map of different scales are input into the first semantic embedding module. The fourth feature map of different sizes is convolved and bilinearly interpolated to obtain the restored feature map. The restored feature map is then multiplied and fused with the third feature map of different scales element by element to obtain the third semantic feature map. The third semantic feature map and the fourth feature map are spatially superimposed and then input into the second deconvolution layer to obtain the third feature map; The third feature map at different scales and the second feature map at different scales are input into the second semantic embedding module to obtain the second semantic feature map; The second semantic feature map and the third feature map are spatially superimposed and then input into the third deconvolution layer to obtain the second feature map; The second feature map at different scales and the first feature map at different scales are input into the third semantic embedding module to obtain the first semantic feature map; The first semantic feature map and the second feature map are spatially superimposed and then input into the fourth deconvolution layer to obtain the sample prediction result.

5. The method according to claim 2, characterized in that, The semantic supervision network includes nine residual correction modules; All the feature maps of different sizes and the multi-scale context information feature maps are input into the semantic supervision network to obtain semantic boundary feature maps, including: The multi-scale context information feature map and the fourth feature map of different sizes are respectively input into the first correction residual module and the second correction residual module, and the output features of the two correction residual modules are superimposed and fused in the spatial channel to obtain the fourth fused feature; The fourth fusion feature and the third feature map of different sizes are respectively input into the third correction residual module and the fourth correction residual module, and the output features of the two correction residual modules are superimposed and fused in the spatial channel to obtain the third fusion feature; The third fusion feature and the second feature map of different sizes are respectively input into the fifth correction residual module and the sixth correction residual module, and the output features of the two correction residual modules are superimposed and fused in the spatial channel to obtain the second fusion feature; The second fusion feature and the first feature map of different sizes are respectively input into the seventh correction residual module and the eighth correction residual module, and the output features of the two correction residual modules are superimposed and fused in the spatial channel to obtain the first fusion feature; The first fused feature is input into the ninth corrected residual module to obtain the semantic boundary feature map.

6. The method according to claim 5, characterized in that, The residual correction module includes: a point convolutional layer, two 3*3 convolutional layers, and one batch normalization layer; The first fused feature is input into the ninth corrected residual module to obtain a semantic boundary feature map, including: The first fused feature is input into the point convolutional layer of the ninth corrected residual module to obtain the point convolutional feature; The point convolution features are input into the first 3*3 convolutional layer, and the resulting convolution is input into the batch normalization layer for normalization. The normalized result is activated by the ReLU function and then input into the second 3*3 convolutional layer to obtain the residual branch features. The semantic boundary feature map is obtained by summing the point convolution features and the residual branch features and then activating them with the ReLU function.

7. A device for detecting top and bottom lines based on image context information, the device being used to detect top and bottom lines using the method for detecting top and bottom lines based on image context information as described in claim 1; characterized in that, The device includes: The training sample determination module is used to label the acquired preprocessed outdoor scene natural image dataset to obtain training samples; A module for constructing a top-to-bottom line detection model is included. This model comprises an encoder, a decoder, and a semantic supervision network. The encoder consists of a main network composed of the first four convolutional modules of a VGG16 network, and a context module. The main network downsamples the training samples using the four convolutional modules, outputting four feature maps of different sizes. The context module processes the feature maps of different sizes output after the fourth convolutional module operation using dilated convolution, global average pooling, and channel attention mechanisms to obtain multi-scale context information feature maps. The semantic supervision network fuses all feature maps of different sizes and multi-scale context information feature maps using bilinear interpolation and a residual correction module to obtain semantic boundary feature maps. The decoder fuses the multi-scale context information feature maps and feature maps of different sizes at various levels using deconvolution and a semantic embedding module to obtain sample prediction results. The sky-ground dividing line detection model training module is used to train the sky-ground dividing line detection model based on the semantic boundary feature map, the annotation of the training samples, and the sample prediction results obtained by inputting the training samples into the sky-ground dividing line detection model, so as to obtain a trained sky-ground dividing line detection model. The top-to-bottom line detection and labeling module is used to input the acquired image to be detected into the trained top-to-bottom line detection model to obtain the prediction result of the image to be detected. If the prediction result of the image to be detected contains a top-to-bottom line, the top-to-bottom line is labeled.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

Images