Road recognition extraction method, device, equipment and medium

Abstract

The application relates to the technical field of image processing, and discloses a road recognition extraction method, device, equipment and medium; the method comprises the following steps: acquiring an inspection image; inputting the inspection image into a road recognition extraction model to obtain a road recognition extraction result; wherein the road recognition extraction model comprises an encoder, a decoder and an output layer; the decoder comprises a plurality of up-sampling modules; the up-sampling module comprises an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used for extracting low-layer features; the multi-scale feature extraction layer comprises a plurality of convolution layers; the plurality of convolution layers are used for performing at least one convolution processing on the low-layer features at different scales respectively; and the up-sampling module is used for fusing the outputs of the original feature extraction layer and the plurality of convolution layers, and then fusing the high-layer features. The above scheme can better integrate feature information at different levels, and improves the accuracy of road recognition extraction.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a road recognition and extraction method, apparatus, device, and medium. Background Technology

[0002] With the continuous expansion of the highway network and the sustained increase in traffic flow, highway maintenance and management face unprecedented challenges. Traditional manual inspection methods are not only time-consuming and labor-intensive, but also inefficient.

[0003] In recent years, drone technology has made continuous progress and its application in various industries has become increasingly widespread, making numerous applications in the low-altitude economy possible. Drone inspection solutions based on deep learning intelligent recognition provide a safe and efficient solution for highway inspection scenarios. Drones can quickly cover large areas of highways and take detailed pictures of road surface conditions, greatly shortening the inspection cycle. Deep learning semantic segmentation technology can perform pixel-level classification of drone-captured images, effectively improving the accuracy and reliability of highway inspections. However, the accuracy of existing semantic segmentation models for road recognition remains relatively low. Summary of the Invention

[0004] In view of this, this application provides a road identification and extraction method, apparatus, equipment and medium to improve the accuracy of road segmentation. The technical solution is as follows.

[0005] Firstly, a road identification and extraction method is provided, the method comprising:

[0006] Acquire inspection images; the inspection images are obtained by the UAV capturing images of the road.

[0007] The inspection image is input into the road recognition and extraction model to obtain the road recognition and extraction result;

[0008] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. These convolutional layers are used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features.

[0009] In one alternative implementation, each of the convolutional layers includes a first convolutional kernel and a second convolutional kernel of the same size;

[0010] The first convolutional kernel is used to perform the first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map;

[0011] The second convolutional kernel is used to perform a second convolution on the intermediate features and restore the number of feature channels to the number of input channels corresponding to the low-level features, so as to obtain the output feature map of the convolutional layer.

[0012] In one optional implementation, the plurality of convolutional layers include at least two convolutional kernels of different sizes; and at least two of the plurality of convolutional layers have convolutional kernels of the same size.

[0013] In one optional implementation, the upsampling module is used to:

[0014] The output features of the aforementioned convolutional layers are summed to obtain the first feature map;

[0015] The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer.

[0016] The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature;

[0017] The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

[0018] In one optional implementation, the topology of the road identification and extraction model is a D-LinkNet network structure.

[0019] Secondly, a road identification and extraction method is provided, the method comprising:

[0020] Acquire sample road images; the sample road images are obtained by drones capturing images of roads; the sample road images include road markings and background markings;

[0021] Based on the sample road images, the road recognition and extraction model is iteratively trained to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection images.

[0022] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. Each convolutional layer is used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features.

[0023] In one alternative implementation, each of the convolutional layers includes a first convolutional kernel and a second convolutional kernel of the same size;

[0024] The first convolutional kernel is used to perform the first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map;

[0025] The second convolutional kernel is used to perform a second convolution on the intermediate features and restore the number of feature channels to the number of input channels corresponding to the low-level features, so as to obtain the output feature map of the convolutional layer.

[0026] In one optional implementation, the plurality of convolutional layers include at least two convolutional kernels of different sizes; and at least two of the plurality of convolutional layers have convolutional kernels of the same size.

[0027] In one optional implementation, the upsampling module is used to:

[0028] The output features of the aforementioned convolutional layers are summed to obtain the first feature map;

[0029] The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer.

[0030] The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature;

[0031] The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

[0032] In one optional implementation, the topology of the road identification and extraction model is a D-LinkNet network structure.

[0033] Thirdly, a road recognition and extraction device is provided, the device comprising:

[0034] The inspection image acquisition module is used to acquire inspection images; the inspection images are obtained by the UAV from the road.

[0035] The road recognition and extraction module is used to input the inspection image into the road recognition and extraction model to obtain the road recognition and extraction result;

[0036] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. Each convolutional layer is used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features.

[0037] Fourthly, a road recognition and extraction device is provided, the device comprising:

[0038] The sample acquisition module is used to acquire sample road images; the sample road images are obtained by a drone capturing images of the road; the sample road images include road markings and background markings;

[0039] The model training module is used to iteratively train the road recognition and extraction model based on the sample road images to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection images.

[0040] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. Each convolutional layer is used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features.

[0041] Fifthly, a computer device is provided, the computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the above-described road identification and extraction method by executing the computer instructions.

[0042] In a sixth aspect, a computer-readable storage medium is provided, on which computer instructions are stored, the computer instructions being used to cause a computer to perform the road identification and extraction method described above.

[0043] In a seventh aspect, a computer program product or computer program is provided, including computer instructions for causing a computer to perform the road identification and extraction method described above.

[0044] The technical solution provided in this application may include the following beneficial effects:

[0045] After acquiring inspection images of roads collected by a drone, these images can be input into a road recognition and extraction model for processing. This model consists of an encoder, a decoder, and an output layer. The decoder includes several upsampling modules, each containing an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, which perform at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers, and then fuses them with the high-level features. The model then processes the features output from each upsampling module, and finally outputs the road recognition and extraction result through the output layer. This approach, through multi-scale convolution operations, allows the model to extract features at different scales, capturing more detailed information and aiding in the recovery of spatial location information in the image. Furthermore, fusing high-level features with low-level features extracted through a multi-scale attention mechanism helps the model better integrate feature information from different levels, improving network recognition errors and road connectivity issues, and enhancing the accuracy of road recognition and extraction. Attached Figure Description

[0046] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram illustrating the structure of a road recognition and extraction system according to an exemplary embodiment.

[0048] Figure 2 This is a flowchart illustrating a road identification and extraction method according to an exemplary embodiment.

[0049] Figure 3 This is a flowchart illustrating a road identification and extraction method according to an exemplary embodiment.

[0050] Figure 4 This is a flowchart illustrating a road identification and extraction method according to an exemplary embodiment.

[0051] Figure 5A schematic diagram of a sample road image according to an embodiment of this application is shown.

[0052] Figure 6 A schematic diagram of an upsampling module according to an embodiment of this application is shown.

[0053] Figure 7 This is a schematic diagram of the structure of a road recognition and extraction device provided in an embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the structure of a road recognition and extraction device provided in an embodiment of this application.

[0055] Figure 9 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0058] Figure 1 This is a schematic diagram illustrating the structure of a road recognition and extraction system according to an exemplary embodiment. For example... Figure 1 As shown, the road recognition and extraction system includes a terminal device 110 and a server 120.

[0059] Optionally, the server 120 can be loaded with a road recognition and extraction model. In this case, the user can transmit the inspection image to be processed to the server 120 via a wired or wireless network through the terminal device 110. The server 120 will process the inspection image through the road recognition and extraction model.

[0060] Optionally, the terminal device 110 may be loaded with a road recognition and extraction model. In this case, after receiving the inspection image, the terminal device 110 can directly process the inspection image through its own deployed road recognition and extraction model to obtain the road recognition and extraction result.

[0061] Optionally, the road recognition and extraction model can be trained in the server 120. In this case, the server 120 can deploy the road recognition and extraction model on itself, or it can send the parameters of the road recognition and extraction model to the terminal device 110, so that the terminal device 110 has the road recognition and extraction model deployed, and thus can process the inspection image.

[0062] Optionally, the aforementioned server can be a server cluster or a distributed system consisting of multiple physical servers, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms and other technology computing services.

[0063] Optionally, the system may also include a management device for managing the system (such as managing the connection status between each module and the server), and the management device is connected to the server via a communication network. Optionally, the communication network may be a wired network or a wireless network.

[0064] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any other network, including but not limited to any combination of local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or virtual private networks (VPNs). In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), VPNs, and Internet Protocol (IP) security can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0065] Figure 2 This is a flowchart illustrating a road recognition and extraction method according to an exemplary embodiment. The method is applied to a computer device, which may be as follows: Figure 1 The terminal device or server shown. For example... Figure 2 As shown, the road identification and extraction method may include the following steps:

[0066] Step 201: Obtain inspection images; these inspection images are collected by the UAV from the road.

[0067] Specifically, the inspection images are taken by aerial photography of the road using a high-resolution camera mounted on a drone, capturing images that include complex backgrounds (such as vegetation, buildings, and shadows) and road targets (such as asphalt pavement, cracks, and road markings).

[0068] Step 202: Input the inspection image into the road recognition and extraction model to obtain the road recognition and extraction results.

[0069] Since road targets are affected by factors such as changes in lighting, occlusion, and tilted viewing angle, the road recognition and extraction model needs to be highly robust.

[0070] Specifically, the road recognition and extraction model includes an encoder, a decoder, and an output layer;

[0071] In this encoder, a convolutional neural network can be used to downsample features and extract high-level semantic features. During the downsampling process, the spatial resolution can be gradually reduced by pooling or stride convolution.

[0072] The decoder includes several upsampling modules; each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer; the raw feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolutional layers; the convolutional layers are used to perform at least one convolutional process on the low-level features at different scales; the upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with the high-level features.

[0073] The output layer can then (e.g., using 1×1 convolution) map the fused features to road segmentation results.

[0074] In summary, after acquiring inspection images of roads collected by a drone, these images can be input into a road recognition and extraction model for processing. This model consists of an encoder, a decoder, and an output layer. The decoder includes several upsampling modules, each containing an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, which perform at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers with the high-level features. The model then processes the features output from each upsampling module, finally outputting the road recognition and extraction result through the output layer. This approach, through multi-scale convolution operations, enables the model to extract features at different scales, capturing more detailed information and aiding in the recovery of spatial location information in the image. Furthermore, fusing high-level features with low-level features extracted through a multi-scale attention mechanism helps the model better integrate feature information from different levels, improving network recognition errors and road connectivity issues, and enhancing the accuracy of road recognition and extraction.

[0075] Figure 3This is a flowchart illustrating a road recognition and extraction method according to an exemplary embodiment. The method is applied to a computer device, which may be as follows: Figure 1 The terminal device or server shown. For example... Figure 3 As shown, the road identification and extraction method may include the following steps:

[0076] Step 301: Obtain a sample road image; the sample road image is obtained by a drone collecting data on the road; the sample road image includes road markings and background markings.

[0077] Step 302: Based on the sample road image, the road recognition and extraction model is iteratively trained to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image.

[0078] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. Each convolutional layer is used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with the high-level features.

[0079] In summary, after acquiring inspection images of roads collected by a drone, these images can be input into a road recognition and extraction model for processing. This model consists of an encoder, a decoder, and an output layer. The decoder includes several upsampling modules, each containing an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, which perform at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers with the high-level features. The model then processes the features output from each upsampling module, finally outputting the road recognition and extraction result through the output layer. This approach, through multi-scale convolution operations, enables the model to extract features at different scales, capturing more detailed information and aiding in the recovery of spatial location information in the image. Furthermore, fusing high-level features with low-level features extracted through a multi-scale attention mechanism helps the model better integrate feature information from different levels, improving network recognition errors and road connectivity issues, and enhancing the accuracy of road recognition and extraction.

[0080] Figure 4This is a flowchart illustrating a road recognition and extraction method according to an exemplary embodiment. The method is applied to a computer device, which may be as follows: Figure 1 The terminal device or server shown. For example... Figure 4 As shown, the road identification and extraction method may include the following steps:

[0081] Step 401: Obtain a sample road image; the sample road image is obtained by a drone collecting data on the road; the sample road image includes road markings and background markings.

[0082] In a real-world scenario, 5,000 images of roads inspected by drones can be collected and calibrated. For example, roads can be categorized as one type, while buildings, vegetation, vehicles, and other objects can be grouped into the background category, thus obtaining road and background annotations for the sample road images.

[0083] To further increase the sample data volume, data augmentation processing can be performed on the aforementioned sample road images, expanding the dataset from 5000 drone images to 30000 images, thus completing the dataset creation. In deep neural networks, a dataset with too small a sample size can lead to overfitting during training, meaning the network fits well on the current training dataset but performs poorly on the validation and test sets. To avoid overfitting, data augmentation methods such as color dithering, image shifting, scaling, rotation, and flipping (including horizontal, vertical, and diagonal flipping), as well as combinations of these five methods, are used to expand the dataset sample size. Specifically, color dithering randomly changes the colors of the image; image shifting translates the image and label up, down, left, and right by a certain number of pixels; scaling randomly enlarges or reduces the image by a factor of 1 to 5; rotation randomly rotates the image and label clockwise by a certain angle (0° to 360°); and flipping flips the image horizontally, vertically, or diagonally. Please refer to [reference needed]. Figure 5 This illustrates a schematic diagram of a sample road image related to an embodiment of this application. Figure 5 As shown, after color dithering, image shifting, and flipping the original image, more sample road images can be generated.

[0084] Step 402: Based on the sample road image, the road recognition and extraction model is iteratively trained to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image.

[0085] In this embodiment, the road recognition and extraction model includes an encoder, a decoder, and an output layer. In one possible implementation, the road recognition and extraction model has a D-LinkNet network topology.

[0086] Furthermore, the encoder in the original D-LinkNet algorithm is ResNet, which has a large receptive field. However, because the input and output features are directly added in the residual module, feature distortion is easily caused when the input and output sizes are mismatched, and it performs poorly on small datasets. In this embodiment, to improve the feature distortion problem of ResNet and enhance the network's segmentation accuracy on small datasets, the encoder is replaced with EfficientNet. EfficientNet can simultaneously optimize the network's depth, width, and resolution through a composite scaling strategy, exhibiting strong generalization ability. Therefore, it can effectively improve the network's segmentation accuracy under small dataset conditions in highway inspection scenarios.

[0087] In this embodiment, the original D-LinkNet algorithm extracts features at a single scale during the feature extraction stage, failing to give sufficient attention and weight to important detailed features, resulting in the loss of a large amount of detailed information and causing network recognition errors and road connectivity problems. To improve the network's ability to perceive detailed information, a multi-scale attention mechanism is introduced. An upsampling module based on the multi-scale attention mechanism is proposed, which selects low-level features and fuses them with high-level features to improve the spatial location information recovery capability of UAV road images.

[0088] Specifically, the decoder of the road recognition extraction model in this application embodiment includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the multi-scale feature extraction layer includes several convolutional layers; the several convolutional layers are used to perform at least one convolutional process on the low-level features at different scales; the original feature extraction layer is used to extract low-level features.

[0089] In this embodiment, the original feature extraction layer can be a 1x1 convolutional layer. While the low-level features output from the shallow layer of the encoder contain rich details, they suffer from channel redundancy and noise interference. By adjusting the number of channels using 1x1 convolutions (e.g., aligning high-level feature channels), and combining this with batch normalization and ReLU activation functions, the channel dimensions of features at different levels can be unified, avoiding dimensionality conflicts during fusion. Furthermore, the activation function can improve the expressive power of the features, thereby making the low-level data input for subsequent multi-scale processing more accurate.

[0090] Multi-scale feature extraction layers can extract features from images at different scales using convolutional kernels of multiple scales, making the model more adaptable to complex scenes such as scale changes (e.g., foreground and background roads) and occlusion (e.g., tree shadows).

[0091] Specifically, each of these convolutional layers includes a first convolutional kernel and a second convolutional kernel of the same size;

[0092] The first convolutional kernel is used to perform the first convolution on the low-level feature and compress the number of feature channels to obtain an intermediate feature map.

[0093] The second convolutional kernel is used to perform a second convolution on the intermediate feature and restore the number of feature channels to the number of input channels corresponding to the lower-level feature, so as to obtain the output feature map of the convolutional layer.

[0094] In this embodiment, each convolutional layer can perform two convolution operations on the input low-level features. For example, in a certain convolutional layer, two consecutive 3*3 convolution operations can be used to perform convolution operations on the input low-level features. Furthermore, during the first convolution, the number of channels can be compressed, allowing the network to filter out important information in a low-dimensional space and reduce interference caused by redundant features. Then, in order to fuse features of multiple scales, the number of feature channels needs to be restored during the second convolution to facilitate subsequent element-wise addition or concatenation.

[0095] Furthermore, the convolutional layers include at least two types of convolutional kernels; and at least two of the convolutional layers have kernels of the same size.

[0096] In this embodiment of the application, taking a 3*3 convolution operation as an example, the upsampling module can be set with two convolutional layers. In either of the two convolutional layers, the first convolutional kernel and the second convolutional kernel are used to perform a 3*3 convolution operation. However, the parameters of the convolutional kernels in the two convolutional layers can be different, so that the upsampling module can notice more diverse features at the 3*3 feature size, thereby improving the feature extraction capability of the upsampling module.

[0097] At this point, the upsampling module is specifically used for:

[0098] The output features of these convolutional layers are summed to obtain the first feature map;

[0099] The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer.

[0100] The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature;

[0101] The attention feature is fused with the high-level feature to obtain the upsampling result of the upsampling module.

[0102] Please refer to Figure 6 The diagram illustrates a structural schematic of an upsampling module according to an embodiment of this application. Figure 6As shown, the upsampling module based on the multi-scale attention mechanism consists of three parts. First, the original low-level features are processed by 1×1 convolution, batch normalization, and ReLU activation. Next, a multi-scale feature extraction layer is implemented, consisting of two 3×3 convolutions, two 5×5 convolutions, and two 7×7 convolutions, with ReLU activation. To accelerate computation, the feature channels are compressed to 1 / 4 of the input channels during the first convolution, and then restored to the input channel count during the second convolution. Third, the results of each convolution are summed, deconvolved to the image size of the intermediate branch, and multiplied with the result of the intermediate branch to generate pixel-level attention. Finally, the high-level features are fused with the low-level features extracted by the multi-scale attention mechanism. The main principle of the upsampling module based on the multi-scale attention mechanism is to use high-level features to select low-level features, which can effectively improve the network's ability to extract details of image spatial information, thereby improving network recognition errors and road connectivity problems.

[0103] The UAV image dataset was divided into a training set (80%) and a validation set (20%) for testing. Experimental results show that, under the same number of iterations, compared to the original D-LinkNet network, the improved D-LinkNet mIOU (mean intersection-over-union ratio) increased from 80.5% to 82.6%, and the F1 score increased from 78.1% to 80.4%.

[0104] Step 403: Obtain the inspection image; the inspection image is obtained by the drone collecting data on the road.

[0105] Step 404: Input the inspection image into the road recognition and extraction model to obtain the road recognition and extraction results.

[0106] The specific implementation methods and execution principles of steps 403 and 404 are similar to those of step 402, and will not be repeated here.

[0107] In summary, after acquiring inspection images of roads collected by a drone, these images can be input into a road recognition and extraction model for processing. This model consists of an encoder, a decoder, and an output layer. The decoder includes several upsampling modules, each containing an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, which perform at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers with the high-level features. The model then processes the features output from each upsampling module, finally outputting the road recognition and extraction result through the output layer. This approach, through multi-scale convolution operations, enables the model to extract features at different scales, capturing more detailed information and aiding in the recovery of spatial location information in the image. Furthermore, fusing high-level features with low-level features extracted through a multi-scale attention mechanism helps the model better integrate feature information from different levels, improving network recognition errors and road connectivity issues, and enhancing the accuracy of road recognition and extraction.

[0108] This application also provides a road recognition and extraction device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0109] This application provides a road identification and extraction device. Figure 7 This is a schematic diagram of a road recognition and extraction device provided in an embodiment of this application. The device includes:

[0110] The inspection image acquisition module 701 is used to acquire inspection images; the inspection images are obtained by the UAV from the road.

[0111] The road recognition and extraction module 702 is used to input the inspection image into the road recognition and extraction model to obtain the road recognition and extraction result;

[0112] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer extracts low-level features. The multi-scale feature extraction layer includes several convolutional layers. These convolutional layers perform at least one convolutional process on the low-level features at different scales. The upsampling module fuses the outputs of the raw feature extraction layer and the convolutional layers, and then fuses them with higher-level features.

[0113] In one alternative implementation, each of the convolutional layers includes a first convolutional kernel and a second convolutional kernel of the same size;

[0114] The first convolutional kernel is used to perform the first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map;

[0115] The second convolutional kernel is used to perform a second convolution on the intermediate features and restore the number of feature channels to the number of input channels corresponding to the low-level features, so as to obtain the output feature map of the convolutional layer.

[0116] In one optional implementation, the plurality of convolutional layers include at least two convolutional kernels of different sizes; and at least two of the plurality of convolutional layers have convolutional kernels of the same size.

[0117] In one optional implementation, the upsampling module is used to:

[0118] The output features of the aforementioned convolutional layers are summed to obtain the first feature map;

[0119] The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer.

[0120] The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature;

[0121] The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

[0122] In one optional implementation, the topology of the road identification and extraction model is a D-LinkNet network structure.

[0123] This application also provides a road identification and extraction device. Figure 7 This is a schematic diagram of a road recognition and extraction device provided in an embodiment of this application. The device includes:

[0124] The device includes:

[0125] The sample acquisition module 801 is used to acquire sample road images; the sample road images are obtained by a drone capturing images of the road; the sample road images include road markings and background markings;

[0126] The model training module 802 is used to iteratively train the road recognition and extraction model based on the sample road images to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection images.

[0127] The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. Each convolutional layer is used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features.

[0128] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0129] The device in this embodiment is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0130] This invention also provides a computer device having the above-described features. Figure 7 or Figure 8 The apparatus shown.

[0131] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 9 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information in a graphical user interface on an external input / output device (such as a display device coupled to the interface). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.

[0132] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0133] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0134] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0136] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0137] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0138] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0139] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A road identification and extraction method, characterized in that, The method includes: Acquire inspection images; the inspection images are obtained by the UAV capturing images of the road. The inspection image is input into the road recognition and extraction model to obtain the road recognition and extraction result; The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. These convolutional layers are used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features. The upsampling module is used for: The output features of the aforementioned convolutional layers are summed to obtain the first feature map; The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer. The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature; The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

2. The method according to claim 1, characterized in that, Each of the convolutional layers includes a first convolutional kernel and a second convolutional kernel of the same size; The first convolutional kernel is used to perform the first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map; The second convolutional kernel is used to perform a second convolution on the intermediate features and restore the number of feature channels to the number of input channels corresponding to the low-level features, so as to obtain the output feature map of the convolutional layer.

3. The method according to claim 2, characterized in that, The plurality of convolutional layers include at least two types of convolutional kernels; and at least two of the plurality of convolutional layers have convolutional kernels of the same size.

4. The method according to claim 3, characterized in that, The road identification and extraction model uses a D-LinkNet network structure.

5. A road recognition and extraction method, characterized in that, The method includes: Acquire sample road images; the sample road images are obtained by drones capturing images of roads; the sample road images include road markings and background markings; Based on the sample road images, the road recognition and extraction model is iteratively trained to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection images. The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. These convolutional layers are used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features. The upsampling module is used for: The output features of the aforementioned convolutional layers are summed to obtain the first feature map; The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer. The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature; The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

6. A road recognition and extraction device, characterized in that, The device includes: The inspection image acquisition module is used to acquire inspection images; the inspection images are obtained by the UAV from the road. The road recognition and extraction module is used to input the inspection image into the road recognition and extraction model to obtain the road recognition and extraction result; The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. These convolutional layers are used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features. The upsampling module is used for: The output features of the aforementioned convolutional layers are summed to obtain the first feature map; The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer. The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature; The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

7. A road identification and extraction device, characterized in that, The device includes: The sample acquisition module is used to acquire sample road images; the sample road images are obtained by a drone capturing images of the road; the sample road images include road markings and background markings; The model training module is used to iteratively train the road recognition and extraction model based on the sample road images to obtain the trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection images. The road recognition and extraction model includes an encoder, a decoder, and an output layer. The decoder includes several upsampling modules. Each upsampling module includes a raw feature extraction layer and a multi-scale feature extraction layer. The raw feature extraction layer is used to extract low-level features. The multi-scale feature extraction layer includes several convolutional layers. These convolutional layers are used to perform at least one convolutional process on the low-level features at different scales. The upsampling module is used to fuse the outputs of the raw feature extraction layer and the convolutional layers, and then fuse them with high-level features. The upsampling module is used for: The output features of the aforementioned convolutional layers are summed to obtain the first feature map; The first feature is deconvolved to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer. The second feature map is multiplied by the image output from the original feature extraction layer to obtain the attention feature; The attention features are fused with the high-level features to obtain the upsampling result of the upsampling module.

8. A computer device, characterized in that, The computer device includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the road identification and extraction method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the road identification and extraction method as described in any one of claims 1 to 5.

Images