Road recognition and extraction method, device, equipment and medium

Abstract

The invention relates to the technical field of image processing, and discloses a road recognition and extraction method, device and equipment and a medium. The method comprises the following steps: acquiring an inspection image; inputting the inspection image into a road identification and extraction model to obtain a road identification and extraction result; wherein the road identification extraction model comprises an encoder, a decoder and an output layer; the decoder comprises a plurality of up-sampling modules; the up-sampling module comprises an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used for extracting low-layer features; the multi-scale feature extraction layer comprises a plurality of convolutional layers; the plurality of convolution layers are used for carrying out at least one time of convolution processing on the low-layer features at different scales; and the up-sampling module is used for fusing the output of the original feature extraction layer and the output of the plurality of convolutional layers and then fusing with the high-level features. According to the scheme, feature information of different levels can be better integrated, and the accuracy of road recognition and extraction is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a road recognition and extraction method, device, equipment and medium. Background Art

[0002] With the continuous expansion of the highway network and the continuous increase in traffic volume, highway maintenance and management are facing unprecedented challenges. Traditional manual inspection methods are not only time-consuming and labor-intensive, but also inefficient.

[0003] In recent years, drone technology has continuously advanced, with its application becoming increasingly widespread across multiple industries, making numerous applications within the low-altitude economy possible. Drone inspection solutions based on deep learning intelligent recognition offer a safe and efficient solution for highway inspections. Drones can quickly cover large sections of highways and capture detailed images of road conditions, significantly shortening inspection cycles. Deep learning semantic segmentation technology enables pixel-level classification of drone-captured images, effectively improving the accuracy and reliability of highway inspections. However, existing semantic segmentation models still have low accuracy for road recognition. Summary of the Invention

[0004] In view of this, the present application provides a road recognition and extraction method, device, equipment and medium to improve the accuracy of road segmentation. The technical solution is as follows.

[0005] In a first aspect, a road identification and extraction method is provided, the method comprising:

[0006] Obtaining inspection images; the inspection images are obtained by collecting images of the road by a drone;

[0007] Inputting the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result;

[0008] The road recognition and extraction model includes an encoder, a decoder, and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolutional layers; the several convolutional layers are used to perform at least one convolution process on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolutional layers, and then fuse them with the high-level features;

[0009] In an optional embodiment, each convolution layer includes a first convolution kernel and a second convolution kernel of the same scale;

[0010] The first convolution kernel is used to perform a first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map;

[0011] The second convolution kernel is used to perform a second convolution on the intermediate feature and restore the number of feature channels to the number of input channels corresponding to the low-level feature to obtain the output feature map of the convolution layer.

[0012] In an optional embodiment, the plurality of convolutional layers include convolution kernels of at least two scales; and the convolution kernels of at least two convolutional layers among the plurality of convolutional layers have the same scale.

[0013] In an optional implementation, the upsampling module is configured to:

[0014] Adding the output features of the plurality of convolutional layers to obtain a first feature map;

[0015] Performing deconvolution on the first feature to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer;

[0016] Multiplying the second feature map by the image output by the original feature extraction layer to obtain an attention feature;

[0017] The attention feature is fused with the high-level feature to obtain the upsampling result of the upsampling module.

[0018] In an optional implementation, the topological structure of the road recognition and extraction model is a D-LinkNet network structure.

[0019] In a second aspect, a road identification and extraction method is provided, the method comprising:

[0020] Acquire a sample road image; the sample road image is obtained by capturing the road using a drone; the sample road image includes road annotations and background annotations;

[0021] Iteratively training a road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image;

[0022] Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

[0023] In an optional embodiment, each convolution layer includes a first convolution kernel and a second convolution kernel of the same scale;

[0024] The first convolution kernel is used to perform a first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map;

[0025] The second convolution kernel is used to perform a second convolution on the intermediate feature and restore the number of feature channels to the number of input channels corresponding to the low-level feature to obtain the output feature map of the convolution layer.

[0026] In an optional embodiment, the plurality of convolutional layers include convolution kernels of at least two scales; and the convolution kernels of at least two convolutional layers among the plurality of convolutional layers have the same scale.

[0027] In an optional implementation, the upsampling module is configured to:

[0028] Adding the output features of the plurality of convolutional layers to obtain a first feature map;

[0029] Performing deconvolution on the first feature to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer;

[0030] Multiplying the second feature map by the image output by the original feature extraction layer to obtain an attention feature;

[0031] The attention feature is fused with the high-level feature to obtain the upsampling result of the upsampling module.

[0032] In an optional implementation, the topological structure of the road recognition and extraction model is a D-LinkNet network structure.

[0033] In a third aspect, a road identification and extraction device is provided, the device comprising:

[0034] An inspection image acquisition module is used to acquire inspection images; the inspection images are acquired by a drone on the road;

[0035] A road recognition and extraction module is used to input the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result;

[0036] Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

[0037] In a fourth aspect, a road identification and extraction device is provided, the device comprising:

[0038] A sample acquisition module is used to acquire a sample road image; the sample road image is acquired by using a drone to capture the road; the sample road image includes road annotations and background annotations;

[0039] A model training module is used to iteratively train the road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image;

[0040] Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

[0041] In a fifth aspect, a computer device is provided, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned road recognition and extraction method by executing the computer instructions.

[0042] In a sixth aspect, a computer-readable storage medium is provided, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the above-mentioned road recognition and extraction method.

[0043] In a seventh aspect, a computer program product or a computer program is provided, comprising computer instructions for enabling a computer to execute the above-mentioned road identification and extraction method.

[0044] The technical solution provided by this application may have the following beneficial effects:

[0045] After obtaining inspection images captured by drones, they can be fed into a road recognition and extraction model for processing. This model consists of an encoder-decoder and an output layer. The decoder includes several upsampling modules, each of which includes an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, each of which performs at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers, and then fuses them with high-level features. The model then processes the features output by each upsampling module and finally outputs the road recognition and extraction results through the output layer. Through multi-scale convolution operations, the model can extract features at different scales, capturing more detailed information and helping to recover the spatial location information of the image. Furthermore, the fusion of high-level features with low-level features extracted through the multi-scale attention mechanism helps the model better integrate feature information from different levels, improve network recognition errors and road connectivity issues, and enhance the accuracy of road recognition and extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 The figure is a schematic structural diagram of a road recognition and extraction system according to an exemplary embodiment.

[0048] Figure 2 The figure is a flowchart of a road recognition and extraction method according to an exemplary embodiment.

[0049] Figure 3 The figure is a flowchart of a road recognition and extraction method according to an exemplary embodiment.

[0050] Figure 4 The figure is a flowchart of a road recognition and extraction method according to an exemplary embodiment.

[0051] Figure 5A schematic diagram of a sample road image involved in an embodiment of the present application is shown.

[0052] Figure 6 A schematic structural diagram of an upsampling module involved in an embodiment of the present application is shown.

[0053] Figure 7 It is a structural diagram of a road recognition and extraction device provided in an embodiment of the present application.

[0054] Figure 8 It is a structural diagram of a road recognition and extraction device provided in an embodiment of the present application.

[0055] Figure 9 It is a structural diagram of a computer device provided by an optional embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0057] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0058] Figure 1 FIG. 1 is a schematic diagram showing the structure of a road recognition and extraction system according to an exemplary embodiment. Figure 1 As shown, the road recognition and extraction system includes a terminal device 110 and a server 120 .

[0059] Optionally, the server 120 may be loaded with a road recognition and extraction model. In this case, the user may transmit the inspection image to be processed to the server 120 via a wired or wireless network through the terminal device 110, and the server 120 will process the inspection image using the road recognition and extraction model.

[0060] Optionally, the terminal device 110 may be loaded with a road recognition and extraction model. In this case, after receiving the inspection image, the terminal device 110 may directly process the inspection image through the road recognition and extraction model deployed by itself, thereby obtaining a road recognition and extraction result.

[0061] Optionally, the road recognition and extraction model can be trained in the server 120. In this case, the server 120 can deploy the road recognition and extraction model on itself, or send the parameters of the road recognition and extraction model to the terminal device 110, so that the road recognition and extraction model is deployed in the terminal device 110, so that the inspection image can be processed.

[0062] Optionally, the above-mentioned server can be a server cluster or a distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms and other technical computing services.

[0063] Optionally, the system may further include a management device for managing the system (such as managing the connection status between each module and the server, etc.), and the management device and the server are connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0064] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is typically the Internet, but may also be any other network, including but not limited to any combination of a local area network, a metropolitan area network, a wide area network, a mobile, a limited or wireless network, a private network, or a virtual private network. In some embodiments, the data exchanged over the network is represented using technologies and / or formats including hypertext markup language, extensible markup language, etc. In addition, conventional encryption technologies such as secure socket layer, transport layer security, virtual private network, internet protocol security, etc. may be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may also be used to replace or supplement the above-mentioned data communication technologies.

[0065] Figure 2 This is a flow chart of a method for road identification and extraction according to an exemplary embodiment. The method is applied to a computer device, which may be a computer device such as Figure 1 The terminal device or server shown. Figure 2 As shown, the road recognition and extraction method may include the following steps:

[0066] Step 201: Obtain an inspection image; the inspection image is obtained by collecting data from a drone on the road.

[0067] Specifically, the inspection image is obtained by aerial photography of the road using a high-resolution camera carried by a drone, collecting images containing complex backgrounds (such as vegetation, buildings, shadows) and road targets (such as asphalt pavement, cracks, and markings).

[0068] Step 202: Input the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result.

[0069] Since road targets are affected by factors such as lighting changes, occlusion, and viewing angle tilt, the road recognition and extraction model needs to be highly robust.

[0070] Specifically, the road recognition and extraction model includes an encoder, a decoder, and an output layer;

[0071] In this encoder, a convolutional neural network can be used for feature downsampling to extract high-level semantic features. During the downsampling process, the spatial resolution can be gradually reduced through pooling or strided convolution.

[0072] The decoder includes several upsampling modules; the upsampling modules include an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution process on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with the high-level features.

[0073] The output layer can then map the fused features into road segmentation results (for example, using 1×1 convolution).

[0074] In summary, after obtaining inspection images captured by drones, they can be input into a road recognition and extraction model for processing. This model consists of an encoder-decoder and an output layer. The decoder includes several upsampling modules, each of which includes an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, each of which performs at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers, and then fuses them with high-level features. The model then processes the features output by each upsampling module and finally outputs the road recognition and extraction results through the output layer. Through multi-scale convolution operations, the model can extract features at different scales, capturing more detailed information and helping to recover the spatial location information of the image. Furthermore, the fusion of high-level features with low-level features extracted through the multi-scale attention mechanism helps the model better integrate feature information from different levels, improve network recognition errors and road connectivity issues, and enhance the accuracy of road recognition and extraction.

[0075] Figure 3This is a flow chart of a method for road identification and extraction according to an exemplary embodiment. The method is applied to a computer device, which may be a computer device such as Figure 1 The terminal device or server shown. Figure 3 As shown, the road recognition and extraction method may include the following steps:

[0076] Step 301: Acquire a sample road image; the sample road image is acquired by capturing the road using a drone; the sample road image includes road annotations and background annotations.

[0077] Step 302: Iteratively train the road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image.

[0078] Among them, the road recognition and extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with the high-level features.

[0079] In summary, after obtaining inspection images captured by drones, they can be input into a road recognition and extraction model for processing. This model consists of an encoder-decoder and an output layer. The decoder includes several upsampling modules, each of which includes an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, each of which performs at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers, and then fuses them with high-level features. The model then processes the features output by each upsampling module and finally outputs the road recognition and extraction results through the output layer. Through multi-scale convolution operations, the model can extract features at different scales, capturing more detailed information and helping to recover the spatial location information of the image. Furthermore, the fusion of high-level features with low-level features extracted through the multi-scale attention mechanism helps the model better integrate feature information from different levels, improve network recognition errors and road connectivity issues, and enhance the accuracy of road recognition and extraction.

[0080] Figure 4This is a flow chart of a method for road identification and extraction according to an exemplary embodiment. The method is applied to a computer device, which may be a computer device such as Figure 1 The terminal device or server shown. Figure 4 As shown, the road recognition and extraction method may include the following steps:

[0081] Step 401: Acquire a sample road image; the sample road image is acquired by capturing the road using a drone; the sample road image includes road annotations and background annotations.

[0082] In a real-world scenario, a drone can collect 5,000 road images of a drone inspection scene and perform calibration. For example, roads can be labeled as one category, while objects such as buildings, vegetation, and vehicles can be classified as background. This will yield both road and background annotations for the sample road images.

[0083] In order to further increase the amount of sample data, the above sample road images can be enhanced, and the 5,000 drone images can be expanded to a data set of 30,000 images to complete the data set production. In deep neural networks, a small sample size of the data set will cause overfitting of the network training, that is, it fits well to the current training data set, but performs poorly on the validation set and test set. To avoid overfitting, color jittering, image shifting, scaling, rotation, flipping (including horizontal flipping, vertical flipping, diagonal flipping) and five combined superposition methods are used to enhance the data and expand the number of data set samples. Specifically, color jittering is to randomly change the color of the image; image shifting is to translate the image and label up, down, left, and right by a number of pixels; scaling is to randomly enlarge or reduce the image by a number of times (1 to 5 times); rotation is to randomly rotate the image and label clockwise by a number of angles (0° to 360°); flipping is to flip the image horizontally, vertically, or diagonally. Please refer to Figure 5 , which shows a schematic diagram of a sample road image involved in the embodiment of the present application. Figure 5 As shown in FIG, more sample road images can be generated after color dithering, image shifting and flipping the original image.

[0084] Step 402: Iteratively train the road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image.

[0085] In an embodiment of the present application, the road recognition and extraction model includes an encoder, a decoder, and an output layer. In a possible implementation, the topology of the road recognition and extraction model is a D-LinkNet network structure.

[0086] Furthermore, the encoder in the original D-LinkNet algorithm is ResNet, which has a large receptive field. However, since the input features and output features in the residual module are directly added, when the input and output sizes do not match, it is easy to cause feature distortion and perform poorly on small-scale data sets. In order to improve the feature distortion problem of ResNet and improve the segmentation accuracy of the network on small-scale data sets in the embodiment of the present application, the encoder is replaced by EfficientNet from ResNet. EfficientNet can optimize the depth, width and resolution of the network at the same time through a composite scaling strategy, and has strong generalization capabilities. Therefore, it can effectively improve the network segmentation accuracy under the conditions of small-scale data sets in highway inspection scenarios.

[0087] In this application's examples, the original D-LinkNet algorithm extracts feature information at a single scale during the feature extraction phase, failing to prioritize and prioritize important details. This results in significant loss of detail, leading to network recognition errors and road connectivity issues. To improve the network's ability to perceive detailed information, a multi-scale attention mechanism is introduced. An upsampling module based on this mechanism is proposed. This module selects low-level features and fuses them with high-level features, improving the ability to recover spatial position information from drone road images.

[0088] Specifically, the decoder of the road recognition extraction model of the embodiment of the present application includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales respectively; the original feature extraction layer is used to extract low-level features.

[0089] In an embodiment of the present application, the original feature extraction layer can adopt a 1*1 convolution layer. Although the low-level features output by the shallow layer of the encoder contain rich details, there are problems of channel redundancy and noise interference. By adjusting the number of channels through 1×1 convolution (such as aligning high-level feature channels), combined with batch normalization (BatchNorm) and ReLU activation function, the channel dimensions of features at different levels can be unified to avoid dimensional conflicts during fusion, and the activation function can improve the expressive power of the features, thereby making the low-level data input by subsequent multi-scale processing more accurate.

[0090] The multi-scale feature extraction layer can extract features from images at different scales through convolution kernels of multiple scales, making the model more adaptable to complex scenes such as scale changes (such as foreground and background roads) and occlusions (such as tree shadows).

[0091] Specifically, each convolution layer includes a first convolution kernel and a second convolution kernel of the same scale;

[0092] The first convolution kernel is used to perform the first convolution on the low-level feature and compress the number of feature channels to obtain an intermediate feature map;

[0093] The second convolution kernel is used to perform a second convolution on the intermediate feature and restore the number of feature channels to the number of input channels corresponding to the low-level feature to obtain the output feature map of the convolution layer.

[0094] In the embodiment of the present application, each convolutional layer can perform two convolution operations on the low-level features of the input. For example, in a certain convolutional layer, two 3*3 convolution operations can be used consecutively to perform convolution operations on the low-level features of the input. In addition, during the first convolution, the number of channels can be compressed, allowing the network to filter out important information in the low-dimensional space and reduce the interference caused by redundant features. Then, in order to allow the multi-scale features to be integrated with each other, it is necessary to restore the number of feature channels during the second convolution process for subsequent element-by-element addition or splicing.

[0095] Furthermore, the plurality of convolutional layers include convolution kernels of at least two scales; and among the plurality of convolutional layers, at least two convolutional layers have convolution kernels of the same scale.

[0096] In an embodiment of the present application, taking a 3*3 convolution operation as an example, two convolution layers can be set in the upsampling module. In any of the two convolution layers, the first convolution kernel and the second convolution kernel are used to perform a 3*3 convolution operation, but the parameters of the convolution kernels in the two convolution layers can be different, so that the upsampling module can notice more diverse features at the 3*3 feature size, thereby improving the feature extraction capability of the upsampling module.

[0097] At this point, the upsampling module is specifically used to:

[0098] Add the output features of the several convolutional layers to obtain a first feature map;

[0099] Deconvolution is performed on the first feature to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer;

[0100] Multiplying the second feature map by the image output by the original feature extraction layer to obtain an attention feature;

[0101] The attention feature is fused with the high-level feature to obtain the upsampling result of the upsampling module.

[0102] Please refer to Figure 6 , which shows a schematic diagram of the structure of an upsampling module involved in an embodiment of the present application. Figure 6As shown in the figure, the upsampling module based on the multi-scale attention mechanism consists of three parts. First, the original low-level features undergo 1×1 convolution, batch normalization, and ReLU activation. This is followed by a multi-scale feature extraction layer consisting of two layers of 3×3 convolution, two layers of 5×5 convolution, and two layers of 7×7 convolution, with ReLU activation. To accelerate computation, the two convolutions in each layer compress the feature channels to 1 / 4 of the input channels in the first convolution and restore them to the input channels in the second convolution. Third, the results of each convolution layer are summed, deconvolved to the image size of the middle branch, and multiplied with the result of the middle branch to generate pixel-level attention. Finally, the high-level features are fused and added with the low-level features extracted by the multi-scale attention mechanism. The main principle of the upsampling module based on the multi-scale attention mechanism is to use high-level features to select low-level features. This can effectively improve the network's ability to extract details of image spatial information, thereby improving network recognition errors and road connectivity issues.

[0103] 80% of the drone image dataset was divided into a training set and 20% into a validation set for testing. Experimental results show that, compared to the original D-LinkNet network, the improved D-LinkNet's mIOU (mean intersection-over-union) improved from 80.5% to 82.6%, and its F1 score increased from 78.1% to 80.4%, with the same number of iterations.

[0104] Step 403: Obtain an inspection image; the inspection image is obtained by collecting data on the road using a drone.

[0105] Step 404: Input the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result.

[0106] The specific implementation of the above steps 403 and 404 and the execution principle of the model are similar to those of step 402 and will not be repeated here.

[0107] In summary, after obtaining inspection images captured by drones, they can be input into a road recognition and extraction model for processing. This model consists of an encoder-decoder and an output layer. The decoder includes several upsampling modules, each of which includes an original feature extraction layer and a multi-scale feature extraction layer. The multi-scale feature extraction layer is composed of several convolutional layers, each of which performs at least one convolution on low-level features at different scales. The upsampling module then fuses the outputs of the original feature extraction layer and the convolutional layers, and then fuses them with high-level features. The model then processes the features output by each upsampling module and finally outputs the road recognition and extraction results through the output layer. Through multi-scale convolution operations, the model can extract features at different scales, capturing more detailed information and helping to recover the spatial location information of the image. Furthermore, the fusion of high-level features with low-level features extracted through the multi-scale attention mechanism helps the model better integrate feature information from different levels, improve network recognition errors and road connectivity issues, and enhance the accuracy of road recognition and extraction.

[0108] In the embodiments of the present application, a road identification and extraction device is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0109] The embodiment of the present application provides a road recognition and extraction device, Figure 7 : is a schematic structural diagram of a road recognition and extraction device provided in an embodiment of the present application, the device comprising:

[0110] Inspection image acquisition module 701, used to acquire inspection images; the inspection images are acquired by drones capturing images of the road;

[0111] A road recognition and extraction module 702 is configured to input the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result;

[0112] The road recognition and extraction model includes an encoder, a decoder, and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolutional layers; the several convolutional layers are used to perform at least one convolution process on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolutional layers, and then fuse them with high-level features. The road recognition and extraction model includes an encoder, a decoder, and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolutional layers; the several convolutional layers are used to perform at least one convolution process on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolutional layers, and then fuse them with high-level features.

[0113] In an optional embodiment, each convolution layer includes a first convolution kernel and a second convolution kernel of the same scale;

[0114] The first convolution kernel is used to perform a first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map;

[0115] The second convolution kernel is used to perform a second convolution on the intermediate feature and restore the number of feature channels to the number of input channels corresponding to the low-level feature to obtain the output feature map of the convolution layer.

[0116] In an optional embodiment, the plurality of convolutional layers include convolution kernels of at least two scales; and the convolution kernels of at least two convolutional layers among the plurality of convolutional layers have the same scale.

[0117] In an optional implementation, the upsampling module is configured to:

[0118] Adding the output features of the plurality of convolutional layers to obtain a first feature map;

[0119] Performing deconvolution on the first feature to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer;

[0120] Multiplying the second feature map by the image output by the original feature extraction layer to obtain an attention feature;

[0121] The attention feature is fused with the high-level feature to obtain the upsampling result of the upsampling module.

[0122] In an optional implementation, the topological structure of the road recognition and extraction model is a D-LinkNet network structure.

[0123] The embodiment of the present application also provides a road recognition and extraction device, Figure 7 : is a schematic structural diagram of a road recognition and extraction device provided in an embodiment of the present application, the device comprising:

[0124] The device comprises:

[0125] The sample acquisition module 801 is used to acquire a sample road image; the sample road image is acquired by using a drone to capture the road; the sample road image includes road annotations and background annotations;

[0126] A model training module 802 is configured to iteratively train a road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image;

[0127] Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

[0128] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0129] The device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0130] The embodiment of the present invention further provides a computer device having the above Figure 7 or Figure 8 The device shown.

[0131] See also Figure 9 , Figure 9 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 9 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information in the graphical user interface on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 9 A processor 10 is taken as an example.

[0132] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0133] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0134] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0136] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0137] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0138] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0139] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A road recognition and extraction method, characterized in that: The method comprises: Obtaining inspection images; the inspection images are obtained by collecting images of the road by a drone; Inputting the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result; Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

2. The method according to claim 1, characterized in that Each convolution layer includes a first convolution kernel and a second convolution kernel of the same scale; The first convolution kernel is used to perform a first convolution on the low-level features and compress the number of feature channels to obtain an intermediate feature map; The second convolution kernel is used to perform a second convolution on the intermediate feature and restore the number of feature channels to the number of input channels corresponding to the low-level feature to obtain the output feature map of the convolution layer.

3. The method according to claim 2, characterized in that The plurality of convolutional layers include convolution kernels of at least two scales; and among the plurality of convolutional layers, at least two convolutional layers have convolution kernels of the same scale.

4. The method according to any one of claims 1 to 3, characterized in that: The upsampling module is used to: Adding the output features of the plurality of convolutional layers to obtain a first feature map; Performing deconvolution on the first feature to obtain a second feature map; the image size of the second feature map is the same as the image size output by the original feature extraction layer; Multiplying the second feature map by the image output by the original feature extraction layer to obtain an attention feature; The attention feature is fused with the high-level feature to obtain the upsampling result of the upsampling module.

5. The method according to claim 4, characterized in that The topological structure of the road recognition and extraction model is a D-LinkNet network structure.

6. A road recognition and extraction method, characterized in that: The method comprises: Acquire a sample road image; the sample road image is obtained by capturing the road using a drone; the sample road image includes road annotations and background annotations; Iteratively training a road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image; Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

7. A road recognition and extraction device, characterized in that: The device comprises: An inspection image acquisition module is used to acquire inspection images; the inspection images are acquired by a drone on the road; A road recognition and extraction module is used to input the inspection image into a road recognition and extraction model to obtain a road recognition and extraction result; Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

8. A road recognition and extraction device, characterized in that: The device comprises: A sample acquisition module is used to acquire a sample road image; the sample road image is acquired by using a drone to capture the road; the sample road image includes road annotations and background annotations; A model training module is used to iteratively train the road recognition and extraction model based on the sample road image to obtain a trained road recognition and extraction model; the trained road recognition and extraction model is used to perform road recognition and extraction on the inspection image; Among them, the road recognition extraction model includes an encoder, a decoder and an output layer; the decoder includes several upsampling modules; the upsampling module includes an original feature extraction layer and a multi-scale feature extraction layer; the original feature extraction layer is used to extract low-level features; the multi-scale feature extraction layer includes several convolution layers; the several convolution layers are used to perform at least one convolution processing on the low-level features at different scales; the upsampling module is used to fuse the outputs of the original feature extraction layer and the several convolution layers, and then fuse them with high-level features.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the road recognition and extraction method as described in any one of claims 1 to 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the road identification and extraction method according to any one of claims 1 to 6.

Images