Road rockfall detection method, device and electronic equipment based on deep learning

Abstract

The present application discloses a road rockfall detection method, device, electronic device and program product based on deep learning. The detection method training is implemented through a completed rockfall detection model. The backbone network of the rockfall detection model is provided with an SDF module for fusing image information extracted by basic convolution and detail information extracted by SPD convolution, which enables the rockfall detection model to not only process low-resolution images or images containing small targets, but also to adaptively fuse multi-angle and multi-level feature information to alleviate the recognition problem caused by changes in image perspective; an SLLGA module is set in the neck network to realize adaptive weighted fusion of global features and local features, so that the model can dynamically adapt to different focus points and improve the rockfall detection effect in complex scenes; a GE‑SPP module is set after the last scale of the backbone network to fuse multi-scale features, global background features and edge features to improve the accuracy of rockfall detection in different background environments and different scales.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a road rockfall detection method, road rockfall detection device, electronic equipment and computer program product based on deep learning. Background Art

[0002] With the acceleration of urbanization, the expansion of road traffic network has greatly facilitated travel, but rockfall accidents have increasingly become a major hidden danger threatening driving safety, which may cause traffic congestion and even casualties, resulting in serious economic losses and. Thanks to the rapid development of information technology and artificial intelligence, rockfall detection technology is gradually transforming from manual inspection to intelligent and automated, which can achieve all-weather monitoring and risk assessment, effectively improve the level of road safety management, and provide important support for the construction of smart transportation.

[0003] However, in road rockfall detection, images with low resolution or containing small-sized rocks have poor detection effects, which can easily reduce the accuracy and reliability of rockfall detection. Summary of the invention

[0004] The present application provides a road rockfall detection method, road rockfall detection device, electronic device and computer program product based on deep learning, which can effectively deal with images with low resolution or containing small targets, and significantly improve the accuracy and reliability of rockfall detection.

[0005] In a first aspect, the present application provides a road rockfall detection method based on deep learning, comprising:

[0006] The backbone network of the pre-trained rockfall detection model extracts features from the image to be detected to obtain image features; the image to be detected includes road rockfall;

[0007] The neck network based on the rockfall detection model fuses the image features to obtain fused features;

[0008] The detection network based on the rockfall detection model detects the fused features and obtains the detection result of the rockfall on the road in the image to be detected;

[0009] Among them, the backbone network is equipped with an SDF module, which is used to fuse the semantic features and detail features extracted by different convolutional sub-modules.

[0010] In a second aspect, the present application provides a road rockfall detection device, comprising:

[0011] An extraction module is used to extract features of an image to be detected based on a backbone network of a pre-trained rockfall detection model to obtain image features; the image to be detected includes road rockfall;

[0012] A fusion module is used to fuse image features based on the neck network of the rockfall detection model to obtain fusion features;

[0013] A detection module is used to detect the fused features based on the detection network of the rockfall detection model to obtain the detection result of the rockfall on the road in the image to be detected;

[0014] Among them, the backbone network is equipped with an SDF module, which is used to fuse the semantic features and detail features extracted by different convolutional sub-modules.

[0015] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0018] Compared with the prior art, the present application has the following beneficial effects: the semantic detail fusion module (SDF module) is introduced into the backbone network, which can effectively fuse the semantic features and detail features extracted by different convolutional layers. Even when processing low-resolution images or images containing small targets, key information can be fully retained, and the representation ability of the backbone network can be enhanced, thereby improving the adaptability of the rockfall detection model to multiple scenarios. The image features extracted by this backbone network have both rich contextual information and detail information, so that the feature fusion based on it can more comprehensively include effective information related to rockfall, and even in low-resolution or small-target scenes, the accuracy and reliability of rockfall detection can be significantly improved.

[0019] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a schematic diagram of the network structure of the rockfall detection model provided in the embodiment of the present application;

[0022] Figure 2 is an example diagram of an image in the Rockfalls dataset provided in an embodiment of the present application;

[0023] Figure 3 It is a flowchart of a road rockfall detection method based on deep learning provided in an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of the network structure of the SDF module provided in an embodiment of the present application;

[0025] Figure 5 Schematic diagram of the network structure of the SLLGA module provided in the embodiment of the present application;

[0026] Figure 6 is a schematic diagram of the network structure of the GE-SPP module provided in an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of the structure of a road rockfall detection device provided in an embodiment of the present application;

[0028] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0030] In the backbone network, the convolution module (ConvModule) is one of the most widely used feature extraction tools in the field of deep learning. Its core advantage is that it can efficiently capture local features and semantic information in the input data while maintaining the spatial hierarchy of the data. In addition, the convolution module also has the characteristic of parameter sharing, which not only reduces the number of model parameters and reduces the risk of overfitting, but also improves computational efficiency.

[0031] However, the present application found that in the detection of road rockfall, if the image resolution is low or contains small-sized rocks, the performance of the traditional Convolutional Neural Network (CNN) architecture will be significantly reduced. This is because the strided convolution and pooling layers commonly used in traditional CNNs are prone to the loss of fine-grained information, which affects the model's learning of effective feature representations. When the image quality is poor or the target object is very small, these details become scarce, causing the model to have poor feature learning effects due to the loss of fine-grained information, thereby reducing the accuracy and reliability of rockfall detection.

[0032] In order to solve this problem, this application proposes a rockfall detection model, which includes a backbone network, a neck network and a detection head. Specifically, the backbone network is used to extract features from the input image to obtain image features; the neck network is used to fuse the image features to obtain fused features; the detection head is used to detect the fused features to obtain the detection results of road rockfall in the image to be processed.

[0033] Among them, in order to deal with low-resolution images or images containing small targets, the present application sets an SDF module in the backbone network, which can effectively fuse the semantic features and detail features extracted by different convolutional layers. Even when processing low-resolution images or images containing small targets, key information can be fully retained, and the representation ability of the backbone network can be enhanced, thereby improving the adaptability of the rockfall detection model to multiple scenarios. The image features extracted by this backbone network have both rich contextual information and detail information, so that the feature fusion based on it can more comprehensively include effective information related to rockfall, even in low-resolution or small target scenes, it can still significantly improve the accuracy and reliability of rockfall detection.

[0034] In some embodiments, the SDF module includes two different convolution submodules, namely a basic convolution submodule and an SPD convolution submodule. The basic convolution submodule extracts the semantic information of the image through standard convolution operations, which helps to capture the semantic patterns and key features in the image and lay the foundation for the understanding of the overall image. The SPD convolution submodule focuses on the image detail information. The SPD convolution submodule transfers the spatial information to the depth dimension, which can reduce the spatial resolution while retaining the fine-grained information, thereby reducing the computational requirements of subsequent convolution layers.

[0035] In order to efficiently utilize semantic information and detail information, the SDF module can efficiently fuse image semantic information and detail information through corresponding fusion methods. This design not only overcomes the limitations of traditional CNN in processing low-resolution or small target tasks, but also can show better performance in a variety of application scenarios. The flexibility of the SDF module allows the model to dynamically adjust the importance of the two features according to the needs of specific tasks, thereby improving the robustness and generalization ability of the model while ensuring computational efficiency.

[0036] In some embodiments, the fusion structure in the SDF module can effectively enhance the SDF module's ability to express features by introducing an attention mechanism to weight both detail information and semantic information. This can adaptively fuse multi-angle and multi-level feature information, thereby enhancing the model's capture and understanding of rockfall features from different perspectives, and improving the model's detection capability and adaptability to rockfall images from different perspectives.

[0037] Exemplarily, it is assumed that the rockfall detection model is improved based on the YOLO series model. Exemplarily, for YOLOv5s / YOLOv8s, the convolution module of the backbone network of YOLOv5s / YOLOv8s will be replaced with the SDF module.

[0038] In related technologies, in addition to image quality issues and small target tasks, rockfall detection also faces new requirements on the generalization ability and adaptability of the model due to scene diversity and complexity.

[0039] In some embodiments, in order to enable the rockfall detection model to meet this requirement, an adaptive global-local attention mechanism module (Self-Learning Local-Global Attention Module, SLLGA module) can be set in the neck network. The SLLGA module performs adaptive weighted fusion on the global features and local features of the input features.

[0040] The global attention mechanism integrates the information of the entire image to highlight the key areas and enhance the model's ability to understand the context. Its advantage is that it can capture long-distance dependencies and improve the richness of feature expression. The local attention mechanism focuses on specific parts of the image and finely extracts detailed information, which helps to improve the model's sensitivity to local features.

[0041] By integrating the advantages of the two attention mechanisms, the SLLGA module can dynamically balance the learning of local features and global features, taking advantage of the guidance of global information while retaining the accuracy of local details, so that the model can represent features more comprehensively and have stronger generalization capabilities. This strategy is particularly effective in complex scenarios because it can balance attention points at different scales, improving the performance of the model.

[0042] In the Yolo series of models, even YOLOv8s, the SPPF module (Spatial Pyramid Pooling Fast module) in its backbone network still has room for improvement. Although the SPPF module is an improved spatial pyramid pooling (SPP) module, it uses three 5×5 maximum pooling layers for feature extraction. On the one hand, it can obtain more feature information and improve the recognition accuracy of the network; on the other hand, it can reduce the amount of calculation while ensuring similar accuracy, thereby improving the network recognition speed. However, the SPPF module may ignore background information due to excessive focus on edge information, making it difficult to cope with multi-scale rockfall detection in complex scenes. This limitation may cause false detection or missed detection, reducing the detection accuracy and reliability of the model.

[0043] In some embodiments, in order to improve the accuracy of the model for rockfall detection of different scales in complex scenes, a global edge spatial pyramid pooling module (GE-SPP module) can be set in the backbone network. Based on the SPPF module, the GE-SPP module introduces a global average pooling layer and a global maximum pooling layer, which can obtain global perspective information and reduce the impact of different scales. By concatenating the three (Concat, that is, performing feature connection operations in the channel dimension), the global background information and edge information are better integrated to form a comprehensive feature representation. This can not only effectively handle rockfall targets of different sizes, but also reduce the impact of the background environment on the detection results to a certain extent, thereby improving the detection accuracy of the model.

[0044] For example, if the YOLOv8s basic network is improved through the SDF module, SLLGA module and GE-SPP module, the network structure of the rockfall detection model can be found in Figure 1 .

[0045] The backbone network of the rockfall detection model is equipped with an SDF module, which is used to fuse the image information extracted by the basic convolution and the detail information extracted by the SPD convolution. This enables the rockfall detection model to not only process low-resolution images or images containing small targets, but also adaptively fuse multi-angle and multi-level feature information to alleviate the recognition problem caused by changes in image perspective; an SLLGA module is set in the neck network to realize the adaptive weighted fusion of global features and local features, so that the network can dynamically adapt to different focus points and improve the rockfall detection effect in complex scenes; a GE-SPP module is set after the last scale of the backbone network to fuse multi-scale features, global background features and edge features, so as to improve the accuracy of rockfall detection in different background environments and at different scales.

[0046] In some embodiments, in order to ensure that the trained rockfall detection model can meet expectations and accurately understand and detect road rockfalls of different scales in different scenarios, a special dataset can be created for training the rockfall detection model. Specifically, a dataset called Rockfalls is created. The dataset is collected by a vehicle camera under various environmental conditions and includes 3,000 images with a resolution of 1280x720, and each image is manually annotated. Figure 2 , Figure 2 An example image is shown.

[0047] For example, in order to provide a high-quality dataset, in addition to ensuring the number of images, the collected images must also cover different scenes such as urban roads, highways, and mountain roads.

[0048] Exemplarily, each scene may include multiple time periods and weather conditions such as day, night, sunny, rainy and foggy days to increase the complexity and comprehensiveness of the scene. Each image may be configured with a corresponding txt label file that details the location and category of the rockfall.

[0049] Such datasets can support research and applications in areas such as intelligent transportation systems, autonomous vehicles, and road safety monitoring to improve road safety and traffic efficiency.

[0050] In order to ensure the effectiveness of model training and the reliability of evaluation results, each image in the dataset of this application is divided into a training set and a validation set in a ratio of 8:2.

[0051] Preferably, in the process of dividing the data set, special attention should be paid to maintaining the balance of sample distribution between the training set and the validation set, ensuring that the two have similar statistical characteristics in terms of rockfall type, size, background environment, etc. This can avoid evaluation bias caused by uneven sample distribution and make the validation results more representative and credible.

[0052] In some embodiments, in order to comprehensively and accurately measure the performance of each version of the rockfall detection model, after at least one version of the rockfall detection model is converged based on the training set, the converged model can be evaluated through the validation set to avoid model overfitting, verify the generalization of the model, and ensure that the model can run stably after deployment.

[0053] Specifically, the performance of the rockfall detection model on the validation set can be evaluated according to preset conditions.

[0054] For example, these preset conditions may include performance indicators such as the intersection-over-union ratio, detection accuracy and recall rate of the facility detection device for road defects.

[0055] IoU is an indicator to evaluate the degree of overlap between the predicted border and the true border. It calculates the ratio of the intersection and union of the predicted border and the true border. It plays a key role in determining whether it is a correct detection. The calculation formula of IoU can be written as:

[0056]

[0057] Precision (p), also known as the precision rate, refers to the proportion of correct positive predictions to all positive predictions, as shown in the formula:

[0058]

[0059] Recall (R), also known as recall rate, refers to the proportion of correctly predicted positives to all actual positives, as shown in the formula:

[0060]

[0061] The average precision (AP) is calculated from the precision and recall. A line graph of the precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:

[0062]

[0063] The mean average precision (mAP) refers to the average value of the average precision AP of C different road disease categories, as shown in the formula:

[0064]

[0065] That is to say, after verifying each version of the rockfall detection model through the validation set, the rockfall detection model of each version can be comprehensively evaluated based on the above-mentioned indicators, so as to determine the rockfall detection model with the best performance from each version as the trained rockfall detection model.

[0066] Based on the network structure of the implementation detection model of the previous embodiment, this application proposes a road rockfall detection method based on deep learning.

[0067] The deep learning-based road rockfall detection method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0068] In order to illustrate the technical solution proposed in this application, each embodiment will be described below with an electronic device as the execution subject.

[0069] Figure 3 A schematic flow chart of a road rockfall detection method based on deep learning provided by the present application is shown, and the road rockfall detection method based on deep learning includes:

[0070] Step 310: The electronic device extracts features of the image to be detected based on the backbone network of the pre-trained rockfall detection model.

[0071] Step 320: The electronic device fuses the image features based on the neck network of the rockfall detection model to obtain fused features.

[0072] Step 330: The electronic device detects the fused features based on the detection network of the rockfall detection model to obtain a detection result of the rockfall on the road in the image to be detected.

[0073] In this embodiment, the improved backbone network can effectively fuse the semantic features and detail features extracted by different convolutional layers thanks to the introduction of the SDF module. Even when processing low-resolution images or images containing small targets, key information can be fully retained, enhancing the representation ability of the backbone network, thereby improving the adaptability of the rockfall detection model to multiple scenarios. The image features extracted by this backbone network have both rich contextual information and detail information, so that the feature fusion based on it can more comprehensively include effective information related to rockfall, and even in low-resolution or small-target scenes, it can still significantly improve the accuracy and reliability of rockfall detection.

[0074] In some embodiments, the SDF module includes a basic convolution submodule and an SPD convolution submodule; for the first feature input to each SDF module, the electronic device may perform the following operations:

[0075] Step A1: The electronic device inputs the first feature into the basic convolution submodule to extract the semantic feature of the first feature.

[0076] Step A2: The electronic device inputs the first feature into the SPD convolution submodule to extract the first semantic detail feature.

[0077] Step A3: The electronic device performs feature fusion based on the semantic feature and the detail feature to obtain an output feature corresponding to the first feature.

[0078] The basic convolution submodule performs a standard convolution operation, such as a convolution operation with a kernel of 3 and a stride of 2. Through this convolution operation, the semantic patterns and information in the image can be captured to obtain semantic features.

[0079] The SPD convolution submodule first downsamples through the SPD convolution structure to transfer the spatial information to the depth dimension, compressing the spatial dimension while retaining all the information in the channel dimension, avoiding the loss of fine-grained information, thereby reducing the spatial resolution and the computational requirements of subsequent convolutions. Then, a standard convolution operation is performed on the output of the SPD convolution structure to adjust the number of channels and obtain detailed features to meet the subsequent processing requirements.

[0080] The SDF module fuses the outputs of the two sub-modules together, and can achieve efficient fusion of image semantic information and detail information without sacrificing the advantages of either side, so as to overcome the limitations of existing detection methods in dealing with low-resolution or small target tasks, thereby showing better performance in various application scenarios.

[0081] In some embodiments, in order to accurately detect fallen rocks from different viewing angles, the aforementioned step A3 specifically includes:

[0082] Step A31: The electronic device splices the semantic features after the compression channel and the detail features after the compression channel to obtain a first splicing feature.

[0083] Step A32: The electronic device performs a normalization operation on the first splicing feature after channel mapping to obtain a first weight of the semantic feature and a second weight of the detail feature.

[0084] Step A33: The electronic device performs weighted fusion on the semantic feature and the detail feature based on the first weight and the second weight to obtain a first fused feature.

[0085] Step A34: The electronic device performs channel expansion on the first fusion feature to obtain an output feature corresponding to the first feature.

[0086] For example, see Figure 4 , Figure 4 The network structure of the SDF module is shown. Figure 4The network structure of the semantic feature and the detail feature is respectively compressed through a convolution layer to perform preliminary channel dimension reduction and feature compression. Then the two compressed features are concatenated to obtain the concatenated feature. In order to calculate the weights of the semantic feature and the detail feature, the concatenated feature is first mapped to a preset number, such as 2, through a convolution layer. Then the concatenated feature with the preset number of channels is input into the normalized ( Softmax ) layer, calculates the first weight of the semantic feature and the second weight of the detail feature respectively, and the normalization layer performs a weighted fusion operation on the semantic feature and the detail feature based on the two weights, and outputs a first fused feature; the first fused feature is subjected to a channel expansion operation to obtain the output feature corresponding to the first feature.

[0087] For example, assuming that the first feature ,in B is the batch size, C 1 is the number of input channels, H and W are the height and width of the feature respectively, and the number of output channels is recorded as C 2, the number of channels after compression is recorded as =64.

[0088] In the basic convolution submodule, X 1 Through a convolution operation with a kernel size of 3×3 and a stride of 2, the input feature size is halved and the number of channels is reduced from C 1 maps to , get semantic features Y conv . This operation can be expressed as:

[0089]

[0090] In the SPD convolution submodule, first X 1 performs a space-to-depth rearrangement operation, X The size of 1 is halved, resulting in a combination of four sub-regions X spd ,let X The number of channels of 1 is from C 1 maps to 4 C 1, and then through the 1×1 convolution operation X spd Convert to Y spd .

[0091]

[0092] Furthermore, a submodule can be regarded as a branch, and the outputs of the two branches are compressed to 0.5 through their respective 1x1 convolutional layers. , and concatenate the compressed features in the channel direction, and use a 3x3 convolution layer to increase the number of channels from Mapped to 2, and then normalized to generate attention weights w 0，1 , that is, the first weight and the second weight.

[0093]

[0094] according to w 0 and w 1 Perform weighted fusion on the outputs of the two branches to obtain the first fusion feature Y fused .

[0095]

[0096] First fusion feature Y fused Then expand it back through the 1x1 convolution layer C 1 channel, final output X The output corresponding to 1 O .

[0097]

[0098] In this embodiment, the entire SDF module can be regarded as a convolutional structure containing a multi-branch information fusion mechanism. By weighting the details and semantic information, the feature expression capability can be significantly enhanced. The unique structural design of the SDF module can adaptively fuse multi-angle and multi-level features, effectively capture the rockfall features under different perspectives, improve the feature extraction capability of the model, and alleviate the recognition problem caused by the change of perspective.

[0099] In some embodiments, in order to balance the global features and the local features, the neck network is provided with a SLLGA module, and the SLLGA module is used to perform adaptive weighted fusion on the global features and the local features of the input features.

[0100] In some embodiments, the SLLGA module includes a local attention mechanism submodule and a global attention mechanism submodule; with respect to the second feature input to the SLLGA module, the electronic device may perform the following operations:

[0101] Step B1: The electronic device extracts the second feature through the local attention mechanism submodule to obtain a local feature and a third weight of the local feature.

[0102] Step B2: The electronic device extracts the second feature through the global attention mechanism submodule to obtain a global feature and a fourth weight of the global feature.

[0103] Step B3: The electronic device performs weighted fusion based on the third weight, the fourth weight, the local feature and the global feature to obtain a second fused feature.

[0104] Step B4: The electronic device multiplies the second fusion feature after channel expansion by the second feature to obtain an output feature corresponding to the second feature.

[0105] For example, see Figure 5 , Figure 5 The network structure of the SLLGA module is shown. The electronic device first performs a local average pooling operation on the second feature through the local attention mechanism submodule, then adjusts the dimension through the Reshape operation, then performs a one-dimensional convolution operation, restores the dimension of the feature through the convolution operation again, and activates it through the Sigmoid function to obtain the global feature and the fourth weight of the global feature. The electronic device first performs a local average pooling operation on the second feature through the global attention mechanism submodule, then adjusts the dimension through the Reshape operation, then performs a one-dimensional convolution operation, restores the dimension of the feature through the Reshape operation again, and activates it through the Sigmoid function to obtain the global feature and the fourth weight of the global feature.

[0106] The electronic device performs weighted fusion of the local features and the global features based on two weight values ​​to obtain a second fused feature, which can be regarded as the final weight after upsampling. By multiplying it with the second feature, the output feature corresponding to the second feature can be obtained.

[0107] Assume that the second feature is ,in B is the batch size, C is the number of input channels, H and W are the height and width of the feature, respectively.

[0108] In the local attention mechanism submodule, the input features are first extracted to obtain the local average pooling operation, and the dimension change operation is performed through View and Transpose, followed by a one-dimensional convolution operation, and finally an inverse dimension change operation is performed through upsampling to restore the dimension and perform Sigmoid activation.

[0109]

[0110] Among them, ALAP stands for adaptive local average pooling; V L Represents a series of dimension change operations in the local attention branch; It represents the corresponding inverse dimension change operation; Represents the Sigmoid activation function in the local attention branch.

[0111] In all attention mechanism sub-modules, the input features are firstly extracted to obtain the local average pooling operation, and the dimension change operation is performed through View and Transpose, followed by a one-dimensional convolution operation, and finally the inverse dimension change operation is performed through View and Transpose to restore the dimension and perform Sigmoid activation.

[0112]

[0113] Among them, AGAP represents adaptive global average pooling; V G Represents a series of dimension-changing operations in the global attention branch; It represents the corresponding inverse dimension change operation; Represents the Sigmoid activation function in the global attention branch.

[0114] GA is upsampled to the same size as LA, and GA and LA are adaptively weighted fused according to the third weight and second weight 1-α and α learned by the network. Then, it is upsampled to the same size as the input feature and multiplied with the feature as the final weight. This process helps the model better focus on important features by adaptively combining global and local attention mechanisms.

[0115]

[0116] in, O 2 means X 2 corresponds to the output features; U represents the upsampling operation (Unsample).

[0117] In this embodiment, the introduction of the SLLGA module can capture both local details and global contextual information of the image. Compared with the traditional single attention mechanism, it can not only understand the input data more comprehensively and capture long-distance dependencies, but also enable the network to dynamically adapt to different focus points, so that the model has better generalization ability and adaptability in different scenarios. Applying it to road rockfall detection helps to improve the accuracy of implementation recognition in complex scenarios.

[0118] In some embodiments, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2. Figure 1 , m=3, then a GE-SPP module is set after the third feature extraction module; the GE-SPP module is used to fuse multi-scale features, global background features and edge features.

[0119] In some embodiments, for the third feature of the input GE-SPP module:

[0120] Step C1: The electronic device performs a multi-scale pooling operation on the third feature after the compression channel to obtain a multi-scale feature.

[0121] Step C2: The electronic device performs an edge feature extraction operation on the third feature after the compression channel to obtain an edge feature.

[0122] Step C3: The electronic device performs a global feature extraction operation on the third feature after the compression channel to obtain a global background feature.

[0123] Step C4: The electronic device splices the multi-scale features, the edge features and the global background features to obtain a second splicing feature.

[0124] Step C5: The electronic device performs channel expansion on the second splicing feature to obtain an output feature corresponding to the third feature.

[0125] Based on the SPPF module, the GE-SPP module introduces the global average pooling layer and the global maximum pooling layer, which can obtain global perspective information and reduce the impact of different scales. By splicing the outputs of the three (SPPF module, global average pooling layer and global maximum pooling layer), it better integrates the global background information and edge information, thus forming a comprehensive feature representation.

[0126] See also Figure 6 , Figure 6 The network structure of the GE-SPP module is shown. Assuming the third feature is ,

[0127] Where B is the batch size, C is the number of input channels, and H and W are the height and width of the features, respectively.

[0128] For the GE-SPP module, firstly, a 1x1 convolution is performed to halve the number of channels of the input feature to obtain a new feature. This operation can be expressed as:

[0129]

[0130] Next, three 5×5 maximum pooling layers are used in sequence to perform multi-scale pooling operations to extract features, which can be expressed as:

[0131]

[0132] in, Represents maximum pooling with a pooling kernel size of 5×5, a stride of 1, and a padding of 2.

[0133] At the same time, the global average pooling layer and the global maximum pooling layer are introduced to obtain the global perspective information and reduce the impact of different scales.

[0134]

[0135] in, AFMP Represents adaptive global maximum pooling; AGAP Represents adaptive global average pooling.

[0136] Finally, the outputs of all pooling layers are concatenated in the channel dimension and a 1x1 convolution is performed to adjust the number of channels to the same as X 3. Consistency.

[0137] In this application, the introduction of the GE-SPP module provides an effective solution to solve the multi-scale change problem and alleviate the interference of the complex background environment. By fusing global background information and edge information, this module can not only effectively process rockfall targets of different sizes, but also reduce the impact of the background environment on the detection results to a certain extent, thereby improving the detection accuracy of the model.

[0138] In some embodiments, the hardware environment in which the rockfall detection model improved in the above three aspects runs includes: Intel (R) Xeon (R) Platinum 8255C CPU processor, 314G memory, NVIDIA Tesla V100 32G graphics card, CentOS 8.5.2 64-bit operating system, and the deep neural network is built on the PyTorch deep learning framework. The input image size of the network is [640, 640], and a multi-scale training mode is adopted. The data input batch is 64, the training epoch is 300, and the AdamW optimizer with an initial learning rate of 0.01 and a cosine decay strategy are adopted. The loss function used in this paper is the weighted sum of BCELoss, VFLLoss and CIoULoss.

[0139] It has been verified that the SDF module is set in the backbone network of the detection method of the present application to fuse the image information extracted by the basic convolution with the detail information extracted by the SPD convolution, so that the model can process low-resolution images or images containing small targets, and adaptively fuse multi-angle and multi-level feature information to alleviate the recognition problem caused by changes in viewing angle; the SLLGA module is introduced in the neck network to realize the adaptive weighted fusion of global and local features, so that the network dynamically adapts to different focus points and improves the detection effect in complex scenes; the GE-SPP module is added after the last scale of the backbone network to fuse multi-scale features, global background features and edge features to improve the detection accuracy of the model in different background environments and scales.

[0140] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0141] Corresponding to the road rockfall detection method based on deep learning in the above embodiment, Figure 7 A structural block diagram of a road rockfall detection device 7 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0142] Reference Figure 7 , the road rockfall detection device 7 comprises:

[0143] An extraction module 71 is used to extract features of an image to be detected based on a backbone network of a pre-trained rockfall detection model to obtain image features; the image to be detected includes road rockfall;

[0144] A fusion module 72, used for fusing image features based on the neck network of the rockfall detection model to obtain fused features;

[0145] A detection module 73, used to detect the fused features based on the detection network of the rockfall detection model, and obtain the detection result of the rockfall on the road in the image to be detected;

[0146] Among them, the backbone network is equipped with an SDF module, which is used to fuse the image information extracted by different convolution sub-modules.

[0147] Optionally, the SDF module includes a basic convolution submodule and an SPD convolution submodule; the extraction module 71 includes a first extraction unit, which is used to:

[0148] For the first feature of the input SDF module:

[0149] Inputting the first feature into the basic convolution submodule to extract the semantic feature of the first feature;

[0150] Input the first feature into the SPD convolution submodule to extract the detail feature of the first semantics;

[0151] Feature fusion is performed based on the semantic feature and the detail feature to obtain the output feature corresponding to the first feature.

[0152] Optionally, the extraction unit is specifically used for:

[0153] Splicing the semantic features after the compression channel and the detail features after the compression channel to obtain a first splicing feature;

[0154] Performing a normalization operation on the first concatenated feature after channel mapping to obtain a first weight of the semantic feature and a second weight of the detail feature;

[0155] Performing weighted fusion on the semantic feature and the detail feature based on the first weight and the second weight to obtain a first fused feature;

[0156] Perform channel expansion on the first fusion feature to obtain the output feature corresponding to the first feature.

[0157] Optionally, the neck network is provided with a SLLGA module, and the SLLGA module is used to perform adaptive weighted fusion on the global features and local features of the input features.

[0158] Optionally, the fusion module 72 is specifically used for:

[0159] Second feature for input SLLGA module:

[0160] The second feature is extracted through the local attention mechanism submodule to obtain the local feature and the third weight of the local feature;

[0161] The second feature is extracted through the global attention mechanism submodule to obtain the global feature and the fourth weight of the global feature;

[0162] Perform weighted fusion based on the third weight, the fourth weight, the local feature and the global feature to obtain a second fused feature;

[0163] The second fused feature after channel expansion is multiplied by the second feature to obtain the output feature corresponding to the second feature.

[0164] Optionally, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2. A GE-SPP module is arranged after the mth feature extraction module; the GE-SPP module is used to fuse multi-scale features, global background features and edge features.

[0165] Optionally, the extraction module 71 includes a second extraction unit, configured to:

[0166] For the third feature of the input GE-SPP module:

[0167] Perform a multi-scale pooling operation on the third feature after the compression channel to obtain a multi-scale feature;

[0168] Performing edge feature extraction operation on the third feature after the compression channel to obtain edge features;

[0169] Performing a global feature extraction operation on the third feature after the compression channel to obtain a global background feature;

[0170] The multi-scale features, edge features and global background features are spliced ​​to obtain a second spliced ​​feature;

[0171] Channel expansion is performed on the second concatenated feature to obtain an output feature corresponding to the third feature.

[0172] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0173] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one processor is shown in the figure), a memory 81, and a computer program 82 stored in the memory 81 and executable on at least one processor 80. When the processor 80 executes the computer program 82, the steps in any of the above-mentioned embodiments of the method for detecting falling rocks on a road based on deep learning are implemented, for example Figure 1 Steps 110-130 are shown.

[0174] The processor 80 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0175] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 8.

[0176] Furthermore, the memory 81 may include both an internal storage unit of the electronic device 8 and an external storage device. The memory 81 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0177] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0178] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0179] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0181] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0182] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0183] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A road rockfall detection method based on deep learning, characterized in that: include: Based on the backbone network of the pre-trained rockfall detection model, feature extraction is performed on the image to be detected to obtain image features; The image to be detected includes fallen rocks on the road; Based on the neck network of the rockfall detection model, the image features are fused to obtain fused features; The detection network based on the rockfall detection model detects the fused features to obtain a detection result of the rockfall on the road in the image to be detected; Wherein, the backbone network is provided with an SDF module, and the SDF module is used to fuse the semantic features and detail features extracted by different convolution submodules; The SDF module includes a basic convolution submodule and an SPD convolution submodule; for the first feature of the input SDF module: Inputting the first feature into the basic convolution submodule to extract semantic features of the first feature; Inputting the first feature into the SPD convolution submodule to extract detail features of the first feature; Performing feature fusion based on the semantic feature and the detail feature to obtain an output feature corresponding to the first feature; The performing feature fusion based on the semantic feature and the detail feature to obtain an output feature corresponding to the first feature includes: Splicing the semantic features after the compression channel and the detail features after the compression channel to obtain a first splicing feature; Performing a normalization operation on the first splicing feature after channel mapping to obtain a first weight of the semantic feature and a second weight of the detail feature; Performing weighted fusion on the semantic feature and the detail feature based on the first weight and the second weight to obtain a first fused feature; Channel expansion is performed on the first fusion feature to obtain an output feature corresponding to the first feature.

2. The road rockfall detection method according to claim 1, characterized in that: The neck network is provided with a SLLGA module, and the SLLGA module is used to perform adaptive weighted fusion on the global features and local features of the input features.

3. The road rockfall detection method as claimed in claim 2, characterized in that: The SLLGA module includes a local attention mechanism submodule and a global attention mechanism submodule; for the second feature of the input SLLGA module: Performing feature extraction on the second feature through the local attention mechanism submodule to obtain a local feature and a third weight of the local feature; Performing feature extraction on the second feature through the global attention mechanism submodule to obtain a global feature and a fourth weight of the global feature; Perform weighted fusion based on the third weight, the fourth weight, the local feature and the global feature to obtain a second fused feature; The second fused feature after channel expansion is multiplied by the second feature to obtain an output feature corresponding to the second feature.

4. The road rockfall detection method according to claim 1, characterized in that: The backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; a GE-SPP module is arranged after the mth feature extraction module; the GE-SPP module is used to fuse multi-scale features, global background features and edge features.

5. The road rockfall detection method as claimed in claim 4, characterized in that: Regarding the third feature of the GE-SPP module: Performing a multi-scale pooling operation on the third feature after the channel compression to obtain the multi-scale feature; Performing an edge feature extraction operation on the third feature after the channel compression to obtain the edge feature; Performing a global feature extraction operation on the third feature after the channel compression to obtain the global background feature; Splicing the multi-scale feature, the edge feature and the global background feature to obtain a second splicing feature; Channel expansion is performed on the second concatenated feature to obtain an output feature corresponding to the third feature.

6. A road rockfall detection device, characterized in that: include: An extraction module is used to extract features of the image to be detected based on the backbone network of the pre-trained rockfall detection model to obtain image features; The image to be detected includes fallen rocks on the road; A fusion module, used for fusing the image features based on the neck network of the rockfall detection model to obtain fusion features; A detection module, used to detect the fused features based on the detection network of the rockfall detection model to obtain a detection result of the road rockfall in the image to be detected; Wherein, the backbone network is provided with an SDF module, and the SDF module is used to fuse the semantic features and detail features extracted by different convolution submodules; The SDF module includes a basic convolution submodule and an SPD convolution submodule; the extraction module includes a first extraction unit, and for the first feature input into the SDF module, the first extraction unit is used to: Inputting the first feature into the basic convolution submodule to extract semantic features of the first feature; Inputting the first feature into the SPD convolution submodule to extract detail features of the first feature; Performing feature fusion based on the semantic feature and the detail feature to obtain an output feature corresponding to the first feature; The first extraction unit is specifically used for: Splicing the semantic features after the compression channel and the detail features after the compression channel to obtain a first splicing feature; Performing a normalization operation on the first splicing feature after channel mapping to obtain a first weight of the semantic feature and a second weight of the detail feature; Performing weighted fusion on the semantic feature and the detail feature based on the first weight and the second weight to obtain a first fused feature; Channel expansion is performed on the first fusion feature to obtain an output feature corresponding to the first feature.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the road rockfall detection method based on deep learning as described in any one of claims 1 to 5 is implemented.

8. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the computer program is executed by a processor, the road rockfall detection method based on deep learning as described in any one of claims 1 to 5 is implemented.

Images