Road slope rockfall detection method based on improved FCOS

Abstract

The invention relates to a road slope rockfall detection method based on improved FCOS, and the method specifically comprises the following steps: 1, improving an FCOS algorithm, introducing a lightweight backbone network to replace an original feature extraction network lightweight network model, reducing network parameters, and training the network model to obtain a network model weight file; 2, quantifying the model into an edge end model of a calculation precision int8 type by using a PTQ quantification method, and deploying the edge end model to a horizon RDK X3 Module module; according to the method, a lightweight backbone network is used as a feature extraction network, network parameters are reduced, a model can be deployed on edge equipment, the model is quantitatively deployed to a horizon RDK X3 Module module for operation, accurate and real-time detection of road slope rockfall is realized, a data frame of an identified rockfall result is uploaded to a server, and the data frame of the rockfall result is analyzed. And the ground quality of the network and the transmission quantity of invalid data are reduced, and the pressure of the server is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of road slope rockfall detection, and in particular, relates to a road slope rockfall detection method based on improved FCOS. Background Art

[0002] my country's terrain is complex and diverse. Some roads or railways are inevitably built next to hillsides. These slopes are affected by factors such as rain and earthquakes, and geological disasters such as landslides and rockfalls may occur, causing traffic jams and even harming vehicles and pedestrians. Therefore, real-time monitoring of rockfall on road slopes is of great practical significance.

[0003] At present, mainstream target detection and recognition algorithms usually rely on cloud servers for model reasoning and prediction. However, this approach is limited by the bandwidth of the communication link, which may lead to high latency and lack of real-time performance in the target detection and recognition process, and fail to respond quickly to sudden road intrusion geological disasters. As the amount of network data continues to grow, a large amount of low-quality or invalid data may exceed the processing capacity of the network and server.

[0004] With the advancement of technology, the performance of embedded microprocessors has been significantly improved, making the research and application of edge computing a hot topic for solving the above problems. Edge computing can process data near the data source, reduce data transmission delays, and improve real-time performance.

[0005] The following technical solutions are also available in the existing technology center to solve the above problems: 1. By using OPPO A5 mobile phone and fill light to collect rock samples in mountainous areas, deep learning models are trained to achieve real-time detection of falling rocks. Finally, the trained model is deployed on the mobile phone, and real-time image processing and rockfall detection are performed through the camera. However, real-time monitoring of falling rocks is not achieved, and manual operation of the smartphone is required to take images, which affects the real-time detection. The manual image shooting method is difficult to achieve long-term continuous monitoring, which limits the continuous application of the monitoring system in rockfall detection; 2. By improving the YOLOv5 model, adding the SimAM attention mechanism and the weighted bidirectional feature pyramid (BiFPN) to improve the detection accuracy, and adding lightweight volume A GSConv lightweight convolution module is introduced at the neck of the model to reduce the amount of model calculation and improve the detection speed, while maintaining the detection accuracy to achieve real-time detection of falling rocks. Although the model has been lightweight, its specific size and resource usage have not been clearly quantified, which makes it difficult to evaluate whether the model is suitable for edge device operation. The model has not yet been deployed and tested on actual edge computing devices, so its performance and reliability in resource-constrained edge environments cannot be determined. The real-time monitoring capability of the model may be limited by the processing power of the edge device, and real-time rockfall monitoring at the edge has not been fully realized.

[0006] In order to overcome the problems existing in the related art, an embodiment of the present invention provides a road slope rockfall detection method based on improved FCOS. Summary of the invention

[0007] The purpose of the present invention is to provide a road slope rockfall detection method based on improved FCOS, which solves the problem of lack of real-time performance of road slope rockfall detection and the problem of continuous growth of network data volume in the prior art.

[0008] The purpose of the present invention can be achieved through the following technical solutions: A road slope detection system based on improved FCOS specifically includes the following steps: Step 1: Improve the FCOS algorithm, introduce a lightweight backbone network to replace the original feature extraction network lightweight network model, reduce network parameters, train the network model to obtain the network model weight file; Step 2: Use the PTQ quantization method to quantize the model into an edge model with a calculation accuracy of int8 and deploy it to the Horizon RDK X3 Module; Step 3: Use a network camera to obtain the road slope video stream, decode the video stream to obtain nv12 format image data, perform model inference on the edge, and obtain the road slope rockfall target detection result; Step 4: Based on the target detection result obtained through the above step 3, if a rockfall is detected, a frame of the detected rockfall is drawn on a frame of the image and the image data is uploaded to the backend server for storage.

[0009] Preferably, in the step 1, the improved FCOS network architecture is replaced by the lightweight backbone network EfficientNet-B0, the backbone feature extraction network Resnet50 of the FCOS network model, the lightweight network model structure, the network parameters are reduced, the maximum number of iterations is set to 200, the batch_size is set to 8, the initial learning rate is 0.001, the classification loss is focal_loss, the positioning loss is giou_loss, the center loss is binary cross entropy loss, the optimizer selects SGD, the learning rate is adaptively adjusted, the weight is saved once every epoch, the model is evaluated with the validation set during training, and is evaluated once every epoch. The training improved FCOS model obtains the pytorch weight file fcos_512 with a calculation accuracy of float32 512.pth.

[0010] Preferably, in step 2, use the PTQ post-training quantization method provided by Horizon to convert the weight file fcos_512 in pytorch format 512.pth is exported as ONNX format opset_version=11 weight file fcos_512 512.onnx, use the horizon hb_mapper checker tool to verify the model to ensure that it meets the support constraints of the horizon processor, prepare 50 calibration sample image inputs in the calibration phase, each sample is an independent data file, and use the hb_mapper makertbin tool to quantize the model into an edge model fcos_512 with a calculation accuracy of int8 type 512.bin, will fcos_512 Copy the 512.bin file to the Horizon RDK X3 Module.

[0011] Preferably, in step 3, use a network cable to connect the wired network port of the Horizon RDK X3 Module and the wired network port of the network camera, configure the network IP address and the camera IP to be in the same network segment, configure the Horizon RDK X3 Module IP address to 192.168.1.100, use the RSTP transmission protocol to pull the camera video stream and decode it to obtain a single frame of image data, convert the image data format to the nv12 image format, perform model forward inference on the nv12 format image data to obtain the target category value, center value and regression box value, and each pixel of the feature map has a corresponding regression target (l, r, t, b) ; Centrality value: .

[0012] Preferably, in step 4, the score value obtained by multiplying the category value cls of each feature pixel by the centerness value and then taking the square root is used as the judgment value of whether the rockfall is detected. The score is a real number between 0 and 1. If the score>0.5, the feature pixel detects the rockfall. After the detection border of the corresponding feature point is drawn on the original image, the image is uploaded to the backend server for storage via the HTTPS protocol. .

[0013] Beneficial effects of the present invention: The present invention proposes a road slope rockfall detection method based on improved FCOS, which uses a lightweight backbone network as a feature extraction network, reduces network parameters, and enables the model to be deployed on edge devices. The model is quantified and deployed to the Horizon RDK X3 Module for operation, thereby achieving accurate and real-time detection of road slope rockfall, uploading data frames with identified rockfall results to a server, reducing network ground quality and invalid data transmission volume, and reducing server pressure. The present invention can effectively improve the accuracy and real-time performance of road slope rockfall detection, and reduce network data transmission volume and server storage pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0015] Figure 1 A flow chart of a road slope rockfall detection method based on improved FCOS according to the present invention; Figure 2 This is a diagram of the improved FCOS network architecture in a road slope rockfall detection method based on improved FCOS in the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1-Figure 2 As shown, the present invention is a road slope rockfall detection method based on improved FCOS, which specifically includes the following steps: Step 1: Improve the FCOS algorithm, introduce a lightweight backbone network to replace the original feature extraction network lightweight network model, reduce network parameters, train the network model to obtain the network model weight file; In the step 1, the improved FCOS network architecture is used to replace the backbone feature extraction network Resnet50 of the FCOS network model with the lightweight backbone network EfficientNet-B0, to lightweight the network model structure, reduce network parameters, set the maximum number of iterations to 200, batch_size to 8, the initial learning rate to 0.001, the classification loss uses focal_loss, the positioning loss uses giou_loss, the center loss uses binary cross entropy loss, the optimizer selects SGD, the learning rate is adaptively adjusted, the weight is saved once every epoch, the model is evaluated with the validation set during training, and is evaluated once every epoch. The pytorch weight file fcos_512 with a calculation accuracy of float32 is obtained by training the improved FCOS model. 512.pth.

[0018] Step 2: Use the PTQ quantization method to quantize the model into an edge model with a calculation accuracy of int8 and deploy it to the Horizon RDK X3 Module; In step 2, the PTQ post-training quantization method provided by Horizon is used to convert the weight file fcos_512 in pytorch format 512.pth is exported as ONNX format opset_version=11 weight file fcos_512 512.onnx, use the horizon hb_mapper checker tool to verify the model to ensure that it meets the support constraints of the horizon processor, prepare 50 calibration sample image inputs in the calibration phase, each sample is an independent data file, and use the hb_mapper makertbin tool to quantize the model into an edge model fcos_512 with a calculation accuracy of int8 type 512.bin, will fcos_512 Copy the 512.bin file to the Horizon RDK X3 Module.

[0019] Step 3: Use a network camera to obtain the road slope video stream, decode the video stream to obtain nv12 format image data, perform model inference on the edge, and obtain the road slope rockfall target detection result; In step 3, use a network cable to connect the wired network port of the Horizon RDK X3 Module and the wired network port of the network camera, configure the network IP address and the camera IP to be in the same network segment, configure the IP address of the Horizon RDK X3 Module to 192.168.1.100, use the RSTP transmission protocol to pull the camera video stream and decode it to obtain single-frame image data, convert the image data format to nv12 image format, perform model forward inference on the nv12 format image data to obtain the target category value, center value and regression box value, and each pixel of the feature map has a corresponding regression target (l, r, t, b) ; Centrality value: .

[0020] Step 4: Based on the target detection result obtained through the above step 3, if a rockfall is detected, a frame of the detected rockfall is drawn on a frame of the image and the image data is uploaded to the backend server for storage.

[0021] In step 4, the score value obtained by multiplying the category value cls of each feature pixel by the centerness value and then taking the square root is used as the judgment value of whether the rockfall is detected. The score is a real number between 0 and 1. If the score>0.5, the feature pixel detects the rockfall. After the detection border of the corresponding feature point is drawn on the original image, the image is uploaded to the backend server for storage through the HTTPS protocol; .

[0022] Example 1 Experimental purpose: To verify whether the improved FCOS algorithm can maintain or improve the performance of the model while reducing network parameters; Experimental steps: Replace the backbone feature extraction network Resnet50 of the FCOS network model with the lightweight backbone network EfficientNet-B0, set the maximum number of iterations to 200, batch_size to 8, the initial learning rate to 0.001, the classification loss uses focal_loss, the positioning loss uses giou_loss, the center loss uses binary cross entropy loss, the optimizer selects SGD and adaptively adjusts the learning rate; use the prepared training data set to train the improved model, save the weights once every epoch, and use the validation set as the evaluation object, and evaluate once every epoch.

[0023] Experimental results: After 200 iterations of training, the pytorch weight file fcos_512 with a calculation accuracy of float32 is obtained 512.pth, compared with the original FCOS model, the network parameters are greatly reduced, while indicators such as accuracy and recall on the validation set remain stable or improve.

[0024] Example 2 Experimental purpose: To verify whether the PTQ quantization method can successfully quantize the model into an edge model of type int8, and ensure that it runs normally on the Horizon RDK X3 Module.

[0025] Experimental steps: Use the PTQ post-training quantization method provided by Horizon to export the weight file fcos_512512.pth in pytorch format to the weight file fcos512512.onnx in ONNX format with opset_version=11. Use the Horizon hb_mapper checker tool to verify the converted model to ensure that it meets the support constraints of the Horizon processor. Prepare 50 calibration sample image inputs and use the hb_mapper makertbin tool to quantize the model to an edge model fcos_512 with a calculation accuracy of int8 type 512.bin. Will fcos_512 Copy the 512.bin file to the Horizon RDKX3 Module.

[0026] Experimental results: After the above steps, the model was successfully quantized and deployed to the Horizon RDK X3 Module. By performing simple tests on the module, the model was able to load and run normally, preparing for subsequent edge inference.

[0027] Example 3 Purpose of the experiment: To verify the accuracy and real-time performance of model reasoning at the edge, as well as the data processing and uploading functions after rockfall is detected.

[0028] Experimental steps: Use a network cable to connect the Horizon RDK X3 Module and the network camera, configure the network IP address so that they are in the same network segment, where the Horizon RDK X3 Module IP address is configured as 192.168.1.100, use the RSTP transmission protocol to pull the camera video stream, decode it to obtain single-frame image data, and convert it into nv12 image format; perform model forward inference on the nv12 format image data to obtain the target category value, centerness value and regression box value; multiply the category value cls of each feature pixel by the centerness value centerness and take the square root to obtain the score value. If the score>0.5, it is considered that the feature pixel detects rockfall, and the detection border of the corresponding feature point is drawn on the original image; upload the picture with the rockfall frame drawn to the backend server for storage through the HTTPS protocol.

[0029] Experimental results: In the actual road slope video stream test, the system can accurately detect falling rocks in real time and upload pictures with falling rock frames to the back-end server in a timely manner. It also shows good stability and robustness under different lighting and weather conditions.

[0030] The technical problems solved by the present invention through the above technical solution are as follows: 1. The FCOS algorithm model was improved, and a lightweight backbone network was introduced to replace the original feature extraction network lightweight network model, reducing network model parameters; 2. The improved FCOS algorithm model is quantitatively deployed to the embedded development board to achieve rapid and accurate target identification, reduce dependence on the central server, and improve the system's response speed and processing capabilities to emergencies; 3. The data results of rockfall detection are uploaded to the back-end server for storage, reducing the amount of invalid data transmission on the network and the storage pressure on the server.

[0031] In the description of the specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0032] The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A road slope rockfall detection method based on improved FCOS, characterized by: The specific steps include: Step 1: Improve the FCOS algorithm, introduce a lightweight backbone network to replace the original feature extraction network lightweight network model, reduce network parameters, train the network model to obtain the network model weight file; Step 2: Use the PTQ quantization method to quantize the model into an edge model with a calculation accuracy of int8 and deploy it to the Horizon RDK X3 Module; Step 3: Use a network camera to obtain the road slope video stream, decode the video stream to obtain nv12 format image data, perform model inference on the edge, and obtain the road slope rockfall target detection result; Step 4: Based on the target detection result obtained through the above step 3, if a rockfall is detected, a frame of the detected rockfall is drawn on a frame of the image and the image data is uploaded to the backend server for storage.

2. The method for detecting rockfall on a road slope based on improved FCOS according to claim 1, characterized in that: In the step 1, the FCOS network architecture is improved, and the backbone feature extraction network Resnet50 of the FCOS network model is replaced with the lightweight backbone network EfficientNet-B0, so as to lighten the network model structure and reduce the network parameters.

3. The method for detecting rockfall on a road slope based on improved FCOS according to claim 2, characterized in that: During the training of the improved FCOS network architecture, the maximum number of iterations is set to 200, the batch_size is set to 8, the initial learning rate is 0.001, the classification loss uses focal_loss, the positioning loss uses giou_loss, the center loss uses binary cross entropy loss, the optimizer selects SGD, the learning rate is adaptively adjusted, the weight is saved once per epoch, the model is evaluated on the validation set during training, and is evaluated once per epoch. The training of the improved FCOS model obtains the pytorch weight file fcos_512 with a calculation accuracy of float32 512.pth.

4. The method for detecting rockfall on road slopes based on improved FCOS according to claim 1, characterized in that: In step 2, the PTQ post-training quantization method provided by Horizon is used to convert the weight file fcos_512 in pytorch format 512.pth is exported as ONNX format opset_version=11 weight file fcos_512 512.onnx, use the horizon hb_mapper checker tool to verify the model to ensure that it meets the support constraints of the horizon processor, prepare 50 calibration sample image inputs in the calibration phase, each sample is an independent data file, and use the hb_mappermakertbin tool to quantize the model into an edge model fcos_512 with a calculation accuracy of int8 type 512.bin, will fcos_512 Copy the 512.bin file to the Horizon RDK X3 Module.

5. The method for detecting rockfall on road slopes based on improved FCOS according to claim 4, characterized in that: Prepare 50 calibration sample image inputs in the calibration phase. Each sample is an independent data file. Use the hb_mappermakertbin tool to quantize the verified model into an edge model fcos_512512.bin of type int8 calculation accuracy, and copy the fcos_512512.bin file to the Horizon RDK X3 Module module.

6. The method for detecting rockfall on road slopes based on improved FCOS according to claim 1, characterized in that: In step 3, use a network cable to connect the wired network port of the Horizon RDK X3 Module and the wired network port of the network camera, configure the network IP address and the camera IP to be in the same network segment, configure the IP address of the Horizon RDK X3 Module to 192.168.1.100, use the RSTP transmission protocol to pull the camera video stream and decode it to obtain single-frame image data, convert the image data format to nv12 image format, perform model forward inference on the nv12 format image data to obtain the target category value, center value and regression box value, and each pixel of the feature map has a corresponding regression target (l, r, t, b) ； Centrality value: 。 7. The method for detecting rockfall on road slopes based on improved FCOS according to claim 1, characterized in that: In step 4, the score value obtained by multiplying the category value cls of each feature pixel by the centerness value and then taking the square root is used as the judgment value of whether the rockfall is detected. The score is a real number between 0 and 1. If the score>0.5, the feature pixel detects the rockfall. After the detection border of the corresponding feature point is drawn on the original image, the image is uploaded to the backend server for storage through the HTTPS protocol; 。

Images