Unmanned aerial vehicle underground personnel tracking method based on state estimation and light sequential serial network

Abstract

This invention discloses a method for tracking personnel underground using unmanned aerial vehicles (UAVs) based on state estimation and lightweight serial networks, belonging to the fields of mine rescue and computer vision technology. This invention addresses the problems of high explosion-proof requirements, limited hardware computing power, strong interference from underground dust and light, and easy target loss for mining UAVs. The specific steps are as follows: S1. Collect images of personnel underground in multiple scenes using a mining UAV, construct a dedicated dataset, and complete segmentation and preprocessing; S2. Build a LiteSerialNet detection model composed of MobileNetV3Small, a channel alignment module, a lightweight SerialFPN, and a localization module, reducing the number of parameters and computation through lightweight design; S3. Train the model using optimized parameters to obtain a stable underground personnel detection model; S4. Deploy the model to an airborne platform for real-time detection of underground personnel. When detection fails, position prediction is performed using Kalman filtering to ensure continuous tracking. This invention enables automatic, accurate, and stable tracking of personnel underground by UAVs, reducing deployment costs and power consumption, and providing efficient and reliable technical support for mine emergency rescue.

Description

Technical Field

[0001] This invention belongs to the field of mine rescue and computer vision technology, specifically relating to a method for tracking personnel underground by unmanned aerial vehicles (UAVs) based on state estimation and light-order serial networks. Background Technology

[0002] As mining depths continue to increase, underground tunnel structures become more complex and working environments harsher, significantly increasing the difficulty of emergency rescue. Tracking underground personnel is a core technology for mine emergency rescue. In the event of a sudden accident, quickly and accurately locating miners and understanding their distribution is a crucial prerequisite for shortening rescue time and improving the success rate of rescue efforts.

[0003] Traditional manual search and rescue methods are inefficient and constrained by the complex underground environment, making it impossible to achieve all-weather, full-coverage searches and failing to meet the actual needs of mine emergency rescue. Drones, with their advantages of maneuverability, wide range, and rapid access to dangerous areas, combined with the non-contact monitoring and real-time capabilities of computer vision, are gradually being applied to the search and location of personnel underground.

[0004] However, existing technologies for tracking personnel in mines still have significant shortcomings: CN116778532B uses the Harris corner detection algorithm for personnel tracking, but suffers from insufficient detection accuracy and poor robustness, making it unable to achieve precise tracking. CN108182413B, based on Faster R-CNN, and CN121564031A, based on YOLOv8, achieve tracking of moving targets in mines; however, when applied to the underground operation scenarios of mining drones, the following prominent problems exist: First, mining drones must meet underground explosion-proof safety requirements, and the onboard chips must pass explosion-proof safety certification, resulting in high design and manufacturing costs and limited computing power and power consumption. The above methods lack lightweight design, have large model sizes and high computational loads, making them difficult to deploy efficiently on explosion-proof drone platforms and exhibiting poor adaptability. Second, the high dust concentration, uneven lighting, and complex environment underground can easily cause target loss and unstable tracking, failing to meet the requirements for use in mine emergency rescue. Summary of the Invention

[0005] The purpose of this invention is to provide a method for tracking personnel in underground mines using unmanned aerial vehicles (UAVs) based on state estimation and LiteSerialNet, in order to solve the technical problems of high hardware cost, high power consumption, easy target loss and poor tracking stability caused by underground dust interference, uneven lighting, and complex occlusion when the existing technology is applied to personnel tracking scenarios of underground explosion-proof UAVs.

[0006] This invention uses the lightweight network MobileNetV3-Small as the backbone feature extraction module, and combines it with the lightweight SerialFPN structure to perform multi-scale semantic feature fusion to construct a lightweight LiteSerialNet network. At the same time, Kalman is introduced to realize temporal state estimation and trajectory compensation, forming a lightweight, efficient, stable and robust tracking architecture, and finally realizing accurate and continuous tracking of personnel by mining drones in complex underground environments.

[0007] To achieve the above-mentioned objectives, the present invention proposes a method for tracking personnel in underground wells using unmanned aerial vehicles (UAVs) based on state estimation and LiteSerialNet, which specifically includes the following steps:

[0008] Step S1: Construct a dedicated dataset for underground personnel using unmanned aerial vehicles (UAVs). Use mining UAVs to perform flight data collection in real underground roadway environments, acquiring visual images of typical interference scenarios such as high dust levels, low light, partial occlusion, and complex roadways. Standardize and organize the underground personnel targets in the images to construct a dedicated dataset for underground personnel suitable for UAV perspective.

[0009] Step S2, build a visual detection model for downhole personnel based on LiteSerialNet network: The model consists of a lightweight feature extraction module MobileNetV3-Small, a channel alignment module, a SerialFPN module and a localization module, and the specific settings of each module are as follows;

[0010] S2.1, the MobileNetV3-Small model adopts a minimized model to adapt to the lightweight deployment requirements of UAVs under explosion-proof conditions. The feature extraction process for underground personnel in this model is as follows: first, a 3×3 Conv2d layer is used for dimensionality reduction and an hswish activation function is applied; then, three 3×3 bneck layers are used for continuous dimensionality reduction and a compression activation mechanism is added in the first layer; then, eight 5×5 bneck layers are used, with each layer having a compression activation mechanism and an hswish activation function; finally, three 1×1 Conv2d layers and one 7×7 pooling layer are used to complete feature extraction. The compression activation mechanism fixes the number of dimensionality-increasing channels that originally varied with the layers to 1 / 4. The MobileNetV3-Small model adopts a lightweight design at both ends. The number of channels in the first 3×3 Conv2d layer is reduced to 16, and the tail layer is first converted to a 1×1 scale by global average pooling before dimensionality increase to reduce the amount of computation.

[0011] The channel alignment module described in S2.2 is used to solve the problem that the inconsistent number of channels in the output layer of MobileNetV3-Small feature extraction makes it difficult to achieve subsequent feature fusion and parameter sharing. This module connects a 1×1 convolutional layer to the four feature maps of different scales output by the previous module, adjusts the number of channels and fixes them to 16 without changing the feature map size information, thereby achieving channel alignment and dimensionality reduction.

[0012] S2.3 The SerialFPN module is a lightweight serial feature pyramid fusion structure. Compared with traditional FPN, it does not use bidirectional sampling to avoid increasing the amount of computation and parameters. The SerialFPN module samples down layer by layer starting from the lowest layer P2. Each level is fused with the original features of the same scale and processed by the enhancement submodule before being used as the input for the next level, forming a unidirectional serial structure P2→P3→P4→P5. P2 to P5 correspond to 1 / 2 to 1 / 5 of the original input scale, respectively. The enhancement submodule is SCConv, which consists of three parallel convolutional branches: a global perception branch k2, an original feature extraction branch k3, and an output transformation branch k4. K2 generates a spatial attention weight map, and K3 obtains the basic feature map. The two types of feature maps are multiplied element-wise to achieve spatial dimension self-calibration and feature enhancement. Finally, the calibrated and enhanced feature map is obtained by the 3×3 convolution output of the output transformation branch k4. The feature fusion submodule adopts an adaptive feature fusion method. First, each input feature map is processed by a 1×1 Convolution normalizes and purifies features while keeping the size and number of channels unchanged. Then, the multi-path features are concatenated along the channel dimension to form aggregated features. The concatenated features are fed into a 1×1 convolution and Softmax layer to output the normalized weights of each feature. Finally, each feature is multiplied by its corresponding weight and then accumulated and fused along the channel to output the final fused feature map.

[0013] S2.4 The localization module is divided into three scales: P3, P4, and P5. It obtains input from the feature enhancement sub-modules corresponding to P3, P4, and P5 of the SerialFPN module and completes parameter analysis respectively.

[0014] Step S3: Set initial training parameters; input the dataset constructed in step S1 into the LiteSerialNet network model designed in step S2 for training and parameter tuning; stabilize the detection accuracy within the preset range to obtain the underground personnel detection model based on LiteSerialNet;

[0015] Step S4: Perform underground personnel tracking; The underground personnel detection model based on LiteSerialNet trained in Step S3 is used to process the real-time underground video input by the UAV, and the processing result is used as the flight control input to drive the UAV to complete the underground personnel tracking; This step first determines whether to start the tracking task. If it is started, it further determines whether the detection model outputs a valid result. If there is a valid detection result, the model output is used directly. If no target is detected, the position of the underground personnel is estimated by Kalman filtering to avoid target loss and the estimated position is output to support the UAV's continuous tracking. Then, the UAV enters the loop tracking of the next frame image.

[0016] The method for predicting the location of personnel in the mine using Kalman filtering is as follows: First, extract the state S of the personnel in the mine. o = [x, y, w, h, dx, dy], and use uniform dynamic motion to estimate the state transition matrix A, realizing the temporal recursion of new states:

[0017]

[0018] in For the new target state of downhole personnel, update the covariance:

[0019]

[0020] in For the new covariance, Let Q be the initial covariance and Q be the process noise. Next, determine if there are any personnel detection results in step S3. If so, first calculate the Kalman gain K:

[0021]

[0022] Where R is the observation noise covariance matrix, B is the observation matrix, and feature fusion is used to correct the new state and covariance:

[0023]

[0024] in The updated target state of the downhole personnel is represented by O, which is the observed state of the downhole personnel based on the LiteSerialNet-based downhole personnel detection model. The covariance is also updated.

[0025]

[0026] Where P NR Let I be the updated covariance, and let I be the identity matrix.

[0027] Compared with the prior art, the present invention has the following positive effects:

[0028] This invention employs the lightweight MobileNetV3-Small model for feature extraction and uses the lightweight serial feature pyramid fusion structure SerialFPN as the semantic understanding module. It abandons the multi-branch mode of repeated upsampling and downsampling in the traditional double pyramid structure. At the same time, it uses the lightweight feature enhancement module SCConv to enhance features. Through multi-level lightweight design, the number of model parameters and computational load are effectively reduced, which can better adapt to the edge deployment requirements of mining drones in high explosion-proof scenarios.

[0029] This invention introduces a time-series state prediction method based on Kalman filtering to predict the location of personnel in underground mines. When the personnel detection model based on LiteSerialNet network fails, the time-series state prediction method can be used to continuously track personnel in underground mines, avoiding the inability of the UAV to continue the tracking task due to target loss. Attached Figure Description

[0030] Figure 1 This invention provides a method for tracking personnel in underground wells using unmanned aerial vehicles (UAVs) based on state estimation and light-order serial networks.

[0031] Figure 2 This is a network structure diagram of LiteSerialNet in this invention;

[0032] Figure 3 This invention presents a time-series state-based method for predicting the location of personnel in underground mines. Detailed Implementation

[0033] To make the technical solutions, beneficial effects, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. These embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments disclosed in this invention without creative effort are within the scope of protection claimed by this invention.

[0034] Unless otherwise defined, the technical or scientific terms used in this invention shall have the meanings commonly understood by one of ordinary skill in the art to which this invention pertains. Example

[0035] As shown in Figure 1, this invention proposes a method for tracking personnel in underground wells using unmanned aerial vehicles (UAVs) based on state estimation and light-order serial networks. The specific implementation process is as follows:

[0036] S1: Dataset Creation for Mine Personnel:

[0037] Mining drones were used to perform daily underground patrols, collecting video data of underground personnel under four typical scenarios: different lighting conditions, different dust obstructions, water mist interference, and other obstructions. Professional technicians screened the raw data, selecting valid videos as the original data. The LabelImg tool was used to label the location and category of underground personnel in the images, constructing an initial dataset containing 13,658 images. The dataset was then divided into training, validation, and test sets in a 7:2:1 ratio, and preprocessing operations such as data augmentation were performed.

[0038] S2: LiteSerialNet network model setup:

[0039] The LiteSerialNet network model structure described in this invention is shown in Figure 2. After the UAV video signal is input, the feature extraction is first completed by the lightweight module MobileNetV3-Small, then the number of feature map channels is unified by the channel alignment module, and then the semantic understanding of the personnel in the well is realized by the SerialFPN module. Finally, the relevant information of the personnel in the well is output by the positioning module, providing data support for UAV tracking.

[0040] Figure 2 shows that the numbers 1-18 within the rectangle are submodule numbers. The format after the rectangle is [Usage: Input submodule number: scale information]. For example, S:5:P2 indicates that SCConv processing is used, the input comes from submodule 5, and the current scale is P2. Other identifiers in the figure include C for Conv downsampling and A for adaptive fusion.

[0041] The feature extraction module MobileNetV3-Small contains 16 operation layers. The figure illustrates feature maps at five scales, 0-4, corresponding to scales P1 to P5. It includes four conv2d layers: a 3×3 convolution in the first layer and three 1×1 convolutions in layers 13, 15-16. Except for the 16th convolution layer, all other layers use the h-swish activation function. The MobileNetV3-Small feature extraction module also includes 11 bneck operation layers and a pooling layer in layer 14. Layers 2-4 consist of three 3×3 convolutional bnecks, and layers 5-12 consist of eight 5×5 convolutional bnecks. All bnecks are equipped with a compression activation mechanism and the h-swish activation function.

[0042] The channel alignment module consists of sub-modules 5-8, and the inputs are the feature maps of scales P2-P5 output by sub-modules 1-4. Each branch uses a 1×1 Conv convolutional layer for transformation to align the number of channels of the feature maps to 16.

[0043] The SerialFPN semantic understanding module consists of submodules 9-18, performing semantic understanding at four scales: P2, P3, P4, and P5. Submodule 9 takes submodule 5 as input, corresponds to the P2 scale, and uses SCConv for feature enhancement. Submodules 10-12 correspond to the P3 scale. Submodule 10 takes submodule 9 as input and uses downsampling Conv to convert from P2 to P3 scales. Submodule 11 uses adaptive weight fusion to fuse the input information from submodules 6 and 10, and submodule 12 performs feature enhancement. Submodules 13-15 correspond to the P4 scale. Submodule 13 takes submodule 12 as input and uses downsampling Conv to convert from P3 to P4 scales. Submodule 14 uses adaptive weight fusion to fuse the input information from submodules 7 and 13, and submodule 15 performs feature enhancement. Submodules 16-18 correspond to the P5 scale. Submodule 16 takes submodule 5 as input and uses downsampling Conv to convert from P3 to P4 scales. Submodule 14 uses adaptive weight fusion to fuse the input information from submodules 7 and 13, and submodule 15 performs feature enhancement. Submodules 16-18 correspond to the P5 scale. Submodule 16 takes submodule 5 as input and uses downsampling Conv to convert from P2 to P4 scales. Submodule 14 uses adaptive weight fusion to fuse the input information from submodules 7 and 13, and submodule 15 performs feature enhancement. The input is the No. 8 submodule, which is converted from P4 to P5 scale through downsampling Conv. The No. 17 submodule uses an adaptive weight fusion method to fuse the input information of the No. 8 and No. 16 submodules, and the No. 18 submodule completes the feature enhancement.

[0044] The positioning module is set with three detection scales: P3, P4, and P5. The input for scale P3 comes from submodule 12, the input for scale P4 comes from submodule 15, and the input for scale P5 comes from submodule 18.

[0045] S3: Training based on the LiteSerialNet network model:

[0046] The LiteSerialNet network model was used for training. The specific training parameters were set as follows: SGD optimizer was selected, batch size was set to 32, total number of training rounds was set to 200, number of workers was set to 2, base learning rate was set to 0.01, momentum parameter was set to 0.937, weight decay coefficient was set to 0.0005, and 5 training warm-up rounds were set to stabilize the model in the early training process.

[0047] S4: Model Deployment and Downhole Personnel Tracking:

[0048] The trained LiteSerialNet network model is deployed to the airborne platform of a mining drone to perform the task of tracking personnel underground. First, it is determined whether the drone has started the tracking task. If it has started, it is further determined whether the LiteSerialNet model has detected personnel underground. If the detection is valid, the detection result is directly output. If no target is detected, the position of personnel underground is estimated by Kalman filtering, and the position and category information of personnel are output. Then, the next frame image is entered and the tracking task status is determined again, forming a loop execution process. If the tracking task terminates, the loop is exited.

[0049] The time-series state-based method for predicting the location of personnel in the well is shown in Figure 3. First, the target state of the personnel in the well is extracted. Then, the new personnel state is obtained through time-series recursion and the covariance is updated. If the LiteSerialNet model outputs a valid detection result, the Kalman gain is calculated and the target state of the personnel in the well is corrected in real time.

[0050] All contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0051] In summary, this invention provides a method for tracking personnel underground using unmanned aerial vehicles (UAVs) based on state estimation and lightweight sequential networks. By building a lightweight network architecture, the deployment cost and power consumption of mining UAVs are reduced. At the same time, a time-series state and position prediction method based on Kalman filtering is used to solve the problem of UAV loss of control due to target loss underground. This method can realize automatic tracking and detection of personnel underground by UAVs in mining environments, providing reliable technical support for emergency rescue in mining accidents.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for tracking personnel in underground wells using unmanned aerial vehicles (UAVs) based on state estimation and light-order serial networks, characterized in that, Including the following steps: S1: Construct a dedicated dataset for mine personnel, using mining drones to collect images of underground personnel and complete annotation, segmentation, and preprocessing; S2: Build the LiteSerialNet network model, which consists of the MobileNetV3Small feature extraction module, the channel alignment module, the SerialFPN semantic understanding module, and the localization module in sequence; S3 trains the LiteSerialNet network model, sets training parameters to complete iterative optimization of the model, and obtains the underground personnel detection model; S4 deploys the detection model to mining drones and performs tracking tasks. During the tracking process, it combines Kalman filtering to achieve temporal state and position prediction, ensuring continuous target tracking.

2. The method according to claim 1, characterized in that, The MobileNetV3Small described in step S2 adopts a lightweight minimum model structure. By optimizing the convolution channel configuration, improving the compression activation mechanism, and using the hswish activation function, it achieves a balance between feature extraction efficiency and model lightweighting, effectively reducing the number of parameters and computational cost.

3. The method according to claim 1, characterized in that, In step S2, the channel alignment module uses a 1×1 convolutional layer to align the number of channels to 16 for the four scale feature maps from P2 to P5, thereby eliminating obstacles to subsequent feature fusion and parameter sharing caused by inconsistent channel numbers.

4. The method according to claim 1, characterized in that, The SerialFPN module described in step S2 is a lightweight serial feature pyramid structure, which adopts a unidirectional serial fusion method of P2→P3→P4→P5, abandoning the traditional bidirectional upsampling and downsampling to reduce the amount of computation and parameters. The module uses SCConv for feature enhancement, and realizes spatial self-calibration and feature enhancement through global perception branch, original feature extraction branch and output transformation branch. It also adopts an adaptive feature fusion method to complete the weighted fusion of multiple features.

5. The method according to claim 1, characterized in that, The positioning module in step S2 uses three detection scales, P3, P4, and P5, to obtain feature inputs from the feature enhancement sub-modules of the corresponding levels of the SerialFPN module, thereby realizing multi-scale downhole personnel positioning output.

6. The method according to claim 1, characterized in that, The time-series state-based downhole personnel position prediction method described in step S4 is as follows: First, extract the state information related to the target position, size, and movement speed of the downhole personnel. Then, perform time-series recursion based on the uniform motion model to update the target state and covariance. When the LiteSerialNet detection model has valid detection results, calculate the Kalman gain and correct the target state. When detection fails, continue to predict the downhole personnel position to avoid target loss and tracking interruption.

Images