Microminiature unmanned aerial vehicle target detection method based on computational energy 1684x and YOLOv11 algorithm

Abstract

The invention relates to a micro unmanned aerial vehicle target detection method based on computational energy 1684x and a YOLOv11 algorithm, and belongs to the technical field of unmanned aerial vehicles. According to the method, a PyTorch framework is adopted to train a YOLOv11 algorithm on an accelerator card by using an unmanned aerial vehicle data set; the trained weight pt file is converted into a bmodel capable of running on a computing energy 1684x accelerator card; a processor adopted by the model reasoning hardware environment is a Feiteng CPU, and a necessary toolkit is installed for realizing reasoning deployment of the YOLOv11 model on a ''Feiteng CPU + energy computing NPU'' platform; and then model reasoning is carried out. According to the invention, a new model YOLOv11 is adapted to a domestic platform for the first time, and real-time detection of the micro unmanned aerial vehicle is realized by adopting the YOLOv11 model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicles, and particularly relates to a micro unmanned aerial vehicle target detection method based on an algorithm 1684x and a YOLOv11 algorithm. BACKGROUND

[0002] With the popularity of small unmanned aerial vehicles, the risk of their misuse in the fields of civil, industrial and security is increasingly prominent, and efficient target detection technology is urgently needed for real-time monitoring and countermeasures. Traditional detection methods mainly rely on radar, radio spectrum analysis or acoustic sensors, but face significant challenges in complex urban environments: radar's reflection signal of micro unmanned aerial vehicles is weak, easily affected by building shielding and electromagnetic interference; radio and acoustic detection are greatly disturbed by environmental noise, have low positioning accuracy and cannot provide visual information. Computer vision-based detection methods have gradually become a research hotspot, but micro unmanned aerial vehicle targets have the characteristics of small pixel ratio (only a few pixels at a distance), fast motion (easy to produce motion blur), and single morphological characteristics (lack of significant texture or color difference), and traditional image processing algorithms have high false detection rates in complex dynamic environments and are difficult to adapt to changes in lighting, similar background interference and other problems.

[0003] In recent years, with the rapid development of deep learning technology, target detection algorithms based on deep learning are a feasible means to realize micro unmanned aerial vehicle detection. There are two types of first-order detection methods and second-order detection methods based on deep learning target detection algorithms. First-order detection methods do not need to find candidate regions separately, so they are faster, such as SSD, YOLO series, etc. The second-order detection method divides the detection algorithm into two steps, first finds the candidate region, and then classifies it. The accuracy is generally better than the first-order method, such as R-CNN, Fast R-CNN, etc. In practical applications, the collected unmanned aerial vehicle video stream needs to be detected in real time, and the second-order algorithm is difficult to meet the real-time requirements, while the YOLO series algorithm has been updated rapidly in recent years, and its test effect on mainstream datasets is better than that of the SSD algorithm. YOLO algorithm does not need to model the target and background complexly, and the construction process is relatively simple, has strong generalization ability, develops very quickly, and has certain advantages in recognition speed and accuracy, and has been applied to multiple scenes, such as initial fire detection, and has become a research hotspot in target detection.

[0004] The current YOLO algorithm has been updated to the 11th version YOLOv11, which introduces a dynamic label assignment strategy, which can adaptively adjust the positive and negative sample matching threshold according to the target scale (such as the pixel ratio of a small unmanned aerial vehicle), avoiding the micro target missing detection problem caused by the traditional fixed threshold; at the same time, the improved multi-scale feature fusion network enhances the detail information (such as unmanned aerial vehicle rotor, outline) of the micro target in the shallow feature map through cross-layer connection and attention mechanism, and improves the detection sensitivity in high-altitude or long-distance scenes. In addition, YOLOv11 can greatly compress the model parameter quantity while maintaining the detection accuracy through the reparameterization network structure and channel dynamic pruning, supporting real-time operation in embedded devices. Combined with the adaptive background suppression mechanism, YOLOv11 can suppress the interference of similar dynamic backgrounds such as clouds and birds through feature map weighting, focus on the discriminative features of the unmanned aerial vehicle target, and reduce the false alarm rate in complex environments.

[0005] Therefore, the application proposes to use the YOLOv11 algorithm to realize real-time detection of small unmanned aerial vehicles, and deploy it on the combination of "Kylin operating system + Feiteng S2500 + algorithm energy 1684x" of domestic hardware. SUMMARY

[0006] (1) Technical problem to be solved

[0007] The technical problem to be solved by the application is how to provide a small unmanned aerial vehicle target detection method based on algorithm energy 1684x and YOLOv11 algorithm to solve the problem of small unmanned aerial vehicle target detection.

[0008] (2) Technical scheme

[0009] In order to solve the above technical problems, the application proposes a small unmanned aerial vehicle target detection method based on algorithm energy 1684x and YOLOv11 algorithm, which comprises the following steps:

[0010] S1, using the PyTorch framework to train the YOLOv11 algorithm on the acceleration card using the unmanned aerial vehicle data set;

[0011] S2, converting the trained weight pt file into a bmodel model that can run on the algorithm energy 1684x acceleration card;

[0012] S3, model inference dependent environment configuration: the processor used in the model inference hardware environment is Feiteng CPU, in order to realize the inference deployment of YOLOv11 model on the "Feiteng CPU + algorithm NPU" platform, install the necessary tool package;

[0013] S4, model inference: model inference includes model loading, data preprocessing, prediction and post-processing.

[0014] (III) Beneficial Effects

[0015] The present application proposes a micro unmanned aerial vehicle target detection method based on the algorithm energy 1684x and YOLOv11 algorithm, the present application proposes a micro unmanned aerial vehicle target detection method based on the algorithm energy 1684x and YOLOv11 algorithm, the present application first adapts the new model YOLOv11 on the domestic platform, wherein not only the YOLOv11 model is converted from the pt file to the bmodel model that can run on the algorithm energy 1684x accelerator card, but also the necessary tool package is installed through cross-compilation and other technical means, so as to realize the deployment of the YOLOv11 model on the combination of the Kirin operating system+Feiteng S2500+algorithm energy 1684x domestic hardware, and then the YOLOv11 model is used to realize the real-time detection of the micro unmanned aerial vehicle, and the detection efficiency and accuracy are improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is the overall architecture diagram of the tpu_mlir of the present application;

[0017] Figure 2 It is the inference effect diagram. DETAILED DESCRIPTION

[0018] In order to make the purpose, content and advantages of the present application clearer, the specific embodiments of the present application are described in further detail below in combination with the drawings and examples.

[0019] The present application proposes a micro unmanned aerial vehicle target detection method based on the algorithm energy 1684x and YOLOv11 algorithm, comprising the following steps:

[0020] S1, using the PyTorch framework to train the YOLOv11 algorithm on the Nvidia acceleration card using the unmanned aerial vehicle data set;

[0021] S2, converting the trained weight pt file into a bmodel model that can run on the algorithm energy 1684x accelerator card;

[0022] S3, model inference dependent environment configuration: the processor used in the model inference hardware environment is Feiteng CPU, in order to realize the inference deployment of the YOLOv11 model on the "Feiteng CPU+algorithm NPU" platform, the necessary tool package is installed;

[0023] S4, model inference: model inference includes: model loading, data preprocessing, prediction and post-processing.

[0024] In step S1, the unmanned aerial vehicle dataset is 114 video clips developed autonomously. The videos are shot in the field, and the content is the process of unmanned aerial vehicle flight, each video is 10s long; 70% (80) of the video clips are randomly selected as the training set, 20% (23) of the video clips are randomly selected as the verification set, and 10% (11) of the video clips are randomly selected as the test set. The preprocessing steps of the dataset include video framing and picture labeling.

[0025] In video framing, OpenCV is used to read a single video, the video frame count interval frequency is set to 1, and the pictures read are saved in a preset folder. Since the frame rate of each video is 30 frames / s and the length of each video is 10 s, 300 pictures can be generated for each video, and the resolution of each picture is 640*512.

[0026] In picture labeling, the labelImg software is used to frame and label the unmanned aerial vehicle in the picture, the labeling format is selected as YOLO mode, each picture corresponds to a txt label file, and the file content format is <object-class> <x> <y> <width> <height>where x, y are the center coordinates of the target, width, height are the width and height of the target. These coordinates are normalized, where x, width are normalized using the width of the original image; and y, height are normalized using the height of the original image.

[0027] In step S1, the training process includes algorithm hyperparameter setting, dataset parameter setting and general parameter setting. The algorithm hyperparameter setting is shown in Table 1. The dataset parameter setting includes num_classes (number of classes) as 1 (i.e. unmanned aerial vehicle drone), image_dir (picture folder), and anno_path (picture label folder). The general parameter setting mainly includes weights (optimal weight saving position after training) and device (hardware acceleration used) as gpu. The weight file obtained after the training of this step is best.pt.

[0028] Table 1 Algorithm hyperparameter setting

[0029]

[0030]

[0031] In step S2, multi-step conversion is involved, i.e. pt—>mlir—>bmodel. mlir provides multi-level intermediate representation (Dialect), which can gradually transition from high-level framework (PyTorch / ONNX) to hardware-specific low-level representation; mlir supports unified conversion of models from PyTorch, ONNX and other frameworks to intermediate representation, and then adapts to different hardware; the Pass mechanism of mlir supports computation graph optimization (such as operator fusion, memory optimization), quantization, pruning, etc., to improve the performance of the model on hardware.

[0032] S21, pt file to mlir

[0033] First, convert the pt file to an ONNX model file. The conversion steps are: input the pt file into YOLOv11 to obtain a model file, and output the model file in ONNX format; that is:

[0034] model = YOLOv11("best.pt")

[0035] model.export(format='ONNX', opset=12, save_dir=". / ")

[0036] S22, install the compiler engineering tpu_mlir of the deep learning processor, which provides a complete tool chain that can convert pre-trained neural networks under different frameworks into model files bmodel / cvimodel that can be efficiently operated on the intelligent visual deep learning processor of the algorithm energy. Figure 1

[0037] The installation step of tpu_mlir is pip install tpu_mlir[ONNX].

[0038] If the model is a picture input, the model needs to be preprocessed before conversion. If the model uses a preprocessed npz file as input, preprocessing is not required. The preprocessing process is expressed by the formula as follows:

[0039] y = (x-mean)*scale

[0040] In the formula, x represents the input, and the yolo input picture is in rgb format, each value is multiplied by 1 / 255, the mean and scale parameters correspond to the three channel mean and normalization scaling coefficient of the input image, here mean is set to 0.0, 0.0, 0.0, which means that the three channels are not centered; To accelerate model convergence and avoid numerical instability, the pixel value of the input image is scaled from the range [0, 255] to the range [0, 1], that is, scale is 1 / 255, 1 / 255, 1 / 255, that is, 0.0039216, 0.0039216, 0.0039216.

[0041] S23, use the model_transform command to convert the model from the ONNX format to the mlir format, the steps are: (1) parse the computation graph of the ONNX model; (2) map the operator to the MLIR ONNX Dialect; (3) according to the hardware characteristics of the algorithm energy 1684x, convert the ONNX Dialect to the Krnl Dialect (low-level computation graph); (4) generate the.mlir file in the MLIR format.

[0042] The conversion command is:

[0043] ​model_transform --model_name YOLOv11s --model_def.. / YOLOv11s.onnx --input_shapes [[1, 3, 640, 640]] --mean 0.0, 0.0, 0.0 --scale 0.0039216, 0.0039216, 0.0039216 --keep_aspect_ratio --pixel_format rgb --output_names 350, 498, 646 --test_input.. / image / dog.jpg --test_result YOLOv11s_top_outputs.npz --mlir YOLOv11s.mlir

[0044] The main parameters of model_transform are as follows:

[0045]

[0046]

[0047]

[0048] After converting to an mlir file, a ${model_name}_in_f32.npz file will be generated, which is the input file of the model.

[0049] S24, mlir file to bmodel: use model_deploy command to convert mlir file to f16 bmodel, operation method as follows:

[0050] model_deploy --mlir YOLOv11s.mlir --quantize F16 --processor bm1684x --test_input YOLOv11s_in_f32.npz --test_reference YOLOv11s_top_outputs.npz --model YOLOv11s_1684_f16.bmodel

[0051] The main parameters of model_deploy command are as follows:

[0052]

[0053]

[0054] After compilation, a file named YOLOv11s_1684_f16.bmodel will be generated. This model file will be used to accelerate model inference on the card platform.

[0055] In step S3, the processor used by the model inference hardware environment is Feiteng S2500. To realize the inference deployment of the YOLOv11 model on the "Feiteng CPU + algorithm NPU" platform, it is necessary to install libsophon, sophon-ffmpeg, sophon-opencv, and sophon-sail toolkits.

[0056] S31, install libsophon

[0057] Download the installation package libsophon_0.5.1_aarch64.tar.gz suitable for this platform from the algorithm website and install it by the following steps:

[0058] tar-xzvflibsophon_0.5.1_aarch64.tar.gz

[0059] sudo cp-rlibsophon_0.5.1_aarch64 / *

[0060] sudo ln-s / opt / sophon / libsophon-0.5.1 / opt / sophon / libsophon-current

[0061] S32, build a driver compilation environment

[0062] Download the driver installation package sophon-driver_0.5.1_arm64.deb and decompress it, then do the following operations:

[0063] sudo ln-s / opt / sophon / driver-0.5.1 / bm1684_ddr.bin_v3.1.1-63a86\14d-220906 / lib / firmware / bm1684_ddr_firmware.bin

[0064] sudo ln-s / opt / sophon / driver-0.5.1 / bm1684_tcm.bin_v3.1.1-63a86\14d-220906 / lib / firmware / bm1684_tcm_firm\ware.bin

[0065] cd / opt / sophon / driver-0.5.1

[0066] Perform driver compilation (here independent of dkms)

[0067]

[0068] Finally some configuration work, add library and executable file path:

[0069] sudo cp / opt / sophon / libsophon-current / data / libsophon.conf / etc / ld.so.conf.d /

[0070] sudo ldconfig

[0071] sudo cp / opt / sophon / libsophon-current / data / libsophon-bin-path.sh / etc / profile.d /

[0072] Execute the following command in the terminal, or log out and log in as the current user to use commands such as bm-smi:

[0073] source / etc / profile

[0074] Add cmake config file:

[0075] sudo mkdir -p / usr / lib / cmake / libsophon

[0076] sudo cp / opt / sophon / libsophon-current / data / libsophon-config.cmake / usr / lib / cmake / libsophon /

[0077] S33, install sophon-ffmpeg and sophon-opencv

[0078] Download sophon-mw_0.10.0_aarch64.tar.gz from the official website of Algorithmic Energy, and install it through the following steps:

[0079] tar -xzvf sophon-mw_0.10.0_aarch64.tar.gz

[0080] sudo cp -r sophon-mw_0.10.0_aarch64 / * /

[0081] sudo ln -s / opt / sophon / sophon-ffmpeg_0.10.0 / opt / sophon / sophon-ffmpeg-latest

[0082] sudo ln -s / opt / sophon / sophon-opencv_0.10.0 / opt / sophon / sophon-opencv-latest

[0083] sudo ln -s / opt / sophon / sophon-sample_0.10.0 / opt / sophon / sophon-sample-latest

[0084] sudo sed -i "s / usr\ / local / opt\ / sophon\ / sophon-ffmpeg-latest / g" / opt / sophon / sophon-ffmpeg-latest / lib / pkgconfig / *.pc

[0085] sudo sed -i "s / ^prefix=.*$ / prefix= / opt / sophon / sophon-opencv-latest / g" / opt / sophon / sophon-opencv-latest / lib / pkgconfig / opencv4.pc

[0086] Finally, install the bz2 libc6 libgcc dependencies, then add library and executable file paths:

[0087] sudo cp / opt / sophon / sophon-ffmpeg-latest / data / 01_sophon-ffmpeg.conf / etc / ld.so.conf.d /

[0088] sudo cp / opt / sophon / sophon-opencv-latest / data / 02_sophon-opencv.conf / etc / ld.so.conf.d /

[0089] sudo ldconfig

[0090] sudo cp / opt / sophon / sophon-ffmpeg-latest / data / sophon-ffmpeg-autoconf.sh / etc / profile.d /

[0091] sudo cp / opt / sophon / sophon-opencv-latest / data / sophon-opencv-autoconf.sh / etc / profile.d /

[0092] sudo cp / opt / sophon / sophon-sample-latest / data / sophon-sample-autoconf.sh / etc / profile.d /

[0093] source / etc / profile

[0094] S34, cross-compile and install sophon-sail

[0095] This project uses Python 3.8.2, located in 'python_3.8.2 / bin / python3', and its dynamic library directory is 'python_3.8.2 / lib'. Installing sophon-sail on an ARM machine requires cross-compilation. This means compiling sophon-sail on an x86 machine to generate the installation package, then installing it on the ARM machine.

[0096] Download the SOPHON-SAIL source code sophon-sail_3.8.0.tar.gz, unzip it and go to its source directory; create a compilation folder build and go to the build folder: mkdirbuild&&cdbuild

[0097] Execute the compilation command on an x86 machine:

[0098] cmake -DBUILD_TYPE=arm_pcie\

[0099] DCMAKE_TOOLCHAIN_FILE=.. / cmake / BM168x_ARM_PCIE / ToolChain_aarch64_linux.cmake\

[0100] DPYTHON_EXECUTABLE=python_3.8.2 / bin / python3\

[0101] DCUSTOM PY LIBDIR = python_3.8.2 / lib

[0102] DLIBSOPHON BASIC PATH = libsophon_0.5.1_aarch64 / opt / sophon / libsophon-0.4.1

[0103] DFFMPEG BASIC PATH = sophon-mw_0.10.0_aarch64 / opt / sophon / sophon-ffmpeg_0.4.1

[0104] DOPENCV BASIC PATH = sophon-mw_0.10.0_aarch64 / opt / sophon / sophon-opencv_0.4.1..

[0105] make pysail

[0106] Packaging generates python wheel, the path of the generated wheel package is ‘python / arm_pcie / dist’, and the file name is

[0107] ‘sophon_arm_pcie-3.8.0-py3-none-any.whl’

[0108] cd.. / python / arm_pcie

[0109] chmod+x sophon_arm_pcie_whl.sh

[0110] . / sophon_arm_pcie_whl.sh

[0111] Copy ‘sophon_arm_pcie-3.8.0-py3-none-any.whl’ to the target ARM host, and then execute the following installation command

[0112] pip3 install sophon_arm_pcie-3.8.0-py3-none-any.whl --force-reinstall

[0113] In step S4, the model inference module involves model loading (load), data preprocessing (preprocess), prediction (predict), and post-processing (postprocess).

[0114] In terms of model loading, the Engine tool in the sail package is adopted, and the Engine can realize the loading and management of bmodel, is the main module for realizing model reasoning, and the calling mode is sail.Engine(bmodel_path:str, tpu_id:int, mode:sail.IOMode), wherein the bmodel_path specifies the path of the bmodel file, the tpu_id specifies the id of the intelligent visual deep learning processor used by the Engine instance, and the mode specifies the memory location of the input / output Tensor.

[0115] In terms of data preprocessing, efficient BMCVAPI is adopted instead of OpenCVAPI interface to scale image data, and BMCVAPI contains all image processing acceleration interfaces that can be accelerated by hardware (here, image processing hardware acceleration includes hardware image processing VPP module acceleration and image processing functions realized by borrowing other hardware modules).

[0116] In terms of prediction, the process interface of the Engine tool is used for forward reasoning on a specific computation graph. The calling mode is process(graph_name:str, input_tensors:dict{str:sail.Tensor}, output_tensors:dict{str:sail.Tensor}, core_list:list[int]). Wherein, graph_name is an input parameter, indicating a specific computation graph name; input_tensors is an input parameter, indicating all input Tensor data, which is input by sail.Tensor; output_tensors is an output parameter, indicating all output Tensor data, which is returned by sail.Tensor; core_list is an input parameter, which is only valid for processors supporting multi-core reasoning, and the core used for reasoning can be selected. If the core_list is empty, N cores starting from core0 are used for reasoning; if the length of the core_list is greater than N, the first N cores in the core_list are used for reasoning.

[0117] In post-processing, there are non-maximum suppression (NMS) and image scaling operations, which are relatively conventional and will not be described here.

[0118] Embodiment 1:

[0119] The application is a kind of micro unmanned aerial vehicle target detection method based on algorithm 1684x and YOLOv11 algorithm. The steps are as follows:

[0120] (1) Using the PyTorch framework to train the YOLOv11 algorithm on a Nvidia acceleration card using a UAV dataset;

[0121] (2) Convert the trained weight pt file into a bmodel model that can run on an Intel® oneMCP® 1684x acceleration card;

[0122] (3) Model inference depends on environment configuration;

[0123] (4) Model inference.

[0124] In step (1), the UAV dataset is 114 video clips developed independently. The videos were shot in the field, and the content was the flight process of a UAV, each video being 10 seconds long; 70% (80) of the video clips were randomly selected as the training set, 20% (23) of the video clips were randomly selected as the validation set, and 10% (11) of the video clips were randomly selected as the test set. The preprocessing steps of the dataset include video frame division and image labeling.

[0125] In video frame division, OpenCV was used to read a single video, the video frame count interval frequency was set to 1, and the read images were saved in a preset folder. Since the frame rate of each video is 30 frames / second and the duration of each video is 10 seconds, 300 images can be generated for each video, and the resolution of each image is 640*512. In image labeling, the labelImg software was used to frame and label the UAV in the image, the labeling format was selected as YOLO mode, and each image corresponded to a txt label file. The file content format was <object-class> <x> <y> <width> <height>where x, y are the center coordinates of the target, width, height are the width and height of the target. These coordinates are normalized, where x, width are normalized using the width of the original image; and y, height are normalized using the height of the original image.

[0126] In step (1), the training process includes algorithm hyperparameter setting, dataset parameter setting, and general parameter setting. The algorithm hyperparameter setting is shown in Table 1. The dataset parameter setting includes num_classes (number of classes) as 1 (i.e. unmanned aerial vehicle drone), image_dir (picture folder), and anno_path (picture label folder). The general parameters mainly include weights (optimal weight saving position after training), and device (hardware acceleration used) as gpu.

[0127] Table 1 Algorithm hyperparameter setting

[0128]

[0129] Step (2) involves multi-step conversion, i.e. pt—>mlir—>bmodel.

[0130] (2.1) pt file to mlir

[0131] First, convert the pt file to an ONNX model file, and the conversion steps are as follows:

[0132] model = YOLOv11("best.pt")

[0133] model.export(format='ONNX', opset=12, save_dir=". / ")

[0134] Second, install the compiler project tpu_mlir for algorithm deep learning processors. This project provides a complete tool chain, which can convert pre-trained neural networks under different frameworks into model files bmodel / cvimodel that can be efficiently operated on algorithm intelligent visual deep learning processors. The overall architecture of tpu_mlir is shown in Figure 1

[0135] The installation steps of tpu_mlir are pip install tpu_mlir[ONNX].

[0136] If the model is an image input, the pre-processing of the model needs to be understood before the model is converted. If the model uses a pre-processed npz file as input, the pre-processing does not need to be considered. The pre-processing process is expressed by the formula as follows (x represents the input):​

[0137] y = (x - mean) * scale

[0138] The picture of yolo input is in rgb format, each value is multiplied by 1 / 255, converted to mean and scale corresponding to 0.0, 0.0, 0.0 and 0.0039216, 0.0039216, 0.0039216.

[0139] The command for converting the model from ONNX format to mlir format is:

[0140] model_transform--model_name YOLOv11s--model_def.. / YOLOv11s.ONNX--input_shapes[[1,3,640,640]]--mean 0.0,0.0,0.0--scale 0.0039216,0.0039216,0.0039216--keep_aspect_ratio--pixel_formatrgb--output_names 350,498,646--test_input.. / image / dog.jpg--test_resultYOLOv11s_top_outputs.npz--mlirYOLOv11s.mlir

[0141] The main parameters of model_transform are as follows:

[0142]

[0143]

[0144]

[0145] After converting to mlir file, a ${model_name}_in_f32.npz file will be generated, which is the input file of the model.

[0146] (2.2) mlir file to bmodel,

[0147] Convert the mlir file to f16 bmodel, the operation method is as follows:

[0148] model_deploy--mlir YOLOv11s.mlir--quantize F16--processor bm1684x--test_input YOLOv11s_in_f32.npz--test_reference YOLOv11s_top_outputs.npz--model YOLOv11s_1684_f16.bmodel

[0149]

[0150]

[0151] The main parameters of model_deploy are as follows:

[0152] After compilation, a file named YOLOv11s_1684_f16.bmodel will be generated. This model file will be used for model inference on the algorithmic computing card platform.

[0153] Step (3) The processor used in the model inference hardware environment is Feiteng S2500. To realize the inference deployment of YOLOv11 model on the "Feiteng CPU + algorithmic NPU" platform, it is necessary to install libsophon, sophon-ffmpeg, sophon-opencv and sophon-sail toolkits.

[0154] (3.1) Install libsophon

[0155] Download the installation package suitable for this platform from the algorithmic computing official website libsophon_0.5.1_aarch64.tar.gz, and install it through the following steps:

[0156] tar-xzvflibsophon_0.5.1_aarch64.tar.gz

[0157] sudo cp-rlibsophon_0.5.1_aarch64 / *

[0158] sudo ln-s / opt / sophon / libsophon-0.5.1 / opt / sophon / libsophon-current

[0159] (3.2) Build a driver compilation environment

[0160] Download the driver installation package sophon-driver_0.5.1_arm64.deb and decompress it, and then do the following operations:

[0161] sudo ln -s / opt / sophon / driver-0.5.1 / bm1684_ddr.bin_v3.1.1-63a86\14d-220906 / lib / firmware / bm1684_ddr_firmware.bin

[0162] sudo ln -s / opt / sophon / driver-0.5.1 / bm1684_tcm.bin_v3.1.1-63a86\14d-220906 / lib / firmware / bm1684_tcm_firm\ware.bin

[0163] cd / opt / sophon / driver-0.5.1

[0164] Make driver compile (here not dependent on dkms)

[0165]

[0166] Finally some configuration work, add library and executable file path:

[0167] sudo cp / opt / sophon / libsophon-current / data / libsophon.conf / etc / ld.so.conf.d /

[0168] sudo ldconfig

[0169] sudo cp / opt / sophon / libsophon-current / data / libsophon-bin-path.sh / etc / profile.d /

[0170] Execute the following command in the terminal, or log out and log in as the current user to use commands such as bm-smi:

[0171] source / etc / profile

[0172] Add cmake config file:

[0173] sudo mkdir -p / usr / lib / cmake / libsophon

[0174] sudo cp / opt / sophon / libsophon-current / data / libsophon-config.cmake / usr / lib / cmake / libsophon /

[0175] (3.3) Install sophon-ffmpeg and sophon-opencv

[0176] Download sophon-mw_0.10.0_aarch64.tar.gz from the official website of Algorithm Energy, and install it through the following steps:

[0177] tar -xzvf sophon-mw_0.10.0_aarch64.tar.gz

[0178] sudo cp -r sophon-mw_0.10.0_aarch64 / *

[0179] sudo ln -s / opt / sophon / sophon-ffmpeg_0.10.0 / opt / sophon / sophon-ffmpeg-latest

[0180] sudo ln -s / opt / sophon / sophon-opencv_0.10.0 / opt / sophon / sophon-opencv-latest

[0181] sudo ln -s / opt / sophon / sophon-sample_0.10.0 / opt / sophon / sophon-sample-latest

[0182] sudo sed -i "s / usr\ / local / opt\ / sophon\ / sophon-ffmpeg-latest / g" / opt / sophon / sophon-ffmpeg-latest / lib / pkgconfig / *.pc

[0183] sudo sed -i "s / ^prefix=.*$ / prefix= / opt / sophon / sophon-opencv-latest / g" / opt / sophon / sophon-opencv-latest / lib / pkgconfig / opencv4.pc

[0184] Finally, install bz2 libc6 libgcc dependent libraries, and then add library and executable file path:

[0185] sudo cp / opt / sophon / sophon-ffmpeg-latest / data / 01_sophon-ffmpeg.conf / etc / ld.so.conf.d /

[0186] sudo cp / opt / sophon / sophon-opencv-latest / data / 02_sophon-opencv.conf / etc / ld.so.conf.d /

[0187] sudo ldconfig

[0188] sudo cp / opt / sophon / sophon-ffmpeg-latest / data / sophon-ffmpeg-autoconf.sh / etc / profile.d /

[0189] sudo cp / opt / sophon / sophon-opencv-latest / data / sophon-opencv-autoconf.sh / etc / profile.d /

[0190] sudo cp / opt / sophon / sophon-sample-latest / data / sophon-sample-autoconf.sh / etc / profile.d /

[0191] source / etc / profile

[0192] (3.4) Compile and install sophon-sail

[0193] The python3 version used in this project is python3.8.2, the path is 'python_3.8.2 / bin / python3', and the dynamic library directory of python3 is 'python_3.8.2 / lib'.

[0194] Download the SOPHON-SAIL source code sophon-sail_3.8.0.tar.gz, and enter its source code directory after decompression; create a compilation folder build, and enter the build folder: mkdir build && cd build

[0195] Execute the compilation command:

[0196] cmake -DBUILD_TYPE=arm_pcie\

[0197] -DCMAKE_TOOLCHAIN_FILE=.. / cmake / BM168x_ARM_PCIE / ToolChain_aarch64_linux.cmake\

[0198] -DPYTHON_EXECUTABLE=python_3.8.2 / bin / python3\

[0199] -DCUSTOM_PY_LIBDIR=python_3.8.2 / lib\

[0200] -DLIBSOPHON_BASIC_PATH=libsophon_0.5.1_aarch64 / opt / sophon / libsophon-0.4.1

[0201] -DFFMPEG_BASIC_PATH=sophon-mw_0.10.0_aarch64 / opt / sophon / sophon-ffmpeg_0.4.1

[0202] -DOPENCV_BASIC_PATH=sophon-mw_0.10.0_aarch64 / opt / sophon / sophon-opencv_0.4.1..

[0203] make pysail

[0204] Packaging python wheel, the path of the generated wheel package is 'python / arm_pcie / dist', and the file name is

[0205] ‘sophon_arm_pcie-3.8.0-py3-none-any.whl’

[0206] cd.. / python / arm_pcie

[0207] chmod+x sophon_arm_pcie_whl.sh

[0208] . / sophon_arm_pcie_whl.sh

[0209] Copy'sophon_arm_pcie-3.8.0-py3-none-any.whl' to the target ARM host, and then execute the following installation command

[0210] pip3 install sophon_arm_pcie-3.8.0-py3-none-any.whl --force-reinstall

[0211] Step (4) Model inference module involves model loading, data preprocessing, prediction, and post-processing.

[0212] In terms of model loading, the Engine tool in the sail package is used. Engine can realize the loading and management of bmodel and is the main module for implementing model inference. The calling method is sail.Engine(bmodel_path:str, tpu_id:int, mode:sail.IOMode), where bmodel_path specifies the path of the bmodel file, tpu_id specifies the id of the intelligent visual deep learning processor used by the Engine instance, and mode specifies the memory location of the input / output Tensor.

[0213] In terms of data preprocessing, efficient BMCVAPI is used instead of OpenCVAPI interface to scale image data. BMCVAPI contains all image processing acceleration interfaces that can be accelerated by hardware (here, image processing hardware acceleration includes hardware image processing VPP module acceleration and image processing functions implemented by borrowing other hardware modules).

[0214] In the prediction aspect, the process interface of the Engine tool carries out forward inference on a specific calculation graph. The calling mode is process(graph_name:str, input_tensors:dict{str:sail.Tensor}, output_tensors:dict{str:sail.Tensor}, core_list:list[int]). Among them, graph_name is an input parameter, indicating a specific calculation graph name; input_tensors is an input parameter, indicating the data of all input Tensors, which is transmitted by using sail.Tensor; output_tensors is an output parameter, indicating the data of all output Tensors, which is returned by using sail.Tensor; core_list is an input parameter, which is only valid for processors supporting multi-core inference, and the cores used for inference can be selected. Assuming that the number of cores corresponding to the bmodel is N, if the core_list is empty, N cores starting from core0 are used for inference; if the length of the core_list is greater than N, the first N cores in the core_list are used for inference.

[0215] In the post-processing aspect, there are non-maximum suppression (NMS) and image scaling operations, which are relatively conventional and will not be described here.

[0216] Embodiment 2

[0217] Figure 2 The inference picture of the YOLOv11 algorithm running on the combination of the "Kylin operating system + Feiteng S2500 + algorithm 1684x" domestic hardware, wherein the unmanned aerial vehicle is marked by a red frame, and the number on the side of the frame represents the confidence, and the upper left corner is the number of unmanned aerial vehicles.

[0218] The present application proposes a kind of based on algorithm 1684x and YOLOv11 algorithm Micro unmanned aerial vehicle target detection method, the present application adapts new model YOLOv11 on domestic platform first, wherein not only YOLOv11 model is converted from pt file to bmodel model that can be run on algorithm 1684x acceleration card, but also by cross-compiled etc. Necessary toolkit is installed by technical means, so as to realize YOLOv11 model deployment to the combination of "Kylin operating system + Feiteng S2500 + algorithm 1684x" domestic hardware, and then YOLOv11 model is used to realize the real-time detection of Micro unmanned aerial vehicle, improve detection efficiency and accuracy.

[0219] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.< / height> < / width> < / y> < / x> < / object-class> < / height> < / width> < / y> < / x> < / object-class>

Claims

1. A micro unmanned aerial vehicle target detection method based on the algorithms of Energy 1684x and YOLOv11, characterized in that, The method comprises the following steps: S1, using a PyTorch framework to train a YOLOv11 algorithm on an acceleration card using a UAV dataset; S2, converting the trained weight pt file into a bmodel model that can run on an algorithm 1684x acceleration card; S3, model inference depends on environment configuration: the processor used in the model inference hardware environment is a Feiteng CPU, and in order to realize the inference deployment of the YOLOv11 model on the "Feiteng CPU+algorithm NPU" platform, the necessary tool package is installed; S4, model inference: model inference includes model loading, data preprocessing, prediction and post-processing.

2. The micro unmanned aerial vehicle target detection method based on the algorithm 1684x and YOLOv11 algorithm of claim 1, wherein, In S1, the UAV dataset is a video clip; The video is shot in the field, and the content is the flight process of the UAV. 70% of the video clips are randomly selected as the training set, 20% of the video clips are randomly selected as the validation set, and 10% of the video clips are randomly selected as the test set. The preprocessing steps of the dataset include video framing and picture labeling; In video framing, OpenCV is used to read a single video, the video frame count interval frequency is set to 1, and the read picture is saved in a preset folder. In picture labeling, the labelImg software is used to frame the unmanned aerial vehicle in the picture for labeling, the labeling format is selected as the YOLO mode, one txt label file corresponds to each picture, and the file content format is <object-class> <x> <y> <width> <height>Wherein x, y are the center coordinates of the target, width, height are the width and height of the target; these coordinates are normalized, wherein x, width are normalized using the width of the original image; and y, height are normalized using the height of the original image.< / height> < / width> < / y> < / x> < / object-class> 3. The micro unmanned aerial vehicle target detection method based on the EC1684x and YOLOv11 algorithms of claim 2, wherein, In S1, the training process includes algorithm hyperparameter setting, dataset parameter setting and general parameter setting; wherein the algorithm hyperparameter setting includes the number of training rounds, the learning rate, the batch size, whether to shuffle the picture order, the number of CPU cores used to load data and the picture resolution of the input model; the dataset parameter setting includes the number of categories, the folder where the pictures are located and the folder where the picture labels are located; the general parameter setting includes the optimal weight saving position after training and the hardware acceleration used; the weight file obtained after this step of training is best.pt.

4. The micro unmanned aerial vehicle target detection method based on the EC1684x and YOLOv11 algorithms of claim 2, wherein, In S2, multiple conversion steps are involved, namely pt--->mlir--->bmodel, wherein mlir provides a multi-level intermediate representation Dialect, which gradually transitions from high-level frameworks such as PyTorch and ONNX to low-level representations specific to hardware; mlir supports converting models from PyTorch and ONNX frameworks into intermediate representations, and then adapting them to different hardware.

5. The micro unmanned aerial vehicle target detection method based on the EC1684x and YOLOv11 algorithms of claim 4, wherein, S2 specifically comprises the following steps: S21, convert pt file to mlir First, convert the pt file to an ONNX model file, the conversion steps are: input the pt file into YOLOv11 to obtain a model file, and output the model file in ONNX format; S22, install the compiler project tpu_mlir of the algorithm deep learning processor; S23, use the model_transform command to convert the model from ONNX format to mlir format; S24, mlir file to bmodel: use the model_deploy command to convert the mlir file to the f16 bmodel; after compilation, a file named YOLOv11s_1684_f16.bmodel will be generated; this model file will be used to model inference on the algorithm energy acceleration card platform.

6. The micro unmanned aerial vehicle target detection method based on the EC1684x and YOLOv11 algorithms of claim 5, wherein, In the S22, The installation step of tpu_mlir is pip install tpu_mlir[ONNX]; If the model is a picture input, the model needs to be preprocessed before the model is converted; if the model uses a preprocessed npz file as input, preprocessing does not need to be considered; The preprocessing process is expressed by the formula as follows: y=(x-mean)*scale In the formula, x represents input, the picture input by yolo is in rgb format, each value is multiplied by 1 / 255, the mean and scale parameters correspond to the mean of the three channels of the input image and the normalization scaling coefficient, here mean is set to 0.0, 0.0, 0.0, which means that the three channels are not centered; in order to accelerate model convergence and avoid numerical instability, the pixel value of the input image is scaled from the range [0, 255] to the range [0, 1], that is, scale is 1 / 255, 1 / 255, 1 / 255, that is, 0.0039216, 0.0039216, 0.0039216.

7. The micro unmanned aerial vehicle target detection method based on the EC1684x and YOLOv11 algorithms of claim 5, wherein, The S23 includes: (1) parsing the computation graph of the ONNX model; (2) mapping the operator to the MLIR ONNXDialect; (3) converting the ONNX Dialect to the Krnl Dialect according to the hardware characteristics of the algorithm energy 1684x; (4) generating an.mlir file in MLIR format; after converting to an mlir file, a ${model_name}_in_f32.npz file will be generated, which is the input file of the model.

8. The micro unmanned aerial vehicle target detection method based on the algorithm 1684x and YOLOv11 algorithm of claim 1, wherein, The S3 includes: S31, install libsophon Download the libsophon installation package suitable for this platform from the algorithm energy official website and install it; S32, build a driver compilation environment Download the driver installation package sophon-driver, decompress it, compile the driver, add the library and executable file path, log out and log in as the current user, and then you can use it; S33, install sophon-ffmpeg and sophon-opencv Download the sophon-mw installation package from the algorithm energy official website and install it, and then install sophon-ffmpeg and sophon-opencv, install bz2 libc6 libgcc dependent libraries, and then add the library and executable file path; S34, cross-compile and install sophon-sail After compiling sophon-sail on an x86 machine to generate an installation package, install the package on an ARM machine.

9. The micro unmanned aerial vehicle target detection method based on the EC1684x and YOLOv11 algorithms of claim 8, wherein, The S34 specifically includes: Download SOPHON-SAIL source code, decompress it and enter its source code directory; create a compilation folder build and enter the build folder: mkdir build && cd build; Package to generate python wheel, the path of the generated wheel package is 'python / arm_pcie / dist'; the file name is'sophon_arm_pcie-3.8.0-py3-none-any.whl'; Copy'sophon_arm_pcie-3.8.0-py3-none-any.whl' to the target ARM host, and then execute the installation command.

10. The micro unmanned aerial vehicle target detection method based on the algorithm 1684x and YOLOv11 algorithm of claim 1, wherein, The S4 comprises: The model inference module involves model loading, data preprocessing, prediction and post-processing; In terms of model loading, the Engine tool in the sail package is adopted, and the calling mode is sail.Engine(bmodel_path:str, tpu_id:int, mode:sail.IOMode), wherein bmodel_path specifies the path of the bmodel file, tpu_id specifies the id of the intelligent visual deep learning processor used by the Engine instance, and mode specifies the memory location of the input / output Tensor; In terms of data preprocessing, the efficient BMCVAPI is adopted instead of the OpenCVAPI interface to scale image data, and the BMCVAPI contains all image processing acceleration interfaces that can be accelerated by hardware; In terms of prediction, the process interface of the Engine tool is used to perform forward inference on a specific computation graph; the calling mode is process(graph_name:str, input_tensors:dict{str:sail.Tensor}, output_tensors:dict{str:sail.Tensor}, core_list:list[int]); wherein graph_name is an input parameter, indicating a specific computation graph name; input_tensors is an input parameter, indicating the data of all input Tensors, which is input by using sail.Tensor; output_tensors is an output parameter, indicating the data of all output Tensors, which is returned by using sail.Tensor; core_list is an input parameter, which is valid only for processors supporting multi-core inference, and is used to select the core used for inference; assuming that the number of cores corresponding to the bmodel is N, if core_list is empty, N cores starting from core0 are used for inference; if the length of core_list is greater than N, the first N cores in core_list are used for inference; In terms of post-processing, non-maximum suppression and image scaling operations are included.

Images