Ground object recognition method and device, equipment and storage medium

Abstract

The invention provides a ground object recognition method and device, equipment and a storage medium, and the method comprises the steps: carrying out the data preprocessing of a to-be-recognized image, and obtaining a preprocessed image; a trained deep learning model deployed on an edge device in advance is obtained, the edge device is a development board, the deep learning model is a lightweight feature processing target detection model, the programming code is run to execute a reasoning task, the deep learning model is used for reasoning the preprocessed picture, and an initial result is output; and performing redundancy processing on the initial result to obtain a ground object recognition result. The method comprises the following steps: deploying a pre-trained deep learning model on an edge device in a development board form; a lightweight feature processing target detection model CRE-YOLO-E is used as a deep learning model for task execution, the overall calculation amount is reduced while the model structure is optimized, and deployment on edge equipment is more efficient.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a ground object recognition method, device, equipment and storage medium. Background Art

[0002] Object recognition refers to the process of automatically detecting, classifying or identifying various objects or features on the surface of the earth using remote sensing images or geographic information system data. The purpose is to extract effective information from remote sensing data to assist monitoring planning or evaluation.

[0003] In related technologies, traditional ground object recognition usually relies on large-scale models and large amounts of data, resulting in high computing requirements. However, due to hardware limitations, it is difficult for recognition platforms deployed on high-altitude aircraft to achieve improvements in recognition speed. In addition, the model used for target detection itself brings certain computing consumption. For example, the FPN module and PAN module in YOLOv5 need to perform feature fusion at multiple scales, introduce a large number of convolutional layers and learning parameters, and increase the training time and resource consumption of the model.

[0004] Based on the above analysis of the development status of this technology field, the existing technology lacks solutions for using lightweight models to optimize target detection models and perform inference on edge devices. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, equipment and storage medium for identifying ground objects, aiming to solve the above-mentioned problems in the prior art.

[0006] According to a first aspect of an embodiment of the present invention, a method for identifying a ground object is provided, comprising:

[0007] Perform data preprocessing on the image to be identified to obtain a preprocessed image;

[0008] Obtain a deep learning model that has been pre-deployed and trained on an edge device, where the edge device is a development board and the deep learning model is a target detection model for lightweight feature processing. Run the programming code to perform the inference task, use the deep learning model to infer the pre-processed image, and output the initial result.

[0009] The initial results are processed redundantly to obtain the ground object recognition results.

[0010] According to a second aspect of an embodiment of the present invention, there is provided a ground object recognition device, comprising:

[0011] A preprocessing module, used for performing data preprocessing on the image to be identified to obtain a preprocessed image;

[0012] The recognition module is used to obtain the deep learning model pre-deployed and trained on the edge device, where the edge device is a development board, and the deep learning model is a target detection model for lightweight feature processing. The programming code is run to perform the inference task, and the deep learning model is used to infer the pre-processed image and output the initial result.

[0013] The processing module is used to perform redundant processing on the initial result to obtain the ground object recognition result.

[0014] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for identifying ground objects provided in the first aspect of the present disclosure are implemented.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the method for identifying ground objects provided in the first aspect of the present disclosure are implemented.

[0016] The technical solution provided by the embodiment of the present invention includes the following beneficial effects: deploying a pre-trained deep learning model on an edge device in the form of a development board; using the lightweight feature processing target detection model CRE-YOLO-E as the deep learning model for performing tasks, optimizing the model structure while reducing the overall computational complexity, making its deployment on the edge device more efficient.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a method for identifying a ground object according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a CRE-YOLO-E model according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a neck network CCFM according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a ground object recognition result according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of the comparison of F1 values ​​of the embodiments of the present invention;

[0024] Figure 6 is a schematic diagram of the inference duration of an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of a complete implementation architecture of an embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of a ground object recognition device according to an embodiment of the present invention;

[0027] Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0029] Method Embodiment

[0030] According to an embodiment of the present invention, a method for identifying a ground object is provided. Figure 1 is a flow chart of a method for identifying ground objects according to an embodiment of the present invention. Figure 1 As shown, the ground object recognition method according to an embodiment of the present invention specifically includes:

[0031] In step S110, data preprocessing is performed on the image to be identified to obtain a preprocessed image, which specifically includes:

[0032] The image to be recognized is resized, converted into a floating point type, and normalized in turn to obtain a preprocessed image;

[0033] The letterbox function is used to adjust the size of the image to be recognized to reduce excessive background interference information in the inference input. The data format is converted to np.float16 format, which represents 16-bit floating point numbers. To avoid multiple memory allocation logic, it can be expressed as img = np.ascontiguousarray (img / 255.0, dtype = np.float16); efficient data preparation methods provide optimized input for hardware accelerated inference. Format conversion and normalization also help memory consumption and computational burden.

[0034] In step S120, a deep learning model pre-deployed and trained on an edge device is obtained, wherein the edge device is a development board, and the deep learning model is a target detection model for lightweight feature processing. The programming code is run to perform the inference task, and the deep learning model is used to infer the preprocessed image and output the initial result, which specifically includes:

[0035] In the embodiment of the present invention, the development board uses Huawei Atlas 200IDK A2 development board because the development board can use the Ascend 310AI processor for accelerated reasoning, and the optimized deep learning model can significantly improve the reasoning speed. At the same time, the development board has multiple interfaces and supports H.264 / H.265Decoder hardware decoding, supporting up to 2-channel 4K (3840x2160) 75FPS, which provides the possibility for real-time recognition;

[0036] In the embodiment of the present invention, the development board is accessed by SSH, the development board is connected to the computer by type-C, the development board driver is updated and the IP address is set through the device manager, and then the MobaXterm tool is started. After entering the development board IP and password in the tool, the login is successful, and the pre-trained deep learning model can be obtained and the reasoning task can be performed;

[0037] Execute the recommendation engine class InferSession in the programming code. The recommendation engine class InferSession is a class from ais_bench.infer.interface. It enables the inference task to be executed on the Ascend AI processor of the development board instead of the CPU, thereby significantly improving the inference speed and reducing latency.

[0038] By calling the recommendation engine class, the reasoning task is executed on the processor in the development board, that is, the deep learning model is used to identify the objects in the image to be identified. Enter python main.py in the command line of the MobaXterm tool to run the code.

[0039] The deep learning model is a pre-trained lightweight feature processing target detection model. The training process is carried out on a platform with an Intel I5 12600KF processor equipped with 32GB of running memory and an RTX2080TI graphics card. The data source during the training process is obtained by taking pictures with the E2000S Pegasus drone equipped with an E-CAM2000 imaging system. The training set includes 743 samples.

[0040] The traditional deep learning model YOLOv5s for target detection is large in scale and will occupy a lot of space when deployed in edge devices. Therefore, YOLOv5s needs to be lightweight. Therefore, based on the original model architecture of YOLOv5s, the CCFM lightweight module is used to replace the Head Neck Network in YOLOv5s, that is, to replace the FPN+PAN structure.

[0041] Therefore, the deep learning model includes a YOLOv5s target detection model (You Only Look Once version 5small) consisting of an input layer, a backbone network, a neck network, and a detection head. The backbone network adds an ECA channel attention mechanism for enhanced capture to the CBS feature extraction module and the SPPF feature fusion module. The neck network is a CCFM lightweight module (Cross-Scale Feature Fusion Module), which is also called a cross-scale feature fusion module. The RepDWBlock convolution module is used in the CCFM lightweight module to replace the original Fusion Block fusion module.

[0042] The ECA channel attention mechanism can enhance the model's ability to express key features and focus on important features by dynamically adjusting channel weights, which is especially suitable for capturing in complex scenes. The RepDWBlock convolution module can improve feature fusion performance, integrating the deep separable convolution DWConv and the multi-branch convolution RepConv, and replacing the FusionBlock fusion module to reduce the amount of calculation and increase the inference speed.

[0043] According to the above description, ECA is an improvement on the backbone network in YOLOv5s, RepDWBlock is an improvement on the CCFM module, and the improved CCFM is used to replace the original neck network in the original YOLOv5s to improve the neck network. In the embodiment of the present invention, the improved target detection model is named CRE-YOLO-E, namely Cross-Scale Feature Fusion with RepDWBlock and Efficient Channel Attention for Object Detection On Edge device.

[0044] Using deep learning models to reason about preprocessed images specifically includes:

[0045] The backbone network of the ECA channel attention mechanism is introduced to extract multi-scale features. The neck network of the RepDWBlock convolution module is introduced to enhance the multi-scale features from the bottom up and reduce the amount of calculation. The initial result of reasoning is output through the detection head. The internal architecture and functions of other modules of YOLOv5s are consistent with the original model. Figure 2 is a schematic diagram of the CRE-YOLO-E model of an embodiment of the present invention, such as Figure 2 As shown in the figure, the CCFM module introduces the RepDWBlock module, and the backbone network introduces the ECA module. Figure 2 All English words in the text have the existing meanings, including Focus, Detect, etc.; Figure 3 Schematic diagram of the neck network CCFM of an embodiment of the present invention, Figure 3 As shown, the figure is Figure 2 correspond;

[0046] Preferably, when the deep learning model is used to start reasoning on the preprocessed image, timing is started to count the time taken for reasoning and evaluate the model performance.

[0047] In step S130, the initial result is subjected to redundancy processing to obtain a ground object recognition result, which specifically includes:

[0048] The initial result is subjected to non-maximum suppression processing, and low-confidence or overlapping bounding boxes are filtered out to obtain the ground object recognition result; preferably, the non-maximum suppression processing usually uses a fixed IoU to determine whether the detection boxes overlap. If the IoU of two detection boxes exceeds the threshold, the box with lower confidence will be directly discarded. In the embodiment of the present invention, the confidence of the overlapping detection boxes can be gradually reduced instead of being directly discarded, thereby reducing the possibility of missed detection.

[0049] The restored detection frame size and category information are annotated in the image to be identified for visual display.

[0050] Preferably, after obtaining the ground object recognition result, the result needs to be saved, and the folder used to save the result needs to be checked to see if it exists. If it does not exist, the folder needs to be created, and then the ground object recognition result is stored in the folder. The category information, location, confidence, etc. can also be saved as a CSV file;

[0051] Figure 4 FIG. 1 is a schematic diagram of the ground object recognition result of an embodiment of the present invention. Figure 4 As shown, the recognition box results are shown; Figure 5Schematic diagram of the comparison of F1 values ​​of the embodiments of the present invention. Figure 5 As shown in the figure, the F1 value comparison between the CRE-YOLO-E model and the original YOLOv5s model is shown. The horizontal axis is the confidence. After optimization, the maximum F1 value is 0.75, which is higher than the original 0.71. The CRE-YOLO-E model performs better when weighing precision and recall. At the same time, the maximum F1 value of the CRE-YOLO-E model appears at the confidence threshold of 0.588, while the original model appears at 0.566, indicating that the model of the present invention has better stability.

[0052] Figure 6 Schematic diagram of the inference duration of an embodiment of the present invention; Figure 6 As shown in the figure, the inference time of the CRE-YOLO-E model is 12.29% shorter than that of the original model, and the speed is significantly improved;

[0053] The method further comprises:

[0054] In step S140, during the deployment process, the trained deep learning model is converted into a cross-platform format, and the converted deep learning model is deployed in the edge device, specifically including:

[0055] This process is a pre-deployment work to prepare for the sequential execution of the ground object recognition task;

[0056] During the deployment process, the deep learning model is first converted into the ONNX format and uploaded to the development board for smooth transmission; then the deep learning model is further converted into the OM format that can be run by the development board.

[0057] Figure 7 is a schematic diagram of a complete implementation architecture of an embodiment of the present invention, such as Figure 7 As shown, the process of model improvement and training, as well as deployment and inference is demonstrated.

[0058] To sum up, in response to the existing problems, the method for identifying ground objects invented this time deploys the pre-trained deep learning model on an edge device in the form of a development board; the development board uses the Huawei Atlas200IDK A2 development board to use its AI acceleration processor for accelerated reasoning; the recommendation engine class InferSession is set to ensure that the reasoning task runs on the AI ​​acceleration processor of the development board instead of on the CPU; a lightweight target detection model is used as the deep learning model for executing the task; ECA is introduced in the backbone network part to better capture and express the relationship between model channels, thereby enhancing the model's ability to express key features; the RepDWBlock convolution module is introduced inside CCFM, and the neck FPN+PAN structure in the original YOLOv5s is replaced with the optimized CCFM module, thereby improving the feature fusion performance; while optimizing the model structure, the overall computational effort is reduced, making its deployment on edge devices more efficient; the overall solution improves the execution efficiency of ground object recognition tasks from both software and hardware perspectives.

[0059] Device Embodiment

[0060] According to an embodiment of the present invention, a ground object recognition device is provided. Figure 8 is a schematic diagram of a ground object recognition device according to an embodiment of the present invention. Figure 8 As shown, the ground object recognition device according to an embodiment of the present invention specifically includes:

[0061] The preprocessing module 80 is used to perform data preprocessing on the image to be identified to obtain a preprocessed image, and is specifically used to:

[0062] The image to be recognized is resized, converted to floating-point type, and normalized in turn to obtain a preprocessed image.

[0063] The recognition module 82 is used to obtain a deep learning model pre-deployed and trained on an edge device, where the edge device is a development board, and the deep learning model is a target detection model for lightweight feature processing, run programming code to perform reasoning tasks, use the deep learning model to reason on the pre-processed image, and output the initial result, which is specifically used for:

[0064] Execute the recommendation engine class in the programming code, and execute the reasoning task on the processor in the development board by calling the recommendation engine class;

[0065] The deep learning model specifically includes:

[0066] The YOLOv5s target detection model is composed of an input layer, a backbone network, a neck network, and a detection head. The backbone network adds an ECA channel attention mechanism for enhanced capture to the CBS feature extraction module and the SPPF feature fusion module. The neck network is a CCFM lightweight module. The CCFM lightweight module uses a RepDWBlock convolution module to replace the original Fusion Block fusion module.

[0067] The backbone network with ECA channel attention mechanism is introduced to extract multi-scale features, and the neck network with RepDWBlock convolution module is introduced to enhance the multi-scale features from bottom to top, reduce the amount of calculation, and output the initial reasoning results through the detection head.

[0068] The processing module 84 is used to perform redundancy processing on the initial result to obtain a ground object recognition result.

[0069] Perform non-maximum suppression processing on the initial result to obtain the ground object recognition result;

[0070] The restored detection frame size and category information are annotated in the image to be identified for visual display.

[0071] To sum up, in response to the existing problems, the device for identifying ground objects invented this time deploys a pre-trained deep learning model on an edge device in the form of a development board; the development board uses Huawei Atlas200IDK A2 development board to use its AI acceleration processor for accelerated reasoning; the recommendation engine class InferSession is set to ensure that the reasoning task runs on the AI ​​acceleration processor of the development board instead of on the CPU; a lightweight target detection model is used as the deep learning model for executing the task; ECA is introduced in the backbone network part to better capture and express the relationship between model channels, thereby enhancing the model's ability to express key features; the RepDWBlock convolution module is introduced inside CCFM, and the neck FPN+PAN structure in the original YOLOv5s is replaced by the optimized CCFM module, thereby improving the feature fusion performance; while optimizing the model structure, the overall computational effort is reduced, making its deployment on edge devices more efficient; the overall solution improves the execution efficiency of ground object recognition tasks from both software and hardware perspectives.

[0072] Electronic device embodiment

[0073] Fig. 9Schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 900 may include at least one processor 910 and a memory 920. The processor 910 may execute instructions stored in the memory 920. The processor 910 is connected to the memory 920 through a data bus. In addition to the memory 920, the processor 910 may also be connected to an input device 930, an output device 940, and a communication device 950 through a data bus.

[0074] The processor 910 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0075] The memory 920 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0076] In the embodiment of the present disclosure, executable instructions are stored in the memory 920, and the processor 910 can read the executable instructions from the memory 920 and execute the instructions to implement all or part of the steps of any of the ground object recognition methods in the above exemplary embodiments.

[0077] Computer Readable Storage Medium Embodiments

[0078] In addition to the above-mentioned methods and devices, the exemplary embodiments of the present disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions, which can be executed by a processor to implement all or part of the steps described in any of the ground object recognition methods in the above-mentioned exemplary embodiments.

[0079] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0080] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: a static random access memory (SRAM) with one or more wires electrically connected, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk, or any suitable combination of the above.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying a ground object, characterized in that: include: Perform data preprocessing on the image to be identified to obtain a preprocessed image; Obtain a deep learning model pre-deployed and trained on an edge device, wherein the edge device is a development board, and the deep learning model is a target detection model for lightweight feature processing, run programming code to perform an inference task, use the deep learning model to infer the pre-processed image, and output an initial result; Redundancy processing is performed on the initial result to obtain a ground object recognition result.

2. The method according to claim 1, characterized in that The method further comprises: During the deployment process, the trained deep learning model is converted into a cross-platform format, and the deep learning model in the converted format is deployed in the edge device.

3. The method according to claim 1, characterized in that The data preprocessing of the image to be identified to obtain the preprocessed image specifically includes: The image to be identified is resized, converted into a floating point type, and normalized in turn to obtain the preprocessed image.

4. The method according to claim 1, characterized in that: The running of the programming code to perform the reasoning task specifically includes: Execute the recommendation engine class in the programming code, and enable the reasoning task to be executed on the processor in the development board by calling the recommendation engine class.

5. The method according to claim 1, characterized in that The deep learning model specifically includes: The invention comprises a YOLOv5s target detection model composed of an input layer, a backbone network, a neck network and a detection head, wherein the backbone network adds an ECA channel attention mechanism for enhanced capture in a CBS feature extraction module and an SPPF feature fusion module, the neck network is a CCFM lightweight module, and the CCFM lightweight module uses a RepDWBlock convolution module to replace an original Fusion Block fusion module; Using the deep learning model to infer the preprocessed image specifically includes: Multi-scale features are extracted by introducing the backbone network of the ECA channel attention mechanism, and bottom-up feature enhancement is performed on the multi-scale features by introducing the neck network of the RepDWBlock convolution module, and the initial result of reasoning is output through the detection head.

6. The method according to claim 1, characterized in that The performing redundancy processing on the initial result to obtain the ground object recognition result specifically includes: Performing non-maximum suppression processing on the initial result to obtain a ground object recognition result; The restored detection frame size and category information are annotated in the image to be identified for visual display.

7. A ground object recognition device, characterized in that: include: A preprocessing module, used to perform data preprocessing on the image to be identified to obtain a preprocessed image; The recognition module is used to obtain a deep learning model pre-deployed and trained on an edge device, wherein the edge device is a development board, and the deep learning model is a target detection model for lightweight feature processing, run programming code to perform reasoning tasks, use the deep learning model to reason on the pre-processed image, and output an initial result; The processing module is used to perform redundancy processing on the initial result to obtain a ground object recognition result.

8. The device according to claim 7, characterized in that The identification module is specifically used for: Executing the recommendation engine class in the programming code, and causing the reasoning task to be executed on the processor in the development board by calling the recommendation engine class; The deep learning model specifically includes: The invention comprises a YOLOv5s target detection model composed of an input layer, a backbone network, a neck network and a detection head, wherein the backbone network adds an ECA channel attention mechanism for enhanced capture in a CBS feature extraction module and an SPPF feature fusion module, the neck network is a CCFM lightweight module, and the CCFM lightweight module uses a RepDWBlock convolution module to replace an original Fusion Block fusion module; Multi-scale features are extracted by introducing the backbone network of the ECA channel attention mechanism, and bottom-up feature enhancement is performed on the multi-scale features by introducing the neck network of the RepDWBlock convolution module, and the initial result of reasoning is output through the detection head.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for identifying a ground object as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the method for identifying a ground object as described in any one of claims 1 to 6 are implemented.

Images