Target detection method and device applied to micro-nano unmanned platform

Abstract

The invention provides a target detection method and device applied to a micro-nano unmanned platform, and relates to the technical field of unmanned platforms. A target detection method applied to a micro-nano unmanned platform comprises the steps that an event frame picture data set is acquired through an event camera, and the event frame picture data set comprises a training set and a test set; constructing a target detection model; based on the training set, training the target detection model to update parameters of the target detection model; and based on the test set, obtaining target detection information through the trained target detection model. According to the embodiment of the invention, the real-time detection of the target can be realized under low light, and the memory occupation and reasoning time of the micro-nano unmanned platform are reduced.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned platforms, and more specifically, to a target detection method and device applied to a micro-nano unmanned platform. Background Art

[0002] Micro-nano unmanned platforms are an ultra-low power unmanned platform that has developed rapidly in recent years. The weight of the entire machine is usually less than 50 grams, and the memory and computing power of the adaptable AI chips are relatively small.

[0003] At present, the target detection technology widely used in unmanned systems mostly captures images through traditional visible light sensors or infrared sensors. However, due to the limitations of light intensity, temperature and other conditions, the sensor imaging quality is not high and the target detection performance is significantly reduced. Although the visible light and infrared fusion target detection method can cover the advantages of both sensors, it requires too much computing and storage resources, and the additional cost is high, which is not suitable for micro-nano unmanned platforms. Summary of the invention

[0004] According to one aspect of the present application, a target detection method applied to a micro-nano unmanned platform is provided, comprising: obtaining an event frame image dataset through an event camera, wherein the event frame image dataset includes a training set and a test set; constructing a target detection model; based on the training set, training the target detection model to update the parameters of the target detection model; based on the test set, obtaining target detection information through the trained target detection model.

[0005] According to some embodiments, obtaining an event frame image data set through an event camera includes: obtaining event data collected by the event camera; generating a spatial data matrix according to timestamps based on the event data, wherein the timestamps correspond one-to-one to the spatial data matrix; and fusing the event data in the spatial data matrix according to a preset time window to obtain an event frame image corresponding to the spatial data matrix.

[0006] According to some embodiments, the target detection model includes an event frame feature extraction module and an event frame detection module, wherein the event frame feature extraction module includes multiple convolutional network layers, normalization layers and pooling layers; the event frame detection module includes multiple convolutional network layers, normalization layers, pooling layers and downsampling layers.

[0007] According to some embodiments, the target detection model is trained based on a training set to update the parameters of the target detection model, including: randomly initializing the parameters of the target detection model; and training the target detection model whose parameters have been initialized based on the training set.

[0008] According to some embodiments, a target detection model with initialized parameters is trained based on a training set, including: performing forward propagation on the target detection model with initialized parameters based on the training set; and performing back propagation on the target detection model that has completed forward propagation to update the parameters of the target detection model.

[0009] According to some embodiments, backpropagation is performed on the target detection model that has completed forward propagation to update the parameters of the target detection model, including: calculating the loss function of the target detection model; and updating the parameters of the target detection model according to the gradient of the loss function.

[0010] According to some embodiments, target detection information is obtained based on a test set through a trained target detection model, including: inputting the test set into the trained target detection model to obtain an output result; and suppressing the output result to generate target detection information.

[0011] According to one aspect of the present application, there is provided a target detection device for a micro-nano unmanned platform, comprising: a data acquisition unit for acquiring an event frame image data set through an event camera, wherein the event frame image data set comprises a training set and a test set; a model building unit for building a target detection model; based on the training set, training the target detection model to update the parameters of the target detection model; and a testing unit for acquiring target detection information through a trained target detection model based on the test set.

[0012] According to one aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0013] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the aforementioned method is implemented.

[0014] According to the embodiments of the present application, the target brightness changes can be captured in real time by an event camera sensor to form an event frame picture, which is less affected by ambient light and temperature. In addition, the target detection model based on the deep neural network is used for inference calculation, which reduces the amount of calculation, the amount of parameters and the data storage space, and shortens the real-time inference time.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.

[0017] Figure 1 A flow chart of a target detection method applied to a micro-nano unmanned platform according to an exemplary embodiment of the present application is shown.

[0018] Figure 2 A schematic diagram of an event frame picture according to an exemplary embodiment of the present application is shown.

[0019] Figure 3 A schematic diagram showing the composition of a target detection model according to an exemplary embodiment of the present application is shown.

[0020] Figure 4 A block diagram of a target detection device applied to a micro-nano unmanned platform according to an exemplary embodiment of the present application is shown.

[0021] Figure 5 A block diagram of an electronic device according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0023] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application may be put into practice without one or more of these specific details, or other modes, components, materials, devices or operations may be adopted. In these cases, known structures, methods, devices, realizations, materials or operations will not be shown or described in detail.

[0024] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0025] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0026] The present application provides a target detection method and device applied to a micro-nano unmanned platform, which can improve the performance of the micro-nano unmanned platform when performing target detection.

[0027] A target detection method and device for a micro-nano unmanned platform according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flow chart of a target detection method applied to a micro-nano unmanned platform according to an exemplary embodiment of the present application is shown.

[0029] like Figure 1 As shown, in step S110, the target detection device obtains an event frame image data set through an event camera.

[0030] For example, in step S110, the target detection device obtains event data through an event camera and generates an event frame image data set based on the event data.

[0031] The target detection device obtains event data of the current detection target in multiple scenes collected by the event camera.

[0032] According to some embodiments, the scene where the detection target is located includes daytime and nighttime, and the event data collected by the event camera is time-asynchronous data in RAW format.

[0033] Based on the event data acquired by the event camera, the target detection device generates a spatial data matrix corresponding to the timestamp.

[0034] For example, assuming that the resolution of the event camera is 320*320, the target detection device generates a 1*320*320 spatial data matrix according to the timestamp, where each timestamp corresponds to a spatial data matrix. According to the time corresponding to the event data, the target detection device puts the event data into the spatial data matrix corresponding to its timestamp.

[0035] The target detection device performs fusion processing on the event data in the spatial data matrix according to a preset time window to obtain event frame images corresponding to all spatial data matrices generated according to timestamps.

[0036] According to some embodiments, since the frequency of event cameras is usually high (e.g., 100,000 Hz), the amount of event data generated by them is relatively large, and the storage and calculation of event data on micro-nano unmanned platforms are easily subject to resource constraints. Therefore, it is necessary to use the target detection device to fuse the event data within a preset time window and generate an event frame image.

[0037] For example, the preset time window may be the time window of the visible light sensor. Assuming that the frequency of the visible light sensor is 5 Hz, the corresponding time window is 0.2 s. The target detection device accumulates and fuses the event data collected by the event camera every 0.2 s according to its positive and negative polarity to generate a grayscale image, i.e., an event frame image.

[0038] like Figure 2 As shown, the fused event frame image can more clearly present the object edges and internal lines of the detection target captured in the field of view.

[0039] The target detection device aggregates the event frame images generated after the fusion processing to form an event frame image data set, which includes a training set and a test set.

[0040] In step S120, the target detection device constructs a target detection model.

[0041] For example, in step S120, the target detection device constructs a target detection model based on the current micro-nano unmanned platform.

[0042] In response to the operator's instructions, the target detection device constructs a target detection model supported by the current micro-nano unmanned platform.

[0043] According to some embodiments, based on the operator types supported by the current micro-nano unmanned platform, the operators used by the target detection model include a convolution operator, a maximum pooling operator, a batch normalization operator, a LeakyRELU operator, and a downsampling operator.

[0044] According to some embodiments, the target detection model includes an event frame feature extraction module and an event frame detection module. The event frame feature extraction module includes multiple convolutional network layers, normalization layers, and pooling layers, and the event frame detection module includes multiple convolutional network layers, normalization layers, pooling layers, and downsampling layers.

[0045] In step S130, based on the training set in the event frame image data set, the object detection device trains the object detection model to update the parameters of the object detection model.

[0046] For example, in step S130, based on the training set in the event frame image data set, the target detection device initializes and trains the target detection model to update the parameters of the target detection model.

[0047] The target detection device randomly initializes the parameters of the constructed target detection model.

[0048] According to some embodiments, the parameters of the target detection model include weights and biases between network neuron nodes of the target detection model.

[0049] Based on the training set in the generated event frame image data set, the target detection device performs forward propagation on the target detection model whose parameters have been initialized.

[0050] According to some embodiments, after randomly initializing the parameters including the weights and biases between the network neuron nodes of the target detection model, the target detection device performs forward propagation on the target detection model according to the training set. During the forward propagation process, the target detection device increases the nonlinear relationship between the networks of the target detection model through the activation function, and generates a nonlinear mapping relationship between the event frame input and the output result of the target detection.

[0051] Furthermore, the target detection device performs backpropagation on the target detection model that has completed forward propagation to update the parameters of the target detection model.

[0052] According to some embodiments, the target detection device calculates the loss function L of the target detection model, and the calculation formula thereof is as follows.

[0053] L=L box +L obj +L cls (1)

[0054] Among them, L box is the loss function of the detection box, L obj is the loss function of the target object, L cls is the loss function of the target object category.

[0055] According to some embodiments, within a preset time, the target detection device keeps the parameters of the target detection model unchanged and calculates the gradient of the loss function of the output layer of the target detection model.

[0056] Furthermore, when the current number of network layers of the target detection model is f=F-1, F-2, ..., 2, the target detection device calculates the gradient of the loss function corresponding to each layer of the target detection model, where F is the total number of network layers of the target detection model.

[0057] According to some embodiments, the target detection device updates the weights and biases in the network of the target detection model according to the gradient descent method.

[0058] In step S140, based on the test set in the event frame image data set, the target detection device obtains target detection information through the trained target detection model.

[0059] For example, in step S140, the target detection device inputs the test set in the event frame image data set into the trained target detection model, and obtains the target detection information output by the target detection model.

[0060] The target detection device uses the test set in the event frame image data set as input data, inputs it into the trained target detection model, and obtains the output result of the target detection model.

[0061] The target detection device performs suppression processing on the output result of the target detection model to obtain target detection information of the detected target.

[0062] According to some embodiments, the target detection device may use a non-maximum suppression (NMS) method to suppress the output result of the target detection model.

[0063] According to some embodiments, the target detection information of the detected target includes target detection box coordinates, target confidence and target category.

[0064] According to the embodiments of the present application, the technical solution of the present application can realize real-time detection of targets, and further reduce the amount of model calculation and parameter through the single-channel data processing method of the event camera and the structural design of the target detection model, reduce data storage space, and shorten the real-time reasoning time of the algorithm. In addition, the technical solution of the present application can be implemented on most end-side chip computing platforms without the need for additional hardware support.

[0065] Figure 3 A schematic diagram showing the composition of a target detection model according to an exemplary embodiment of the present application is shown.

[0066] like Figure 3 As shown, the target detection model 100 includes an event frame feature extraction module 110 and an event frame detection module 120 .

[0067] The event frame feature extraction module 110 includes multiple convolutional network layers, normalization layers and pooling layers.

[0068] According to some embodiments, Figure 3 As shown, the event frame feature extraction module 110 is composed of 12 combination blocks and 2 pooling layers, wherein each combination block includes a convolutional network layer, a batch normalization layer and an activation function.

[0069] The event frame detection module 120 includes a plurality of convolutional network layers, normalization layers, pooling layers, and downsampling layers.

[0070] According to some embodiments, Figure 3 As shown, the event frame detection module 120 is composed of 5 combination blocks, 3 pooling layers and 1 downsampling layer. Each combination block includes 1 convolutional network layer, 1 batch normalization layer and an activation function.

[0071] The input data (i.e., the event frame image data in the event frame image data set) is input into the target detection model 100 through the event frame feature extraction module 110, and after calculation and reasoning, the result (i.e., target detection information) is output through the event frame detection module 120.

[0072] According to some embodiments, the resolution of the event frame image in the input data may be 320*256, the computational complexity of the target detection model 100 is 658 MFLOPS, and the parameter quantity is 0.28M.

[0073] Figure 4 A block diagram of a target detection device applied to a micro-nano unmanned platform according to an exemplary embodiment of the present application is shown.

[0074] like Figure 4 As shown, the target detection device 200 includes a data acquisition unit 210 , a model building unit 220 and a testing unit 230 .

[0075] The data collection unit 210 obtains event data of the current detection target in multiple scenes collected by the event camera.

[0076] According to the event data acquired by the event camera, the data acquisition unit 210 generates a spatial data matrix corresponding to the time stamp.

[0077] The data acquisition unit 210 performs fusion processing on the event data in the spatial data matrix according to a preset time window to obtain event frame images corresponding to all spatial data matrices generated according to timestamps.

[0078] The data acquisition unit 210 aggregates the event frame images generated after the fusion processing to form an event frame image data set, which includes a training set and a test set.

[0079] In response to the operator's instructions, the model building unit 220 builds a target detection model supported by the current micro-nano unmanned platform, wherein the target detection model includes a plurality of convolutional network layers, a normalization layer, a pooling layer, and a downsampling layer.

[0080] The model building unit 220 randomly initializes the parameters of the built target detection model.

[0081] Based on the training set in the generated event frame image dataset, the model building unit 220 performs forward propagation on the target detection model whose parameters have been initialized.

[0082] The model building unit 220 performs back-propagation on the target detection model that has completed forward propagation to update the parameters of the target detection model.

[0083] The testing unit 230 uses the test set in the event frame image data set as input data, inputs it into the trained target detection model, and obtains the output result of the target detection model.

[0084] The testing unit 230 performs suppression processing on the output result of the target detection model to obtain target detection information of the detected target.

[0085] Figure 5 A block diagram of an electronic device according to an exemplary embodiment of the present application is shown.

[0086] like Figure 5 As shown, the electronic device 600 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0087] like Figure 5 As shown, the electronic device 600 is in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 610 may execute the following: Figure 1 The method shown in .

[0088] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0089] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0090] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0091] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0092] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by combining software with necessary hardware. The technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal or a network device, etc.) to execute the method according to the embodiment of the present application.

[0093] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0094] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0095] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0096] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0097] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0098] The embodiments of the present application are described in detail above, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the idea of ​​the present application, the specific implementation method and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as limiting the present application.

Claims

1. A target detection method applied to a micro-nano unmanned platform, characterized in that: include: Acquire an event frame image dataset through an event camera, wherein the event frame image dataset includes a training set and a test set; Build an object detection model; Based on the training set, training the target detection model to update parameters of the target detection model; Based on the test set, target detection information is obtained through a trained target detection model.

2. The method according to claim 1, characterized in that Obtain event frame image datasets through event cameras, including: Acquire event data collected by the event camera; Generate a spatial data matrix according to the event data and by time stamp, wherein the time stamp corresponds to the spatial data matrix one by one; The event data in the spatial data matrix are fused according to a preset time window to obtain an event frame image corresponding to the spatial data matrix.

3. The method according to claim 1, characterized in that The target detection model includes an event frame feature extraction module and an event frame detection module, wherein: The event frame feature extraction module includes a plurality of convolutional network layers, a normalization layer and a pooling layer; The event frame detection module includes multiple convolutional network layers, normalization layers, pooling layers and downsampling layers.

4. The method according to claim 1, characterized in that: Based on the training set, training the target detection model to update parameters of the target detection model includes: Randomly initializing the parameters of the target detection model; Based on the training set, the target detection model with initialized parameters is trained.

5. The method according to claim 4, characterized in that Based on the training set, the target detection model with initialized parameters is trained, including: Based on the training set, forward propagation is performed on the target detection model whose parameters have been initialized; Back-propagation is performed on the target detection model that has completed forward propagation to update the parameters of the target detection model.

6. The method according to claim 5, characterized in that Back-propagation is performed on the target detection model that has completed forward propagation to update the parameters of the target detection model, including: Calculating a loss function of the target detection model; According to the gradient of the loss function, the parameters of the target detection model are updated.

7. The method according to claim 1, characterized in that Based on the test set, target detection information is obtained through the trained target detection model, including: Inputting the test set into the trained target detection model to obtain an output result; The output result is subjected to suppression processing to generate the target detection information.

8. A target detection device applied to a micro-nano unmanned platform, characterized in that: include: A data acquisition unit, configured to acquire an event frame image data set through an event camera, wherein the event frame image data set includes a training set and a test set; A model building unit, used to build a target detection model; based on the training set, the target detection model is trained to update the parameters of the target detection model; The testing unit is used to obtain target detection information based on the test set through a trained target detection model.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images