Event camera-based target detection network self-supervised training method and device

Abstract

The present application discloses a method and device for self-supervised training of a target detection network based on an event camera, the method comprising: obtaining an RGB image and a DVS event sequence of a target; annotating the RGB image and converting the RGB image into a simulated event sequence; characterizing the DVS event sequence and the simulated event sequence; migrating the label of the RGB image to simulated event representation data, and pre-training the target detection network model using the labeled simulated event representation data; predicting the DVS event representation data using the pre-trained target detection network model to obtain a pseudo-label of the DVS event sequence; and fine-tuning the weights of the pre-trained target detection network model using the labeled simulated event grid data and the pseudo-labeled DVS event grid data. The present application increases the amount of data and data diversity, improves the robustness of the target detection algorithm, and reduces the cost of manual labeling.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to a method and device for training a target detection network based on an event camera. Background Art

[0002] Object detection is one of the important research directions in the field of computer vision, and has a wide range of application scenarios in the fields of human-computer interaction, abnormal behavior recognition, intelligent monitoring, virtual reality, etc. The main task of object detection is to identify all objects of interest in an image and determine their position and size.

[0003] In recent years, with the popularization and development of CMOS active pixel sensing (APS) and deep learning, APS-based computer vision has made great progress in the field of target detection technology. However, problems such as limited frame rate, high inter-frame redundancy, and low dynamic range of APS cameras have greatly restricted the rapid and widespread application of target detection technology in actual products.

[0004] Event cameras have shown great application potential in challenging scenarios in traditional industrial vision fields such as high speed, high dynamics, and low latency. For example, Chinese patent application 202010198845.1 discloses a high-dynamic target detection method based on event cameras.

[0005] However, the current object detection technology based on event data is still a very challenging task. First of all, in actual engineering applications, due to the lack of a large number of labeled event training data sets and the lack of a general evaluation standard for event data, the only way is to convert data slices into event graphs and use manual annotation methods to construct labeled event data sets, which has very high labor costs. At the same time, traditional deep learning has significant performance in the field of computer vision, but due to the asynchronous and sparsity of event data, the characteristics of directly learning events are extremely complex, making the information encoding of event data different from traditional computer vision and unable to be directly adapted to traditional networks. In addition, the asynchronous output based on event cameras cannot be directly applied to traditional deep learning. After obtaining event stream data, the existing technology accumulates events within a fixed time interval to generate an event graph, but this only considers the number of event points, and does not take into account the unique time characteristics and positive and negative polarities of event cameras compared to ordinary cameras, which loses a lot of effective information. Summary of the invention

[0006] The present application aims to disclose a method and device for training a target detection network based on an event camera, which uses a supervised learning method to effectively solve the problem of few-sample target detection when there is insufficient labeled event data set, improve the performance of few-sample target detection, and reduce data annotation costs.

[0007] In the first aspect of the present application, a method for training a target detection network model is disclosed, comprising:

[0008] Get the target's RGB image and DVS event sequence;

[0009] Annotating the RGB image and converting the RGB image into a simulated event sequence;

[0010] Characterizing the DVS event sequence and the simulated event sequence;

[0011] Transfer the labels of RGB images to simulated event representation data, and use the labeled simulated event representation data to pre-train the target detection network model;

[0012] Predicting the DVS event representation data using a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence;

[0013] The pre-trained object detection network model weights are fine-tuned using the labeled simulated event grid data and the pseudo-labeled DVS event grid data.

[0014] In some examples, the characterizing the DVS event sequence and the simulation event sequence includes: converting event data into a three-layer grid representation, wherein three channels are represented as positive polarity event points, negative polarity event points, and a normalized average timestamp, respectively.

[0015] In some examples, before performing the grid representation, the method further includes removing background noise and / or salt and pepper noise of the DVS event sequence and the simulation event sequence.

[0016] In some examples, a threshold method is used to remove background noise of the DVS event sequence and the simulation event sequence, and / or a morphological method is used to remove the salt and pepper noise.

[0017] In some examples, the target detection network model uses a yolov3 network to pre-train the target detection network model in a supervised manner.

[0018] In some examples, when fine-tuning the pre-trained object detection network model weights using labeled simulated event grid data and pseudo-labeled DVS event grid data, the total loss is a weighted sum of labeled and unlabeled loss terms.

[0019] The second aspect of the present application discloses a method for generating pseudo labels for an image, comprising:

[0020] Get the target's RGB image and DVS event sequence;

[0021] Annotating the RGB image and converting the RGB image into a simulated event sequence;

[0022] Characterizing the DVS event sequence and the simulated event sequence;

[0023] Transfer the labels of RGB images to simulated event representation data, and use the labeled simulated event representation data to pre-train the target detection network model;

[0024] The DVS event representation data is predicted using a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence.

[0025] In a third aspect of the present application, a device for generating pseudo-image labels is disclosed, comprising:

[0026] Image acquisition module, used to acquire the RGB image and DVS event sequence of the target;

[0027] A label generation module, used to label the RGB image;

[0028] An event simulator, used for converting the RGB image into a simulated event sequence;

[0029] A characterization module, used for characterizing the DVS event sequence and the simulation event sequence;

[0030] The first input module is used to transfer the label of the RGB image to the simulated event representation data, and use the labeled simulated event representation data to pre-train the target detection network model;

[0031] The second input module is used to input the DVS event representation data into a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence.

[0032] In some examples, the image acquisition module includes a CeleX event camera that outputs DVS data and RGB data in a patrol manner.

[0033] In a fourth aspect of the present application, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the described method when executed by a processor.

[0034] This application uses mixed data (i.e., event sequences and RGB images) as input, which increases the amount and diversity of data and improves the robustness of the target detection algorithm.

[0035] In addition, by using the present application to obtain pseudo labels for DVS events, the process of labeling event grid data can be skipped, thus reducing the cost of manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0037] Figure 1 is a schematic diagram of the image fusion method according to an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a device for generating pseudo-image labels according to an embodiment of the present application;

[0039] Figure 3 A flowchart of a target detection network model training method according to an embodiment of the present application;

[0040] Figure 4 An example of event characterization for the present invention;

[0041] Figure 5 The figure is a schematic diagram of the effect of using the present invention to perform target detection. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] Due to the lack of a large number of labeled event datasets in actual industrial scenarios, in order to increase the number and diversity of samples, the present application discloses a method for generating image pseudo-labels.

[0045] Figure 1 The following is a schematic flow chart of a method for generating pseudo-labels for an image according to an embodiment of the present application. As shown in the figure, the method includes:

[0046] S101, acquiring the RGB image and DVS event sequence of the target;

[0047] The DVS event camera is a bio-inspired data-driven sensor whose output depends on the amount of motion or changes in brightness in the scene. The faster the motion, the more events are generated per second. The event camera generates events with microsecond resolution and transmits them with sub-millisecond latency, which allows the event camera to react quickly to visual stimuli.

[0048] In this application, a target image acquisition system equipped with a DVS event camera and a traditional RGB camera is used to simultaneously acquire RGB image data and a DVS event sequence to obtain mixed data. Specifically, the DVS event camera and the RGB camera are set at the same position, for example, the two are constructed as a module or component, and the time synchronization and data synchronization of the two are achieved through image registration.

[0049] Alternatively, the CeleX event camera of CelePixel can be used. The camera can output grayscale images and event data in a round-robin manner, that is, output DVS data and RGB data in a round-robin manner. This can omit the camera alignment step and save data image acquisition time.

[0050] Although the CeleX camera patrol output of grayscale images and event data can save the camera registration step, it will also cause a slight time deviation between the two types of data, so the data needs to be time-corrected.

[0051] S102, annotating the RGB image and converting the RGB image into a simulation event sequence;

[0052] For example, LabelImg is used to mark the bounding box (label) for the RGB image and convert it to a uniform resolution. Alternatively, other labeling tools can be used, which are mature technologies and will not be described in detail here.

[0053] Using an event simulator (such as the ESIM event camera simulator), the RGB image is converted into a simulated event sequence by linear interpolation and Gaussian noise perturbation.

[0054] S103, characterizing the DVS event sequence and the simulation event sequence;

[0055] In some examples, the DVS event sequence acquired by the DVS camera and the simulated event sequence converted by the event simulator are converted into a three-layer grid representation, in which the three channels represent the number of positive polarity event points, the number of negative polarity event points and the normalized average timestamp, respectively, so as to preserve the integrity of the data as much as possible.

[0056] In some other examples, methods such as event accumulation, time surface, and time histogram may also be used to characterize the DVS event sequence and the simulation event sequence.

[0057] In an event camera, background noise occurs when an event that is not caused by an event source occurs at a certain pixel. In addition, salt and pepper noise (impulse noise) may also exist in the image, which is a random white or black point, which may be a black pixel in a bright area or a white pixel in a dark area. These noises will greatly interfere with target detection. Therefore, before converting the event sequence (including DVS event sequence and simulation event sequence) into a grid representation, noise needs to be removed.

[0058] In some examples, a threshold method is used to remove background noise from the simulated event sequence and the DVS event sequence, and a morphological method is used to remove salt and pepper noise from the simulated event sequence and the DVS event sequence.

[0059] S104, transferring the label of the RGB image to the simulated event representation data, and using the simulated event representation data with labels to pre-train the target detection network model;

[0060] In some examples, the target detection network model uses the YOLOv3 network. The YOLOv3 network has very good target detection performance in RGB images. In other examples, the target detection network model can also use other networks such as YOLOv5 and SSD.

[0061] By migrating the labels of RGB images to simulated event representation data (such as grid representation), the process of labeling simulated events can be skipped, thereby reducing the cost of manual labeling.

[0062] The object detection network model is pre-trained in a supervised manner using labeled simulated event representation data.

[0063] S105, predicting the DVS event representation data using a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence.

[0064] The DVS event representation data (e.g., grid representation) is input into the pre-trained target detection network model to predict the unlabeled DVS events. If the predicted probability P is greater than the threshold T, it is considered a high-confidence sample and its result is used as a pseudo label.

[0065] Figure 2 The following is a schematic diagram of a device for generating pseudo-image labels according to an embodiment of the present application. In this embodiment, the device includes:

[0066] An image acquisition module 201 is used to acquire an RGB image and a DVS event sequence of a target;

[0067] A label generation module 202, used to label the RGB image;

[0068] An event simulator 203, used to convert the RGB image into a simulated event sequence;

[0069] A characterization module 204 is used to characterize the DVS event sequence and the simulation event sequence; for example, converting the DVS event sequence and the simulation event sequence into a grid representation;

[0070] The first input module 205 is used to transfer the label of the RGB image to the simulated event representation data, and use the simulated event representation data with labels to pre-train the target detection network model;

[0071] For example, labeled simulated event grid data is input into an object detection network model to pre-train the model.

[0072] The second input module 206 is used to input the DVS event representation data into a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence.

[0073] For example, unlabeled DVS event grid data is input into a pre-trained target detection network model for prediction to obtain a pseudo label of the DVS event sequence.

[0074] The present application obtains pseudo labels of DVS events through the above method or device, which can skip the process of labeling event grid data, reduce the cost of manual labeling, and then quickly obtain a large amount of labeled DVS event data for building a data set for training target detection networks.

[0075] Figure 3 Schematic diagram of the target detection network model training method according to the embodiment of the present application. The method first uses the aforementioned image pseudo label generation method to obtain the pseudo label of the DVS event sequence, specifically referring to the aforementioned steps S101 to S105 ( Figure 3 The corresponding steps are S301 to S305), which will not be repeated here. And also include:

[0076] S306, fine-tuning the weights of the pre-trained target detection network model using the labeled simulated event representation data and the pseudo-labeled DVS event representation data. During the training process, S305 and S306 are repeated to adjust the target detection network weights until an optimal model is obtained.

[0077] In some examples, labeled simulated event grid data and pseudo-labeled DVS event grid data are added to the training set at the same time, and the weights of the pre-trained target detection network model are fine-tuned, that is, the pre-trained target detection network model is applied to the new DVS data set. Because the new DVS data and the simulated data are relatively similar, only the latter several layers of the model network are adjusted.

[0078] Among them, when fine-tuning the weights of the pre-trained object detection network model, the total loss is the weighted sum of the labeled and unlabeled loss terms, expressed as follows:

[0079]

[0080] Where n and n' represent the minimum batch of minimum gradient descent for labeled and unlabeled data, respectively; y and y' represent the true labels of m samples in labeled data and the pseudo labels of m samples in unlabeled data, respectively; and They represent the output set of m samples in labeled data and the output set of m samples in unlabeled data respectively; α(t) represents the balance coefficient:

[0081]

[0082] α(t) has a great impact on network performance. If it is too small, the pseudo-label data will have no effect on the network. If it is too large, it will interfere with the labeled data. To ensure that the model has learned enough information from the labeled data, this parameter is set to 0 in the initial T1 epochs, and then slowly increases and remains unchanged after T2 epochs. That is, when t>T2, α f Remain unchanged (e.g. α f 3), which can avoid local minima as much as possible, making the pseudo labels of unlabeled data as similar to the true labels as possible.

[0083] Figure 4 An example of event characterization for the present invention is: Figure 4 (a) is the original image. Figure 4 (b) is the characterization result.

[0084] Due to the limited frame rate, high inter-frame redundancy, and high power consumption of ordinary cameras, their effectiveness is greatly limited. Event cameras are very suitable for special applications such as high-speed detection, recognition, and obstacle avoidance due to their high speed, high dynamics, and low latency. However, the labeled event data is very limited, especially in the field of target detection. The relevant target detection algorithms are still under development, and there is no more efficient target detection algorithm.

[0085] This application uses mixed data (i.e., event sequences and RGB images) as input, which increases the amount and diversity of data and improves the robustness of the target detection algorithm. Figure 5 The figure shows the effect of performing target detection using the target detection model trained by the present invention.

[0086] According to an embodiment of the present application, a non-transient computer-readable storage medium is also disclosed for storing non-transient software programs, non-transient computer executable programs and modules, such as programs or instructions corresponding to the image pseudo-label generation method or the target detection network model training method in the aforementioned embodiment of the present application. The processor implements the image pseudo-label generation method or the target detection network model training method in the above method embodiment by running the non-transient software program or instruction.

[0087] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips.

[0088] Although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the inventive concept of the present application, the technical solutions of the embodiments of the present application may be modified or replaced by equivalents, which should not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target detection network model training method, characterized in that: include: Get the target's RGB image and DVS event sequence; Annotating the RGB image and converting the RGB image into a simulated event sequence; Characterizing the DVS event sequence and the simulated event sequence, including: converting the event data into a three-layer grid representation, wherein three channels are represented as positive polarity event points, negative polarity event points, and a normalized average timestamp, respectively; Transfer the labels of RGB images to simulated event representation data, and use the labeled simulated event representation data to pre-train the target detection network model; Predicting the DVS event representation data using a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence; The pre-trained object detection network model weights are fine-tuned using the labeled simulated event grid data and the pseudo-labeled DVS event grid data.

2. The target detection network model training method according to claim 1, characterized in that: Before performing the grid representation, the process also includes removing background noise and / or salt and pepper noise of the DVS event sequence and the simulation event sequence.

3. The target detection network model training method according to claim 2, characterized in that: A threshold method is used to remove background noise of the DVS event sequence and the simulation event sequence, and / or a morphological method is used to remove the salt and pepper noise.

4. The target detection network model training method according to claim 1, characterized in that: The target detection network model uses the yolov3 network to pre-train the target detection network model in a supervised manner.

5. The target detection network model training method according to claim 1, characterized in that: When fine-tuning the pre-trained object detection network model weights using labeled simulated event grid data and pseudo-labeled DVS event grid data, the total loss is a weighted sum of labeled and unlabeled loss terms.

6. A method for generating pseudo-labels for an image, characterized in that: include: Get the target's RGB image and DVS event sequence; Annotating the RGB image and converting the RGB image into a simulated event sequence; Characterizing the DVS event sequence and the simulated event sequence, including: converting the event data into a three-layer grid representation, wherein three channels are represented as positive polarity event points, negative polarity event points, and a normalized average timestamp, respectively; Transfer the labels of RGB images to simulated event representation data, and use the labeled simulated event representation data to pre-train the target detection network model; The DVS event representation data is predicted using a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence.

7. An image pseudo-label generation device, characterized in that: include: Image acquisition module, used to acquire the RGB image and DVS event sequence of the target; A label generation module, used to label the RGB image; An event simulator, used for converting the RGB image into a simulated event sequence; A characterization module is used to characterize the DVS event sequence and the simulation event sequence, including: converting the event data into a three-layer grid representation, wherein the three channels are represented as the number of positive polarity event points, the number of negative polarity event points, and the normalized average timestamp, respectively; The first input module is used to transfer the label of the RGB image to the simulated event representation data, and use the simulated event representation data with labels to pre-train the target detection network model; The second input module is used to input the DVS event representation data into a pre-trained target detection network model to obtain a pseudo label of the DVS event sequence.

8. The image pseudo-label generating device according to claim 7, characterized in that: The image acquisition module includes a CeleX event camera, which outputs DVS data and RGB data in a patrol mode.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 6.

Images