Method and device for realizing target segmentation based on event sequence training neural network

Abstract

The present invention discloses a method and device for realizing target segmentation based on event sequence training neural network. The segmentation method comprises: obtaining the event sequence of the target, converting the event sequence into event frames using a time window of a predetermined length; inputting the event frame into a trained target segmentation network to obtain a target segmentation map; and segmenting the target segmentation map into a binary map based on a set threshold. The present invention uses an event camera to obtain the target event sequence, and completes high-speed target segmentation by training a neural network. It only needs to mark and train a few types of targets to complete the segmentation of untrained targets, and has a category-independent characteristic. In addition, it can effectively avoid the interference of noise, and at the same time has strong environmental adaptability, and also has good robustness in dim light.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and specifically relates to a method and device for realizing target segmentation based on event sequence training of neural network. Background Art

[0002] In some intelligent agent obstacle avoidance tasks, it is necessary to detect oncoming obstacles and estimate the depth of the target area in order to make obstacle avoidance plans in a timely manner. Obstacle detection and target area extraction can be achieved by using target detection or target segmentation methods.

[0003] When an intelligent agent detects a target, it needs to collect data of that target and train it for each target, which is very costly. In addition, the obstacles encountered by the intelligent agent in the actual environment are diverse and it is impossible to collect all of them in advance. Traditional image segmentation is mainly based on RGB images. Since RGB images are shot at a fixed frame rate, motion blur will be formed when capturing high-speed moving targets, and it is difficult to detect targets under low light conditions.

[0004] Event cameras can capture high-speed moving targets asynchronously and are also robust in low light conditions, which can solve the above problems to a certain extent. For event cameras, the threshold segmentation method is generally used to segment moving targets, that is, the target is segmented by taking advantage of the fact that the number of events of high-speed moving targets is greater than the background. However, when there is background noise (such as background shaking), it is difficult to accurately segment irrelevant targets, so the environmental adaptability is weak. Summary of the invention

[0005] The present invention aims to disclose a target segmentation method and device based on an event camera, so as to improve the anti-noise interference capability and environmental adaptability of the target segmentation.

[0006] According to a first aspect of the present invention, a target segmentation method is disclosed, comprising:

[0007] Acquire an event sequence of a target, and convert the event sequence into an event frame using a time window of a predetermined length;

[0008] Inputting the event frame into a trained object segmentation network to obtain an object segmentation map; and

[0009] The target segmentation map is segmented into a binary map based on a set threshold.

[0010] In some other examples, each event frame includes a first channel, a second channel and a third channel; wherein, for any time window, all events with p being positive in the time window are accumulated, and the cumulative number of events with p being positive at each position (x, y) is used as the pixel value at the position (x, y) to obtain the first channel of the event frame; all events with p being negative in the time window are accumulated, and the cumulative number of events with p being negative at each position (x, y) is used as the pixel value at the position (x, y) to obtain the second channel of the event frame; and the average time plane is used as the third channel of the event frame.

[0011] In some other examples, inputting the event frame into a trained target segmentation network includes: the event frame input into the target segmentation network includes a first channel, a second channel, a third channel, or only includes the third channel, or only includes the first channel and the second channel.

[0012] In some other examples, the object segmentation network first uses a convolutional layer to extract features from the input event frame, and then uses a deconvolution layer to restore the image size.

[0013] In some other examples, the target segmentation network consists of an encoder and a decoder, the encoder consists of multiple convolutional layers, the decoder consists of multiple deconvolutional layers, and each convolutional layer and deconvolution layer is connected with a BN layer and a RELU layer.

[0014] In some other examples, the output of the convolutional layer and the deconvolution layer with the same output size are stacked and input into the next deconvolution layer.

[0015] In some other examples, the first layer does not adopt a lightweight network structure, and is followed by a BN layer and a RELU layer; other layers all adopt a lightweight network structure, each convolution layer consists of a depth-separable convolution layer and a 1x1 convolution layer, each deconvolution layer consists of a depth-separable deconvolution layer and a 1x1 convolution layer, and each 1x1 convolution layer is connected to a BN layer and a RELU layer.

[0016] According to the second aspect of the present invention, there is also disclosed an object segmentation device, comprising:

[0017] A data acquisition module is used to acquire the event sequence of the target;

[0018] A conversion module, converting the event sequence into event frames using a time window of a predetermined length;

[0019] A data preprocessing module, used for preprocessing the converted event frames; and

[0020] The target segmentation module uses a trained target segmentation network to calculate the preprocessed event frame to obtain a target segmentation map, and divides the target segmentation map into binary maps according to a set threshold.

[0021] According to the third aspect of the present invention, a target segmentation network training method is also disclosed, comprising:

[0022] Obtaining an event sequence of a target, converting the event sequence into event frames according to the length of a time window, marking target regions and categories for the event frames, and constructing a data set for training a target segmentation network;

[0023] The target segmentation network is trained using the dataset.

[0024] In some other examples, the event frame used to construct the data set includes a first channel, a second channel, and a third channel.

[0025] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the method described in any one of the above schemes when executed by a processor.

[0026] Compared with the existing segmentation methods, the present invention uses an event camera to obtain the target event sequence, completes high-speed target segmentation by training a neural network, and can complete the segmentation of untrained targets by only marking and training a few types of targets, which has the characteristics of category independence. In addition, it can effectively avoid noise interference, has strong environmental adaptability, and also has good robustness in dim light. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0028] Figure 1 1. It is a schematic diagram of the flow of a method for object segmentation based on an event camera according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a target segmentation network structure according to an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a target segmentation network structure according to another embodiment of the present invention;

[0031] Figure 4 A schematic diagram of a target segmentation network structure according to yet another embodiment of the present invention;

[0032] Figure 5 This is an example of target segmentation results obtained by using the present invention;

[0033] Figure 6 A schematic diagram of a target segmentation network training method according to an embodiment of the present invention;

[0034] Figure 7 Schematic diagram of the structure of a target segmentation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0036] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] When using the threshold segmentation method to segment targets based on event data streams, for example, using the histogram peak as the segmentation threshold for target segmentation is easily affected by noise, such as background shaking, or other moving objects, resulting in weak environmental adaptability of the target segmentation method.

[0038] To solve this problem, the present invention discloses a target segmentation method based on event camera, such as Figure 1 As shown, the method comprises the following steps:

[0039] S101, acquiring an event sequence of a target, and converting the event sequence into event frames using a time window of a predetermined length;

[0040] The event camera is used to obtain the event stream data of the target, that is, the event sequence. For example, the event sequence is (x, y, p, t), where (x, y) is the location coordinate of the event; p is positive or negative polarity, and positive polarity means that the light intensity at the current moment is stronger than the light intensity at the previous moment, and the change exceeds a certain threshold; negative polarity means that the light intensity becomes weaker; t represents the time information of the current event.

[0041] In the present invention, the event sequence is converted into an event frame according to the length of the time window (for example, 5ms). Each event frame includes a first channel, a second channel, and a third channel. Specifically, for any time window, all events with p being positive in the time window are accumulated, and the cumulative number of events with p being positive at each position (x, y) is used as the pixel value at the position (x, y) to obtain the first channel of the event frame; all events with p being negative in the time window are accumulated, and the cumulative number of events with p being negative at each position (x, y) is used as the pixel value at the position (x, y) to obtain the second channel of the event frame; and the average time plane is used as the third channel of the event frame. Among them, the cumulative number of events (that is, including positive and negative polarities) at each position (x, y) in the time window is I x,y , then the average time surface is Among them, t is the time information of each event.

[0042] The present invention combines three channels to more comprehensively express the characteristics of the event sequence in a combined description. At the same time, since the RGB image has three channels, the present invention can simulate the three-channel processing method by using this method. S102, input the event frame into a trained target segmentation network to obtain a target segmentation map;

[0043] In some examples, before the event frame is input into the target segmentation network, the converted event frame may be preprocessed, including one or a combination of flipping, translating, rotating, and scaling the event frame. For example, the converted event frame is preprocessed as follows, including:

[0044] (i) Center cropping step, for example, cropping the center of the image to a multiple of 8 to facilitate GPU parallel computing.

[0045] (ii) Normalization step. Considering that the pixel values ​​of the three-channel event frames are in the range of 0-255 when they are saved as images, the following method is used to normalize the event frame I to map the pixel values ​​to [-1, 1]:

[0046]

[0047] The event frame input into the target segmentation network includes the first channel, the second channel, the third channel, or only the third channel, or only the first channel and the second channel.

[0048] In particular, by combining the cumulative description of positive and negative events with the average time surface description, the characteristics of the event sequence can be described more richly.

[0049] The target segmentation network structure used in the present invention is as follows: Figure 2As shown in the figure, the target segmentation network consists of an encoder and a decoder, where conv represents a convolutional layer and deconv represents a deconvolutional layer. Each convolutional layer and deconvolutional layer is followed by a BN layer and a RELU layer. The target segmentation network first uses the convolutional layer to extract features from the input event frame, and then uses the deconvolutional layer to restore the image size to obtain the target segmentation result.

[0050] It can be understood that in the target segmentation network, the number of convolutional layers and deconvolutional layers can be selected according to the actual application, for example Figure 2 As shown, the numbers of the two are the same, or the numbers of the two can be set to be different. However, generally, the number of convolutional layers and deconvolutional layers is greater than or equal to 3.

[0051] In addition, it is possible to Figure 2 The target segmentation network shown in the figure is lightweight and its structure is as follows: Figure 3 As shown in the figure, a lightweight design is performed using depthwise separable convolution. Among them, dwconv represents a depthwise separable convolution layer, and dwdeconv represents a depthwise separable deconvolution layer. In this lightweight neural network, the first layer is not lightweight, and is followed by a BN layer and a RELU layer. The other five layers are lightweight, and each convolution layer consists of a depthwise separable convolution layer and a 1x1 convolution layer, and each deconvolution layer consists of a depthwise separable deconvolution layer and a 1x1 convolution layer. Each 1x1 convolution layer is followed by a BN layer and a RELU layer.

[0052] In some other examples, you can also use Figure 4 The target segmentation network structure shown in Figure 2 Based on the network structure shown, the output of the first convolution layer is superimposed with the output of the second deconvolution layer and then input into the third deconvolution layer, and the output of the second convolution layer is superimposed with the output of the first deconvolution layer and then input into the second deconvolution layer.

[0053] It can be understood that the superimposed layers can also be selected according to the actual application, as long as the output size of the superimposed convolutional layer and the deconvolutional layer is consistent. For example, only one or part of the convolutional layer and the deconvolutional layer are superimposed and input into the next deconvolutional layer.

[0054] Among them, you can follow Figure 3 The lightweight structure shown is Figure 5 The object segmentation network shown is designed to be lightweight.

[0055] Since the characteristic of event sequence is to capture high-speed moving targets asynchronously and is not limited by frame rate, the segmentation network has high processing efficiency to meet the asynchronicity. However, traditional segmentation networks such as FCN and DeepLab have large structures. On the one hand, they have many parameters, and the actual unmanned embedded system memory may not be able to meet the requirements; on the other hand, the processing time is long, so that the drone may not have time to react and avoid obstacles when it moves quickly.

[0056] The network structure of the present invention is simple, the processing speed is fast, and the number of parameters is small, so it can be applied to the embedded system of unmanned aerial vehicles. The traditional segmentation network has tens of millions of parameters, while the basic network of the present invention has only tens of thousands of parameters, and after lightweighting, it can have only more than 3,000 parameters, which is more suitable for use on carriers with small onboard memory and limited memory, and can achieve fast obstacle avoidance during fast movement, such as flying.

[0057] At the same time, the present invention adopts a network structure of convolutional layer connected to deconvolutional layer, which can further improve the segmentation accuracy without increasing the number of parameters. Compared with the traditional image segmentation network (such as deeplab, etc.), the segmentation network used in the present invention has fewer layers, and through further lightweight design, the complexity of target segmentation is greatly reduced, especially suitable for the asynchronous characteristics of event cameras.

[0058] S103: Segment the target segmentation map into binary maps based on a set threshold.

[0059] The result directly output by the above target segmentation network is not a binary image, therefore, the target segmentation image is segmented into a binary image according to a set threshold. Figure 5 This is an example of a target segmentation result obtained using the above method.

[0060] The following describes the training process of the target segmentation network, wherein the target segmentation network adopts the following Figure 2-Figure 4 Any of the neural networks shown or their lightweight design. Figure 6 As shown, the training method comprises the following steps:

[0061] S601, obtaining a target event sequence and constructing a data set for training a target segmentation network;

[0062] The event camera is used to obtain the event stream data of the target, that is, the event sequence. The event sequence is represented as (x, y, p, t), where (x, y) is the location coordinate of the event; p is the positive and negative polarity; t represents the time information of the current event.

[0063] The event sequence is converted into event frames according to the time window length (eg, 5 ms), wherein each event frame includes a first channel, a second channel, and a third channel.

[0064] Specifically, for any time window, all events with positive polarity p in the time window are accumulated, and the accumulated number of events at each position (x, y) is used as the pixel value at the position (x, y) to obtain the first channel of the event frame; all events with negative polarity p in the time window are accumulated, and the accumulated number of events at each position (x, y) is used as the pixel value at the position (x, y) to obtain the second channel of the event frame; and the average time plane is used as the third channel of the event frame. Among them, the accumulated number of events at each position (x, y) in the time window (i.e., including positive and negative polarity) is I x,y , then the average time surface is Among them, t is the time information of each event.

[0065] The target area and category are labeled for the event frame, each event frame corresponds to one label, and the labeled data is used to construct a training set, a validation set, and a test set, for example, in a ratio of 60%, 20%, and 20%.

[0066] The event frame used to construct the data set includes the first channel, the second channel, and the third channel, or only includes the third channel, or only includes the first channel and the second channel.

[0067] S602, training a target segmentation network using the data set;

[0068] The training set is input into the target segmentation network for training. The loss function adopts the cross entropy loss function, or adopts the loss function such as dice loss and focal loss. The batch size (the number of samples trained at one time), optimizer and learning rate can be set as needed. For example, the batch size can be selected as 64, the optimizer can be selected as Adam, and the learning rate is set to 0.001.

[0069] In addition, before the training set is input into the target segmentation network, the event frames in the training set may be preprocessed, including one or a combination of flipping, translating, rotating, and scaling the event frames. For example, the event frames are preprocessed as follows, including:

[0070] (i) A center cropping step, for example, cropping the center of the image to a size that is a multiple of 8.

[0071] (ii) Normalization step.

[0072] The present invention uses an event camera to obtain a target event sequence, and completes high-speed target segmentation by training a neural network. It only needs to mark and train a few types of targets to complete the segmentation of untrained targets, and has a category-independent characteristic.

[0073] Compared with the threshold segmentation method, the present invention adopts a neural network for training, which can effectively avoid the interference of noise.

[0074] In addition, since the event camera is used to acquire target data, the present invention has strong environmental adaptability and also has good robustness in dim light.

[0075] Figure 7 FIG. 1 is a schematic diagram of a target segmentation device based on an event camera according to an embodiment of the present invention. In this embodiment, the device includes:

[0076] A data acquisition module 701, such as an event camera, is used to acquire a target event sequence;

[0077] The conversion module 702 converts the event sequence into an event frame using a time window of a predetermined length; wherein each event frame includes a first channel, a second channel, and a third channel. Specifically, for any time window, all events with p being positive in the time window are accumulated, and the accumulated number of events with p being positive at each position (x, y) is used as the pixel value at the position (x, y) to obtain the first channel of the event frame; all events with p being negative in the time window are accumulated, and the accumulated number of events with p being negative at each position (x, y) is used as the pixel value at the position (x, y) to obtain the second channel of the event frame; and the average time plane is used as the third channel of the event frame.

[0078] The data preprocessing module 703 is used to preprocess the event frame obtained by conversion; the preprocessing includes one or a combination of flipping, translating, rotating and scaling the event frame. For example, the event frame obtained by conversion is preprocessed as follows, including:

[0079] (i) A center cropping step, for example, cropping the center of the image to a size that is a multiple of 8.

[0080] (ii) Normalization step.

[0081] The target segmentation module 704 uses the trained target segmentation network to calculate the preprocessed event frame to obtain a target segmentation map, and divides the target segmentation map into binary maps according to a set threshold.

[0082] The target segmentation network adopts Figure 2 The neural network shown in Figure 3 The lightweight neural network shown in Figure 4 A neural network with outputs superimposed as shown.

[0083] According to an embodiment of the present invention, a non-transient computer-readable storage medium is also disclosed for storing non-transient software programs, non-transient computer executable programs and modules, such as programs or instructions corresponding to the event camera-based target segmentation method or target segmentation network training method in the aforementioned embodiment of the present invention. The processor implements the target segmentation method or target segmentation network training method in the above method embodiment by running the non-transient software program or instruction.

[0084] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips.

[0085] Although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the inventive concept of the present invention, the technical solutions of the embodiments of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target segmentation method, characterized in that: include: Acquire the event sequence of the target, and convert the event sequence into an event frame using a time window of a predetermined length; wherein the event sequence is represented as (x, y, p, t), (x, y) is the position coordinate of the event, p is positive or negative polarity, and t represents the time information of the current event; each event frame includes a first channel, a second channel, and a third channel; wherein, for any time window, accumulate all events with p being positive polarity in the time window, and use the accumulated number of events with p being positive polarity at each position (x, y) as the pixel value at the position (x, y) to obtain the first channel of the event frame; accumulate all events with p being negative polarity in the time window, and use the accumulated number of events with p being negative polarity at each position (x, y) as the pixel value at the position (x, y) to obtain the second channel of the event frame; and use the average time plane as the third channel of the event frame; Inputting the event frame into a trained target segmentation network to obtain a target segmentation map; wherein the target segmentation network is composed of an encoder and a decoder, the encoder is composed of multiple convolutional layers, the decoder is composed of multiple deconvolutional layers, and each convolutional layer and deconvolutional layer is connected to a BN layer and a RELU layer; and The target segmentation map is segmented into a binary map based on a set threshold.

2. The target segmentation method according to claim 1, characterized in that: In the target segmentation network, the outputs of the convolutional layer and the deconvolution layer with the same output size are superimposed and then input into the next deconvolution layer.

3. The target segmentation method according to claim 1, characterized in that: The first layer of the target segmentation network does not adopt a lightweight network structure, and is subsequently connected to a BN layer and a RELU layer; the other layers all adopt a lightweight network structure, each convolution layer consists of a depth-separable convolution layer and a 1x1 convolution layer, each deconvolution layer consists of a depth-separable deconvolution layer and a 1x1 convolution layer, and each 1x1 convolution layer is connected to a BN layer and a RELU layer.

4. A target segmentation device, characterized in that: include: A data acquisition module is used to acquire the event sequence of the target; A conversion module converts the event sequence into an event frame using a time window of a predetermined length; wherein the event sequence is represented as (x, y, p, t), (x, y) is the position coordinate of the event, p is positive or negative polarity, and t represents the time information of the current event; each event frame includes a first channel, a second channel, and a third channel; wherein, for any time window, all events with p being positive in the time window are accumulated, and the accumulated number of events with p being positive at each position (x, y) is used as the pixel value at the position (x, y) to obtain the first channel of the event frame; all events with p being negative in the time window are accumulated, and the accumulated number of events with p being negative at each position (x, y) is used as the pixel value at the position (x, y) to obtain the second channel of the event frame; and the average time plane is used as the third channel of the event frame; A data preprocessing module, used for preprocessing the converted event frames; and The target segmentation module uses a trained target segmentation network to calculate the preprocessed event frame to obtain a target segmentation map, and divides the target segmentation map into a binary map according to a set threshold; wherein the target segmentation network is composed of an encoder and a decoder, the encoder is composed of multiple convolutional layers, and the decoder is composed of multiple deconvolutional layers, and each convolutional layer and deconvolution layer is connected to a BN layer and a RELU layer.

5. The target segmentation device according to claim 4, characterized in that: In the target segmentation network, the outputs of the convolutional layer and the deconvolution layer with the same output size are superimposed and then input into the next deconvolution layer.

6. The target segmentation device according to claim 4, characterized in that: The first layer of the target segmentation network does not adopt a lightweight network structure, and is subsequently connected to a BN layer and a RELU layer; the other layers all adopt a lightweight network structure, each convolution layer consists of a depth-separable convolution layer and a 1x1 convolution layer, each deconvolution layer consists of a depth-separable deconvolution layer and a 1x1 convolution layer, and each 1x1 convolution layer is connected to a BN layer and a RELU layer.

7. A target segmentation network training method, characterized in that: include: Obtain the event sequence of the target, convert the event sequence into event frames according to the length of the time window, mark the target area and category for the event frame, and construct a data set for training the target segmentation network; wherein the event sequence is represented as (x, y, p, t), (x, y) is the position coordinate of the event, p is positive or negative polarity, and t represents the time information of the current event; each event frame includes a first channel, a second channel, and a third channel; wherein, for any time window, all events with p being positive polarity in the time window are accumulated, and the cumulative number of events with p being positive polarity at each position (x, y) is used as the pixel value at the position (x, y) to obtain the first channel of the event frame; all events with p being negative polarity in the time window are accumulated, and the cumulative number of events with p being negative polarity at each position (x, y) is used as the pixel value at the position (x, y) to obtain the second channel of the event frame; and the average time plane is used as the third channel of the event frame; The target segmentation network is trained using the data set; wherein the target segmentation network consists of an encoder and a decoder, the encoder consists of multiple convolutional layers, the decoder consists of multiple deconvolutional layers, and each convolutional layer and deconvolutional layer is connected to a BN layer and a RELU layer.

8. The target segmentation network training method according to claim 7, characterized in that: In the target segmentation network, the outputs of the convolutional layer and the deconvolution layer with the same output size are superimposed and then input into the next deconvolution layer.

9. The target segmentation network training method according to claim 7, characterized in that: The first layer of the target segmentation network does not adopt a lightweight network structure, and is subsequently connected to a BN layer and a RELU layer; the other layers all adopt a lightweight network structure, each convolution layer consists of a depth-separable convolution layer and a 1x1 convolution layer, each deconvolution layer consists of a depth-separable deconvolution layer and a 1x1 convolution layer, and each 1x1 convolution layer is connected to a BN layer and a RELU layer.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1-3 or 7-9.

Images