Image segmentation method, device and storage medium

Abstract

The embodiment of the present invention discloses an image segmentation method, device and storage medium, the method comprising: acquiring target event data of a target scene based on an event camera, and obtaining a target event tensor through voxel gridding according to the target event data; inputting the target event tensor into a trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene; wherein the target autonomous driving image segmentation model is obtained by training an initial autonomous driving image segmentation model based on a sample event tensor corresponding to a sample scene and a sample label image, and the target autonomous driving image segmentation model is a deep generative adversarial network with a codec network structure including an attention mechanism. Through the technical solution of the embodiment of the present invention, the technical effect of automatically and accurately segmenting the surrounding scene during autonomous driving is achieved.

Description

Technical Field

[0001] Embodiments of the present invention relate to autonomous driving scene monitoring technology, and more particularly to an image segmentation method, device, and storage medium. Background Art

[0002] Unmanned driving image data refers to the road condition information and car image data collected during the driving process of the car. The foreground car and background driving environment area can be extracted through the image segmentation algorithm and sent to the unmanned driving algorithm to improve the performance of the unmanned driving algorithm.

[0003] At present, image segmentation algorithms are mainly divided into segmentation algorithms based on traditional technologies and segmentation algorithms based on deep learning.

[0004] However, segmentation algorithms based on traditional technologies usually require manual intervention, have a low degree of automation, and are mostly based on shallow texture structures for image segmentation, which are easily disturbed by noise and have poor segmentation accuracy. Segmentation algorithms based on deep learning are often accompanied by deficiencies such as imbalanced attention between deep and shallow features and inability to capture long-distance correlation information, which affects the accuracy of image segmentation. Summary of the invention

[0005] The embodiments of the present invention provide an image segmentation method, device and storage medium to achieve the technical effect of automatically and accurately segmenting the image of the surrounding scene during autonomous driving.

[0006] In a first aspect, an embodiment of the present invention provides an image segmentation method, the method comprising:

[0007] Acquire target event data of a target scene based on an event camera, and obtain a target event tensor by voxel gridding according to the target event data;

[0008] The target event tensor is input into the trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene; wherein the target autonomous driving image segmentation model is trained on an initial autonomous driving image segmentation model based on a sample event tensor and a sample label image corresponding to a sample scene, and the target autonomous driving image segmentation model is a deep generative adversarial network with an encoding / decoding network structure that includes an attention mechanism.

[0009] In a second aspect, an embodiment of the present invention further provides an image segmentation device, the device comprising:

[0010] An event acquisition module, used for acquiring target event data of a target scene based on an event camera, and obtaining a target event tensor by voxel gridding according to the target event data;

[0011] An image segmentation module is used to input the target event tensor into a trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene; wherein the target autonomous driving image segmentation model is obtained by training an initial autonomous driving image segmentation model based on a sample event tensor and a sample label image corresponding to a sample scene, and the target autonomous driving image segmentation model is a deep generative adversarial network with an encoding and decoding network structure that includes an attention mechanism.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation method as described in any one of the embodiments of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image segmentation method as described in any one of the embodiments of the present invention.

[0017] The technical solution of the embodiment of the present invention obtains target event data of the target scene based on an event camera, obtains a target event tensor through voxel gridding processing based on the target event data, and inputs the target event tensor into a trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene, thereby solving the problems of insufficient information and poor image segmentation effect during image segmentation, and achieving the technical effect of automatically and accurately performing image segmentation on surrounding scenes during autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a flow chart of an image segmentation method provided in Embodiment 1 of the present invention;

[0020] Figure 2 A flowchart of a method for training a target autonomous driving image segmentation model provided in Embodiment 2 of the present invention;

[0021] Figure 3 A schematic diagram of a flow chart of an image segmentation method provided in Embodiment 3 of the present invention;

[0022] Figure 4 A schematic diagram of event data representation provided by Embodiment 3 of the present invention;

[0023] Figure 5 A schematic diagram of a deep adversarial network structure with an attention mechanism and multi-scale spatial pooling provided in Embodiment 3 of the present invention;

[0024] Figure 6 A schematic diagram of the structure of an attention mechanism provided in Embodiment 3 of the present invention;

[0025] Figure 7 A schematic diagram of the structure of an image segmentation device provided by Embodiment 4 of the present invention;

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Embodiment 1

[0029] Figure 1 This is a flow chart of an image segmentation method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation where image segmentation of surrounding scenes is performed during autonomous driving. The method can be executed by an image segmentation device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device. Optionally, the electronic device can be a mobile terminal, a PC, a vehicle-mounted terminal, etc.

[0030] like Figure 1 The method of this embodiment specifically includes the following steps:

[0031] S110 , acquiring target event data of a target scene based on an event camera, and obtaining a target event tensor through voxel gridding processing according to the target event data.

[0032] Among them, the event camera can be an acquisition device, such as: DAVIS event camera, etc. The event camera can output an asynchronous signal by measuring the brightness change of each pixel. The target scene can be the scene around the autonomous driving vehicle, and the scene can include the foreground car and the background driving environment area. The target event data can be the data collected by the event camera, and can include: light and dark changes, encoding time and position information, etc. The voxel grid processing can be a Voxel Grid method, that is, a method of event representation of the target event data. The target event tensor can be the event tensor of the target event data after event representation, that is, a tensor that takes into account the encoding time, position information, and light and dark changes in the target event data.

[0033] Specifically, the target scene is captured based on the event camera to obtain target event data of the target scene, the target event data is processed into voxels, and the processed result is used as the target event tensor.

[0034] Optionally, the target event tensor can be obtained by performing voxel gridding processing through the following steps:

[0035] Step 1: segment the target event data according to the number of preset time periods to obtain target sub-data corresponding to each target time period.

[0036] The preset number of time periods may be a number preset for segmentation processing, and the target time period may be each time period evenly segmented according to the preset number of time periods.

[0037] Specifically, the encoding time in the target event data can be evenly divided according to the preset number of time periods to obtain multiple target time periods. Further, according to each target time period, the target event data is divided according to each target time period to obtain target sub-data corresponding to each target time period.

[0038] For example, the target event data is divided into B segments (preset number of time segments) according to the time dimension; the encoding time in the target event data is time-normalized, that is, (0, 0.004, ..., t N ) is mapped to (0,1,2,…,B), t N Represents the encoding time corresponding to the last target event data. Then, the data in each target time period after division is used as target sub-data.

[0039] Step 2: For each target time period, merge the target sub-data corresponding to the target time period to obtain the target sub-tensor corresponding to the target time period.

[0040] The target sub-tensor may be the result of merging each target sub-data within a target time period.

[0041] Specifically, for each target time period, each target sub-data falling within the target time period may be merged, and the merged result is used as the target sub-tensor.

[0042] Step 3: Determine the target event tensor corresponding to the target event data based on each target sub-tensor.

[0043] Specifically, after obtaining each target sub-tensor, each target sub-tensor can be spliced ​​according to the target time period to obtain a target event tensor corresponding to the target event data.

[0044] S120. Input the target event tensor into the trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene.

[0045] The target autonomous driving image segmentation model is used to segment the target scene, such as labeling edge information. The target autonomous driving image segmentation model is obtained by training the initial autonomous driving image segmentation model based on the sample event tensor corresponding to the sample scene and the sample label image. The target autonomous driving image segmentation model is a deep generative adversarial network with an encoder-decoder network structure including an attention mechanism. The sample event tensor can be an event tensor used for training, and the sample label image can be an image with segmentation boundaries marked corresponding to each sample event tensor. The target label image can be an image corresponding to the target scene after the segmentation boundaries are marked.

[0046] Specifically, by inputting the target event tensor into the trained target autonomous driving image segmentation model, the result of processing the target event tensor, that is, the model output result, can be obtained. The model output result can be determined as a target label image corresponding to the target scene.

[0047] Optionally, the encoder-decoder network structure including the attention mechanism includes: an encoder, a decoder, and an attention mechanism module.

[0048] Among them, the encoder includes at least two downsampling layers, each downsampling layer includes a convolution layer, a regularization layer and a linear rectification function, and each downsampling layer corresponds to a different scale; the decoder includes an upsampling layer corresponding to each downsampling layer, and each upsampling layer includes a transposed convolution layer, a regularization layer and a linear rectification function; the attention mechanism module includes a linear rectification function, a Sigmoid function and a resampling layer; the image segmentation discriminator in the initial autonomous driving image segmentation model includes a multi-scale pooling layer and a convolution layer.

[0049] Exemplarily, the encoder includes 4 downsampling layers, each of which uses convolution, regularization, and ReLU activation functions, and the decoder includes 4 upsampling layers, each of which uses transposed convolution, regularization, and ReLU activation functions to ensure that the input data and the output data have the same size.

[0050] It should be noted that the attention mechanism module extracts the long-distance dependencies between different pixels in the feature map (i.e., the target event tensor), so that the network pays more attention to the part that needs to be extracted, while suppressing the feature information of other areas, and improving the feature extraction ability of the convolution operation process as a whole. The linear rectification function can be a ReLU activation function, i.e., a rectified linear unit. The ReLU activation function is a commonly used activation function in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0051] It should also be noted that the image segmentation generator in the initial autonomous driving image segmentation model is an encoder-decoder network structure that includes an attention mechanism. The image segmentation discriminator in the initial autonomous driving image segmentation model can be a discriminator that processes at multiple scales, and the image segmentation discriminator can include a multi-scale pooling layer and a convolution layer for accurate discrimination.

[0052] On the basis of the above embodiments, the upsampling layer and the corresponding downsampling layer can be connected by a skip connection, so that the feature information of the downsampling layer can be transmitted to the corresponding upsampling layer to optimize the segmentation result. Specifically, the skip connection can be performed by the following steps:

[0053] Step 1: for each down-sampling layer, determine a down-sampled image output by the down-sampling layer and a first up-sampled image output by the up-sampling layer corresponding to the down-sampling layer.

[0054] The downsampled image may be an output image corresponding to the downsampled layer, and each downsampled layer may downsample the input image input to the downsampled layer to obtain a downsampled image. The first upsampled image may be an output image corresponding to the upsampled layer, and each upsampled layer may upsample the input image input to the upsampled layer to obtain a first upsampled image.

[0055] Specifically, for each down-sampling layer, an image output by the down-sampling layer is used as a down-sampling image, and an image output by an up-sampling layer corresponding to the down-sampling layer is used as a first up-sampling image.

[0056] Step 2: Process the first up-sampled image and the down-sampled image through the attention mechanism module to obtain a second up-sampled image corresponding to the up-sampled layer.

[0057] The second up-sampled image may be an image obtained after comprehensive processing of the first up-sampled image and the down-sampled image by the attention mechanism module.

[0058] Specifically, the first up-sampled image and the down-sampled image are used as two inputs of the attention mechanism module, and the output result of the attention mechanism module is used as the second up-sampled image corresponding to the up-sampling layer.

[0059] Step 3: Concatenate the first upsampled image and the second upsampled image to obtain an upsampled output image corresponding to the upsampled layer.

[0060] The upsampled output image is the upsampled input image of the next upsampled layer corresponding to the upsampled layer.

[0061] Based on the above embodiments, the attention mechanism module may be used to process the first upsampled image and the downsampled image in the following manner:

[0062] Step 1: Add the first up-sampled image and the down-sampled image element by element to obtain a sampled and image.

[0063] Exemplarily, the first up-sampled image is recorded as x, and the down-sampled image is recorded as g. The down-sampled image records the global feature vector, and (x+g) is taken as the sampled image.

[0064] Step 2: Process the sample and image through the linear rectification function, Sigmoid function and resampling layer in turn to obtain the attention image.

[0065] Specifically, the sample and image are respectively passed through the ReLU activation function and the Sigmoid activation function, and then processed to the original scale through the resampling layer, and the processed image is used as the attention image.

[0066] Step 3: Multiply the attention image by the first upsampled image element by element to obtain a second upsampled image corresponding to the upsampled layer.

[0067] Exemplarily, the sampled image (x+g) is element-wise multiplied with the first upsampled image x to obtain the output of the attention mechanism module, which is the second upsampled image.

[0068] The technical solution of the embodiment of the present invention obtains target event data of the target scene based on an event camera, obtains a target event tensor through voxel gridding processing based on the target event data, and inputs the target event tensor into a trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene, thereby solving the problems of insufficient information and poor image segmentation effect during image segmentation, and achieving the technical effect of automatically and accurately performing image segmentation on surrounding scenes during autonomous driving.

[0069] Embodiment 2

[0070] Figure 2 This is a flow chart of a method for training a target autonomous driving image segmentation model provided in the second embodiment of the present invention. This embodiment is based on any optional technical solution in the embodiment of the present invention. Optionally, the training method for the target autonomous driving image segmentation model can refer to the technical solution of this embodiment. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0071] like Figure 2 As shown, the method of this embodiment may specifically include:

[0072] S210: Obtain a sample event tensor corresponding to the sample scene, and determine a sample label image corresponding to the sample event tensor.

[0073] The sample scene may be a scene used for model training. The sample event tensor may be a tensor obtained by voxel-gridding the sample event data obtained by photographing the sample scene with an event camera. The sample label image may be an image corresponding to the sample event tensor that annotates the segmentation boundary.

[0074] Specifically, a certain amount of sample event data can be obtained by shooting a sample scene with an event camera. Then, voxel gridding can be performed on each sample event data to obtain a sample event tensor corresponding to the sample event data. For each sample event data corresponding to the sample event tensor, the image segmentation boundary is annotated to obtain a sample label image corresponding to the sample event tensor.

[0075] Optionally, the sample label image corresponding to the sample event tensor can be determined as follows:

[0076] The sample event data corresponding to the sample scene is obtained, and the sample event tensor is obtained by voxel gridding according to the sample event data.

[0077] Specifically, a certain amount of sample event data is obtained by photographing a sample scene with an event camera, and voxel gridding can be performed on each sample event data to obtain a sample event tensor.

[0078] According to the sample grayscale image corresponding to the sample event data, a sample label image corresponding to the sample event tensor is determined based on a labeling tool.

[0079] The sample grayscale image may be a grayscale image of a sample scene corresponding to the sample event data. The annotation tool may be a tool for annotating image segmentation boundaries in the grayscale image.

[0080] Specifically, for each sample event data, a sample grayscale image corresponding to the sample event image is determined, and the image segmentation boundaries in the sample grayscale image are annotated by an annotation tool to obtain a sample label image corresponding to the sample event tensor.

[0081] S220. Construct a training sample set based on the sample event tensor and the sample label image corresponding to the sample event tensor, train the initial autonomous driving image segmentation model according to the sample event tensor in the training sample set and the sample label image corresponding to the sample event tensor, and use the trained image segmentation generator as the target autonomous driving image segmentation model.

[0082] The initial autonomous driving image segmentation model includes a generative adversarial network, and the initial autonomous driving image segmentation model includes an image segmentation generator and an image segmentation discriminator. The image segmentation generator can be an encoding and decoding network structure including an attention mechanism, and the image segmentation discriminator can be any discriminator model, such as a fully connected network, a network including convolution, etc., and can include multi-scale pooling layers and convolution layers, etc. The training sample set can be a data set consisting of multiple sample event tensors and sample label images corresponding to each sample event tensor, which is used for subsequent training to obtain a target autonomous driving image segmentation model.

[0083] Specifically, the sample event tensor and the sample label image corresponding to the sample event tensor are combined to obtain a training sample set. The initial autonomous driving image segmentation model is trained according to the sample event tensor in the training sample set and the sample label image corresponding to the sample event tensor, which may be to train the image segmentation generator and the image segmentation discriminator separately, and then use the trained image segmentation generator as the target autonomous driving image segmentation model.

[0084] Optionally, the initial autonomous driving image segmentation model may be trained to obtain a target autonomous driving image segmentation model through the following steps:

[0085] Step 1: Input the sample event tensor in the training sample set into the image segmentation generator in the initial autonomous driving image segmentation model to obtain a sample generated image.

[0086] Specifically, the sample event tensors in the training sample set are input into the image segmentation generator used to generate the initial autonomous driving image segmentation model. Through the processing of the image segmentation generator, a sample generated image corresponding to each sample event tensor can be obtained.

[0087] Step 2: Adjust the network parameters of the image segmentation generator according to the sample event tensor and the sample label image corresponding to the sample event tensor.

[0088] Specifically, the image segmentation generator is trained according to the sample event tensor and the sample label image corresponding to the sample event tensor, and the network parameters of the image segmentation generator are continuously adjusted during the training process to improve the training effect.

[0089] Step three: train the image segmentation discriminator according to the sample generated image and the sample label image corresponding to the sample generated image, and determine whether to end the adjustment of the image segmentation generator based on the discrimination result of the trained image segmentation discriminator on the sample generated image.

[0090] Specifically, the image segmentation discriminator is trained according to the sample generated image and the sample label image corresponding to the sample generated image, so that the image segmentation discriminator can more effectively distinguish the sample generated image from the sample label image corresponding to the sample generated image. Based on the discrimination result of the trained image segmentation discriminator on the sample generated image, it can be determined whether the sample generated image and the sample label image corresponding to the sample generated image are distinguishable. If they are obviously distinguishable, it indicates that the effect of the sample generated image is not good, and the parameters of the image segmentation generator need to be readjusted, and the adjustment of the image segmentation generator cannot be terminated; if they are already difficult to distinguish, it indicates that the effect of the sample generated image is good and close to the sample label image, and the adjustment of the image segmentation generator can be terminated.

[0091] Step 4: If yes, the adjusted image segmentation generator is used as the target autonomous driving image segmentation model.

[0092] Specifically, if the discrimination result of the sample-generated image by the trained image segmentation discriminator determines that the adjustment of the image segmentation generator can be ended, it means that the effect of the adjusted image segmentation generator is stable, and the adjusted image segmentation generator can be used as the target autonomous driving image segmentation model.

[0093] The technical solution of this embodiment obtains the sample event tensor corresponding to the sample scene, determines the sample label image corresponding to the sample event tensor, and constructs a training sample set based on the sample event tensor and the sample label image corresponding to the sample event tensor. The initial autonomous driving image segmentation model is trained according to the sample event tensor in the training sample set and the sample label image corresponding to the sample event tensor, and the trained image segmentation generator is used as the target autonomous driving image segmentation model, which solves the problem of poor segmentation effect of the image segmentation model and achieves the technical effect of improving the accuracy and stability of the image segmentation model.

[0094] Embodiment 3

[0095] As an optional implementation scheme of the above embodiments, Figure 3 FIG. 1 is a flow chart of an image segmentation method provided by Embodiment 3 of the present invention. Figure 3 The method of this embodiment specifically includes:

[0096] First, the real scene (sample scene) is used to obtain sparse asynchronous event data (sample event data) and the corresponding grayscale image (APS frame) (sample grayscale image) through the DAVIS event camera, which is used as the label (GT) (sample label image) for supervised training.

[0097] Subsequently, the time data of the sparse and asynchronous event data is represented by Voxel Grid (voxel grid processing), thus obtaining a dense event tensor (sample event tensor), which can match the deep adversarial network (target autonomous driving image segmentation model) used subsequently.

[0098] After that, the dense event tensor is fed into the deep adversarial network, and its input is used together with the APS frame to calculate the loss function, and then fed back to the network for iterative training. Finally, when the network converges, the output obtained on the test set is the semantic segmentation result.

[0099] The following will explain the representation method of event data and the specific architecture of the proposed deep adversarial network.

[0100] like Figure 4 The event data representation diagram shown in the figure represents the input event data through the Voxel Grid method. The specific process is as follows:

[0101] First, the file format of the input event data is txt. Each event data can be a four-tuple, such as (x, y, t, p), where x, y are the corresponding coordinates of the event data in the image, t represents the time when the event occurs, and p represents the polarity (0 represents a negative event, 1 represents a positive event). A positive event means that at time t, the brightness at the (x, y) position increases, and a negative event means that the brightness decreases.

[0102] Subsequently, the sequence of event data is converted into a visual 3D point cloud format, and then into a Voxel Grid format. Finally, it is input into the deep adversarial network. The steps for converting 3D point cloud into Voxel Grid are:

[0103] 1) Divide each event data sequence into B segments (preset number of time periods) according to the time dimension, such as Figure 4 in timebin;

[0104] 2) Regularize the time, that is, (0, 0.004, ..., t N ) is mapped to (0,1,2,…,B), t N Represents the time when the last event data occurred;

[0105] 3) All events at each location falling within the corresponding time period (target time period) are merged into the closest Voxel.

[0106] The purpose of using Voxel Grid event representation is to take into account the event time information while making full use of the event polarity, so as to more comprehensively represent the event information. Therefore, sending it into the deep adversarial network can achieve a more accurate segmentation effect.

[0107] like Figure 5As shown in the figure, it is a schematic diagram of the deep adversarial network structure with attention mechanism and multi-scale spatial pooling. Specifically, the event data is input into an encoder-decoder structure with an attention mechanism. Among them, the encoder contains 4 downsampling layers (each layer uses convolution, regularization and ReLU activation functions), and the decoder contains 4 upsampling layers (each layer uses transposed convolution, regularization and ReLU activation functions) to ensure that the input and output sizes are the same. The corresponding layers of upsampling and downsampling are connected by jump connections, so that the feature information of the downsampling layer can be transmitted to the corresponding upsampling to optimize the segmentation results. Subsequently, through a convolution layer, the output of the generator (image segmentation generator) is obtained. On the other hand, the output of the generator and the sample label image are respectively sent to the discriminator (image segmentation discriminator), and the final output is obtained after the multi-scale pooling layer and the convolution layer. This output is a real value between 0-1, which is used to determine whether the input of the discriminator comes from GT (value close to 1) or from the output result of the generator (value close to 0). In Figure 5 In the figure, the downward arrow represents the multi-scale pooling operation, which downsamples the input to 1, 4, and 9, making its length and width become 1, 1 / 4, and 1 / 9 of the original, and then feeds it into multiple layers of convolution, finally obtaining an output value between 0 and 1. Subsequently, the loss value is fed back for training, and the model is trained using adversarial learning as a whole until the model converges.

[0108] Figure 6 It is a schematic diagram of the structure of the attention mechanism, which is used to extract the long-distance dependencies between different pixels in the feature map, so that the network pays more attention to the parts that need to be extracted, while suppressing the feature information of other areas, and improving the feature extraction ability of the convolution operation process as a whole. Among them, x is the output of each layer in the upsampling layer process, and g is the global feature vector of the corresponding downsampling layer. After adding them element by element, they are sent to the attention mechanism together, and then pass through the ReLU and Sigmoid activation functions respectively, and then resampled to the original scale, and then multiplied element by element with the input x to get the final output of the attention mechanism. The attention mechanism corresponds to Figure 5 The "A" in .

[0109] The technical solution of this embodiment improves the segmentation performance of the model by adopting adversarial learning for training, through attention mechanism and multi-scale pooling operation, with event data as input.

[0110] Embodiment 4

[0111] Figure 7 This is a structural schematic diagram of an image segmentation device provided in Embodiment 4 of the present invention. The device includes: an event acquisition module 310 and an image segmentation module 320.

[0112] Among them, the event acquisition module 310 is used to acquire the target event data of the target scene based on the event camera, and obtain the target event tensor through voxel gridding processing according to the target event data; the image segmentation module 320 is used to input the target event tensor into the trained target autonomous driving image segmentation model to obtain the target label image corresponding to the target scene; wherein, the target autonomous driving image segmentation model is obtained by training the initial autonomous driving image segmentation model based on the sample event tensor corresponding to the sample scene and the sample label image, and the target autonomous driving image segmentation model is a deep generative adversarial network with a codec network structure including an attention mechanism.

[0113] Optionally, the event acquisition module 310 is specifically used to segment the target event data according to a preset number of time periods to obtain target sub-data corresponding to each target time period; for each target time period, merge the target sub-data corresponding to the target time period to obtain a target sub-tensor corresponding to the target time period; and determine a target event tensor corresponding to the target event data based on each target sub-tensor.

[0114] Optionally, the initial autonomous driving image segmentation model includes an image segmentation generator and an image segmentation discriminator, and the device also includes: a model training module, which is used to train the target autonomous driving image segmentation model based on the following method: obtaining a sample event tensor corresponding to a sample scene, and determining a sample label image corresponding to the sample event tensor; constructing a training sample set based on the sample event tensor and the sample label image corresponding to the sample event tensor, training the initial autonomous driving image segmentation model according to the sample event tensors in the training sample set and the sample label images corresponding to the sample event tensor, and using the trained image segmentation generator as the target autonomous driving image segmentation model.

[0115] Optionally, the model training module is also used to obtain sample event data corresponding to the sample scene, and obtain a sample event tensor based on the sample event data through voxel gridding; and determine the sample label image corresponding to the sample event tensor based on a labeling tool according to the sample grayscale image corresponding to the sample event data.

[0116] Optionally, the model training module is also used to input the sample event tensor in the training sample set into the image segmentation generator in the initial autonomous driving image segmentation model to obtain a sample generated image; adjust the network parameters of the image segmentation generator according to the sample event tensor and the sample label image corresponding to the sample event tensor; train the image segmentation discriminator according to the sample generated image and the sample label image corresponding to the sample generated image, and determine whether to end the adjustment of the image segmentation generator based on the discrimination result of the trained image segmentation discriminator on the sample generated image; if so, use the adjusted image segmentation generator as the target autonomous driving image segmentation model.

[0117] Optionally, the codec network structure including the attention mechanism includes: an encoder, a decoder and an attention mechanism module; wherein the encoder includes at least two downsampling layers, each of which includes a convolution layer, a regularization layer and a linear rectification function, and each downsampling layer corresponds to a different scale; the decoder includes an upsampling layer corresponding to each downsampling layer, each of which includes a transposed convolution layer, a regularization layer and a linear rectification function; the attention mechanism module includes a linear rectification function, a Sigmoid function and a resampling layer; the image segmentation discriminator in the initial autonomous driving image segmentation model includes a multi-scale pooling layer and a convolution layer.

[0118] Optionally, the device also includes: an attention analysis module, used to determine, for each downsampling layer, a downsampled image output by the downsampling layer, and a first upsampled image output by the upsampling layer corresponding to the downsampling layer; processing the first upsampled image and the downsampled image through the attention mechanism module to obtain a second upsampled image corresponding to the upsampling layer; splicing the first upsampled image and the second upsampled image to obtain an upsampled output image corresponding to the upsampling layer; wherein the upsampled output image is an upsampled input image of the next upsampling layer corresponding to the upsampling layer.

[0119] Optionally, the attention analysis module is also used to add the first up-sampled image and the down-sampled image element by element to obtain a sample and an image; process the sample and image in sequence through a linear rectification function, a Sigmoid function and a resampling layer to obtain an attention image; and multiply the attention image and the first up-sampled image element by element to obtain a second up-sampled image corresponding to the up-sampling layer.

[0120] The technical solution of the embodiment of the present invention obtains target event data of the target scene based on an event camera, obtains a target event tensor through voxel gridding processing based on the target event data, and inputs the target event tensor into a trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene, thereby solving the problems of insufficient information and poor image segmentation effect during image segmentation, and achieving the technical effect of automatically and accurately performing image segmentation on surrounding scenes during autonomous driving.

[0121] The image segmentation device provided in the embodiment of the present invention can execute the image segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0122] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0123] Embodiment 5

[0124] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 8 A block diagram of an exemplary electronic device 40 suitable for implementing exemplary embodiments of the present invention is shown. Figure 8 The electronic device 40 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0125] like Figure 8 As shown, the electronic device 40 is in the form of a general computing device. The components of the electronic device 40 may include but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0126] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0127] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0128] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 403 via one or more data medium interfaces. The system memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0129] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in system memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0130] The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboards, pointing devices, displays 410, etc.), one or more devices that enable a user to interact with the electronic device 40, and / or any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an I / O interface (input / output interface) 411. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 412. As shown, the network adapter 412 communicates with other modules of the electronic device 40 through the bus 403. It should be understood that although Figure 8Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] The processing unit 401 executes various functional applications and data processing by running the programs stored in the system memory 402, such as implementing the image segmentation method provided in the embodiment of the present invention.

[0132] Embodiment 6

[0133] Embodiment 6 of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to perform an image segmentation method when executed by a computer processor, the method comprising:

[0134] Acquire target event data of a target scene based on an event camera, and obtain a target event tensor by voxel gridding according to the target event data;

[0135] The target event tensor is input into the trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene; wherein the target autonomous driving image segmentation model is trained on an initial autonomous driving image segmentation model based on a sample event tensor and a sample label image corresponding to a sample scene, and the target autonomous driving image segmentation model is a deep generative adversarial network with an encoding / decoding network structure that includes an attention mechanism.

[0136] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0137] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0139] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Acquire target event data of a target scene based on an event camera, and obtain a target event tensor by voxel gridding according to the target event data; The target event tensor is input into the trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene; wherein the target autonomous driving image segmentation model is obtained by training an initial autonomous driving image segmentation model based on a sample event tensor and a sample label image corresponding to a sample scene, and the target autonomous driving image segmentation model is a deep generative adversarial network with an encoder-decoder network structure including an attention mechanism; The step of obtaining a target event tensor by voxel gridding according to the target event data includes: Segmenting the target event data according to the number of preset time periods to obtain target sub-data corresponding to each target time period; For each target time period, merging the target sub-data corresponding to the target time period to obtain a target sub-tensor corresponding to the target time period; According to each target sub-tensor, a target event tensor corresponding to the target event data is determined.

2. The method according to claim 1, characterized in that The initial autonomous driving image segmentation model includes an image segmentation generator and an image segmentation discriminator, and the target autonomous driving image segmentation model is trained based on the following method: Obtaining a sample event tensor corresponding to a sample scene, and determining a sample label image corresponding to the sample event tensor; A training sample set is constructed based on a sample event tensor and a sample label image corresponding to the sample event tensor, the initial autonomous driving image segmentation model is trained according to the sample event tensor in the training sample set and the sample label image corresponding to the sample event tensor, and the trained image segmentation generator is used as a target autonomous driving image segmentation model.

3. The method according to claim 2, characterized in that The step of obtaining a sample event tensor corresponding to the sample scene and determining a sample label image corresponding to the sample event tensor includes: Acquire sample event data corresponding to the sample scene, and obtain a sample event tensor by voxel gridding according to the sample event data; According to the sample grayscale image corresponding to the sample event data, a sample label image corresponding to the sample event tensor is determined based on a labeling tool.

4. The method according to claim 2, characterized in that: The step of training the initial autonomous driving image segmentation model according to the sample event tensor in the training sample set and the sample label image corresponding to the sample event tensor, and using the trained image segmentation generator as the target autonomous driving image segmentation model, includes: Inputting the sample event tensor in the training sample set into the image segmentation generator in the initial autonomous driving image segmentation model to obtain a sample generated image; Adjusting the network parameters of the image segmentation generator according to the sample event tensor and the sample label image corresponding to the sample event tensor; Training an image segmentation discriminator according to the sample generated image and a sample label image corresponding to the sample generated image, and determining whether to end the adjustment of the image segmentation generator based on a discrimination result of the trained image segmentation discriminator on the sample generated image; If yes, the adjusted image segmentation generator is used as the target autonomous driving image segmentation model.

5. The method according to claim 1, characterized in that The encoding and decoding network structure including the attention mechanism includes: an encoder, a decoder and an attention mechanism module; Among them, the encoder includes at least two downsampling layers, each of which includes a convolution layer, a regularization layer and a linear rectification function, and each downsampling layer corresponds to a different scale; the decoder includes an upsampling layer corresponding to each downsampling layer, each of which includes a transposed convolution layer, a regularization layer and a linear rectification function; the attention mechanism module includes a linear rectification function, a Sigmoid function and a resampling layer; the image segmentation discriminator in the initial autonomous driving image segmentation model includes a multi-scale pooling layer and a convolution layer.

6. The method according to claim 5, characterized in that Also includes: For each down-sampling layer, determine a down-sampled image output by the down-sampling layer and a first up-sampled image output by an up-sampling layer corresponding to the down-sampling layer; Processing the first up-sampled image and the down-sampled image through the attention mechanism module to obtain a second up-sampled image corresponding to the up-sampled layer; The first up-sampled image and the second up-sampled image are concatenated to obtain an up-sampled output image corresponding to the up-sampled layer; wherein the up-sampled output image is an up-sampled input image of a next up-sampled layer corresponding to the up-sampled layer.

7. The method according to claim 6, characterized in that The step of processing the first up-sampled image and the down-sampled image through the attention mechanism module to obtain a second up-sampled image corresponding to the up-sampled layer includes: Adding the first up-sampled image and the down-sampled image element by element to obtain a sampled image; The samples and images are processed sequentially by a linear rectification function, a Sigmoid function, and a resampling layer to obtain an attention image; The attention image is element-wise multiplied by the first up-sampled image to obtain a second up-sampled image corresponding to the up-sampled layer.

8. An image segmentation device, characterized in that: include: An event acquisition module, used for acquiring target event data of a target scene based on an event camera, and obtaining a target event tensor by voxel gridding according to the target event data; An image segmentation module, used for inputting the target event tensor into a trained target autonomous driving image segmentation model to obtain a target label image corresponding to the target scene; wherein the target autonomous driving image segmentation model is obtained by training an initial autonomous driving image segmentation model based on a sample event tensor and a sample label image corresponding to a sample scene, and the target autonomous driving image segmentation model is a deep generative adversarial network with an encoder-decoder network structure including an attention mechanism; Among them, the event acquisition module is specifically used to segment the target event data according to the preset number of time periods to obtain target sub-data corresponding to each target time period; for each target time period, merge the target sub-data corresponding to the target time period to obtain a target sub-tensor corresponding to the target time period; based on each target sub-tensor, determine the target event tensor corresponding to the target event data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image segmentation method as described in any one of claims 1 to 7 is implemented.

Images