A scene recognition method and system based on fusion event camera

Abstract

The application provides a scene recognition method and system based on a fusion event camera, and belongs to the technical field of image processing, and comprises the following steps: obtaining a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream, pre-processing the preset high-speed high-dynamic image to obtain a pre-processed event stream, fusing the pre-processed event stream and the preset high-speed high-dynamic image to obtain fusion features, obtaining a preset reference image, extracting reference image features in the preset reference image, and performing retrieval matching on the fusion features and the reference image features to obtain a reference image recognition result. Through the proposed visual scene recognition framework of the fusion event camera, the low time delay and high dynamic nature of the event stream can be used to implicitly enhance the image quality, and scene recognition can be completed, so that the problem of poor scene recognition performance of the traditional scene recognition method in a high-speed high-dynamic scene is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a scene recognition method and system based on a fused event camera. Background Technology

[0002] In traditional scene recognition, cameras capture images at a fixed frame rate, exposing all pixels in the image frame simultaneously and recording the absolute brightness information of the scene, directly reflecting rich scene information. However, due to hardware limitations, traditional cameras often have relatively low frame rates and small dynamic ranges. Therefore, when objects move at high speeds, motion blur is easily produced. When the scene brightness exceeds the recording range of the camera's image sensor, the output image from the optical camera exhibits degradation phenomena such as underexposure and overexposure, resulting in the loss of a significant amount of information and affecting the performance of image-based visual scene recognition algorithms.

[0003] The emergence of event cameras has brought new options to the field of computer vision. Based on bio-inspired sensors, event cameras possess a high dynamic range (approximately 130 dB) and can generate asynchronous events with extremely low latency (1 μs), recording the coordinates, time, and polarity of brightness changes in a scene. Therefore, event cameras are virtually unaffected by motion blur and scene brightness range, making them highly advantageous and attractive for visual scene recognition tasks. However, existing event camera-based visual scene recognition methods only utilize intramodal matching between events or cross-modal matching between events and a reference image database. Events have extremely low signal-to-noise ratios and contain only sparse edge features, limiting the scene recognition performance of event camera-based visual scene recognition methods. Furthermore, there are currently no methods that simultaneously utilize optical cameras and event cameras for visual scene recognition.

[0004] Therefore, a new method is needed for scene recognition using event cameras to overcome the above-mentioned shortcomings. Summary of the Invention

[0005] This invention provides a scene recognition method and system based on fused event cameras to address the shortcomings of existing technologies.

[0006] In a first aspect, the present invention provides a scene recognition method based on a fused event camera, comprising:

[0007] A preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream are acquired. The preset high-speed high-dynamic image is preprocessed to obtain a preprocessed event stream. The preprocessed event stream is fused with the preset high-speed high-dynamic image to obtain a fused feature.

[0008] Obtain a preset reference image and extract reference image features from the preset reference image;

[0009] The fused features are searched and matched with the features of the reference image to obtain the reference image recognition result.

[0010] According to a scene recognition method based on a fused event camera provided by the present invention, the step of acquiring a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream includes:

[0011] Acquire a reference image sequence and a corresponding event stream, wherein the reference image sequence includes any frame of a preset clear exposure image, and the corresponding event stream includes the event stream within the exposure range of the preset clear exposure image;

[0012] The blurred image sequence is obtained by summing and averaging multiple consecutive reference images in the reference image sequence.

[0013] The preset high-speed, high-dynamic event stream is obtained by taking the union of the event streams corresponding to the reference image sequence based on the consecutive multi-frame reference images;

[0014] The photometric linear transformation parameters and the photometric linear transformation interval are determined, and the preset high-speed high dynamic range image is obtained based on the photometric linear transformation parameters, the photometric linear transformation interval, and the blurred image sequence.

[0015] According to a scene recognition method based on a fused event camera provided by the present invention, the step of preprocessing the preset high-speed, high-dynamic image to obtain a preprocessed event stream includes:

[0016] Determine the event stream within the exposure range corresponding to any frame image in the preset high-speed high dynamic range image, and determine the exposure time interval of the event stream within the exposure range;

[0017] Obtain the number of exposure events within the exposure time interval, and obtain an exposure event quadruple based on the number of exposure events. The exposure event quadruple includes the timestamp of any exposure event, the x-axis of any exposure event, the y-axis of any exposure event, and the polarity identifier of any exposure event.

[0018] A zero matrix is ​​constructed by dividing the preset high-speed, high-dynamic image into equal parts based on its width, height, and exposure duration. The zero matrix is ​​then filled with the exposure event quadruples to obtain the preprocessed event stream.

[0019] According to a scene recognition method based on a fusion event camera provided by the present invention, the step of fusing the preprocessed event stream with the preset high-speed, high-dynamic image to obtain fusion features includes:

[0020] A fusion feature extraction network is determined. The preset high-speed and high-dynamic image is sequentially input into the Conv layer and the DConv layer to generate the first image tensor feature. The preprocessed event stream is sequentially input into the Conv layer and the DConv layer to generate the first event tensor feature. The first image tensor feature and the first event tensor feature are concatenated and then input into the ResBlock layer to obtain the first fusion feature.

[0021] The first image tensor feature corresponding to the first scale is input into the DConv layer to generate the second image tensor feature. The first event tensor feature corresponding to the first scale is input into the DConv layer to generate the second event tensor feature. The first fusion feature corresponding to the first scale is input into the DConv layer to generate the second fusion feature. The second image tensor feature, the second event tensor feature and the second fusion feature are concatenated and then input into the ResBlock layer to obtain the third fusion feature.

[0022] The second image tensor feature, the second event tensor feature, and the third fusion feature corresponding to the second scale are concatenated and then input into the ResBlock layer to obtain the initial multi-scale fusion feature.

[0023] The initial multi-scale fusion features are filtered through a CBAM layer to obtain the fusion features;

[0024] The Conv layer includes a single convolutional layer with a stride of 1, the DConv layer includes a combination of a downsampling convolutional layer with a stride of 2 and a convolutional layer with a stride of 1, the ResBlock layer includes a residual connection layer, the kernel size of all convolutional layers is 3×3, and all convolutional layers are connected to the ReLU activation function.

[0025] According to a scene recognition method based on a fused event camera provided by the present invention, the step of acquiring a preset reference image and extracting reference image features from the preset reference image includes:

[0026] A network for extracting image features is determined by sequentially inputting the preset reference image into a Conv layer, three DConv layers, and a ResBlock layer to obtain the features of the reference image.

[0027] The Conv layer includes a single convolutional layer with a stride of 1, the DConv layer includes a combination of a downsampling convolutional layer with a stride of 2 and a convolutional layer with a stride of 1, the ResBlock layer includes a residual connection layer, the kernel size of all convolutional layers is 3×3, and all convolutional layers are connected to the ReLU activation function.

[0028] According to a scene recognition method based on a fused event camera provided by the present invention, after acquiring a preset reference image and extracting reference image features from the preset reference image, the method further includes:

[0029] Construct training triplets of query image, positive example image and negative example image. The query image includes the preset high-speed high dynamic range image and the preprocessed event stream. The positive example image includes images with the same orientation as the query image and within a preset query distance range. The negative example image includes images with a different orientation than the query image and outside the preset query distance range.

[0030] A maximum loss function is constructed from the training triples, the vector distance function, and the similarity distance constant.

[0031] Based on the maximum loss function, the adaptive learning rate of the fusion feature extraction network and the image feature extraction network is determined by a preset optimization algorithm, and the parameters of the neural network convolutional layer are updated by the training triples to obtain the optimized fusion feature extraction network and the optimized image feature extraction network.

[0032] The optimization parameters for each convolutional layer in the optimized fusion feature extraction network and the optimized image feature extraction network are different.

[0033] Secondly, the present invention also provides a scene recognition system based on a fused event camera, comprising:

[0034] The first processing module is used to acquire a preset high-speed high dynamic range image and a preset high-speed high dynamic range event stream, preprocess the preset high-speed high dynamic range image to obtain a preprocessed event stream, and fuse the preprocessed event stream with the preset high-speed high dynamic range image to obtain a fused feature.

[0035] The second processing module is used to acquire a preset reference image and extract reference image features from the preset reference image;

[0036] The recognition module is used to search and match the fused features with the reference image features to obtain the reference image recognition result.

[0037] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scene recognition method based on the fusion event camera as described above.

[0038] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scene recognition method based on a fused event camera as described above.

[0039] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the scene recognition method based on a fused event camera as described above.

[0040] The scene recognition method and system based on fusion event cameras provided by this invention proposes a visual scene recognition framework based on fusion event cameras, which can implicitly enhance image quality by utilizing the low latency and high dynamic properties of event streams and complete scene recognition, thus solving the problem of poor performance of traditional scene recognition methods in high-speed and high-dynamic scenes. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts illustrating the scene recognition method based on fused event cameras provided by the present invention;

[0043] Figure 2 This is the second flowchart of the scene recognition method based on fused event cameras provided by the present invention;

[0044] Figure 3 This is a structural diagram of the image feature extraction network and the fusion feature extraction network provided by the present invention;

[0045] Figure 4 This is a reference image of the scene recognition results provided by the present invention;

[0046] Figure 5 This is a schematic diagram of the scene recognition system based on fused event cameras provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] Figure 1This is one of the flowcharts illustrating the scene recognition method based on fused event cameras provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0050] Step 100: Acquire a preset high-speed high dynamic range image and a preset high-speed high dynamic range event stream; preprocess the preset high-speed high dynamic range image to obtain a preprocessed event stream; fuse the preprocessed event stream with the preset high-speed high dynamic range image to obtain a fused feature.

[0051] Step 200: Obtain a preset reference image and extract reference image features from the preset reference image;

[0052] Step 300: The fused features are searched and matched with the reference image features to obtain the reference image recognition result.

[0053] This invention generates an image set and its event flow within the exposure range of a high-speed, high-dynamic scene by using a set of well-exposed and clear reference images and the event flow within their exposure range. It constructs an image feature extraction network and a fusion feature extraction network, which are used to generate image features of the well-exposed and clear reference images, and fusion features of the image set and its event flow within the exposure range of the high-speed, high-dynamic scene, respectively. It constructs query, positive, and negative triplets, and uses triplet loss to supervise the training of the image feature extraction network and the fusion feature extraction network, further optimizing the resulting image feature extraction network.

[0054] Comparison Figure 2 The process shown involves the following steps: the fusion feature extraction module fuses the images from high-speed, high-dynamic scenes with the events within the exposure range of the images from high-speed, high-dynamic scenes after the event preprocessing module, to obtain fusion features; the image feature extraction module extracts reference images and their feature databases from the reference image database; and then the fusion features are searched and matched with the reference images and their feature databases to obtain the reference image.

[0055] Specifically, firstly, in scenes with good brightness and limited motion speed, an optical camera and an event camera simultaneously capture a sequence of reference images with the same field of view and their corresponding event streams. Then, multiple consecutive reference images are summed and averaged to obtain a blurred image; the event stream corresponding to this image is the union of the exposure intervals of the multiple frames used for averaging. Finally, the image undergoes dynamic range transformation and cropping to obtain an image for a high-speed, high-dynamic scene.

[0056] Then, the event stream is preprocessed to convert the events within the exposure range of the image in the high-speed, high-dynamic scene into a multi-channel image form, thus obtaining the preprocessed event stream; an image feature extraction network is constructed to generate reference image features for scenes with limited speed and good exposure; and a fusion feature extraction network is constructed to generate fusion features of the image and its events in the high-speed, high-dynamic scene.

[0057] Furthermore, training triplets for query, positive examples, and negative examples are constructed. The query is an image in a high-speed, high-dynamic scene and its preprocessed event stream within its exposure time interval. Positive examples are images with a similar field of view to the query image, and negative examples are images with a different scene from the query image. All triplets constructed from the dataset are used for network training, and supervision is provided through triplet loss, thereby obtaining the optimized image feature extraction network and fusion feature extraction network.

[0058] Finally, images and event streams with field of view alignment in high-speed, high-dynamic scenes are captured by optical cameras and event cameras. The event streams are preprocessed to obtain preprocessed event streams. The images and event streams in high-speed, high-dynamic scenes are input into an optimized fusion feature extraction network to obtain fusion features. The reference image database is input into an optimized image feature extraction network to obtain a reference image feature database. Finally, based on the feature vector similarity between the fusion features and the image features, the reference image that is closest to the field of view of the input high-speed, high-dynamic image is retrieved from the reference image database.

[0059] This invention proposes a visual scene recognition framework that integrates event cameras. It can implicitly enhance image quality and complete scene recognition by utilizing the low latency and high dynamics of event streams, thus solving the problem of poor performance of traditional scene recognition methods in high-speed and high-dynamic scenes.

[0060] Based on the above embodiments, step 100, acquiring the preset high-speed high-dynamic image and the preset high-speed high-dynamic event stream, includes:

[0061] Acquire a reference image sequence and a corresponding event stream, wherein the reference image sequence includes any frame of a preset clear exposure image, and the corresponding event stream includes the event stream within the exposure range of the preset clear exposure image;

[0062] The blurred image sequence is obtained by summing and averaging multiple consecutive reference images in the reference image sequence.

[0063] The preset high-speed, high-dynamic event stream is obtained by taking the union of the event streams corresponding to the reference image sequence based on the consecutive multi-frame reference images;

[0064] The photometric linear transformation parameters and the photometric linear transformation interval are determined, and the preset high-speed high dynamic range image is obtained based on the photometric linear transformation parameters, the photometric linear transformation interval, and the blurred image sequence.

[0065] Specifically, in this embodiment, a DAVIS346 event camera is selected to capture field-of-view aligned grayscale images and event streams in a scene with limited speed and good exposure. The resolution of both the captured grayscale images and the event streams is 346×260. Then, seven well-exposed and clear reference images are summed and averaged to generate a blurred image. The exposure time of this simulated blurred image is defined as the time from the first frame to the last frame of the seven images. Finally, the blurred image undergoes dynamic range transformation and cropping to obtain an image under high-speed, high-dynamic scene conditions. The set of events within the exposure time of this image is used as the corresponding event stream to obtain the event stream under high-speed, high-dynamic scene conditions. The captured and generated datasets are divided into a training set and a test set. The training set contains 28,800 reference images and their generated images under high-speed, high-dynamic scene conditions, along with their event streams within their exposure ranges.

[0066] Set the reference image sequence and its corresponding event stream as:

[0067] I[f], f∈{1,2,…,L}

[0068] E[f], f∈{1,2,…,L}

[0069] Where I[f] represents the well-exposed, clear image of the f-th frame of the reference image sequence, L is the number of images in the image sequence, and E[f] represents the event stream within the exposure range of the f-th frame of the reference image.

[0070] The blurred image sequence is as follows:

[0071]

[0072] in, The f-th frame is a blurred image generated from multiple reference images. The method uses a weighted average of the f-th frame reference image and the M images before and after it (a total of 2M+1 images) to obtain the f-th frame blurred image. In this embodiment, M=3.

[0073] The event stream corresponding to the blurred image sequence is:

[0074]

[0075] Where ε[f] is the event stream corresponding to the blurred image of the f-th frame, and is the union of the event streams within the exposure range of the 2M+1 consecutive images. The definition of M is the same as described above.

[0076] The image sequence in high-speed, high-dynamic scenes is as follows:

[0077]

[0078] in, This refers to the f-th blurred image in the previously generated blurred image sequence, where α is the photometric linear transformation parameter, which is multiplied by the blurred image to achieve a linear photometric change. clip The function restricts the result of the linear photometric transformation to the interval [L]. l ,L h The luminance value is rounded down, and B[f] is the f-th frame of the high-speed, high-dynamic-range image sequence in a high-speed, high-dynamic-range scene. In this embodiment, the values ​​of α are 0.25 and 4 for low light intensity and high light intensity, respectively. l =10,L h =255.

[0079] Based on the above embodiments, step 100, which involves preprocessing the preset high-speed, high-dynamic image to obtain a preprocessed event stream, includes:

[0080] Determine the event stream within the exposure range corresponding to any frame image in the preset high-speed high dynamic range image, and determine the exposure time interval of the event stream within the exposure range;

[0081] Obtain the number of exposure events within the exposure time interval, and obtain an exposure event quadruple based on the number of exposure events. The exposure event quadruple includes the timestamp of any exposure event, the x-axis of any exposure event, the y-axis of any exposure event, and the polarity identifier of any exposure event.

[0082] A zero matrix is ​​constructed by dividing the preset high-speed, high-dynamic image into equal parts based on its width, height, and exposure duration. The zero matrix is ​​then filled with the exposure event quadruples to obtain the preprocessed event stream.

[0083] Specifically, in this embodiment, the event stream is preprocessed to convert the events within the exposure range of the image in a high-speed, high-dynamic scene into a multi-channel, such as a 96-channel image format, thereby obtaining the preprocessed event stream.

[0084] For the f-th frame of an image sequence in a high-speed, high-dynamic scene, its corresponding exposure range [t] f , t f The event flow within [+T], where t f T is the timestamp of image B[f] in a high-speed, high-dynamic scene, and T is the exposure time length.

[0085] For all events (t) within the exposure time interval w u w v w p w), w = 1, 2, 3, ..., num, where num is the number of all events within the exposure time interval, and t w Let w be the timestamp of the w-th event within the exposure time interval, (u w v w Let p be the coordinates of the w-th event within the exposure time interval. w Let p be the polarity of the w-th event within the exposure time interval, and p w ={0, 1}.

[0086] Define a zero matrix of size 2C×Width×Height, where Width is the width of the image B[f] in a high-speed, high-dynamic scene, and Height is the height of the image B[f] in a high-speed, high-dynamic scene. Divide the exposure time T into C parts. If the time interval of an event falls into the nth interval and the event polarity is p... w Then in the matrix (2n+p) w u w v w The value is increased by 1 at the position of ), and so on. In this embodiment, C = 48, Width = 346, and Height = 260.

[0087] Based on the above embodiments, step 200 includes:

[0088] A network for extracting image features is determined by sequentially inputting the preset reference image into a Conv layer, three DConv layers, and a ResBlock layer to obtain the features of the reference image.

[0089] The Conv layer includes a single convolutional layer with a stride of 1, the DConv layer includes a combination of a downsampling convolutional layer with a stride of 2 and a convolutional layer with a stride of 1, the ResBlock layer includes a residual connection layer, the kernel size of all convolutional layers is 3×3, and all convolutional layers are connected to the ReLU activation function.

[0090] Specifically, such as Figure 3 As shown, an image feature extraction network (IFE) is constructed for extracting features from a well-exposed and sharp reference image. Its input is the reference image I.

[0091] Its input layer is a Conv layer, which is a combination of a convolutional layer and an activation layer. The convolutional reference image can be obtained through this layer. The parameter variable to be optimized in the convolutional layer here is the convolution kernel parameter F0.

[0092] Then, through 3 DConv layers, the DConv layer is a combination of a downsampling convolutional layer with a stride of 2 and a convolutional layer with a stride of 1. Each convolutional layer is connected to an activation layer. The parameter variable to be optimized for several convolutional layers here is the convolutional kernel parameter F1.

[0093] Finally, the features F of the reference image are obtained through the ResBlock layer. I The ResBlock layer contains a convolutional layer and an activation layer with skip connections, and the parameter variable to be optimized in the convolutional layer is the convolutional kernel parameter F2.

[0094] Based on the above embodiments, step 100, fusing the preprocessed event stream with the preset high-speed, high-dynamic image to obtain fusion features, includes:

[0095] A fusion feature extraction network is determined. The preset high-speed and high-dynamic image is sequentially input into the Conv layer and the DConv layer to generate the first image tensor feature. The preprocessed event stream is sequentially input into the Conv layer and the DConv layer to generate the first event tensor feature. The first image tensor feature and the first event tensor feature are concatenated and then input into the ResBlock layer to obtain the first fusion feature.

[0096] The first image tensor feature corresponding to the first scale is input into the DConv layer to generate the second image tensor feature. The first event tensor feature corresponding to the first scale is input into the DConv layer to generate the second event tensor feature. The first fusion feature corresponding to the first scale is input into the DConv layer to generate the second fusion feature. The second image tensor feature, the second event tensor feature and the second fusion feature are concatenated and then input into the ResBlock layer to obtain the third fusion feature.

[0097] The second image tensor feature, the second event tensor feature, and the third fusion feature corresponding to the second scale are concatenated and then input into the ResBlock layer to obtain the initial multi-scale fusion feature.

[0098] The initial multi-scale fusion features are filtered through a CBAM layer to obtain the fusion features;

[0099] The Conv layer includes a single convolutional layer with a stride of 1, the DConv layer includes a combination of a downsampling convolutional layer with a stride of 2 and a convolutional layer with a stride of 1, the ResBlock layer includes a residual connection layer, the kernel size of all convolutional layers is 3×3, and all convolutional layers are connected to the ReLU activation function.

[0100] Specifically, such as Figure 3 As shown, a fusion feature extraction network is constructed for extracting fusion features from images and their corresponding preprocessed event streams in high-speed, high-dynamic scenes. Its input consists of an image B from a high-speed, high-dynamic scene and its corresponding preprocessed event stream T. ε .

[0101] The first step is the pre-fusion process: combining the image B from the high-speed, high-dynamic scene with its corresponding preprocessed event stream T. ε Input Conv layer and DConv layer to generate image and event tensor features F B0 and F ε0 The features are then concatenated and the fused features F at the initial scale are generated through the residual connection module. F0 Here, the parameter variables to be optimized for several convolutional layers are the convolutional kernel parameters F3;

[0102] Then comes the two-step fusion process: Fn of the image features at the nth (n∈{0,1}) scale. Bn Event tensor features F εn and fusion feature F Fn The image features F at the (n+1)th scale are generated through the DConv layer. B(n+1) and event tensor features F ε(n+1) Then fuse the features F Fn After passing through the DConv layer, it is connected together with the above features, and the fused feature F at the (n+1)th scale is generated through the ResBlock submodule. F(n+1) Here, the parameter variable to be optimized for several convolutional layers is the convolutional kernel parameter F4;

[0103] Finally, the Final-Fusion step involves inputting the image features F at the Nth (N=2) scale. BN Event tensor features F εN and fusion feature F FN First, the above features are concatenated and then passed through a ResBlock layer to obtain the initial multi-scale fused feature F. P0 Then, the effective information at both the feature map location and channel levels is filtered through the CBAM layer to obtain the fused F. p Here, the parameter variables to be optimized for several convolutional layers are the convolutional kernel parameters F5;

[0104] Where Conv is a single convolutional layer with a stride of 1, DConv is a combination of a downsampling convolutional layer with a stride of 2 and a convolutional layer with a stride of 1, and ResBlock is a convolutional layer with residual connections. All the convolutional layers have a kernel size of 3×3, and the activation function following them is ReLU.

[0105] Based on the above embodiments, step 200 is followed by:

[0106] Construct training triplets of query image, positive example image and negative example image. The query image includes the preset high-speed high dynamic range image and the preprocessed event stream. The positive example image includes images with the same orientation as the query image and within a preset query distance range. The negative example image includes images with a different orientation than the query image and outside the preset query distance range.

[0107] A maximum loss function is constructed from the training triples, the vector distance function, and the similarity distance constant.

[0108] Based on the maximum loss function, the adaptive learning rate of the fusion feature extraction network and the image feature extraction network is determined by a preset optimization algorithm, and the parameters of the neural network convolutional layer are updated by the training triples to obtain the optimized fusion feature extraction network and the optimized image feature extraction network.

[0109] The optimization parameters for each convolutional layer in the optimized fusion feature extraction network and the optimized image feature extraction network are different.

[0110] Optionally, this invention further trains the network and adjusts parameters by constructing query, positive example, and negative example training triples. The query is an image in a high-speed, high-dynamic scene and its preprocessed event stream within its exposure time interval; positive examples are images with a similar field of view to the query image; and negative examples are images with a different scene from the query image. All triples constructed from the dataset are used for network training, and supervision is provided through triple loss, thereby further obtaining an optimized image feature extraction network and a fusion feature extraction network.

[0111] It should be noted that the training triples are constructed as follows:

[0112] The query is an image B and its preprocessed event stream T within the exposure time interval of a high-speed, high-dynamic scene. ε , positive example I p It is a reference image similar to the query image's field of view, negative example I. n This is a reference image that differs from the query image B. Positive and negative examples are selected based on the location and orientation of the image. Selected positive examples are close to the query location and have the same orientation, while selected negative examples are far from the query location and have a different orientation. In this embodiment, positive examples are required to be within 1 meter of the query location and have the same orientation, while negative examples are required to be more than 25 meters away from the query location and have a different orientation.

[0113] The corresponding triplet loss is:

[0114]

[0115] Where L is the loss function, F pThese are the fused features obtained from the fusion feature extraction network used in the query. These are the image features obtained by the image feature extraction network for the positive and negative examples, respectively. max(·) takes the larger value of the two inputs, d(·) calculates the L2 distance between the two input vectors, and M is the similarity distance constant.

[0116] The optimization algorithm used in this invention is the Adam optimization algorithm. It designs independent adaptive learning rates for different parameters by calculating the first and second moment estimates of the gradient, and iteratively updates the parameters of the neural network convolutional layers based on the training data. The resulting optimized image feature extraction network is constructed based on the optimized parameters of each convolutional layer, and the optimized fusion feature extraction network is also constructed based on the optimized parameters of each convolutional layer.

[0117] Images and event streams with field-of-view alignment in high-speed, high-dynamic scenes are captured using optical and event cameras. The event streams are preprocessed to obtain a preprocessed event stream. The images and event streams from the high-speed, high-dynamic scene are then input into an optimized fusion feature extraction network to obtain fused features. A reference image database is then input into the optimized image feature extraction network to obtain a reference image feature database. Finally, based on the feature vector similarity between the fused features and the image features, a reference image that most closely matches the field of view of the input high-speed, high-dynamic scene is retrieved from the reference image database. Figure 4 The scene recognition example shown demonstrates multiple sets of data including an image B, an event stream ε, a reference image sequence I, and a recognition result I in a high-speed, high-dynamic scene. m As can be seen, the scene recognition method based on fused event cameras proposed in this invention has strong robustness.

[0118] The scene recognition system based on fused event cameras provided by the present invention will be described below. The scene recognition system based on fused event cameras described below can be referred to in correspondence with the scene recognition method based on fused event cameras described above.

[0119] Figure 5 This is a schematic diagram of the structure of a scene recognition system based on a fused event camera provided in an embodiment of the present invention, as shown below. Figure 5 As shown, it includes: a first processing module 51, a second processing module 52, and a recognition module 53, wherein:

[0120] The first processing module 51 is used to acquire a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream, preprocess the preset high-speed high-dynamic image to obtain a preprocessed event stream, and fuse the preprocessed event stream with the preset high-speed high-dynamic image to obtain a fusion feature; the second processing module 52 is used to acquire a preset reference image and extract reference image features from the preset reference image; the recognition module 53 is used to search and match the fusion feature with the reference image feature to obtain a reference image recognition result.

[0121] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a scene recognition method based on a fused event camera. This method includes: acquiring a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream; preprocessing the preset high-speed high-dynamic image to obtain a preprocessed event stream; fusing the preprocessed event stream with the preset high-speed high-dynamic image to obtain fused features; acquiring a preset reference image; extracting reference image features from the preset reference image; and performing a search and matching operation between the fused features and the reference image features to obtain a reference image recognition result.

[0122] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the scene recognition method based on a fused event camera provided by the above methods. The method includes: acquiring a preset high-speed high dynamic range image and a preset high-speed high dynamic range event stream; preprocessing the preset high-speed high dynamic range image to obtain a preprocessed event stream; fusing the preprocessed event stream with the preset high-speed high dynamic range image to obtain a fused feature; acquiring a preset reference image; extracting reference image features from the preset reference image; and performing a search and matching between the fused feature and the reference image features to obtain a reference image recognition result.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the scene recognition method based on a fusion event camera provided by the above methods. The method includes: acquiring a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream; preprocessing the preset high-speed high-dynamic image to obtain a preprocessed event stream; fusing the preprocessed event stream with the preset high-speed high-dynamic image to obtain fusion features; acquiring a preset reference image; extracting reference image features from the preset reference image; and performing a search and matching between the fusion features and the reference image features to obtain a reference image recognition result.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scene recognition based on a fusion event camera, characterized in that, The method comprises the following steps: obtaining a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream, preprocessing the preset high-speed high-dynamic image to obtain a preprocessed event stream, fusing the preprocessed event stream and the preset high-speed high-dynamic image to obtain fusion features; obtaining a preset reference image and extracting reference image features from the preset reference image; performing retrieval matching on the fusion features and the reference image features to obtain a reference image recognition result; the step of fusing the preprocessed event stream and the preset high-speed high-dynamic image to obtain fusion features comprises: determining a fusion feature extraction network, inputting the preset high-speed high-dynamic image into a Conv layer and a DConv layer in sequence to generate a first image tensor feature, inputting the preprocessed event stream into a Conv layer and a DConv layer in sequence to generate a first event tensor feature, and inputting the first image tensor feature and the first event tensor feature into a ResBlock layer after being connected to obtain a first fusion feature; inputting the first image tensor feature corresponding to a first scale into a DConv layer to generate a second image tensor feature, inputting the first event tensor feature corresponding to the first scale into a DConv layer to generate a second event tensor feature, inputting the first fusion feature corresponding to the first scale into a DConv layer to generate a second fusion feature, and inputting the second image tensor feature, the second event tensor feature and the second fusion feature into a ResBlock layer after being connected to obtain a third fusion feature; connecting the second image tensor feature, the second event tensor feature and the third fusion feature corresponding to a second scale and inputting them into a ResBlock layer to obtain an initial multi-scale fusion feature; screening the initial multi-scale fusion feature through a CBAM layer to obtain the fusion features; The Conv layer comprises a single-layer convolution layer with a step of 1, the DConv layer comprises a combination of a down-sampling convolution layer with a step of 2 and a convolution layer with a step of 1, the ResBlock layer comprises a residual connection layer, and the convolution kernel size of all the convolution layers is , and all the convolution layers are connected with an activation function ReLU. after the step of obtaining a preset reference image and extracting reference image features from the preset reference image, the method further comprises the following steps: constructing a training triple of a query image, a positive example image and a negative example image, the query image comprising the preset high-speed high-dynamic image and the preprocessed event stream, the positive example image comprising an image having the same orientation as the query image and being within a preset query distance range, and the negative example image comprising an image having a different orientation from the query image and being outside the preset query distance range; constructing a maximum loss function from the training triple, an inter-vector distance function and a similarity distance constant; based on the maximum loss function, determining an adaptive learning rate of the fusion feature extraction network and the image feature extraction network by using a preset optimization algorithm, and updating neural network convolution layer parameters by using the training triple to obtain an optimized fusion feature extraction network and an optimized image feature extraction network; wherein each convolution layer optimization parameter in the optimized fusion feature extraction network and the optimized image feature extraction network is different.

2. The fusion event camera based scene identification method of claim 1, wherein, the step of obtaining a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream comprises: Collect a reference image sequence and an event stream corresponding to the reference image sequence, the reference image sequence comprising any frame of preset clear exposure image, and the event stream corresponding to the reference image sequence comprising an event stream in an exposure interval of the any frame of preset clear exposure image; Sum and average a plurality of continuous frames of reference images in the reference image sequence to obtain a blur image sequence; Take a union of the event stream corresponding to the reference image sequence based on the plurality of continuous frames of reference images to obtain a preset high-speed high-dynamic event stream; Determine a photometric linear transformation parameter and a photometric linear transformation interval, and obtain the preset high-speed high-dynamic image based on the photometric linear transformation parameter, the photometric linear transformation interval and the blur image sequence.

3. The fusion event camera based scene identification method of claim 1, wherein, The preprocessing of the preset high-speed high-dynamic image to obtain a preprocessed event stream comprises: Determine an event stream in an exposure interval corresponding to any frame of image in the preset high-speed high-dynamic image, and determine an exposure time interval of the event stream in the exposure interval; Obtain the number of exposure events of the exposure time interval, and obtain an exposure event quadruple based on the number of exposure events, the exposure event quadruple comprising any exposure event timestamp, any exposure event abscissa, any exposure event ordinate and any exposure event polarity identifier; Determine a full zero matrix constructed by dividing the width, height and exposure time length of the preset high-speed high-dynamic image into equal parts, and fill the full zero matrix based on the exposure event quadruple to obtain the preprocessed event stream.

4. The fusion event camera based scene identification method of claim 1, wherein, The preset reference image is obtained, and a reference image feature in the preset reference image is extracted, comprising: Determine an image feature extraction network, and input the preset reference image into a Conv layer, three DConv layers and a ResBlock layer in sequence to obtain the reference image feature. The Conv layer comprises a single-layer convolution layer with a step of 1, the DConv layer comprises a combination of a down-sampling convolution layer with a step of 2 and a convolution layer with a step of 1, the ResBlock layer comprises a residual connection layer, and the convolution kernel size of all the convolution layers is , and all the convolution layers are connected with an activation function ReLU.

5. A fusion event camera based scene recognition system based on the fusion event camera based scene recognition method of any one of claims 1 to 4, characterized in that, Comprise: The first processing module is configured to obtain a preset high-speed high-dynamic image and a preset high-speed high-dynamic event stream, preprocess the preset high-speed high-dynamic image to obtain a preprocessed event stream, and fuse the preprocessed event stream and the preset high-speed high-dynamic image to obtain a fusion feature; The second processing module is configured to obtain a preset reference image and extract a reference image feature in the preset reference image; The identification module is configured to search and match the fusion feature and the reference image feature to obtain a reference image identification result.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the scene identification method based on the fusion event camera according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the scene identification method based on the fusion event camera according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the scene identification method based on the fusion event camera according to any one of claims 1 to 4.

Images