Image processing method, image processing device, and monitoring system

Abstract

Provided are an image processing method, an image processing device, and a monitoring system. The image processing method comprises: capturing changes in a monitored scene; and performing sparse feature calculation on the captured changes in the monitored scene to obtain a sparse feature map. The image processing method, the image processing device, and the monitoring system according to the example embodiments of the present application can improve the image processing speed by utilizing the sparsity of scene changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video monitoring and artificial intelligence, and more particularly to an image processing method, an image processing device and a monitoring system. BACKGROUND

[0002] Video surveillance is an important part of a security and guard system, and generally includes a front-end camera, a transmission medium and a video surveillance platform. It is a comprehensive system with strong prevention capability, and is widely used in many fields due to its intuitive, accurate and timely information content. At present, video surveillance is usually based on CIS (CMOS Image Sensor) cameras with frame imaging, and the content analysis is usually performed by manual analysis by the staff of the video surveillance platform.

[0003] Since security and guard events such as theft and falling may occur at any time, video surveillance needs to be in a 24-hour open state. A 24-hour open video surveillance system will introduce a large amount of data. For example, the amount of data of a video surveillance system with VGA (Video Graphics Array) resolution can reach 74 GB per day. Analyzing and processing such a large amount of data requires a large amount of computing resources, and the performance of the video surveillance system greatly limits the application range of the video surveillance system.

[0004] Massive video data monitored by manual analysis not only consumes a large amount of labor cost, but also inevitably leads to late detection and missed detection due to the limitation of concentration.

[0005] Artificial intelligence is a technology science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Applying artificial intelligence technology to the field of video surveillance can complete automatic video understanding, monitoring and analysis functions, reduce labor costs of the video surveillance platform, and expand the application range of video surveillance.

[0006] Convolution neural network is a representative algorithm of artificial intelligence in the field of vision, and is widely used in the field of video surveillance. It simulates the visual perception mechanism of biology, uses a series of convolution kernels to traverse the entire image multiple times to extract high-dimensional features that can describe the content of the image. However, due to the particularity of the field of video surveillance, the convolution neural network needs to traverse all the information in the monitored video, and the massive data will lead to massive consumption of computing resources.

[0007] Sparse Convolution Neural Network is a special kind of Convolution Neural Network. Compared with traditional Convolution Neural Network which traverses the whole image, Sparse Convolution Neural Network only calculates the region which contains valid information. An efficient Sparse Convolution Neural Network design can greatly improve the performance of the system.

[0008] In the existing Sparse Convolution Neural Network, one is based on Block to determine whether the Block is sparse, and based on the sparse Block to perform convolution calculation. For example, first, divide the input picture into multiple Blocks, determine whether the data in the Block reaches the standard of valid data, then perform convolution on the Block containing valid data, and finally spread the operation result to the output space. In the above method, the sparse Block is selected, extracted and calculated, but there are still a large number of sparsities in the Block, resulting in a large amount of calculation. In addition, there are overlapping regions between Blocks, which will cause additional calculation overhead.

[0009] Another is to analyze the sparsity of the input Feature map of the Convolution Neural Network or the sparsity of the convolution kernel, so that the calculation unit does not process these invalid calculations. As shown in Figure 1 As shown, first, the first non-zero weight vector and the position of the original vector corresponding to the non-zero weight vector are received, and the second non-zero activation vector and the original position of the non-zero activation vector are received, then the result of the non-zero weight vector and the non-zero weight activation vector is calculated through multiplication, that is, the third vector, the position mapping relationship of the non-zero weight vector and the non-zero activation vector is arranged, and the fourth vector is formed, and the third vector is arranged through the position mapping relationship contained in the fourth vector. This method of judging sparsity through input needs to analyze the sparsity of the input Feature map with additional calculation, and needs a special calculation platform to realize the above method.

[0010] A large amount of resources is wasted in the recording analysis of static background in the prior art. SUMMARY

[0011] The exemplary embodiments of the present application aim to provide an image processing method, an image processing device and a monitoring system to overcome at least one of the above-mentioned defects.

[0012] According to an aspect of the exemplary embodiments of the present application, an image processing method is provided, comprising: capturing changes of a monitored scene; performing sparse feature calculation on the captured changes of the monitored scene to obtain a sparse Feature map.

[0013] Optionally, the step of performing sparse feature computation on the captured change of the monitored scene to obtain a sparse feature map can comprise: generating pixel-based events based on the captured change of the monitored scene; projecting the generated events along a time axis, and performing sparse feature computation on a two-dimensional image formed based on the projection to obtain the sparse feature map.

[0014] Optionally, the event can comprise a position of a pixel, a time of occurrence of the event, and a polarity of the pixel, and the polarity can indicate a change in brightness value of the pixel.

[0015] Optionally, the step of generating pixel-based events can comprise: generating pixel-based events based on a change of the monitored scene captured by an event camera, or generating pixel-based events based on frame difference processing on a frame image captured by a normal camera.

[0016] Optionally, the event camera can comprise a dynamic vision sensor and / or a time-asynchronous image sensor.

[0017] Optionally, the earlier the time of occurrence of the event, the smaller the mapping value of the event on the two-dimensional image, and the later the time of occurrence of the event, the larger the mapping value of the event on the two-dimensional image.

[0018] Optionally, a texture of an object generating the events is reconstructed using the pixel-based events.

[0019] Optionally, the two-dimensional image can be a two-dimensional image formed based on events generated by movement of the object, wherein the events generated by movement of the object can be obtained by filtering out events generated by light source flickering and / or by internal noise of a camera used to capture the change of the monitored scene from all generated events.

[0020] Optionally, the step of performing sparse feature computation on the two-dimensional image based on the projection to obtain the sparse feature map can comprise: performing sparse feature computation on the two-dimensional image based on a sparse convolutional neural network to obtain the sparse feature map.

[0021] Optionally, the step of performing sparse feature computation on the two-dimensional image based on the sparse convolutional neural network to obtain the sparse feature map can comprise: determining a feature activation map corresponding to each convolutional layer of the sparse convolutional neural network based on a structure of the sparse convolutional neural network and the two-dimensional image; and obtaining the sparse feature map based on the determined feature activation maps and the two-dimensional image by the sparse convolutional neural network.

[0022] Optionally, the feature activation map corresponding to each convolutional layer can indicate a position of a pixel required to participate in computation in the convolutional layer.

[0023] Optionally, the size of the feature activation map corresponding to each convolutional layer can be determined according to the size of the input feature map of each convolutional layer of the sparse convolutional neural network and the size of the receptive field of the convolution kernel.

[0024] Optionally, in the sparse convolutional neural network, the sparsity of the output feature map of each convolutional layer can be constrained by the sparsity of the input feature map of the next convolutional layer of each convolutional layer.

[0025] Optionally, the output feature map of each convolutional layer in the sparse convolutional neural network can be obtained by: extracting, from the input feature map of each convolutional layer, pixel points that need to participate in calculation based on the feature activation map corresponding to each convolutional layer; performing sparse feature aggregation on the extracted pixel points to form a dense matrix; obtaining a feature aggregation matrix according to the formed dense matrix and the convolution kernel matrix corresponding to each convolutional layer; and dispersively outputting the feature vectors in the obtained feature aggregation matrix to obtain the output feature map of each convolutional layer.

[0026] Optionally, the image processing method can further include generating a position mapping table between the pixel points extracted from the input feature map of each convolutional layer and the data in the dense matrix, wherein the feature vectors in the feature aggregation matrix are mapped based on the position mapping table to obtain the output feature map of each convolutional layer.

[0027] Optionally, the image processing method can further include: performing regional feature extraction on the obtained sparse feature map; identifying an object causing a change in the monitored scene based on the extracted regional features; and if the identified object is a person, determining the position, motion trajectory, behavior and / or identity of the object.

[0028] Optionally, the image processing method can further include: when the identified object is a person, triggering a normal camera for capturing the frame image to capture the object.

[0029] Optionally, the image processing method can further include: fusing the sparse feature map obtained at the current time with a historical sparse feature map to perform regional feature extraction on the fused sparse feature map.

[0030] Optionally, the sparse feature map and the historical sparse feature map are fused using a long short-term memory recurrent neural network.

[0031] According to another aspect of the exemplary embodiments of the present application, a monitoring system is provided, comprising: a front-end camera for capturing changes in a monitored scene; and a processor configured to: perform sparse feature calculation on the captured changes in the monitored scene to obtain a sparse feature map.

[0032] Optionally, the processor can project the pixel-based events generated by capturing the changes of the monitored scene along a time axis, and perform sparse feature calculation based on a two-dimensional image formed by the projection to obtain a sparse feature map.

[0033] Optionally, the event can include a position of the pixel, a time when the event occurs, and a polarity of the pixel, and the polarity can indicate a luminance value change of the pixel.

[0034] Optionally, the front-end camera can include an event camera for directly outputting events or a normal camera for capturing frame images, wherein the event camera generates pixel-based events when capturing changes of the monitored scene, or the processor generates pixel-based events by performing frame difference processing on the frame images captured by the normal camera.

[0035] Optionally, the event camera can include a dynamic vision sensor and / or a time-asynchronous image sensor.

[0036] Optionally, the earlier the event occurs, the smaller the mapping value of the event on the two-dimensional image, and the later the event occurs, the larger the mapping value of the event on the two-dimensional image.

[0037] Optionally, the processor can further use the pixel-based events to reconstruct a texture of an object that generates the events.

[0038] Optionally, the two-dimensional image can be a two-dimensional image formed based on events generated by object motion, wherein the processor can obtain the events generated by object motion by filtering out events generated by light source flickering and / or internal noise of a camera used to capture changes of the monitored scene from all generated events.

[0039] Optionally, the processor can perform sparse feature calculation on the two-dimensional image by a sparse convolutional neural network to obtain the sparse feature map.

[0040] Optionally, the processor can determine a feature activation map corresponding to each convolutional layer of the sparse convolutional neural network based on a structure of the sparse convolutional neural network and the two-dimensional image, and obtain the sparse feature map by the sparse convolutional neural network using the determined feature activation maps and the two-dimensional image.

[0041] Optionally, the feature activation map corresponding to each convolutional layer can indicate a position of a pixel that needs to participate in calculation in each convolutional layer.

[0042] Optionally, a size of the feature activation map corresponding to each convolutional layer can be determined according to a size of an input feature map of each convolutional layer of the sparse convolutional neural network and a size of a receptive field of a convolution kernel.

[0043] Optionally, in the sparse convolutional neural network, the sparsity of the input feature map of a next convolutional layer of any convolutional layer can be used to constrain the sparsity of the output feature map of the any convolutional layer.

[0044] Optionally, the processor can obtain the output feature map of any convolutional layer in the sparse convolutional neural network by: extracting, from the input feature map of the any convolutional layer, pixel points that need to participate in calculation based on a feature activation map corresponding to the any convolutional layer; performing sparse feature aggregation on the extracted pixel points to form a dense matrix; obtaining a feature aggregation matrix according to the formed dense matrix and a convolution kernel matrix corresponding to the any convolutional layer; and dispersively outputting feature vectors in the obtained feature aggregation matrix to obtain the output feature map of the any convolutional layer.

[0045] Optionally, the processor can further generate a position mapping table between the extracted pixel points and data in the dense matrix, and the processor can map the feature vectors in the feature aggregation matrix based on the position mapping table to obtain the output feature map of the any convolutional layer.

[0046] Optionally, the processor can be further configured to: perform region feature extraction on the obtained sparse feature map; identify an object causing a change in the monitored scene based on the extracted region features; and if the identified object is a person, determine a position, a motion trajectory, a behavior and / or an identity of the object.

[0047] Optionally, the front-end camera can include an event camera for directly outputting events and a normal camera for capturing frame images, and the event camera can be used to capture changes in the monitored scene, and the processor can trigger the normal camera to take a picture of the object when the identified object is a person.

[0048] Optionally, the processor further fuses the sparse feature map obtained at the current time with a historical sparse feature map to perform region feature extraction on the fused sparse feature map.

[0049] Optionally, the processor uses a long short-term memory recurrent neural network to fuse the sparse feature map with the historical sparse feature map.

[0050] According to another aspect of the exemplary embodiments of the present application, there is provided an image processing device, comprising: a dynamic event information acquisition module, which captures changes in a monitored scene; and a sparse feature map generation module, which performs sparse feature calculation on the captured changes in the monitored scene to obtain a sparse feature map.

[0051] Optionally, the sparse feature map generation module can comprise: a dynamic event map generation submodule, projecting events based on pixels generated by capturing changes in the monitored scene along a time axis to form a two-dimensional image; and a sparse feature map operation submodule, performing sparse feature calculation based on the two-dimensional image formed by the projection to obtain the sparse feature map.

[0052] Optionally, the events can comprise a location of the pixel, a time of the event, and a polarity of the pixel, and the polarity can indicate a luminance value change of the pixel.

[0053] Optionally, the dynamic event information acquisition module can generate events based on pixels when capturing changes in the monitored scene, or the sparse feature map generation module can further comprise a frame difference processing submodule, which generates events based on pixels by performing frame difference processing on the captured frame images of the monitored scene.

[0054] Optionally, the earlier the time of the event, the smaller the mapping value of the event on the two-dimensional image, and the later the time of the event, the larger the mapping value of the event on the two-dimensional image.

[0055] Optionally, the sparse feature map generation module can further comprise a texture reconstruction submodule, which uses events based on pixels to reconstruct the texture of the object that generates the events.

[0056] Optionally, the sparse feature map generation module can further comprise a dynamic event preprocessing submodule, which filters out events generated by light source flickering and / or internal noise of a camera used to capture changes in the monitored scene from all generated events to obtain events generated by object motion, and the dynamic event map generation submodule can form a two-dimensional image based on the events generated by object motion.

[0057] Optionally, the sparse feature map operation submodule can perform sparse feature calculation on the two-dimensional image by a sparse convolutional neural network to obtain the sparse feature map.

[0058] Optionally, the sparse feature map operation submodule can comprise: a feature activation map generation unit, determining a feature activation map corresponding to each convolutional layer of the sparse convolutional neural network based on a structure of the sparse convolutional neural network and the two-dimensional image; and a sparse feature map operation unit, obtaining the sparse feature map by the sparse convolutional neural network using the determined feature activation maps and the two-dimensional image.

[0059] Optionally, the feature activation map corresponding to each convolutional layer can indicate a location of a pixel that needs to participate in calculation in the convolutional layer.

[0060] Optionally, a size of the feature activation map corresponding to each convolutional layer can be determined according to a size of an input feature map of each convolutional layer of the sparse convolutional neural network and a size of a receptive field of a convolution kernel.

[0061] Optionally, in the sparse convolutional neural network, the sparsity of the input feature map of a next convolutional layer of any convolutional layer is used to constrain the sparsity of the output feature map of the any convolutional layer.

[0062] Optionally, the sparse feature map operation unit can obtain the output feature map of any convolutional layer in the sparse convolutional neural network by: extracting, from the input feature map of the any convolutional layer, pixel points that need to participate in calculation based on the feature activation map corresponding to the any convolutional layer; performing sparse feature aggregation on the extracted pixel points to form a dense matrix; obtaining a feature aggregation matrix according to the formed dense matrix and the convolution kernel matrix corresponding to the any convolutional layer; and dispersively outputting feature vectors in the obtained feature aggregation matrix to obtain the output feature map of the any convolutional layer.

[0063] Optionally, the sparse feature map operation unit can further generate a position mapping table between the pixel points extracted from the input feature map of the any convolutional layer and data in the dense matrix, wherein the sparse feature map operation unit can map the feature vectors in the feature aggregation matrix based on the position mapping table to obtain the output feature map of the any convolutional layer.

[0064] Optionally, the image processing device can further include a feature map analysis module that performs region feature extraction on the obtained sparse feature map, and identifies an object that causes a change in a monitored scene based on the extracted region features, and determines a position, a motion trajectory, a behavior and / or an identity of the object if the identified object is a human.

[0065] Optionally, when the identified object is a human, the feature map analysis module can generate and output a shooting trigger signal to trigger a normal camera for capturing a frame image to shoot the object.

[0066] Optionally, the image processing device can further include a fusion module that fuses the obtained sparse feature map at a current time with a historical sparse feature map, and the feature map analysis module performs region feature extraction on the fused sparse feature map.

[0067] Optionally, the fusion module uses a long short-term memory recurrent neural network to fuse the sparse feature map with the historical sparse feature map.

[0068] In another general aspect, there is provided a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method described above.

[0069] The image processing method, the image processing device and the monitoring system using the above-mentioned exemplary embodiments of the present application improve the image processing speed by using pixel-level sparsity. BRIEF DESCRIPTION OF DRAWINGS

[0070] The above and other objects, features and advantages of the present exemplary embodiments will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:

[0071] Figure 1 A schematic diagram showing a conventional block sparse convolutional neural network;

[0072] Figure 2 A flowchart showing an image processing method according to an exemplary embodiment of the present application;

[0073] Figure 3 A flowchart showing a step of obtaining a sparse feature map according to an exemplary embodiment of the present application;

[0074] Figure 4 A schematic diagram showing generation of a two-dimensional image according to an exemplary embodiment of the present application;

[0075] Figure 5A And Figure 5B A schematic diagram showing a comparison of data amount between a conventional image processing method and an image processing method according to an exemplary embodiment of the present application;

[0076] Figure 6 A flowchart showing a step of obtaining a sparse feature map by a sparse convolutional neural network according to an exemplary embodiment of the present application;

[0077] Figure 7 A flowchart showing a step of generating an output feature map corresponding to any convolutional layer in a sparse convolutional neural network according to an exemplary embodiment of the present application;

[0078] Figure 8 A schematic diagram showing generation of an output feature map corresponding to any convolutional layer in a sparse convolutional neural network according to an exemplary embodiment of the present application;

[0079] Figure 9 A flowchart showing a step of performing scene change analysis based on a sparse feature map according to an exemplary embodiment of the present application;

[0080] Figures 10A to 10F A schematic diagram showing movement of an object in a monitored scene according to an exemplary embodiment of the present application;

[0081] Figure 11 A schematic diagram showing a comparison of data amount and a comparison of computational efficiency between a conventional image processing method and an image processing method according to an exemplary embodiment of the present application;

[0082] Figure 12 A block diagram showing a monitoring system according to an exemplary embodiment of the present application;

[0083] Figure 13A block diagram of an image processing device according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0084] The various example embodiments will now be described more fully with reference to the accompanying drawings in which some example embodiments are illustrated.

[0085] Figure 2 A flowchart of an image processing method according to an example embodiment of the present application is shown.

[0086] Reference Figure 2 In step S10, changes in the monitored scene are captured.

[0087] Since most of the regions in the monitored scene are static, responding to changes in the monitored scene only can greatly reduce the amount of data that needs to be saved and analyzed.

[0088] In step S20, sparse feature calculation is performed on the captured changes in the monitored scene to obtain a sparse feature map.

[0089] In an example embodiment of the present application, changes in the scene can be monitored in real time, and when changes in the scene occur, a sparse feature map can be obtained based on the sparsity of the pixel points, effectively improving the processing speed of the image. The process of obtaining a sparse feature map based on changes in the monitored scene will be described in detail below with reference to Figure 3

[0090] Figure 3 A flowchart of the steps of obtaining a sparse feature map according to an example embodiment of the present application is shown.

[0091] Reference Figure 3 In step S201, when changes in the monitored scene are captured, an event based on the pixel points is generated.

[0092] For example, when the amount of change in the brightness value of any pixel point in the captured scene image of the monitored scene is greater than a predetermined value, an event of the pixel point is generated. As an example, the event can include the location of the pixel point, the time when the event occurred, and the polarity of the pixel point. Here, the polarity can indicate the change in the brightness value of the pixel point, for example, the polarity of the pixel point can indicate whether the change that occurred in the pixel point is an increase in the brightness value or a decrease in the brightness value.

[0093] For example, the event based on the pixel points can be generated in the following manner.

[0094] In one case, the changes in the monitored scene can be captured by an event camera to generate an event based on the pixel points.

[0095] ​Here, the event camera can refer to a camera capable of directly outputting events, as an example, the event camera can include but is not limited to a Dynamic Vision Sensor (DVS) and / or a time-asynchronous image sensor.

[0096] Taking the DVS as an example, the DVS is a new generation of vision sensor. The traditional vision sensor records the image of the scene in units of frames, while the DVS captures the changes in the scene and generates a series of events based on the changes. However, due to the changes in the ambient light, the performance of the sensor itself, and other factors, the events generated by the DVS have the characteristics of being noisy and having a small number (sparse). Events can also be generated due to the flickering of some light sources.

[0097] In the DVS, for a single pixel point, an event (pulse) signal is output only when the received light intensity changes. For example, if the brightness value of a pixel point increases by an amount exceeding a predetermined value, an event of the brightness increase of the pixel point is generated.

[0098] In another case, pixel-based events can be generated by performing frame difference processing on the frame images captured by a general camera.

[0099] Here, the general camera can refer to a traditional frame imaging camera, as an example, the general camera can include but is not limited to a CIS (CMOS Image Sensor) camera based on frame imaging. By performing frame difference processing on the frame images captured by the CIS camera, it is determined whether the monitored scene has changed, and when it is determined that the monitored scene has changed, pixel-based events are generated.

[0100] It should be understood that the method of performing frame difference processing on frame images is well known in the art, and the present application will not repeat the contents of this part.

[0101] Optionally, the at least one camera described above can be arranged at different positions in the monitored scene, and the at least one camera can collect scene information of different monitoring ranges.

[0102] It should be understood that in the existing sparse convolutional neural network, it is generally based on blocks to determine whether a region is sparse, and convolution calculation is performed based on the sparse blocks. However, the block mask still contains a large amount of sparsity in the above processing process, resulting in a still large amount of calculation.

[0103] However, in the exemplary embodiments of the present application, when the change in the monitored scene is captured, one pixel point corresponds to one event, and the subsequent convolution processing is performed using the event corresponding to the pixel point, that is, the convolution processing is performed based on the sparsity of the pixel points, and the convolution acceleration is realized by using the pixel-level sparsity.

[0104] In step S202, the generated events are projected onto a two-dimensional image along the time axis.

[0105] Because sensors capture all scene changes at high speed and asynchronously, generating a large number of events densely sampled along the timeline, processing these events one by one would consume significant computational resources. Therefore, events can be projected along the timeline to form a two-dimensional image, and subsequent processing can be performed based on this two-dimensional image to reduce computational load.

[0106] Here, when a change in the monitored scene is captured, an event is generated. Over time, an event stream is formed. The event stream is then projected onto a two-dimensional image plane in a time-decay manner to generate a two-dimensional image.

[0107] As an example, the process of projecting an event onto a two-dimensional image along the time axis described above can be mathematically expressed as an integral. In this case, the two-dimensional image can be obtained by integrating the generated events along the time axis. For example, by integrating the event stream along the time axis to form a spatial integral image, this integral image is the two-dimensional image formed by the projection.

[0108] Preferably, a two-dimensional image can be obtained by performing a time decay integral on the generated events along the time axis.

[0109] In one example, by performing an exponentially decaying integral based on the difference between the time the event occurred and the current time, the pixel value of a single pixel on the two-dimensional image at the current time can be calculated using the following formula:

[0110]

[0111] In formula (1), I (x,y) T represents the pixel value of the pixel at coordinate (x, y) in a two-dimensional image. last(x,y) T represents the time when the pixel at coordinate position (x, y) appears (i.e., the time when the event occurs). now σ is the current time, and σ is a scaling factor used to control the rate at which the event decays along the time axis.

[0112] In another example, by performing a linear decay integral based on the difference between the time the event occurred and the current time, the pixel value of a pixel on the two-dimensional image at the current time can be calculated using the following formula:

[0113]

[0114] In formula (2), I (x,y) T represents the pixel value of the pixel at coordinate (x, y) in a two-dimensional image. last(x,y) T represents the time when the pixel at coordinate position (x, y) appears (i.e., the time when the event occurs).start Tstart is the start time of the statistical window now Tcurrent is the current time.

[0115] That is, the earlier the event occurs, the smaller the mapping value of the event on the two-dimensional image (i.e., the smaller the pixel value calculated above), and the later the event occurs, the larger the mapping value of the event on the two-dimensional image (i.e., the larger the pixel value calculated above).

[0116] Figure 4 A schematic diagram of generating a two-dimensional image according to an example embodiment of the present application is shown.

[0117] In Figure 4 In the example shown, an event stream is generated by capturing changes in a monitored scene using a dynamic vision sensor, Figure 4 The left image in FIG. 4 is an event stream of changes in a monitored scene captured by a dynamic vision sensor, and the right image is a result of projecting the captured event stream in a predetermined time period on a two-dimensional image plane.

[0118] When there is movement of objects in the monitored scene, the dynamic vision sensor generates a series of event streams (such as the left image in FIG. 4), and the generated event stream can form a spatial integral image (such as the right image in FIG. 4) by integrating the event stream on the time axis. The actual effective information (i.e., the information of the pixel points that need to participate in the calculation) only exists in the non-zero region of the integral image, and the zero value region does not contain effective information and does not need to be saved or participate in subsequent processing and calculation. Figure 4 When there is no movement or change of objects in the monitored scene, the dynamic vision sensor does not image and does not need to save data or perform calculation and analysis. Therefore, when there is only individual movement or small range change of objects in the monitored scene, the data volume and calculation volume of the dynamic vision sensor are much lower than those of the traditional frame imaging sensor.

[0119]

[0120] and Figure 5A A schematic diagram of comparing the data volume of an existing image processing method with that of an image processing method according to an example embodiment of the present application is shown. Figure 5B

[0121] The imaging result of an existing CIS camera based on frame imaging is shown, Figure 5A The integral imaging image based on the dynamic vision sensor under the same scene is shown. From Figure 5B and Figure 5A It can be seen from the comparison of FIGS. 5 and 6 that the video monitoring system based on the dynamic vision sensor does not capture the static background, and thus the data volume is much lower than that of the camera based on frame imaging. Figure 5B

[0122] ​The sensor for monitoring reacts to changes in the scene outside, but due to its high response speed and high dynamic range, changes in the light outside (for example, the flicker of the power frequency of the lighting source) can also cause the sensor to generate response events. In addition, the sensor can also generate sparse events due to internal noise. The events generated by light source flicker or internal noise can be separated from the collected events through preprocessing due to their strong regularity. The separated events can be considered as events generated by object movement.

[0123] In a preferred embodiment, the two-dimensional image can be a two-dimensional image formed based on events generated by object movement.

[0124] For example, events generated by object movement can be obtained by filtering out events generated by light source flicker and / or internal noise of the camera used to capture changes in the monitored scene from all generated events.

[0125] As an example, events generated by light source flicker can be filtered out from all events through a light source flicker detection algorithm, for example, a light source flicker detection algorithm detects a flickering light source region in the scene, and filters out all events in the flickering light source region. And / or, events generated by internal noise of the camera used to capture changes in the monitored scene can be filtered out from all events through a scatter point detection algorithm.

[0126] It should be understood that the above-mentioned interference filtering step can be performed after the event generation step (i.e., step S201) to project events generated by object movement to form a two-dimensional image. However, the present application is not limited thereto, and the above-mentioned interference filtering step can also be performed after the step of forming a two-dimensional image (i.e., step S202), and the two-dimensional image formed based on all events is subjected to the above-mentioned interference filtering processing, and the processed two-dimensional image is the two-dimensional image formed based on events generated by object movement.

[0127] That is, the above-mentioned interference filtering processing can be performed on the generated events, or the above-mentioned interference filtering processing can be performed on the two-dimensional image formed by projection, so as to filter out events caused by light source flicker and / or internal noise of the sensor, and obtain a two-dimensional image formed based on events generated by object movement.

[0128] In a preferred embodiment, the image processing method according to the exemplary embodiments of the present application can further include reconstructing the texture of the object generating the events using pixel-based events to obtain more detailed information of the object, so as to make the edges of the object in the two-dimensional image more detailed.

[0129] Here, the method of texture reconstruction is well known in the art, and the present application will not be repeated here.

[0130] ReturnFigure 3 In step S203, sparse feature calculation is performed on the two-dimensional image formed by the projection to obtain a sparse feature map.

[0131] In a preferred embodiment, sparse feature calculation is performed on the two-dimensional image by a sparse convolutional neural network to obtain the sparse feature map.

[0132] The steps of obtaining the sparse feature map by the sparse convolutional neural network will be described below with reference to Figure 6

[0133] Figure 6 A flowchart showing the steps of obtaining the sparse feature map by the sparse convolutional neural network according to an exemplary embodiment of the present application is shown.

[0134] Referring to Figure 6 In step S2031, the feature activation map corresponding to each convolutional layer of the sparse convolutional neural network is determined based on the structure of the sparse convolutional neural network and the two-dimensional image.

[0135] For example, the two-dimensional image can form the feature activation map of the bottom layer convolution, and the feature activation map corresponding to each convolutional layer of the sparse convolutional neural network can be obtained by scaling according to the receptive field of the sparse convolutional neural network.

[0136] As an example, the feature activation map corresponding to each convolutional layer can indicate the positions of the pixel points that need to participate in the calculation in each convolutional layer. When scaling according to the receptive field of the sparse convolutional neural network, the positions of the events existing in the receptive field can be set to the active state in the feature activation map, and the positions of the events not existing in the receptive field can be set to the inactive state in the feature activation map, the active state indicating that the pixel points at the positions need to participate in the calculation, and the inactive state indicating that the pixel points at the positions do not need to participate in the calculation.

[0137] Since the size of the feature map of the sparse convolutional neural network decreases with the increase of the number of convolutional layers, and the receptive field of the convolution kernel also expands with the increase of the number of convolutional layers, the feature activation map also needs to be adjusted according to the size of the feature map of the sparse convolutional neural network and the change of the receptive field of the convolution kernel to adapt to different convolutional layers, so as to ensure the completeness of the information.

[0138] That is, the size of the feature activation map corresponding to each convolutional layer is determined according to the size of the input feature map of each convolutional layer of the sparse convolutional neural network and the size of the receptive field of the convolution kernel.

[0139] In step S2032, the sparse feature map is obtained by the sparse convolutional neural network using the determined feature activation maps and the two-dimensional image.

[0140] ​For example, a two-dimensional image can be taken as an input feature map of sparse convolutional neural network bottom convolution, and the output feature map of any convolutional layer in the sparse convolutional neural network is the input feature map of the next convolutional layer of the convolutional layer. For any convolutional layer, referring to the feature activation map of the convolutional layer, it is determined which positions of the pixels on the input feature map need to participate in the calculation and which positions of the pixels do not need to participate in the calculation, and the pixels that need to participate in the calculation are subjected to convolution processing to output a sparse feature map.

[0141] In the exemplary embodiments of the present application, the corresponding feature activation map is determined for each convolutional layer, and the sparse information is extended to different convolutional layers to reduce the calculation cost of each convolutional layer.

[0142] The process of obtaining the output feature map of any convolutional layer in the sparse convolutional neural network will be described below in conjunction with Figure 7 and Figure 8 .

[0143] Figure 7 A flowchart showing the steps of generating the output feature map corresponding to any convolutional layer in the sparse convolutional neural network according to the exemplary embodiments of the present application is shown. Figure 8 A schematic diagram showing the generation of the output feature map corresponding to any convolutional layer in the sparse convolutional neural network according to the exemplary embodiments of the present application is shown.

[0144] Referring to Figure 7 , in step S701, the pixels that need to participate in the calculation are extracted from the input feature map of any convolutional layer based on the feature activation map corresponding to the convolutional layer.

[0145] Unlike the prior art method of using the input of the current convolutional layer to determine whether to perform operation, in the sparse convolutional neural network of the exemplary embodiments of the present application, the sparsity of the input feature map of the next convolutional layer of any convolutional layer is used to constrain the sparsity of the output feature map of the convolutional layer, thereby effectively organizing the memory.

[0146] Referring to Figure 8 , in the example shown, 1 is the feature activation map corresponding to any convolutional layer, and the points shown in the feature activation map are the positions of the pixels that need to participate in the calculation, and 2 is the input feature map of the convolutional layer, which shows the area covered by the positions shown in the feature activation map.

[0147] In step S702, the extracted pixels are subjected to sparse feature aggregation to form a dense matrix.

[0148] Referring to Figure 8 , in the example shown, the extracted pixels can be arranged in a column to form a dense matrix 3.

[0149] In the sparse feature aggregation process described above, a position mapping table between the pixel points extracted from the input feature map of any convolution layer and the data in the dense matrix is also generated.

[0150] In step S703, a feature aggregation matrix is obtained according to the formed dense matrix and the convolution kernel matrix corresponding to any convolution layer.

[0151] For example, the formed dense matrix can be multiplied by the convolution kernel matrix corresponding to any convolution layer to obtain the feature aggregation matrix.

[0152] As an example, the existing general matrix multiplication GEMM can be used to operate the dense matrix and the convolution kernel matrix, but the present application is not limited thereto, and other methods can also be used for aggregation.

[0153] In step S704, the feature vectors in the obtained feature aggregation matrix are dispersedly output to obtain the output feature map of any convolution layer. Figure 4 .

[0154] For example, the feature vectors in the feature aggregation matrix can be mapped to the image space based on the position mapping table generated above to obtain the output feature map of any convolution layer. Different columns in the feature aggregation matrix correspond to different output positions on the output feature map, and other positions can be filled with fixed values, for example, assigned to zero, a fixed bias value, or other positions realized by a ground state.

[0155] In the process of obtaining the output feature map described above, the activation positions of events are calculated based on the sparsity of the pixel points, achieving processing acceleration. In the exemplary embodiments of the present application, the sparsity of the sensor input is directly used for acceleration, without the need to analyze the sparsity of the input of each layer of the sparse convolutional neural network.

[0156] It should be understood that after the sparse feature map is obtained by the above-mentioned manner, the cause of the scene change can be analyzed based on the sparse feature map.

[0157] In a preferred embodiment, the sparse feature map obtained at the current time can be fused with the historical sparse feature map to analyze the cause of the scene change based on the fused sparse feature map. Here, the historical sparse feature map can refer to at least one sparse feature map obtained before the current time, that is, the fusion result of the sparse feature maps obtained at multiple times is used for scene change analysis.

[0158] Here, various fusion methods can be used to fuse the sparse feature map and the historical sparse feature map. As an example, the long short-term memory recurrent neural network can be used to fuse the sparse feature map and the historical sparse feature map. However, the present application is not limited thereto, and other methods can also be used for fusion. In addition, the method of using the long short-term memory recurrent neural network for feature map fusion is common knowledge in the art, and the present application will not repeat this part.

[0159] The steps of scene change analysis based on the sparse feature map will be introduced below with reference to Figure 9

[0160] A flowchart of the steps of scene change analysis based on the sparse feature map according to an exemplary embodiment of the present application is shown. Figure 9 Referring to

[0161] In step S30, the obtained sparse feature map is subjected to region feature extraction. Figure 9 For the above-mentioned fusion of the sparse feature map and the historical sparse feature map, the fused sparse feature map can be subjected to region feature extraction in step S30.

[0162] As an example, the region feature can be extracted using the Region of Interest (ROI), but the present application is not limited thereto, and other feature extraction methods can also be used to extract the region feature.

[0163] In step S40, based on the extracted region feature, the object causing the change in the monitored scene is identified.

[0164] Here, various methods can be used to identify whether the object causing the change in the monitored scene is a human or other objects other than a human based on the region feature.

[0165] In step S50, it is determined whether the identified object is a human.

[0166] If the identified object is a human, step S60 is performed to determine the position, motion trajectory, behavior and / or identity of the object.

[0167] Here, various feature map classification or regression algorithms can be used to determine the position, motion trajectory, behavior and / or identity of the object, and the present application is not limited thereto.

[0168] In an exemplary embodiment of the present application, human recognition from event streams can be achieved based on sparse convolution of events.

[0169] If the identified object is not a human, other processing can be performed or step S60 can not be performed.

[0170]

[0171] ​Through the image processing method, the manual cost, data volume and computation volume of the monitoring system can be effectively reduced, and the large-scale deployment of the video monitoring system is facilitated.

[0172] In a preferred embodiment, the recognition result can also be used as a trigger for other sensors for the purpose of reducing computation volume and saving cost.

[0173] For example, when the object is recognized as a person, a normal camera for capturing frame images can be triggered to capture the object. That is, when it is determined that the cause of the change of the monitored scene is the movement of the person, a CIS camera capable of collecting more detailed image information can be triggered to capture.

[0174] For the case where multiple cameras are arranged at different positions in the monitored scene, since the relative positions of each camera are known, the distance of the moving object from each camera, the position of the moving object in the image and the three-dimensional position of the moving object in the scene can also be calculated.

[0175] Here, the distance and position can be obtained by using various distance / position calculation methods, and the present application will not be described in detail.

[0176] Figures 10A to 10F A schematic diagram showing the movement of an object in a monitored scene according to an exemplary embodiment of the present application is shown.

[0177] In this example, a person in the scene walks away from the camera and then walks back to the camera, and the two-dimensional images are arranged in time sequence as shown in 10A-10F.

[0178] Figure 11 A schematic diagram showing the comparison of data volume and computation efficiency between the existing image processing method and the image processing method according to an exemplary embodiment of the present application is shown.

[0179] As shown in Figure 11 The solid line shows the sparsity degree of the dynamic vision sensor DVS, and the dashed line shows the acceleration ratio between sparse and dense convolution of the first convolution layer of the sparse convolutional neural network. Figures 10A to 10F For the object movement example shown in

[0180] Figure 12 A block diagram of a monitoring system according to an exemplary embodiment of the present application is shown.

[0181] As shown in Figure 12As shown, the monitoring system according to the exemplary embodiment of the present application comprises a front-end camera 100 and a processor 200. Here, various transmission media can be used for data interaction between the front-end camera 100 and the processor 200, which is not limited by the present application.

[0182] Specifically, the front-end camera 100 captures changes of the monitored scene. The processor 200 is configured to perform sparse feature calculation on the captured changes of the monitored scene to obtain a sparse feature map.

[0183] Preferably, the processor 200 can project pixel-based events generated by capturing changes of the monitored scene along a time axis, and perform sparse feature calculation based on a two-dimensional image formed by the projection to obtain a sparse feature map.

[0184] As an example, the event can include a position where a pixel is located, a time when the event occurs, and a polarity of the pixel. Here, the polarity can indicate a change in luminance value of the pixel, for example, the polarity of the pixel can indicate whether the change occurring at the pixel is an increase in luminance value or a decrease in luminance value. The pixel-based event can be generated in the following way.

[0185] The front-end camera 100 can include an event camera for directly outputting events and / or a normal camera for capturing frame images.

[0186] In one case, the event camera generates pixel-based events when capturing changes of the monitored scene.

[0187] Here, the event camera can refer to a camera capable of directly outputting events, as an example, the event camera can include but is not limited to a dynamic vision sensor and / or a time-asynchronous image sensor.

[0188] In another case, the normal camera can capture frame images of the monitored scene, and the processor 200 generates pixel-based events by performing frame difference processing on the frame images captured by the normal camera.

[0189] The process of the processor 200 forming a two-dimensional image based on event projection is described below.

[0190] Here, when the front-end camera captures changes of the monitored scene, an event will be generated, and over time, an event stream will be formed, and the processor 200 projects the event stream to a two-dimensional image plane in a time-decay manner to generate a two-dimensional image.

[0191] The earlier the event occurs, the smaller the mapping value of the event projection on the two-dimensional image, and the later the event occurs, the larger the mapping value of the event projection on the two-dimensional image.

[0192] For example, the processor 200 can obtain the two-dimensional image by time-decay integration of the generated events on a time axis.

[0193] In a preferred embodiment, the two-dimensional image can be a two-dimensional image formed based on events generated by object motion.

[0194] For example, the processor 200 can obtain events generated by object motion by filtering out events generated by light source flicker and / or by internal noise of a camera used to capture changes in the monitored scene from all generated events.

[0195] As an example, the processor 200 can filter out events generated by light source flicker from all events by a light source flicker detection algorithm, and / or can filter out events generated by internal noise of a camera used to capture changes in the monitored scene from all events by a scatter detection algorithm.

[0196] In a preferred embodiment, the processor 200 further reconstructs the texture of the object generating the events using pixel-based events to obtain more detailed information of the object, which can make the edges of the object in the two-dimensional image more detailed.

[0197] Preferably, the processor 200 can obtain the sparse feature map by sparse feature calculation of the two-dimensional image through the sparse convolutional neural network.

[0198] Specifically, the processor 200 can determine a feature activation map corresponding to each convolutional layer of the sparse convolutional neural network based on the structure of the sparse convolutional neural network and the two-dimensional image, and obtain the sparse feature map through the sparse convolutional neural network by using the determined feature activation maps and the two-dimensional image.

[0199] For example, the two-dimensional image can form a feature activation map of a bottom convolution, and the processor 200 can obtain the feature activation maps corresponding to each convolutional layer of the sparse convolutional neural network according to the receptive field of the sparse convolutional neural network.

[0200] As an example, the feature activation map corresponding to each convolutional layer indicates the positions of the pixel points that need to participate in the calculation in each convolutional layer. The size of the feature activation map corresponding to each convolutional layer is determined according to the size of the input feature map of each convolutional layer of the sparse convolutional neural network and the size of the receptive field of the convolution kernel.

[0201] Preferably, in the sparse convolutional neural network of the present application, the sparsity of the input feature map of the next convolutional layer of each convolutional layer is used to constrain the sparsity of the output feature map of each convolutional layer.

[0202] The processor 200 can obtain the output feature map of each convolutional layer in the sparse convolutional neural network in the following manner.

[0203] extracts the pixel points participating in the calculation from the input feature map of the any convolution layer based on the feature activation map corresponding to the any convolution layer; performs sparse feature aggregation on the extracted pixel points to form a dense matrix; obtains a feature aggregation matrix according to the formed dense matrix and a convolution kernel matrix corresponding to the any convolution layer; and dispersively outputs the feature vectors in the obtained feature aggregation matrix to obtain the output feature map of the any convolution layer.

[0204] For example, the processor 200 further generates a position mapping table between the pixel points extracted from the input feature map of the any convolution layer and the data in the dense matrix. In this case, the processor 200 maps the feature vectors in the feature aggregation matrix based on the position mapping table to obtain the output feature map of the any convolution layer.

[0205] In the exemplary embodiments of the present application, after obtaining the sparse feature map, the processor 200 can analyze the reason for the change of the monitored scene based on the obtained sparse feature map.

[0206] In a preferred embodiment, the processor 200 can further fuse the sparse feature map obtained at the current time with a historical sparse feature map, and analyze the reason for the change of the scene based on the fused sparse feature map.

[0207] As an example, the processor 200 can use a long short-term memory recurrent neural network to fuse the sparse feature map with the historical sparse feature map.

[0208] Specifically, the processor 200 can be further configured to: perform region feature extraction on the obtained sparse feature map, and identify the object causing the change of the monitored scene based on the extracted region features, and if the identified object is a person, determine the position, motion trajectory, behavior and / or identity of the object. If the identified object is not a person, the processor 200 can not process.

[0209] For the above-mentioned case of fusing the sparse feature map with the historical sparse feature map, the processor 200 can perform region feature extraction on the fused sparse feature map.

[0210] For the case that the front-end camera includes an event camera for directly outputting events and a normal camera for capturing frame images, the event camera can be used to capture the change of the monitored scene, and when the processor 200 identifies that the object is a person, the processor 200 can trigger the normal camera to take a picture of the object to record image or video data of the object for further analysis of the object.

[0211] Figure 13 A block diagram of an image processing device according to an exemplary embodiment of the present application is shown.

[0212] As Figure 13As shown, the image processing device according to the exemplary embodiment of the present application comprises a dynamic event information acquisition module 10 and a sparse feature map generation module 20. It should be understood that these modules can be implemented by a general hardware processor such as a digital signal processor, a field programmable gate array, or the like, or by a special hardware processor such as a special chip, or can be implemented in software completely through a computer program, for example, can be implemented as Figure 12 each module in the processor shown in FIG. 1.

[0213] Specifically, the dynamic event information acquisition module 10 captures changes of a monitored scene.

[0214] The sparse feature map generation module 20 performs sparse feature calculation on the captured changes of the monitored scene to obtain a sparse feature map.

[0215] Optionally, the sparse feature map generation module 20 according to the exemplary embodiment of the present application can comprise a dynamic event map generation submodule 21 and a sparse feature map operation submodule 22.

[0216] For example, the dynamic event map generation submodule 21 can project events based on pixel points generated by capturing changes of the monitored scene along a time axis to form a two-dimensional image.

[0217] As an example, the events can include a position where a pixel point is located, a time when the event occurs, and a polarity of the pixel point. Here, the polarity can indicate a change in luminance value of the pixel point, for example, the polarity of the pixel point can indicate whether the change occurring at the pixel point is an increase in luminance value or a decrease in luminance value.

[0218] The events based on pixel points can be generated in the following manner.

[0219] In one case, the dynamic event information acquisition module 10 can obtain events based on pixel points when capturing changes of the monitored scene.

[0220] For example, the dynamic event information acquisition module 10 obtains events based on pixel points from an event camera by capturing changes of the monitored scene using the event camera.

[0221] In another case, the sparse feature map generation module 20 according to the exemplary embodiment of the present application can further comprise a frame difference processing submodule (not shown in the figure).

[0222] In this case, the dynamic event information acquisition module 10 obtains frame images from a normal camera by capturing frame images of the monitored scene using the normal camera, and the frame difference processing submodule generates events based on pixel points by performing frame difference processing on the captured frame images of the monitored scene.

[0223] The process of forming a two-dimensional image by the dynamic event map generation submodule 21 is described below.

[0224] Here, when a change in the monitored scene is captured, an event is generated, and over time, an event stream is formed, and the dynamic event map generation submodule 21 projects the event stream onto the two-dimensional image plane in a time-decay manner to generate a two-dimensional image.

[0225] The earlier the event occurs, the smaller the mapping value of the event projection on the two-dimensional image, and the later the event occurs, the larger the mapping value of the event projection on the two-dimensional image.

[0226] In a preferred embodiment, the sparse feature map generation module 20 can further include a dynamic event preprocessing submodule (not shown in the figure), which obtains events generated by object motion by filtering out events generated by light source flickering and / or internal noise of the camera used to capture changes in the monitored scene from all generated events.

[0227] In this case, the dynamic event map generation submodule 21 can form a two-dimensional image based on events generated by object motion.

[0228] In a preferred embodiment, the sparse feature map generation module 20 can further include a texture reconstruction submodule that uses pixel-based events to reconstruct the texture of the object that generated the event.

[0229] The sparse feature map operator module 22 can perform sparse feature calculation based on the two-dimensional image formed by projection to obtain a sparse feature map.

[0230] Preferably, the sparse feature map operator module 22 can perform sparse feature calculation on the two-dimensional image by a sparse convolutional neural network to obtain a sparse feature map. The process of obtaining a sparse feature map by the sparse feature map operator module 22 based on a sparse convolutional neural network is described below.

[0231] For example, the sparse feature map operator module 22 according to an exemplary embodiment of the present application can include a feature activation map generation unit 221 and a sparse feature map operation unit 222.

[0232] Specifically, the feature activation map generation unit 221 determines a feature activation map corresponding to each convolutional layer of the sparse convolutional neural network based on the structure of the sparse convolutional neural network and the two-dimensional image.

[0233] For example, the two-dimensional image can form a feature activation map of the bottom layer convolution, and the feature activation map generation unit 221 performs scale adjustment according to the receptive field of the sparse convolutional neural network to obtain a feature activation map corresponding to each convolutional layer of the sparse convolutional neural network.

[0234] As an example, the feature activation map corresponding to each convolutional layer indicates the positions of the pixel points that need to participate in the calculation in each convolutional layer. The size of the feature activation map corresponding to each convolutional layer is determined according to the size of the input feature map of each convolutional layer of the sparse convolutional neural network and the size of the receptive field of the convolution kernel.

[0235] The sparse feature map operation unit 222 obtains the sparse feature map by using the determined feature activation map and the two-dimensional image through the sparse convolutional neural network.

[0236] Preferably, in the sparse convolutional neural network of the present application, the sparsity of the input feature map of the next convolutional layer of each convolutional layer is used to constrain the sparsity of the output feature map of each convolutional layer.

[0237] For example, the sparse feature map operation unit 222 can obtain the output feature map of each convolutional layer in the sparse convolutional neural network in the following way.

[0238] Based on the feature activation map corresponding to each convolutional layer, the pixel points that need to participate in the calculation are extracted from the input feature map of each convolutional layer; the sparse feature aggregation is performed on the extracted pixel points to form a dense matrix; the feature aggregation matrix is obtained according to the formed dense matrix and the convolution kernel matrix corresponding to each convolutional layer; and the feature vectors in the obtained feature aggregation matrix are dispersedly output to obtain the output feature map of each convolutional layer.

[0239] Preferably, the sparse feature map operation unit 222 further generates a position mapping table between the pixel points extracted from the input feature map of each convolutional layer and the data in the dense matrix. In this case, the sparse feature map operation unit 222 can map the feature vectors in the feature aggregation matrix based on the position mapping table to obtain the output feature map of each convolutional layer.

[0240] In the exemplary embodiments of the present application, after obtaining the sparse feature map, the reason for the change of the monitored scene can also be analyzed based on the obtained sparse feature map.

[0241] In this case, the image processing device according to the exemplary embodiments of the present application can further include a feature map analysis module 30 that performs region feature extraction on the obtained sparse feature map, identifies the object that causes the change of the monitored scene based on the extracted region features, and determines the position, motion trajectory, behavior and / or identity of the object if the identified object is a person.

[0242] In a preferred embodiment, the image processing device according to the exemplary embodiments of the present application can further include a fusion module (not shown in the figure) that fuses the sparse feature map obtained at the current time with the historical sparse feature map, and in this case, the feature map analysis module 30 performs region feature extraction on the fused sparse feature map.

[0243] As an example, the fusion module can use a long short-term memory recurrent neural network to fuse the sparse feature map and the historical sparse feature map.

[0244] For the case where an event camera for directly outputting events and a normal camera for capturing frame images are set in the monitored scene, when the object is identified as a person, the feature map analysis module 30 generates and outputs a shooting trigger signal to trigger the normal camera for capturing frame images to shoot the object.

[0245] According to the exemplary embodiments of the present application, a computer-readable storage medium storing a computer program is also provided. The computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to perform the image processing method described above. The computer-readable recording medium is any data storage device that can store data readable by a computer system. Examples of the computer-readable recording medium include a read-only memory, a random access memory, a read-only optical disc, a magnetic tape, a floppy disc, an optical data storage device, and a carrier wave such as data transmission through an internet via a wired or wireless transmission path.

[0246] By employing the image processing method, the image processing device, and the monitoring system according to the exemplary embodiments of the present application, the sparse nature of the sensor is exploited for acceleration, the monitored scene is optimized, and the amount of data and the amount of computation for processing are reduced.

[0247] In addition, by employing the image processing method, the image processing device, and the monitoring system according to the exemplary embodiments of the present application, the cause of the change in the monitored scene can be quickly, accurately, and automatically analyzed based on the change in the monitored scene.

[0248] In addition, by employing the image processing method, the image processing device, and the monitoring system according to the exemplary embodiments of the present application, the sparse convolutional neural network is introduced into the scene understanding based on the change in the scene, the sparse convolution method is optimized according to the characteristics of the sensor, and only the more effective activation positions are calculated through the feature activation map, thereby effectively reducing the amount of computation and the amount of calculation for image processing.

[0249] Although the present application has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. An image processing method, characterized by, comprising: capturing changes in a monitored scene using a dynamic vision sensor; and performing sparse feature computation on the changes in the monitored scene by a sparse convolutional neural network to obtain a sparse feature map, wherein the step of performing sparse feature computation comprises: generating pixel-based events from the changes in the monitored scene; projecting the generated pixel-based events along a time axis and integrating the generated pixel-based events to form a two-dimensional image; and performing sparse feature computation based on the two-dimensional image.

2. The image processing method of claim 1, wherein, The step of generating pixel-based events comprises: capturing the changes in the monitored scene by a first camera to generate the pixel-based events; or, generating the pixel-based events by performing frame difference processing on frame images captured by a second camera.

3. The image processing method of claim 1, wherein, reconstructing texture of an object generating the pixel-based events using the pixel-based events.

4. The image processing method of claim 1, wherein, The two-dimensional image is formed based on events generated by object motion, wherein the events generated by object motion are obtained by filtering out events generated by light source flicker and by internal noise of a camera used to capture the changes in the monitored scene.

5. The image processing method of claim 1, wherein, The step of performing sparse feature computation comprises: performing sparse feature computation based on the two-dimensional image by the sparse convolutional neural network to obtain the sparse feature map.

6. The image processing method of claim 5, wherein, The step of performing sparse feature computation based on the two-dimensional image by the sparse convolutional neural network to obtain the sparse feature map comprises: determining feature activation maps corresponding to each convolutional layer of the sparse convolutional neural network based on a structure of the sparse convolutional neural network and the two-dimensional image; and obtaining the sparse feature map by the sparse convolutional neural network using the feature activation maps and the two-dimensional image.

7. The image processing method of claim 1, wherein, The image processing method further comprises: performing region feature extraction on the sparse feature map; determining an object causing the changes in the monitored scene based on the extracted region features; and if the determined object is a human, determining a position, a motion trajectory, a behavior and / or an identity of the object.

8. A monitoring system, characterized by comprising: a front-end camera configured to capture changes in a monitored scene using a dynamic vision sensor in real time; and a processor configured to perform sparse feature computation on the changes in the monitored scene by a sparse convolutional neural network to obtain a sparse feature map, wherein the processor is configured to: project pixel-based events generated from the changes in the monitored scene along a time axis, integrate the generated pixel-based events to form a two-dimensional image, and perform sparse feature computation based on the two-dimensional image.

9. The monitoring system of claim 8, wherein, The front-end camera comprises a first camera configured to output events directly or a second camera configured to capture frame images, wherein the processor is configured to: generate the pixel-based events from the changes in the monitored scene when the first camera captures the changes in the monitored scene; or, generate the pixel-based events by performing frame difference processing on frame images captured by the second camera.

10. The monitoring system of claim 8, wherein, The processor is configured to: reconstruct texture of an object generating the pixel-based events using the pixel-based events.

11. The monitoring system of claim 8, wherein, The two-dimensional image is formed based on events generated by object motion, wherein the events generated by object motion are obtained by filtering out events generated by light source flicker and by internal noise of a camera used to capture the changes in the monitored scene. The processor is configured to obtain events generated by object motion by filtering out events generated by light source flicker and by internal noise of a camera used to capture changes in the monitored scene.

12. The monitoring system of claim 8, wherein, The processor is configured to: perform sparse feature computation based on the two-dimensional image to obtain a sparse feature map via a sparse convolutional neural network.

13. The monitoring system of claim 12, wherein, The processor is configured to: determine feature activation maps corresponding to each convolutional layer of the sparse convolutional neural network based on the structure of the sparse convolutional neural network and the two-dimensional image, and obtain the sparse feature map via the sparse convolutional neural network using the feature activation maps and the two-dimensional image.

14. The monitoring system of claim 8, wherein, The processor is configured to: perform region feature extraction on the sparse feature map; determine an object causing changes in the monitored scene based on the extracted region features; and if the determined object is a human, determine a position, a motion trajectory, a behavior and / or an identity of the object.

15. An image processing apparatus characterized by comprising: Comprise: a dynamic event information acquisition module that captures changes in a monitored scene; and a sparse feature map generation module that performs sparse feature computation on the changes in the monitored scene using a sparse convolutional neural network to obtain a sparse feature map, wherein the sparse feature map generation module comprises: a dynamic event map generation submodule that projects events based on pixel points generated by capturing changes in the monitored scene along a time axis and integrates the events based on pixel points to form a two-dimensional image; and a sparse feature map operation submodule that performs sparse feature computation based on the two-dimensional image.

16. The image processing device of claim 15, wherein: the dynamic event information acquisition module generates events based on pixel points when capturing changes in the monitored scene, or the sparse feature map generation module further comprises a frame difference processing submodule that generates events based on pixel points by performing frame difference processing on captured frame images of the monitored scene.

17. The image processing apparatus of claim 15, wherein The sparse feature map generation module further comprises a texture reconstruction submodule that reconstructs a texture of an object generating the events based on pixel points using the events based on pixel points.

18. The image processing apparatus of claim 15, wherein The sparse feature map generation module further comprises a dynamic event preprocessing submodule that obtains events generated by object motion by filtering out events generated by light source flicker and by internal noise of a camera used to capture changes in the monitored scene, wherein the dynamic event map generation submodule forms the two-dimensional image based on the events generated by object motion.

19. The image processing apparatus of claim 15, wherein The sparse feature map operation submodule performs sparse feature computation based on the two-dimensional image to obtain a sparse feature map via a sparse convolutional neural network.

20. The image processing apparatus of claim 19, wherein The sparse feature map operation submodule comprises: a feature activation map generation unit that determines feature activation maps corresponding to each convolutional layer of the sparse convolutional neural network based on the structure of the sparse convolutional neural network and the two-dimensional image; and a sparse feature map operation unit that obtains the sparse feature map via the sparse convolutional neural network using the feature activation maps and the two-dimensional image.

21. The image processing apparatus of claim 15, wherein The image processing device further comprises a feature map analysis module, which performs region feature extraction on the obtained sparse feature map, determines an object causing a change in the monitored scene based on the extracted region features, and if the determined object is a human, determines a position, a motion trajectory, a behavior and / or an identity of the object.

22. A computer readable storage medium storing a computer program, wherein the computer program comprises instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 21. The computer program, when executed by a processor, implements the image processing method of any one of claims 1-7.

Images