System and method for occupancy detection using a frame-based sensor

Abstract

The present invention relates to a method for occupancy detection in a visual scene imaged by an image sensor using a spiking neural network. The method comprises obtaining event-based data, wherein the event-based data is a representation of input from the image sensor in the form of one or more discrete events, each event representing a change in the visual scene that passes a predefined threshold. Next, the event-based data is encoded as input spikes, wherein the input spikes are configured to serve as input for the spiking neural network. The spiking neural network produces one or more output spikes by processing the input spikes. The output spikes are decoded into one or more classification results. Using these classification results, the occupancy in the visual scene can be detected.

Description

System and method for occupancy detection using a frame-based sensorTECHNICAL FIELD

[0001] The present invention relates to the field of occupancy detection, and in particular a method and device for generating and / and using event-based data from an image sensor, to be processed via a spiking neural network.BACKGROUND

[0002] Occupancy detection is defined as the task of identifying whether a person is present in a particular room, space or environment, or if the state of occupancy of a room, space or environment changes. Furthermore, additional information may be required from a system capable of performing occupancy detection, such as person or object counting.

[0003] The applications for occupancy detection include building management, wherein an automatic action could be taken when no humans are present in a room or area of the building (for example: turn off ambient lights or air conditioning, activate security cameras or alarm), or graded actions if some number of humans are present (for example: turn the fan of the air conditioner to medium if three persons are present, but turn it to maximum if five or more persons are present).

[0004] Typically, frame-based sensors are used to perform occupancy detection. These are sensors that capture data in discrete frames or snapshots, similar to how a camera captures images. Frame-based sensors can utilize various technologies such as infrared (IR), ultrasonic, microwave, or even visual cameras, and capture data in frames or snapshots at regular intervals. For example, an IR sensor may take snapshots of the heat signature in a room, while a visual camera may capture images of the area being monitored.

[0005] The captured frames are then analyzed to detect changes in occupancy within the monitored area. This analysis can involve detecting movement, spatial and temporal features associated with human presence, heat signatures, or other relevant parameters depending on the sensor technology used. The data from frame-based sensors, for example a frame of RGB- values, is typically processed using algorithms designed to identify patterns associated with occupancy. Machine learning algorithms may also be employed to improve the accuracy of occupancy detection over time.

[0006] One can use frame-based sensors to perform occupancy detection by employing minimal preprocessing, data buffering and coupling the frame-based sensors to a neural network for processing.

[0007] Frame-based sensors can be configured to continuously capture data, such as images or sensor readings, at regular intervals to monitor occupancy within a space. This constant data collection ensures that any changes in occupancy are promptly detected. The data captured by the sensors is then fed into a neural network for analysis. The neural network processes the data to identify patterns associated with occupancy, such as movement, heat signatures, or other relevant features. Since occupancy detection often requires real-time responsiveness, the neural network must process the data quickly to provide timely feedback on occupancy status. This real-time analysis is essential for applications like smart building automation, security systems, and energy management.

[0008] To maintain continuous monitoring and real-time analysis, both the frame-based sensors and the neural network must remain "always on," meaning they are operational and actively processing data at all times. This constant operation consumes processing power and energy, even during periods when occupancy levels are stable.

[0009] The continuous operation of frame-based sensors and neural networks can lead to significant power consumption, especially in battery-powered devices or systems with strict energy efficiency requirements.SUMMARY OF INVENTION

[0010] In order to solve the aforementioned problems, the subject-matter of the present claims is proposed.

[0011] The presented solution introduces a methodology for performing data gathering and dataset assembly to enable training of a performant network. This method enables the realization of a performant presence sensing application capable of operating on a limited power budget.

[0012] According to a first aspect of the disclosure, a method for occupancy detection in a visual scene imaged by an image sensor is disclosed. The method uses a spiking neural network. The method may comprise: obtaining event-based data, wherein the event-based data may be a representation of input from the image sensor in the form of one or more discrete events, each event may represent a change in the visual scene that passes a predefined threshold; encoding the event-based data as input spikes, wherein the input spikes may beconfigured to serve as input for the spiking neural network; processing the input spikes using the spiking neural network, wherein the spiking neural network may produce one or more output spikes by processing the input spikes; decoding the output spikes into one or more classification results; detecting the occupancy in the visual scene using the classification results.

[0013] According to an embodiment of the first aspect, the image sensor may be an eventbased image sensor, which captures the event-based data when there is a change in the visual scene that passes the predefined threshold. Preferably, the event-based image sensor captures individual pixel-level and / or regional pixel-level changes. More preferably, when a change is detected at a particular region in the visual scene beyond the predefined threshold, the eventbased image sensor may transmit event-based data representing the location of the change, optionally along with information about the rate of the change and timing of the change.

[0014] According to an embodiment of the first aspect, the image sensor may be a framebased image sensor which captures images in a frame-by-frame manner, where each frame represents a complete snapshot of the visual scene at a specific moment in time. The step of obtaining event-based data may further comprise: preprocessing the captured images from the frame-based image sensor by computing the difference between a current frame and a background frame, wherein the background frame can be either obtained from one or multiple frames previously captured by the image sensor; checking whether the difference is a change in the visual scene that passes the predefined or an immediately computed threshold; generating event-based data based on the difference if the change passed the predefined threshold.

[0015] According to an embodiment of the first aspect, the event-based data may comprise the location, time, size, shape, and / or intensity of the change in the visual scene.

[0016] According to an embodiment of the first aspect, the predefined threshold may be either a positive threshold or a negative threshold. The positive threshold may be used to check whether an increase in intensity in a particular region of the visual scene is an event. The negative threshold may be used to check whether a decrease in intensity in a particular region of the visual scene is an event.

[0017] According to an embodiment of the first aspect, the step of encoding the event-based data into spike trains may comprise mapping features of the event-based data to the spiking activity of neurons in the spiking neural network, by: mapping the location of a change in the visual scene to a corresponding neuron; mapping the size of a change in the visual scene to the fire rate of a corresponding neuron; mapping the shape of a change in the visual scene to aspecific subset of neurons that corresponds to that shape; mapping the intensity of a change in the visual scene to the timing, frequency or amplitude of spikes; and / or mapping the time of a change in the visual scene to spike timing.

[0018] According to an embodiment of the first aspect, the spiking neural network may be composed of multiple layers of spiking neurons connected in a feedforward fashion, allowing the input spikes to flow from the input layer to the output layer. Recurrent and / or feedback connections may be present, respectively connecting layers to themselves or to upstream layers.

[0019] According to an embodiment of the first aspect, the step of decoding the one or more output spikes may comprise: counting the spiking activity of the output neurons; and / or computing a maximum or softmax function in order to make a selection of which class the network predicts the input belongs to; and / or analyzing the membrane potential of the output neurons, preferably by computing the first derivative of the membrane potential representing a rate of change of spike counts that arrive at the output neurons over time or area under the curve of the membrane potential, to extract information about the timing and dynamics of the output neurons.

[0020] According to an embodiment of the first aspect, the method may further comprise: noting a timestamp of the event, wherein the timestamp represents the timing of the event relative to the input stimulus or the features they represent; correlating the timestamps of the events with the features represented by the output neurons; and determining when and how the network responds to specific input spikes on the basis of the correlation.

[0021] According to an embodiment of the first aspect, the method may further comprise: determining an inter-feature interval and / or a time-averaged count of a feature in a time interval based on the timestamps; measuring the repetitiveness of a feature using the inter-feature interval and / or the intensity of the feature in the time interval using the time-averaged count.

[0022] According to an embodiment of the first aspect, the method may further comprise: taking an action based on the detected occupancy, wherein the action is either intrinsic or extrinsic. The intrinsic actions may be those actions that affect the method by changing control parameters of the image sensor or the spiking neural network, while the extrinsic actions may be those actions that affect an environment of the visual scene.

[0023] According to a second aspect of the disclosure, an occupancy detection device is disclosed for occupancy detection in a visual scene imaged by an image sensor. The occupancy detection device may comprise a processor, an encoder, a spiking neural network and a decoder. The processor may be configured to obtain event-based data. The event-based datamay be a representation of input from the image sensor in the form of one or more discrete events, each event representing a change in the visual scene that passes a predefined threshold. The encoder may be configured to encode the event-based data as input spikes, wherein the input spikes are configured to serve as input for the spiking neural network. The spiking neural network may be configured to produce one or more output spikes by processing the input spikes. The decoder may be configured to decode the output spikes into one or more classification results. The processor may be configured to detect the occupancy in the visual scene using the classification results.

[0024] According to an embodiment of the second aspect, the occupancy detection device may further comprise the image sensor. The image sensor may be an event-based image sensor or a frame-based image sensor.

[0025] The event-based image sensor may capture the event-based data when there is a change in the visual scene that passes the predefined threshold. Preferably, the event-based image sensor captures individual pixel-level and / or regional pixel-level changes. More preferably, when a change is detected at a particular region in the visual scene beyond a predefined threshold, the event-based image sensor may transmit event-based data representing the change, along with information about the location and timing of the event.

[0026] The frame-based image sensor may capture images in a frame-by-frame manner, where each frame represents a complete snapshot of the visual scene at a specific moment in time. The processor may be further configured: to preprocess the captured images from the frame-based image sensor by computing the difference between subsequent frames; to check whether the difference is a change in the visual scene that passes the predefined threshold; and to generate event-based data based on the difference if the change passed the predefined threshold.

[0027] According to an embodiment of the second aspect, the event-based data comprises the location, time, size, shape, and / or intensity of the change in the visual scene.

[0028] According to an embodiment of the second aspect, the encoder is configured to map features of the event-based data to the spiking activity of neurons in the spiking neural network, by: mapping the location of a change in the visual scene to a corresponding neuron; mapping the size of a change in the visual scene to the fire rate of a corresponding neuron; mapping the shape of a change in the visual scene to a specific subset of neurons that corresponds to that shape; mapping the intensity of a change in the visual scene to the timing, frequency or amplitude of spikes; and / or mapping the time of a change in the visual scene to spike timing.

[0029] According to an embodiment of the second aspect, the processor may be configured to take an action based on the detected occupancy. The action may be either intrinsic or extrinsic. The intrinsic actions may be those actions that affect the method by changing control parameters of the image sensor or the spiking neural network, while the extrinsic actions may be those actions that affect an environment of the visual scene.BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Embodiments will now be described, by way of example only, with reference to the accompanying drawings in which corresponding reference symbols indicate corresponding parts, and in which:

[0031] FIG. 1 shows a schematic representation of an embodiment of an occupancy detection system according to the invention; and

[0032] FIG. 2 shows a schematic representation of an embodiment of an occupancy detection method according to the invention.DESCRIPTION OF EMBODIMENTS

[0033] Hereinafter, certain embodiments will be described in further detail. It should be appreciated, however, that these embodiments should not be construed as limiting the scope of protection for the present disclosure.

[0034] FIG. 1 shows a schematic representation of an embodiment of an occupancy detection system according to the invention.

[0035] The occupancy detection system 100 comprises an imaging sensor 101, which may be for example a low-resolution sensor or an event-based sensor, and a processing device 102.

[0036] The image sensor images a visual scene. A visual scene refers to the entirety of what is captured within the field of view of an image sensor at a given moment. It encompasses all the objects, backgrounds, colors, textures, and lighting conditions present in the image. In other words, a visual scene is the specific area monitored by the sensor where the presence or absence of objects or individuals is to be determined, preferably the visual scene is imaged by the imaging sensor to provide the input data from which the spiking neural network will process and analyze to determine whether and where there are occupants. In this context, the field of view is the spatial extent captured by the image sensor, including the edges and corners of the scene. Objects within the field of view may be any tangible items within the scene, such as furniture, vehicles, or individuals. The background is the part of the scene that serves as thebackdrop for the objects, such as walls, floors, or the ground. Textures and colors comprise the surface properties and color variations of objects and backgrounds within the scene. The lighting conditions refer to the illumination within the scene, including natural and artificial light sources and their effects on visibility and shadows.

[0037] Low-resolution sensors capture images at a lower pixel count compared to high- resolution sensors. For example, a low-resolution sensor might have resolutions such as VGA (640x480 pixels) or even lower. These sensors are often used in applications where high image detail is not critical, or where constraints such as cost, power consumption, or processing requirements are limiting factors.

[0038] Event-based sensors, also known as event-driven sensors or neuromorphic sensors, operate differently from traditional frame-based sensors. Instead of capturing entire frames at fixed intervals, they only capture and transmit information when there is a significant change in the scene. Event-based sensors detect and report individual pixel-level changes e.g., in luminance, asynchronously, meaning they only respond to changes in the scene. When a pixel detects a change beyond a predefined threshold, it sends an event signal indicating the change, along with information about the location and timing of the event. Event-based sensors are particularly well-suited for applications requiring low-latency, high-speed, and high-dynamic- range sensing.

[0039] The processing device 102 comprises one or more peripheral interfaces 103 in order to transfer data from the imaging sensor 101 to the processing device 102. For example, a Serial Peripheral Interface (SPI), an Inter-Integrated Circuit (I2C), a MIPI (Mobile Industry Processor Interface), a parallel interface, a USB-port, a Bluetooth interface, HDMI, an ethemet port and / or Low Voltage Differential Signaling (LVDS). The transfer can be done over a wired or wireless connection, either directly or indirectly via e.g. a server.

[0040] The processing device 102 furthermore comprises a microprocessor 104, a memory 105, an interconnect structure 109 and a spiking neural network (SNN) accelerator.

[0041] The microprocessor 104 orchestrates the operation of the system, performs data gathering and generic computations and data manipulations. The microprocessor 104 can preprocess the image data to enhance relevant features and / or to reduce noise. The imaging sensor 101 may be controlled by the microprocessor 104, e.g. by configuring image capture rate, exposure duration, low-power or high-power modes through writes to configuration registers or memory locations contained within the sensor, et cetera. The microprocessor 104places the data from the imaging sensor 101 in the memory 105. The data from the image sensor 101 can be either RGB, Luminosity / Grayscale, events, or another suitable format.

[0042] RGB data represents images in terms of three primary colors: Red, Green, and Blue. Each pixel in the image is typically represented by three values corresponding to the intensity of these three color channels. RGB data is typically represented as a matrix of pixel values. Luminosity or grayscale data represents images in terms of overall brightness or intensity, without color information, where the data is represented as a single channel. Event-based data comprises data of individual events triggered by significant changes in the scene, such as motion or changes in brightness. Each event typically includes information such as the pixel location, timestamp, and polarity (indicating whether the change was an increase or decrease in intensity). Event-based sensors offer high temporal resolution, enabling real-time responses to dynamic scenes with low latency.

[0043] The memory 105 comprises storage components used to temporarily hold data, instructions, and intermediate results during image processing tasks. The memory 105 can comprise for example RAM (Random Access Memory), cache memory, and / or non-volatile memory. The memory 105 can for example be used to store raw image data captured by the imaging sensor 101, processed images generated during image processing algorithms, parameter values to be used by other components of the system 100, and the output from the SNN accelerator.

[0044] The SNN accelerator may comprise an encoder 106, a spiking neural network 107 and a decoder 108.

[0045] The encoder 106 is the initial stage of the SNN accelerator. Its purpose is to convert input data into a format suitable for processing by the spiking neural network. In the context of a spiking neural network, input data could be sensory data, such as images, audio signals, or other types of time-varying signals. The encoder transforms this continuous or analog input into a discrete representation of spikes, which are the fundamental units of communication in spiking neural networks. The encoder outputs the obtained spike train to the spiking neural network 107.

[0046] The spiking neural network (SNN) 107 is the core computational unit of the accelerator. It takes as input the spike train obtained from the encoder 106. It consists of interconnected neurons that communicate with each other through spikes. Each neuron in the SNN integrates input spikes over time and generates output spikes based on certain activation rules, typically modelled after biological neurons. The neurons are connected via synapticelements, which comprise a certain weight which identifies how strong the connection between neurons is. The SNN may process input spikes through layers of neurons, performing computations such as feature extraction, pattern recognition, or classification.

[0047] The decoder 108 is the final stage of the SNN accelerator. Its purpose is to interpret the output spikes generated by the spiking neural network 107 and produce a meaningful output based on the task being performed. Depending on the application, the decoder may perform tasks such as classification, regression, or decision-making based on the patterns of spikes generated by the SNN. The decoder essentially translates the neural activity encoded in spikes back into a form that is understandable or usable by the microprocessor 104.

[0048] There are several reasons why the usage of an SSN is beneficial when one uses eventbased data as its input, the event-based data coming for example from an event-based sensor (or a frame-based sensor for which the data is pre-processed to obtain event-based data).

[0049] Event-based data is inherently asynchronous, since only changes in the scene (events) are reported rather than capturing frames at fixed intervals like traditional cameras. SNNs are well-suited to process such asynchronous data because they operate on the principle of spike timing, mimicking the behavior of neurons in the brain firing in response to input spikes.

[0050] SNNs have event-driven processing, which means they can efficiently handle low- latency data such as event-based data.

[0051] SNNs, inspired by the brain's energy-efficient processing, require lower computational resources compared to traditional artificial neural networks (ANNs). This makes it the ideal combination of event-based sensors, which only consume power when there is a change in the scene.

[0052] Event-based data is generally sparse data, since events only occur when there is significant activity in the scene. SNNs naturally handle sparse data well since they are accustomed to processing information in a sparse and distributed manner.

[0053] Event-based sensors are often more robust to noise and high-dynamic-range scenes compared to traditional frame-based sensors. SNNs, with their ability to encode and process information in the timing of spikes, can further enhance this robustness by focusing on the most relevant information while filtering out noise.

[0054] FIG. 2 shows a schematic representation of an embodiment of an occupancy detection method 200 according to the invention.

[0055] The system 100 may be configured to perform a sequence of operations or steps of the method 200. These sequence of operations 201-206 may be performed as phases in a pipeline of operations.

[0056] The first step of the method comprises data acquisition 201. To maximise the performance of the solution, appropriate data gathering is important. Captured data is transferred from the image sensor 101 to the processing device 102 via the one or more peripheral interfaces 103. As mentioned, this transfer can happen through various types of interfaces. Upon receiving the image data, the microprocessor 104 may process the data to perform tasks such as image compression (if needed), applying filters, adjusting color balance, and performing any other desired image manipulations. The image data, such as events, frames or sequences, can then be stored in memory 105 of the processing device 102.

[0057] If the image sensor is an event-based sensor, steps 202 and 203 below may be skipped; otherwise the following steps are executed in order.

[0058] The second step of the method comprises preprocessing the image data into eventbased data 202. The difference between successive or subsequent frames is computed so that N frames translate to a sequence of N-l difference frames. This can be done using frame differencing, or any other suitable method, for example a method that includes frame differencing.

[0059] Frame differencing works by computing the difference between two consecutive frames captured by the image sensor. To simplify the processing, one can convert both images to grayscale. This may reduce the computation required and simplifies the comparison process. Next, one calculates the absolute difference between corresponding pixel values in the two grayscale images. This operation yields a new image where each pixel represents the absolute difference in intensity between the same pixel locations in the two input images.

[0060] The third step of the method comprises an encoding step 203. A temporal contrast / level sampling encoding is used to translate differences into events. A positive change between subsequent frames is encoded as an event / spike if the difference is larger than a predefined threshold, which can be trained or engineered. An event capturing a negative change would conversely be generated if a difference crosses a predefined negative threshold.

[0061] One may thus apply a threshold to the image showing the absolute difference to identify regions in the image where the change in intensity exceeds a certain positive or negative threshold. This step helps to distinguish significant changes from noise. Next, one may identify connected components in the thresholded image. Each connected componentrepresents a region where a change occurred between the two frames. One may identity each connected component as an “event.” For each identified event, one may extract information such as the location, time, size, shape, and intensity of the change. This information can be stored or transmitted as event-based data.

[0062] Next, the encoder encodes the event-based data such as location, timing, size, shape, and intensity of changes into spike trains, which can be used as input to the SNN. Typically, this may involve mapping each of these features to specific characteristics of the spike trains.

[0063] Spatial information, such as the location of a change in the image, can be encoded using the spatial location of neurons in the input layer of the spiking neural network. Each neuron in the input layer corresponds to a specific location in the image sensor's field of view. When an event occurs at a particular location in the image, the neuron corresponding to that location generates a spike.

[0064] Encoding the time of the change into spike trains involves representing temporal information through spike timing. One might use e.g. temporal coding, rate coding, population coding and / or relative timing coding to encode the timing data of the event.

[0065] In temporal coding, one may use the timing of spikes to encode the time of the change relative to a reference point. Each spike generated by a neuron corresponds to an event occurring at a specific time. The precise timing of spikes relative to the onset of the event can encode temporal information such as the time of occurrence and duration of the change. Earlier spikes might represent changes that occurred earlier in time, while later spikes indicate changes that occurred later. In rate coding, one may use the firing rate of neurons to encode temporal information. Higher firing rates can represent changes occurring over shorter durations or occurring more frequently in time. Lower firing rates can represent changes occurring over longer durations or occurring less frequently. In population coding, one may use populations of neurons to collectively encode temporal information. Different neurons or groups of neurons can represent different time intervals or temporal features. The pattern of activation across the population of neurons can encode complex temporal information about the changes. Lastly, using relative timing coding, one may encode the relative timing of changes between different spatial locations or features. Neurons representing different locations or features in the image can fire at different times relative to each other, encoding the temporal relationships between the changes.

[0066] The size of a change in the image can be encoded using the firing rate of neurons. Neurons that represent larger areas or objects in the image can have higher baseline firing rates,while neurons representing smaller areas can have lower firing rates. The firing rate of neurons can be modulated based on the size of the change detected in their receptive fields.

[0067] Encoding the shape of changes in the image into spike trains can be more complex. One approach is to use a population of neurons with spatially arranged receptive fields that collectively represent different shapes. When a change with a particular shape is detected, it activates a specific subset of neurons in the input layer that corresponds to that shape.

[0068] Intensity information, such as the magnitude of a change in pixel values, can be encoded using the timing or frequency of spikes. Higher-intensity changes can lead to higher spike rates or earlier spike timings, while lower-intensity changes result in lower spike rates or delayed spikes. Alternatively, intensity information can be encoded using the amplitude of spikes, with higher intensities leading to spikes with higher amplitudes.

[0069] Overall, the encoding process involves mapping the various features of the eventbased data to the spiking activity of neurons in the input layer of the spiking neural network. This allows the network to process and interpret the event-based data in a format suitable for spike-based computation. The specific encoding scheme chosen may vary depending on the requirements of the task and the characteristics of the input data.

[0070] A concrete example of the first, second and third steps will now be given. The processing device can acquire data in the first step, for example from a frame-based image sensor. These may be frames with a particular width and height, for example 400 times 400 pixels and a particular color model, for example the RGB color model.

[0071] Next, these frames are preprocessed into event-based data. In this particular example, at any given point in time, a number of past frames are resized (if necessary) to respective luminance-only frames from which a background image is obtained as the average of these luminance-only frames.

[0072] For example, the past 30 frames in their original 400x400 RGB format are resized to 64x64 luminance-only frames, while discarding the bordering 8 pixels and selecting every 6th pixel. This can be done by discarding the bordering 8 pixels, reducing the 400x400 image to 384x384 pixels. Next, the cropped image is down sampled by selecting every 6th pixel to get a 64x64 image. Next, each frame is converted from RGB to a grayscale (luminance) image. To convert an RGB image to grayscale (luminance), one typically uses a weighted sum of the RGB channels to represent the intensity of each pixel. The conversion to luminance can also be done beforehand. Finally, the average of these 30 resized luminance frames is calculated to get the background image.

[0073] As a next step, the latest incoming frame is resized in the same manner, as the number of past frames. So, for example, the latest incoming frame in its original 400x400 RGB format is resized by discarding the bordering 8 pixels, selecting every 6th one to obtain a 64x64 image and converting to a luminance image.

[0074] It is also possible to use the original frames, for example the 400x400 RGB, without downscaling or luminance conversion. This keeps all the original data channels in place, and reduces the time required for preprocessing, but puts higher demands on the inference steps afterwards.

[0075] An absolute integer difference is applied between this processed frame and the previously obtained background frame. On the resulting difference frame, one or multiple iterations of a dilation operation are applied.

[0076] Firstly, the absolute integer difference between the processed frame and the previously obtained background frame is obtained. This operation calculates the absolute difference between corresponding pixels of the two frames. Mathematically, for each pixel (x, y) in the background frame and processed frame, the absolute integer difference is calculated as: Difference (x, y) =1 Processed Frame(x,y) — Background Frame(x,y) I. After computing the absolute integer difference, one obtains the difference frame. This frame highlights areas where significant changes have occurred between the current processed frame and the background frame. Areas with little change will have low intensity values, while areas with significant changes will have higher intensity values.

[0077] As mentioned previously, the difference frame can also be determined as the difference between the current frame and a previous frame, such as the immediate previous frame, so that one does not have to perform the rolling average calculation. The previous frame or the frame obtained by computing the rolling average can also be quantized before calculating the difference with the current frame. Quantization here means rounding the pixel values to the nearest level-crossing point, which is defined as the nearest integer input of the threshold value with a potential constant offset value.

[0078] After obtaining the difference frame, the next step is to apply one or multiple iterations of a dilation operation on this frame. Dilation is a morphological operation that expands regions of foreground pixels (in this case, regions where the difference with the background is significant) based on a structuring element (kernel). One can choose a structuring element (kernel) which determines the neighborhood used for the dilation operation. Common choices include rectangular or circular kernels. For example, one can use a 2x2 kernel, which is a 2x2matrix where all elements are 1. When applying dilation to the difference frame, the goal is to enhance regions where this difference exceeds a certain threshold. The dilation operation will expand these regions of significant change, effectively "highlighting" them compared to the surrounding areas. One can place the kernel at each pixel of the difference frame. If the kernel overlaps with any foreground pixels (where the intensity value is above a certain threshold), the corresponding pixel in the output image is set to foreground.

[0079] Finally, one can apply the dilation operation more times if needed. Each iteration expands the foreground regions further. Each application of the dilation operation expands the foreground objects by the size and shape of the kernel, giving a more pronounced effect. By applying the dilation operation multiple times, one can further expand these foreground objects, which can be useful in various image processing tasks. For example, two iterations of dilation are applied by using the 2x2 kernel.

[0080] Next, the encoding step is performed in this example as the third step of the method.

[0081] The encoding step requires the definition of a threshold and a number of expanded timesteps. Hence, one can choose a threshold that determines how the integer values will be encoded into spikes. This threshold is used to normalize the values for encoding. Next, one defines the total number of expanded timesteps for encoding spikes.

[0082] On the previously obtained processed image, a row-wise and column-wise sum is applied, resulting in integer values that represent the sum of pixel intensities across rows and columns respectively. For example, on the previously obtained 64x64 processed image (e.g., the dilated difference frame), a row-wise and column-wise sum is applied, resulting in 128 integer values.

[0083] These integer values are then encoded to spikes. To do this, a normalization is performed by dividing each of the integer values by the selected threshold. For example, each of the 128 values is divided by the selected threshold so that one obtains the integer quotient of the division.

[0084] This integer quotient is then used to select how many consecutive timesteps (out of the total of the pre-defined expanded timesteps) will contain a spike, the remainder out of the total of the pre-defined expanded timesteps (if any) representing the absence of a spike.

[0085] Thus, for example, for each of the 128 integer values, a spike pattern is generated where the quotient determines how many consecutive timesteps (out of the total expanded timesteps) will have a spike. The remainder of the timesteps will represent the absence of aspike. If the expanded timesteps is set to a particular number of timesteps, it can be determined how many out of the particular number of timesteps will contain a spike.

[0086] Alternatively, the quotient of the row-wise and column-wise sums divided by the threshold can be used as the rate for other kinds of encodings (periodic rate, Poisson rate or integrate & fire encoding), thus generating a spike train across the expanded timesteps by using the aforementioned techniques.

[0087] This process may then be applied for a particular number of consecutive frames, for example 30, resulting in the input to our Spiking Neural Network.

[0088] The fourth step of the method comprises processing the event-based data using a spiking neural network 204.

[0089] The pre-trained Spiking Neural Network (SNN) receives as input the events produced by the appropriate preceding steps. The SNN is composed of multiple layers of spiking neurons connected in a feedforward fashion, allowing the input spikes to flow from the input layer to the output layer. Additional recurrent or feedback connections can be added depending on the final complexity of the application - connecting layers to themselves (recurrent) or to upstream layers (feedback) would have the effect of increasing the ability of the network to identify longer temporal connections between features.

[0090] The spiking neural network terminates in one or more output nodes / neurons, whose activity (spiking events produced or internal states if observable) is interpreted as the instantaneous presence in the input of an abstract feature they’re sensitive to - for example, a particular neuron could be tuned to respond to person outlines, while another could be tuned to respond maximally for features related to pets or curtains.

[0091] To assess the performance of the network, the testing / validation data should be organised in a similar manner as during training.

[0092] The fifth step of the method comprises decoding 205 the state of the output neurons of the SNN. Decoding the state of the output neurons of the spiking neural network (SNN) involves interpreting the spiking activity of these neurons to extract meaningful information about the input data or the network's response to the input. Each output neuron may represent a specific feature, class, or category that the network is trained to recognize or respond to. The spiking activity of these neurons encodes information about the network's decision or response based on the input data.

[0093] The state of the output neurons refers to the spiking activity patterns over time of the output neurons. The state of the output neurons can be processed by thresholding in the activitydomain or first derivative and the timestamp of that event is noted in relation to the feature they represent.

[0094] The step of decoding the one or more output spikes may comprise for example counting the spiking activity of each of the output neurons. Another option is to compute an argmax of the (raw) spike counts per output neuron or the values obtained after applying a softmax function to the (raw) spike counts, wherein the output neurons may each represent an output class of the spiking neural network, in order to make a selection of which class the network predicts the input belongs to. Another option is to analyze the membrane potential of the output neurons, preferably by computing the first derivative of the membrane potential representing a rate of change of spike counts that arrive at the output neurons over time or area under the curve of the membrane potential, to extract information about the timing and dynamics of the output neurons.

[0095] Thresholding in the activity domain involves setting a threshold on the spiking activity of the output neurons. Neurons that exceed the threshold are considered to be activated or "firing," indicating a positive response to the input stimulus or the presence of a particular feature. This thresholding process helps to distinguish significant responses from background activity or noise in the network.

[0096] Alternatively, one can analyse the first derivative of the spike trains to extract information about the timing and dynamics of the spiking activity. The first derivative represents the rate of change of spike counts over time, indicating periods of increased or decreased activity. Peaks or changes in the first derivative can correspond to significant events or transitions in the spiking activity of the neurons.

[0097] Once the spiking activity of the output neurons is processed through thresholding or first derivative analysis, the timestamps of the events are noted. These timestamps represent the timing of the events relative to the input stimulus or the features they represent. By correlating the timestamps of the events with the features represented by the output neurons, one can determine when and how the network responds to specific input stimuli or detects particular features in the data.

[0098] The evidence of which tokens / features are produced over time may be stored in a memory bounded and efficient structure that overwrites itself during normal operation (for example a circular buffer). A circular buffer is a memory of a fixed size and overwrites older data as new data comes in, ensuring that memory usage remains constant.

[0099] The inter-token interval can serve as a measure of repetitiveness of a feature, while the time-averaged count of a feature represents the intensity of the feature in a short time interval. Short inter-token intervals indicate frequent occurrences of the feature, suggesting high repetitiveness, while longer intervals suggest lower repetitiveness. A higher time- averaged count indicates a higher intensity of the feature during that time interval, while a lower count suggests lower intensity.

[0100] The sixth step of the method comprises taking a specific action 206 based on the decoded result. The time between features, as well as the numbers of each feature predicted by the SNN is used to create higher level predictions (e.g. ~5 persons present in the room), and therefore an appropriate action can be taken. Actions can be intrinsic or extrinsic. Intrinsic actions are those that affect the system (e.g. changing exposure time of the camera), while extrinsic actions affect the environment (e.g. turn off the light in the room). The action is then recorded as a mechanism for the system to account for changes in the way the environment is sensed. The action can be initiated or governed by the microprocessor 104 of the processing device. For example, the microprocessor 104 can determine particular parameter values that define the specific action 206. The decoded result of the SNN can also be forwarded directly to a separate device from the processing device 102. The command or decoded result can be forwarded to a different device using the peripheral interface 103.

[0101] Below some data gathering and use considerations are stipulated.

[0102] Training and validation data used to train and validate the spiking neural network should be recorded in continuous sequences, so that realistic temporal patterns can be extracted from the data. This process involves recording multiple frames using the same or similar image sensor mounting position and with gradually changing environmental conditions. The number of frames to be recorded within a sequence would be at least as long as required by the neural network to produce an output, and can be affected by task complexity, network architecture or internal confidence target threshold in individual outputs.

[0103] Data labelling should be performed per frame or in an event-based fashion: when a label changes within a recording, the time and new label is registered for that particular frame.

[0104] Semi-automatic or automatic labelling can be employed with the assistance of a larger model (e.g. larger neural network a neural network optionally combined with some pre- and post-processing) providing the label considered to be ground truth (e.g. a label considered correct for the dataset). The labelling of the dataset can be iteratively improved by investigating correct and incorrect predictions of the smaller neural network through random sampling.

[0105] Recordings should be performed with settings that would yield more accurate / less noisy results to provide a basis for injecting faults or noise, or simply dropping data out.

[0106] The data gathering / assembly method allows for optimising the entire solution pipeline, including heuristics, any necessary preprocessing as well as the neural network to be trained.

[0107] The methodologies presented here and the processing pipeline enable extremely low power vision-based occupancy detection, as well as derivative presence / movement detection where the main constraint is power. This performance is enabled by the evidence-based computational flow and closing the sensor-compute loop.

[0108] The invention can be applied to a multitude of technical fields. For example, in building-management by security camera wake-up and alarm, control of building lighting, heating and other services.

[0109] This work describes a method for data gathering that enables the creation of an optimised pipeline able to perform classification of occupancy status of an arbitrary environment by enforcing that chunks of data have smoothly changing environmental conditions to exploit background similarities during training.

[0110] This work describes a method for producing classifications of occupancy (person present vs. person absent) using level sampling of inputs, presented directly in event form or translated into events / spikes, and processed by a spiking neural network.

[0111] The presented pipeline is sufficiently lightweight and tunable to enable the application of occupancy detection with very low power budgets as the preprocessing involves computing pairwise differences and threshold comparisons, and storage of at most 2 full data frames and encoded sparse events / spikes.

[0112] Note that features of any of the embodiments disclosed herein may be combined in an appropriate manner.

Claims

CLAIMS1. A method for occupancy detection in a visual scene imaged by an image sensor, the method using a spiking neural network and comprising: obtaining event-based data, wherein the event-based data is a representation of input from the image sensor in the form of one or more discrete events, each event representing a change in the visual scene that passes a predefined threshold; encoding the event-based data as input spikes, wherein the input spikes are configured to serve as input for the spiking neural network; processing the input spikes using the spiking neural network, wherein the spiking neural network produces one or more output spikes by processing the input spikes; decoding the output spikes into one or more classification results; detecting the occupancy in the visual scene using the classification results.

2. The method of claim 1, wherein the image sensor is an event-based image sensor, which captures the event-based data when there is a change in the visual scene that passes the predefined threshold, preferably wherein the event-based image sensor captures individual pixel-level and / or regional pixel-level changes, more preferably wherein when a change is detected at a particular region in the visual scene beyond the predefined threshold, the eventbased image sensor transmits event-based data representing the location of the change, optionally along with information about the rate of the change and timing of the change.

3. The method of claim 1, wherein the image sensor is a frame-based image sensor which captures images in a frame-by-frame manner, where each frame represents a complete snapshot of the visual scene at a specific moment in time, and wherein the step of obtaining event-based data further comprises: preprocessing the captured images from the frame-based image sensor by computing the difference between a current frame and a background frame, wherein the background frame can be either obtained from one or multiple frames previously captured by the image sensor; checking whether the difference is a change in the visual scene that passes the predefined or an immediately computed threshold; generating event-based data based on the difference if the change passed the predefined threshold.

4. The method of any one of the preceding claims, wherein the event-based data comprises the location, time, size, shape, and / or intensity of the change in the visual scene.

5. The method of any one of the preceding claims, wherein the predefined threshold is either a positive threshold or a negative threshold, wherein the positive threshold is used to check whether an increase in intensity in a particular region of the visual scene is an event, and wherein the negative threshold is used to check whether a decrease in intensity in a particular region of the visual scene is an event.

6. The method of any one of the preceding claims, wherein the step of encoding the eventbased data into spike trains comprises mapping features of the event-based data to the spiking activity of neurons in the spiking neural network, by: mapping the location of a change in the visual scene to a corresponding neuron; mapping the size of a change in the visual scene to the fire rate of a corresponding neuron; mapping the shape of a change in the visual scene to a specific subset of neurons that corresponds to that shape; mapping the intensity of a change in the visual scene to the timing, frequency or amplitude of spikes; and / or mapping the time of a change in the visual scene to spike timing.

7. The method of any one of the preceding claims, wherein the spiking neural network is composed of multiple layers of spiking neurons connected in a feedforward fashion, allowing the input spikes to flow from the input layer to the output layer, wherein recurrent and / or feedback connections are present, respectively connecting layers to themselves or to upstream layers.

8. The method of any one of the preceding claims, wherein the step of decoding the one or more output spikes comprises: counting the spiking activity of the output neurons; and / or computing a maximum or softmax function in order to make a selection of which class the network predicts the input belongs to; and / oranalyzing the membrane potential of the output neurons, preferably by computing the first derivative of the membrane potential representing a rate of change of spike counts that arrive at the output neurons over time or area under the curve of the membrane potential, to extract information about the timing and dynamics of the output neurons.

9. The method of any one of the preceding claims, wherein the method further comprises: noting a timestamp of the event, wherein the timestamp represents the timing of the event relative to the input stimulus or the features they represent; correlating the timestamps of the events with the features represented by the output neurons; and determining when and how the network responds to specific input spikes on the basis of the correlation.

10. The method of claim 9, wherein the method further comprises: determining an inter-feature interval and / or a time-averaged count of a feature in a time interval based on the timestamps; measuring the repetitiveness of a feature using the inter-feature interval and / or the intensity of the feature in the time interval using the time-averaged count.

11. The method of any one of the preceding claims, wherein the method further comprises: taking an action based on the detected occupancy, wherein the action is either intrinsic or extrinsic, wherein the intrinsic actions are those actions that affect the method by changing control parameters of the image sensor or the spiking neural network, while the extrinsic actions are those actions that affect an environment of the visual scene.

12. An occupancy detection device for occupancy detection in a visual scene imaged by an image sensor, wherein the occupancy detection device comprises a processor, an encoder, a spiking neural network and a decoder, wherein the processor is configured to obtain event-based data, wherein the event-based data is a representation of input from the image sensor in the form of one or more discrete events, each event representing a change in the visual scene that passes a predefined threshold;wherein the encoder is configured to encode the event-based data as input spikes, wherein the input spikes are configured to serve as input for the spiking neural network; wherein the spiking neural network is configured to produce one or more output spikes by processing the input spikes; wherein the decoder is configured to decode the output spikes into one or more classification results; and wherein the processor is configured to detect the occupancy in the visual scene using the classification results13. The occupancy detection device of claim 12, further comprising the image sensor, and wherein the image sensor is an event-based image sensor or a frame-based image sensor; wherein the event-based image sensor captures the event-based data when there is a change in the visual scene that passes the predefined threshold, preferably wherein the eventbased image sensor captures individual pixel-level and / or regional pixel-level changes, more preferably wherein when a change is detected at a particular region in the visual scene beyond a predefined threshold, the event-based image sensor transmits event-based data representing the change, along with information about the location and timing of the event; and wherein the frame-based image sensor captures images in a frame-by-frame manner, where each frame represents a complete snapshot of the visual scene at a specific moment in time, and wherein the processor is further configured: to preprocess the captured images from the frame-based image sensor by computing the difference between subsequent frames; to check whether the difference is a change in the visual scene that passes the predefined threshold; and to generate event-based data based on the difference if the change passed the predefined threshold.

14. The occupancy detection device of claim 12 or 13, wherein the event-based data comprises the location, time, size, shape, and / or intensity of the change in the visual scene; and / or wherein the encoder is configured to map features of the event-based data to the spiking activity of neurons in the spiking neural network, by: mapping the location of a change in the visual scene to a corresponding neuron;mapping the size of a change in the visual scene to the fire rate of a corresponding neuron; mapping the shape of a change in the visual scene to a specific subset of neurons that corresponds to that shape; mapping the intensity of a change in the visual scene to the timing, frequency or amplitude of spikes; and / or mapping the time of a change in the visual scene to spike timing.

15. The occupancy detection device of any one of claims 12-14, wherein the processor is configured to take an action based on the detected occupancy, wherein the action is either intrinsic or extrinsic, wherein the intrinsic actions are those actions that affect the method by changing control parameters of the image sensor or the spiking neural network, while the extrinsic actions are those actions that affect an environment of the visual scene.

Images