Occupancy detection systems and methods using frame-based sensors

Abstract

The invention relates to a method for detecting occupancy of a visual scene acquired by an image sensor by using a spiking neural network. The method includes obtaining event-based data, where the event-based data is an input representation in the form of one or more discrete events from the image sensor, each event representing a change in a visual scene exceeding a predetermined threshold. The event-based data is then encoded as an input pulse, where the input pulse is configured as an input to the spiking neural network. The spiking neural network generates one or more output pulses by processing the input pulses. The output pulses are decoded into one or more classification results. Using these classification results, the occupancy state in the visual scene can be detected.

Description

Technical Field

[0001] This invention relates to the field of occupancy detection, and in particular to a method and apparatus for generating and / or using event-based data from an image sensor, the data being processed by a spiking neural network. Background Technology

[0002] Occupancy detection is defined as the task of identifying whether a person is present in a particular room, space, or environment, or whether the occupancy status of the room, space, or environment has changed. Additionally, the system capable of performing occupancy detection may need to provide extra information, such as a count of people or objects.

[0003] Occupancy detection applications include building management, where automatic actions can be taken when a room or area in a building is unoccupied (e.g., turning off ambient lighting or air conditioning, activating security cameras or alarms), or tiered actions can be taken based on the number of people present (e.g., if there are three people, the air conditioning fan is set to medium; if there are five or more people, it is set to maximum).

[0004] Typically, frame-based sensors are used to perform occupancy detection. These sensors capture data in the form of discrete frames or snapshots, similar to how a camera captures images. Frame-based sensors can utilize a variety of technologies, such as infrared (IR), ultrasound, microwave, and even vision cameras, and capture data periodically in the form of frames or snapshots. For example, an infrared sensor might take a snapshot of the thermal features of a room, while a vision camera might capture an image of the monitored area.

[0005] The captured frames are then analyzed to detect changes in occupancy within the monitored area. This analysis may involve detecting motion and spatiotemporal features related to person presence, thermal characteristics, or other relevant parameters depending on the sensor technology used. Data from frame-based sensors, such as a frame of RGB values, is typically processed using algorithms designed to identify patterns associated with occupancy. Machine learning algorithms can also be employed to improve the accuracy of occupancy detection.

[0006] Occupancy detection can be performed using frame-based sensors by employing minimal preprocessing, data buffering, and coupling the frame-based sensor to a neural network for processing.

[0007] Frame-based sensors can be configured to periodically and continuously capture data, such as images or sensor readings, to monitor occupancy within a space. This continuous data collection ensures that any changes in occupancy are detected promptly. The data captured by the sensors is then fed into a neural network for analysis. The neural network processes the data to identify patterns associated with occupancy, such as motion, thermal features, or other relevant characteristics. Because occupancy detection typically requires a real-time response, the neural network must process the data rapidly to provide timely feedback on the occupancy status. This real-time analysis is crucial for applications such as smart building automation, security systems, and energy management.

[0008] To maintain continuous monitoring and real-time analysis, frame-based sensors and neural networks must be kept "always on," meaning they are constantly running and actively processing data. Even during periods of stable load, this continuous operation consumes processing power and energy.

[0009] The continuous operation of frame-based sensors and neural networks can lead to significant power consumption, especially in battery-powered devices or systems with stringent energy efficiency requirements. Summary of the Invention

[0010] To address the aforementioned problems, this technical solution is proposed.

[0011] The solution proposed in this invention introduces a method for performing data collection and dataset assembly to train high-performance networks. This method enables a high-performance presence sensing application that operates within a limited power budget.

[0012] According to a first aspect of the present invention, a method for occupancy detection of a visual scene imaged by an image sensor using a spiking neural network is disclosed. The method includes the following steps: acquiring event-based data, wherein the event-based data may be an input representation in the form of one or more discrete events from the image sensor, each event representing a change in the visual scene exceeding a predefined threshold; encoding the event-based data into input pulses, wherein the input pulses may be configured as inputs to the spiking neural network; processing the input pulses using the spiking neural network, wherein the spiking neural network can generate one or more output pulses by processing the input pulses; decoding the output pulses into one or more classification results; and detecting occupancy in the visual scene using the classification results.

[0013] According to one embodiment of the first aspect, the image sensor may be an event-based image sensor that captures event-based data when a change exceeding a predefined threshold exists in the visual scene. Preferably, the event-based image sensor captures changes at the individual pixel level and / or the region pixel level. More preferably, when a change exceeding the predefined threshold is detected in a specific region of the visual scene, the event-based image sensor may send event-based data representing the location of the change, optionally along with information about the rate and timing of the change.

[0014] According to one embodiment of the first aspect, the image sensor may be a frame-based image sensor that captures images frame by frame, wherein each frame represents a complete snapshot of the visual scene at a specific moment. The step of acquiring event-based data may further include: preprocessing the captured image from the frame-based image sensor by calculating the difference between the current frame and a background frame, wherein the background frame may be obtained from one or more frames previously captured by the image sensor; checking whether the difference is a change in the visual scene exceeding the predefined threshold or an on-the-fly threshold; and if the change exceeds the predefined threshold, generating event-based data based on the difference.

[0015] According to one embodiment of the first aspect, the event-based data may include the location, time, size, shape, and / or intensity of changes in the visual scene.

[0016] According to one embodiment of the first aspect, the predefined threshold can be a positive threshold or a negative threshold. The positive threshold can be used to check whether an increase in intensity in a specific region of the visual scene is an event. The negative threshold can be used to check whether a decrease in intensity in a specific region of the visual scene is an event.

[0017] According to one embodiment of the first aspect, the step of encoding the event-based data into a pulse sequence may include mapping features of the event-based data to the pulse activity of neurons in the spiking neural network by: mapping the location of the change in the visual scene to the corresponding neuron; mapping the magnitude of the change in the visual scene to the firing rate of the corresponding neuron; mapping the shape of the change in the visual scene to a specific subset of neurons corresponding to the shape; mapping the intensity of the change in the visual scene to the timing, frequency, or amplitude of the pulse; and / or mapping the time of the change in the visual scene to the pulse timing.

[0018] According to one embodiment of the first aspect, the spiking neural network may consist of multiple layers of spiking neurons connected in a feedforward manner, allowing the input pulses to flow from the input layer to the output layer. Looping and / or feedback connections may exist, where layers are connected to themselves or to upstream layers.

[0019] According to one embodiment of the first aspect, the step of decoding the one or more output pulses may include: counting the pulse activity of the output neurons; and / or calculating a maximum value or softmax function to make a selection of which category the input is predicted to belong to by the network; and / or and / or analyzing the membrane potential of the output neurons, preferably by calculating the first derivative of the membrane potential, which represents the rate of change of the pulse count arriving at the output neurons over time or the area under the membrane potential curve, to extract temporal and dynamic information about the output neurons.

[0020] According to one embodiment of the first aspect, the method may further include: recording a timestamp of an event, wherein the timestamp represents the temporal sequence of the event relative to an input stimulus or a representation feature thereof; associating the timestamp of the event with features represented by an output neuron; and determining, based on the association, when and how the network responds to a specific input impulse.

[0021] According to one embodiment of the first aspect, the method may further include: determining an inter-feature time interval and / or a time-averaged count of features within the time interval based on the timestamp; measuring the repeatability of features using the inter-feature time interval; and / or measuring the intensity of the features within the time interval using the time-averaged count.

[0022] According to one embodiment of the first aspect, the method may further include: taking action based on detected occupancy, wherein the action is intrinsic or extrinsic. Intrinsic actions may be those that affect the method by changing control parameters of the image sensor or the spiking neural network, while extrinsic actions may be those that affect the visual scene environment.

[0023] According to a second aspect of this disclosure, an occupancy detection device is disclosed for occupancy detection of a visual scene imaged by an image sensor. The occupancy detection device may include a processor, an encoder, a spiking neural network, and a decoder. The processor may be configured to acquire event-based data. The event-based data may be an input representation in the form of one or more discrete events from the image sensor, each event representing a change in the visual scene exceeding a predefined threshold. The encoder may be configured to encode the event-based data into input pulses, wherein the input pulses are configured as inputs to the spiking neural network. The spiking neural network may be configured to generate one or more output pulses by processing the input pulses. The decoder may be configured to decode the output pulses into one or more classification results. The processor may be configured to detect occupancy in the visual scene using the classification results.

[0024] According to one embodiment of the second aspect, the occupancy detection device may further include the image sensor. The image sensor may be an event-based image sensor or a frame-based image sensor.

[0025] The event-based image sensor can capture event-based data when a change exceeding a predefined threshold exists in the visual scene. Preferably, the event-based image sensor captures changes at the individual pixel level and / or the region pixel level. More preferably, when a change exceeding a predefined threshold is detected in a specific region of the visual scene, the event-based image sensor can transmit event-based data representing the change, along with information about the location and timing of the event.

[0026] The frame-based image sensor can capture images frame by frame, where each frame represents a complete snapshot of the visual scene at a specific moment. The processor can be further configured to: preprocess the captured images from the frame-based image sensor by calculating the difference between subsequent frames; check whether the difference is a change in the visual scene that exceeds the predefined threshold; and if the change exceeds the predefined threshold, generate event-based data based on the difference.

[0027] According to one embodiment of the second aspect, the event-based data includes the location, time, size, shape, and / or intensity of changes in the visual scene.

[0028] According to one embodiment of the second aspect, the encoder is configured to map features of the event-based data to the spiking activity of neurons in the spiking neural network by: mapping the position of changes in the visual scene to corresponding neurons; mapping the magnitude of changes in the visual scene to the firing rate of corresponding neurons; mapping the shape of changes in the visual scene to a specific subset of neurons corresponding to that shape; mapping the intensity of changes in the visual scene to the timing, frequency, or amplitude of pulses; and / or mapping the time of changes in the visual scene to pulse timing.

[0029] According to one embodiment of the second aspect, the processor can be configured to take action based on detected occupancy. The action can be intrinsic or extrinsic. Intrinsic actions can be those that influence the method by changing control parameters of the image sensor or the spiking neural network, while extrinsic actions can be those that influence the visual scene environment. Attached Figure Description

[0030] The embodiments will now be described by way of example only with reference to the accompanying drawings, wherein corresponding reference numerals denote corresponding parts, and in the drawings:

[0031] Figure 1 A schematic diagram of an embodiment of the occupancy detection system according to the present invention is shown; and

[0032] Figure 2 A schematic diagram of an embodiment of the occupancy detection method according to the present invention is shown. Detailed Implementation

[0033] Certain embodiments will be described in more detail below. However, it should be understood that these embodiments should not be construed as limiting the scope of this disclosure.

[0034] Figure 1 A schematic diagram of an embodiment of the occupancy detection system according to the present invention is shown.

[0035] The occupancy detection system 100 includes an imaging sensor 101 (e.g., a low-resolution sensor or an event-based sensor) and a processing device 102.

[0036] An image sensor images a visual scene. A visual scene refers to all content captured by the image sensor within its field of view at a given moment. It includes all objects, background, color, texture, and lighting conditions present in the image. In other words, a visual scene is a specific area monitored by the sensor where it is necessary to determine the presence or absence of an object or person. Preferably, the visual scene is imaged by the imaging sensor to provide input data, which a spiking neural network processes and analyzes to determine the presence and location of an occupant. In this context, the field of view is the spatial extent captured by the image sensor, including the edges and corners of the scene. Objects within the field of view can be any tangible item in the scene, such as furniture, vehicles, or individuals. The background is the part of the scene that serves as the background to objects, such as walls, floors, or ground. Texture and color include the surface properties and color variations of objects and the background within the scene. Lighting conditions refer to the illumination within the scene, including natural and artificial light sources and their effects on visibility and shadows.

[0037] Compared to high-resolution sensors, low-resolution sensors capture images with a lower number of pixels. For example, a low-resolution sensor may have a resolution of VGA (640x480 pixels) or even lower. These sensors are typically used in applications where image detail is not critical, or where limiting factors such as cost, power consumption, or processing requirements are at play.

[0038] Event-based sensors, also known as event-driven sensors or neuromorphic sensors, operate differently from traditional frame-based sensors. Instead of capturing entire frames at fixed time intervals, they capture and transmit information only when significant changes occur in the scene. Event-based sensors asynchronously detect and report individual pixel-level changes, such as brightness, meaning they only respond to changes in the scene. When a pixel detects a change exceeding a predefined threshold, it sends an event signal indicating that change, along with information about the event's location and timing. Event-based sensors are particularly well-suited for applications requiring low latency, high speed, and high dynamic range sensing.

[0039] The processing device 102 includes one or more peripheral interfaces 103 for transmitting data from the imaging sensor 101 to the processing device 102. Examples include a Serial Peripheral Interface (SPI), an Internal Integrated Circuit (I2C), a MIPI (Mobile Industry Processor Interface), a parallel interface, a USB port, a Bluetooth interface, an HDMI port, an Ethernet port, and / or Low Voltage Differential Signaling (LVDS). Transmission can be made via wired or wireless connections, directly or indirectly via, for example, a server.

[0040] The processing device 102 also includes a microprocessor 104, a memory 105, an interconnect structure 109, and a spiking neural network (SNN) accelerator.

[0041] The microprocessor 104 coordinates the operation of the system, performing data collection and general computation and data manipulation. The microprocessor 104 can preprocess image data to enhance relevant features and / or reduce noise. The imaging sensor 101 can be controlled by the microprocessor 104, for example, by writing to configuration registers or storage locations contained within the sensor to configure image capture rate, exposure duration, low-power or high-power modes, etc. The microprocessor 104 places data from the imaging sensor 101 in memory 105. The data from the image sensor 101 can be RGB, luminance / grayscale, events, or another suitable format.

[0042] RGB data represents an image based on three primary colors: red, green, and blue. Each pixel in an image is typically represented by three values ​​corresponding to the intensity of these three color channels. RGB data is usually represented as a matrix of pixel values. Luminance or grayscale data represents an image based on overall brightness or intensity, without color information; the data is represented as a single channel. Event-based data includes data from individual events triggered by significant changes in the scene, such as motion or changes in brightness. Each event typically includes information such as pixel location, timestamp, and polarity (indicating whether the change is an increase or decrease in intensity). Event-based sensors provide high temporal resolution, enabling real-time responses to dynamic scenes with low latency.

[0043] Memory 105 includes storage components for temporarily storing data, instructions, and intermediate results during image processing tasks. Memory 105 may include, for example, RAM (Random Access Memory), cache memory, and / or non-volatile memory. Memory 105 may be used to store, for example, raw image data captured by imaging sensor 101, processed images generated during image processing algorithms, parameter values ​​to be used by other components of system 100, and outputs from an SNN (spiking neural network) accelerator.

[0044] The SNN accelerator may include an encoder 106, a spiking neural network 107, and a decoder 108.

[0045] Encoder 106 is the initial stage of the SNN accelerator. Its purpose is to convert the input data into a format suitable for processing by the spiking neural network. In the context of a spiking neural network, the input data can be sensed data such as images, audio signals, or other types of time-varying signals. The encoder converts this continuous or analog input into a discrete spike representation, which is the basic unit of communication in the spiking neural network. The encoder outputs the obtained spike sequence to the spiking neural network 107.

[0046] The Spiking Neural Network (SNN) 107 is the core computational unit of the accelerator. It receives a sequence of pulses from the encoder 106 as input. It consists of neurons interconnected by pulses. Each neuron in the SNN accumulates input pulses over time and generates output pulses according to certain activation rules (typically mimicking biological neurons). Neurons are connected by synaptic elements, which have weights to indicate the strength of the connection between neurons. The SNN can process input pulses through neuronal layers to perform computations such as feature extraction, pattern recognition, or classification.

[0047] Decoder 108 is the final stage of the SNN accelerator. Its purpose is to interpret the output pulses generated by the spiking neural network 107 and produce meaningful outputs based on the task being performed. Depending on the application, the decoder can perform tasks such as classification, regression, or decision-making based on the pulse patterns generated by the SNN. Essentially, the decoder translates the neural activity encoded in the pulses back into a form that the microprocessor 104 can understand or use.

[0048] There are several beneficial reasons to use SNNs when using event-based data as input, such as data from event-based sensors (or from frame-based sensors whose data has been preprocessed to obtain event-based data).

[0049] Event-based data is inherently asynchronous because it only reports changes (events) in the scene, rather than capturing frames at fixed time intervals like a traditional camera. SNNs are well-suited for handling this type of asynchronous data because they operate based on the principle of impulse timing, simulating the behavior of neurons in the brain firing in response to input impulses.

[0050] SNNs have event-driven processing capabilities, which means they can effectively handle low-latency data such as event-based data.

[0051] Inspired by the brain's efficient processing, SNNs require fewer computational resources compared to traditional artificial neural networks (ANNs). This makes them an ideal complement to event-based sensors, which only consume power when the scene changes.

[0052] Event-based data is typically sparse because events only occur when there is significant activity in the scene. SNNs naturally handle sparse data well because they are accustomed to processing information in a sparse and distributed manner.

[0053] Compared to traditional frame-based sensors, event-based sensors are generally more robust to noise and high dynamic range scenarios. SNNs further enhance this robustness by focusing on the most relevant information while filtering out noise, thanks to their ability to encode and process information in pulse-time sequences.

[0054] Figure 2 A schematic diagram of an embodiment of the occupancy detection method 200 according to the present invention is shown.

[0055] System 100 can be configured to execute a sequence of operations or steps of method 200. These operation sequences 201-206 can be executed as stages in an operation pipeline.

[0056] The first step of this method includes data acquisition 201. Proper data collection is crucial to maximizing the performance of the solution. The captured data is transmitted from the image sensor 101 to the processing device 102 via one or more peripheral interfaces 103. As previously described, this transmission can be performed through various types of interfaces. Upon receiving the image data, the microprocessor 104 can process the data to perform tasks such as image compression (if needed), applying filters, adjusting color balance, and performing any other desired image manipulations. The image data (e.g., events, frames, or sequences) can then be stored in the memory 105 of the processing device 102.

[0057] If the image sensor is an event-based sensor, steps 202 and 203 below can be skipped; otherwise, proceed with the following steps.

[0058] The second step of the method involves preprocessing the image data into event-based data 202. Differences between consecutive or subsequent frames are calculated, such that N frames are transformed into a sequence of N-1 difference frames. This can be accomplished using frame differencing or any other suitable method, such as methods incorporating frame differencing.

[0059] Frame differencing works by calculating the difference between two consecutive frames captured by an image sensor. To simplify processing, both images can be converted to grayscale. This reduces the computational cost and simplifies the comparison process. Next, the absolute difference between corresponding pixel values ​​in the two grayscale images is calculated. This operation produces a new image where each pixel represents the absolute difference in intensity at the same pixel location in the two input images.

[0060] The third step of this method includes encoding step 203. Differences are converted into events using time-contrast / level sampling encoding. If a positive change between subsequent frames exceeds a predefined threshold (which can be trained or designed), it is encoded as an event / pulse. If the difference exceeds a predefined negative threshold, an event capturing the negative change is generated.

[0061] Therefore, a threshold can be applied to images displaying absolute differences to identify regions where intensity changes exceed a certain positive or negative threshold. This step helps distinguish significant changes from noise. Next, connected components can be identified in the thresholded image. Each connected component represents a region where a change occurs between two frames. Each connected component can be identified as an "event." For each identified event, information such as the location, time, size, shape, and intensity of the change can be extracted. This information can be stored or transmitted as event-based data.

[0062] Next, the encoder encodes event-based data, such as the changing location, timing, size, shape, and intensity, into pulse sequences, which can be used as input to the SNN. Typically, this may involve mapping each feature to a specific feature of the pulse sequence.

[0063] Spatial information, such as changing locations in an image, can be encoded using the spatial locations of neurons in the input layer of a spiking neural network. Each neuron in the input layer corresponds to a specific location within the image sensor's field of view. When an event occurs at a specific location in the image, the neuron corresponding to that location generates a spiking pulse.

[0064] Encoding varying time into a pulse sequence involves representing time information through pulse timing. Timing data of events can be encoded using, for example, timing coding, rate coding, group coding, and / or relative timing coding.

[0065] In temporal coding, the timing of pulses can be used to encode the time of change relative to a reference point. Each pulse generated by a neuron corresponds to an event that occurs at a specific time. The precise timing of the pulses relative to the start of the event encodes temporal information, such as the time of occurrence and duration of the change. Earlier pulses may represent earlier changes, while later pulses represent later changes. In rate coding, the firing rate of neurons can be used to encode temporal information. Higher firing rates may represent changes that occur more frequently or within shorter time periods. Lower firing rates may represent changes that occur less frequently or within longer time periods. In population coding, groups of neurons can be used to collectively encode temporal information. Different neurons or groups of neurons can represent different time intervals or temporal features. Activation patterns within a neuronal population can encode complex temporal information about the change. Finally, relative temporal coding can be used to encode the relative temporal sequence of changes between different spatial locations or features. Neurons representing different locations or features in an image can fire at relatively different times, thus encoding the temporal relationship between changes.

[0066] The magnitude of changes in an image can be encoded using the firing rates of neurons. Neurons representing larger regions or objects in an image can have higher baseline firing rates, while neurons representing smaller regions can have lower firing rates. The firing rates of neurons can be modulated based on the magnitude of changes detected in their receptive fields.

[0067] Encoding changing shapes in an image into a sequence of pulses can be more complex. One approach is to use a population of neurons with spatially arranged receptive fields that collectively represent different shapes. When a change with a specific shape is detected, it activates a specific subset of neurons in the input layer that corresponds to that shape.

[0068] Intensity information, such as the magnitude of pixel value changes, can be encoded using the timing or frequency of pulses. Higher intensity changes may result in higher pulse rates or earlier pulse timing, while lower intensity changes result in lower pulse rates or delayed pulses. Alternatively, intensity information can be encoded using the pulse amplitude, with higher intensity resulting in pulses with higher amplitudes.

[0069] In summary, the encoding process involves mapping various features of event-based data to the spiking activity of neurons in the input layer of a spiking neural network. This enables the network to process and interpret event-based data in a format suitable for spiking computation. The specific encoding scheme chosen may vary depending on the task requirements and the characteristics of the input data.

[0070] Specific examples of the first, second, and third steps will now be given.

[0071] The processing device can acquire data in the first step, such as from a frame-based image sensor. These might be frames with a specific width and height, such as 400 by 400 pixels, and a specific color model, such as the RGB color model.

[0072] Next, these frames are preprocessed into event-based data. In this particular example, at any given point in time, a certain number of past frames (if necessary) are resized to the corresponding luminance-only frames, from which the background image is obtained as the average of these luminance-only frames.

[0073] For example, the past 30 frames, in their original 400x400 RGB format, are resized to 64x64 luminance-only frames, discarding 8 boundary pixels and selecting every 6th pixel. This can be done by discarding the 8 boundary pixels, cropping the 400x400 image to 384x384 pixels. Next, the cropped image is downsampled by selecting every 6th pixel to obtain a 64x64 image. Next, each frame is converted from RGB to grayscale (luminance). To convert an RGB image to grayscale (luminance), a weighted sum of the RGB channels is typically used to represent the intensity of each pixel. The luminance conversion can also be pre-done. Finally, the average of these 30 resized luminance frames is calculated to obtain the background image.

[0074] As a next step, the size of the latest incoming frame is adjusted in the same way as the previous frames. For example, the latest incoming frame, in its original 400x400 RGB format, is resized by discarding 8 pixels at the boundary, selecting every 6th pixel to obtain a 64x64 image, and converting it to a luminance image.

[0075] Alternatively, raw frames, such as 400x400 RGB, can be used without downsampling or luminance conversion. This preserves all raw data channels and reduces the time required for preprocessing, but places higher demands on subsequent inference steps.

[0076] An absolute integer difference is applied between the processed frame and the previously obtained background frame. One or more expansion operations are then applied iteratively to the resulting difference frame.

[0077] First, the absolute integer difference between the processed frame and the previously obtained background frame is obtained. This operation calculates the absolute difference between corresponding pixels in the two frames. Mathematically, for each pixel (x,y) in the background frame and the processed frame, the absolute integer difference is calculated as: Difference (x,y) = |Processed Frame(x,y) - Background Frame(x,y)|. After calculating the absolute integer difference, the difference frame is obtained. This frame highlights the areas where significant changes occur between the current processed frame and the background frame. Areas with smaller changes will have lower intensity values, while areas with significant changes will have higher intensity values.

[0078] As mentioned earlier, the difference frame can also be determined as the difference between the current frame and the previous frame (e.g., the immediately preceding frame), thus eliminating the need to perform a rolling average calculation. The previous frame, or the frame obtained by calculating the rolling average, can also be quantized before calculating the difference with the current frame. Quantization here means rounding the pixel values ​​to the nearest level-crossing point, which is defined as the nearest integer input to a threshold with a potentially constant offset value.

[0079] After obtaining the difference frame, the next step is to apply one or more multiple iterations of the dilation operation to that frame. Dilation is a morphological operation that expands the foreground pixel region (in this case, the region that differs significantly from the background) based on a structuring element (kernel). A structuring element (kernel) can be chosen, which determines the neighborhood used for the dilation operation. Common choices include rectangular or circular kernels. For example, a 2x2 kernel, i.e., a 2x2 matrix where all elements are 1, can be used. When applying dilation to the difference frame, the goal is to enhance regions where the difference exceeds a certain threshold. The dilation operation will expand these significantly changed regions, effectively "highlighting" them compared to the surrounding areas. The kernel can be placed at each pixel of the difference frame. If the kernel overlaps with any foreground pixel (where the intensity value is above a certain threshold), the corresponding pixel in the output image is set as the foreground.

[0080] Finally, the expansion operation can be applied multiple times if needed. Each iteration further expands the foreground region. Each application of the expansion operation expands the foreground object according to the size and shape of the kernel, producing a more noticeable effect. For example, two expansion iterations can be applied using a 2x2 kernel.

[0081] Next, in this example, the encoding step is performed as the third step of the method.

[0082] The encoding step requires defining a threshold and the number of expansion time steps. Therefore, a threshold can be chosen to determine how integer values ​​are encoded as pulses. This threshold is used to normalize the encoded values. Next, the total number of expansion time steps used to encode the pulses is defined.

[0083] Applying row and column summation to the previously obtained processed image yields integer values ​​representing the sum of pixel intensities in the rows and columns, respectively. For example, applying row and column summation to a previously obtained 64x64 processed image (e.g., an expanded difference frame) yields 128 integer values.

[0084] These integer values ​​are then encoded as pulses. This is done by normalizing each integer value by dividing it by a selected threshold. For example, dividing each of the 128 values ​​by the selected threshold yields an integer quotient for the division.

[0085] Then use this integer quotient to select how many consecutive time steps (in the predefined total number of extended time steps) will contain a pulse, with the remainder (if any) in the predefined total number of extended time steps representing no pulses.

[0086] Therefore, for example, for each of the 128 integer values, a pulse pattern is generated, where the quotient determines how many consecutive time steps (in the total extended time steps) will have pulses. The remaining time steps will represent no pulses. If the extended time step is set to a specific number of time steps, it is possible to determine how many of those specific time steps will contain pulses.

[0087] Alternatively, the quotient of the sum of the row and column directions divided by the threshold can be used as the rate for other types of coding (periodic rate, Poisson rate, or integral firing coding), thereby generating a pulse sequence over the entire extended time step using the above techniques.

[0088] This process can then be applied to a specific number of consecutive frames, such as 30 frames, to form the input to our spiking neural network.

[0089] The fourth step of this method involves processing event-based data 204 using a spiking neural network.

[0090] A pre-trained spiking neural network (SNN) receives events generated by the appropriate preceding steps as input. This SNN consists of multiple layers of spiking neurons connected in a feedforward manner, allowing input spiking to flow from the input layer to the output layer. Depending on the final complexity of the application, additional recurrent or feedback connections can be added—connecting layers to themselves (recurrent) or to upstream layers (feedback) increases the network's ability to recognize longer temporal connections between features.

[0091] Spiking neural networks terminate at one or more output nodes / neurons, and their activity (generated spiking events or observable internal states) is interpreted as the instantaneous presence of features they are sensitive to in the input—for example, a particular neuron may be tuned to respond to human silhouettes, while another neuron may be tuned to respond maximally to features associated with pets or curtains.

[0092] To evaluate the network's performance, the test / validation data should be organized in a similar way to that used during training.

[0093] The fifth step of this method involves decoding the states of the 205 SNN output neurons. Decoding the states of the spiking neural network (SNN) output neurons involves interpreting the spiking activity of these neurons to extract meaningful information about the input data or the network's response to the input. Each output neuron may represent a specific feature, category, or class that the network was trained to recognize or respond to. The spiking activity of these neurons encodes information about the network's decision-making or response based on the input data.

[0094] The state of an output neuron refers to the pulse activity pattern of the output neuron that changes over time. The state of an output neuron can be controlled by thresholding in the active domain or the first derivative, and by recording the relationship between the timestamp of the event and the feature it represents.

[0095] Decoding one or more output pulses may include, for example, counting the pulse activity of each output neuron. Another option is to calculate the argmax of the (raw) pulse count for each output neuron or the value obtained by applying a softmax function to the (raw) pulse count, where each output neuron may represent an output class of the spiking neural network, in order to select the class to which the network predicts the input belongs. Yet another option is to analyze the membrane potential of the output neurons, preferably by calculating the first derivative of the membrane potential, which represents the rate of change of the pulse count arriving at the output neuron over time, or the area under the membrane potential curve, to extract information about the temporal and dynamic aspects of the output neurons.

[0096] Throttling in the active domain involves setting a threshold for the spiking activity of output neurons. Neurons exceeding the threshold are considered activated or "fired," indicating a positive response to the input stimulus or the presence of a specific characteristic. This thresholding process helps distinguish significant responses from background activity or noise in the network.

[0097] Alternatively, the first derivative of the pulse sequence can be analyzed to extract temporal and dynamic information about pulse activity. The first derivative represents the rate of change of the pulse count over time, indicating periods of increased or decreased activity. Peaks or variations in the first derivative can correspond to significant events or transitions in neuronal pulse activity.

[0098] Once the spiking activity of the output neurons has been processed through thresholding or first-derivative analysis, timestamps of the events are recorded. These timestamps represent the temporal sequence of the events relative to the input stimulus or its representative feature. By correlating the event timestamps with what the output neurons represent, it is possible to determine when and how the network responds to a specific input stimulus or a specific feature in the detection data.

[0099] Evidence of which tags / features are generated over time can be stored in a memory-constrained and efficient structure that overwrites itself during normal operation (e.g., a circular buffer). A circular buffer is a fixed-size memory that overwrites older data as new data arrives, ensuring that memory usage remains constant.

[0100] The inter-label interval can serve as a measure of feature repeatability, while the time-averaged count of a feature represents the intensity of the feature within a short time interval. Shorter inter-label intervals indicate frequent feature occurrences, suggesting high repeatability, while longer intervals indicate lower repeatability. Higher time-averaged counts indicate higher feature intensity within that time interval, while lower counts indicate lower intensity.

[0101] The sixth step of the method involves taking a specific action 206 based on the decoding result. The time between features and the number of each feature predicted by the SNN are used to create a higher-level prediction (e.g., there are approximately 5 people in the room), allowing for appropriate actions to be taken. Actions can be intrinsic or extrinsic. Intrinsic actions are those that affect the system (e.g., changing the camera's exposure time), while extrinsic actions are those that affect the environment (e.g., turning off the lights in the room). This action is then recorded as a mechanism for the system to consider changes in how it perceives the environment. This action can be initiated or controlled by the microprocessor 104 of the processing device. For example, the microprocessor 104 can determine specific parameter values ​​that define the specific action 206. The SNN's decoding result can also be forwarded directly to another device outside of the processing device 102. This command or decoding result can be forwarded to different devices using the peripheral interface 103.

[0102] The following guidelines outline some considerations for data collection and use.

[0103] Training and validation data for spiking neural networks should be recorded as a continuous sequence so that true temporal patterns can be extracted from the data. This process involves recording multiple frames using the same or similar image sensor mounting locations and gradually changing environmental conditions. The number of frames to be recorded in the sequence should be at least as long as required for the neural network to produce its output, and may be affected by task complexity, network architecture, or the target confidence threshold for a single output.

[0104] Data labeling should be done frame-by-frame or in an event-based manner: when the label in a record changes, the time and new label are registered for that specific frame.

[0105] Semi-automatic or automatic labeling can be employed, leveraging a larger model (e.g., a larger neural network, optionally incorporating some preprocessing and post-processing) to provide labels that are considered ground truth (e.g., labels considered correct for the dataset). The labeling of the dataset can be iteratively improved by randomly sampling the correct and incorrect predictions of the smaller neural network.

[0106] The settings for recording should produce more accurate / less noisy results in order to provide a basis for injecting faults or noise, or simply discarding data.

[0107] This data collection / assembly approach allows for optimization of the entire solution pipeline, including heuristics, any necessary preprocessing, and the neural network to be trained.

[0108] The methodology and processing pipeline presented here enable extremely low-power vision-based occupancy detection, as well as presence / motion detection, where the main limitation is power consumption. This performance is achieved through an evidence-based computational flow and a closed sensor-computation loop.

[0109] This invention can be applied to a variety of technical fields. For example, in building management, it can be used for activating and alarming security cameras, and controlling building lighting, heating, and other services.

[0110] The present invention describes a data collection method that creates an optimized pipeline to perform occupancy state classification of arbitrary environments by forcing data blocks to have smoothly varying environmental conditions in order to leverage background similarity during training.

[0111] The present invention describes a method for generating occupancy classifications (person presence and absence) using level sampling of inputs, which are presented directly as events or converted into events / pulses and processed by a spiking neural network.

[0112] The pipeline proposed in this invention is lightweight and adjustable enough to enable occupancy detection applications to run with a very low power budget, because preprocessing involves calculating pairwise differences and threshold comparisons, and storing at most two complete data frames and encoded sparse events / pulses.

[0113] It should be noted that the features of any of the embodiments disclosed herein can be combined in an appropriate manner.

Claims

1. A method for occupancy detection of a visual scene imaged by an image sensor using a spiking neural network, the method comprising the steps of: acquiring event-based data, wherein the event-based data is an input representation in the form of one or more discrete events from the image sensor, each event representing a change in the visual scene that exceeds a predefined threshold; encoding the event-based data into input spikes, wherein the input spikes are configured as input to the spiking neural network; processing the input spikes using the spiking neural network, wherein the spiking neural network generates one or more output spikes by processing the input spikes; decoding the output spikes into one or more classification results; detecting occupancy in the visual scene using the classification results.

2. The method of claim 1, wherein, the image sensor is an event-based image sensor that captures event-based data when a change in the visual scene exceeds the predefined threshold, preferably wherein the event-based image sensor captures single-pixel level and / or area-pixel level changes, more preferably wherein the event-based image sensor sends event-based data representing the location of the change, optionally together with information about the rate and timing of the change, when a change exceeding the predefined threshold is detected in a specific region of the visual scene.

3. The method of claim 1, wherein, the image sensor is a frame-based image sensor that captures images in a frame-by-frame fashion, wherein each frame represents a complete snapshot of the visual scene at a specific moment in time, and wherein the step of acquiring event-based data further comprises: preprocessing the captured images from the frame-based image sensor by computing the difference between a current frame and a background frame, wherein the background frame is obtainable from one or more frames previously captured by the image sensor; checking whether the difference is a change in the visual scene that exceeds the predefined threshold or an instantaneously computed threshold; generating event-based data based on the difference if the change exceeds the predefined threshold.

4. The method of any one of the preceding claims, wherein the event-based data comprises: the location, time, size, shape and / or intensity of the change in the visual scene.

5. The method according to any of the preceding claims, wherein the predefined threshold is a positive threshold or a negative threshold, wherein the positive threshold is used to check whether an increase in intensity in a specific region of the visual scene is an event, and wherein the negative threshold is used to check whether a decrease in intensity in a specific region of the visual scene is an event.

6. The method according to any of the preceding claims, wherein the step of encoding the event-based data into a sequence of spikes comprises mapping features of the event-based data to spiking activity of neurons in the spiking neural network by: mapping the location of the change in the visual scene to a corresponding neuron; mapping the size of the change in the visual scene to the firing rate of the corresponding neuron; mapping the shape of the change in the visual scene to a specific subset of neurons corresponding to the shape; mapping the intensity of the change in the visual scene to the timing, frequency or amplitude of the spikes. and / or mapping a time of change in the visual scene to the pulse timing.

7. The method according to any of the preceding claims, wherein the spiking neural network consists of multiple layers of spiking neurons connected in a feed-forward fashion, allowing the input pulses to flow from an input layer to an output layer, wherein there are recurrent and / or feedback connections connecting a layer to itself or to an upstream layer, respectively.

8. The method according to any of the preceding claims, wherein the step of decoding the one or more output pulses comprises: counting the pulse activity of the output neurons; and / or computing a maximum or a softmax function in order to make a selection of which class the input predicted by the network belongs to; and / or analyzing the membrane potential of the output neurons, preferably by computing the first derivative of the membrane potential, which represents the rate of change of the pulse count or the area under the membrane potential curve reaching an output neuron over time, to extract timing and dynamic information about the output neurons.

9. The method according to any of the preceding claims, wherein the method further comprises the steps of: recording a timestamp of an event, wherein the timestamp represents the timing of the event relative to the input stimulus or a representation thereof; associating the timestamp of an event with a feature represented by an output neuron; and determining, based on the association, when and how the network responded to a particular input pulse.

10. The method according to claim 9, wherein the method further comprises the steps of: determining a feature interval and / or a time-averaged count of a feature over a time interval based on the timestamp; measuring the repetitiveness of the feature using the feature interval and / or the intensity of the feature over the time interval using the time-averaged count.

11. The method according to any of the preceding claims, wherein the method further comprises the step of: taking an action based on the detected occupancy, wherein the action is intrinsic or extrinsic, wherein an intrinsic action is an action that affects the method by changing a control parameter of the image sensor or the spiking neural network, and an extrinsic action is an action that affects the environment of the visual scene.

12. An occupancy detection device for occupancy detection of a visual scene imaged by an image sensor, wherein the occupancy detection device comprising a processor, an encoder, a spiking neural network, and a decoder, wherein the processor is configured to acquire event-based data, wherein the event-based data is an input representation in the form of one or more discrete events from the image sensor, each event representing a change in the visual scene that exceeds a predefined threshold; wherein the encoder is configured to encode the event-based data into input pulses, wherein the input pulses are configured to be inputs to the spiking neural network; wherein the spiking neural network is configured to produce one or more output pulses by processing the input pulses; wherein the decoder is configured to decode the output pulses into one or more classification results; and wherein the processor is configured to detect occupancy in the visual scene using the classification results.

13. The occupancy detection device of claim 12, further comprising the image sensor, and wherein, The image sensor is an event-based image sensor or a frame-based image sensor; wherein the event-based image sensor captures the event-based data when there is a change in the visual scene that exceeds the predefined threshold, preferably wherein the event-based image sensor captures a change at a single pixel level and / or a regional pixel level, more preferably wherein the event-based image sensor transmits event-based data representing the change, along with information about the event location and timing, when a change exceeding a predefined threshold is detected in a specific region in the visual scene; and wherein the frame-based image sensor captures images in a frame-by-frame fashion, wherein each frame represents a complete snapshot of the visual scene at a specific moment in time, and wherein the processor is further configured to: pre-process the captured images from the frame-based image sensor by computing the difference between subsequent frames; check whether this difference is a change in the visual scene that exceeds a predefined threshold; and generate event-based data based on this difference if the change exceeds the predefined threshold.

14. The occupancy detection device of claim 12 or 13, wherein, The event-based data comprises: the location, time, size, shape and / or intensity of the change in the visual scene; and / or wherein the encoder is configured to map the features of the event-based data to the spiking activity of neurons in the spiking neural network by: mapping the location of the change in the visual scene to a corresponding neuron; mapping the size of the change in the visual scene to the firing rate of the corresponding neuron; mapping the shape of the change in the visual scene to a specific subset of neurons corresponding to this shape; mapping the intensity of the change in the visual scene to the timing, frequency or amplitude of the spikes; and / or mapping the time of the change in the visual scene to the timing of the spikes.

15. The occupancy detection device of any one of claims 12-14, wherein the processor is configured to take action based on the detected occupancy, wherein the action is intrinsic or extrinsic, wherein an intrinsic action is an action that affects the method by changing control parameters of the image sensor or the spiking neural network, and an extrinsic action is an action that affects the visual scene environment.

Images