System for tracking walkways

Abstract

A system (100) for locating a person, comprising: an image sensor (110) viewing a part of a walking surface; and a controller (120) arranged for: receiving (205) an image of the image sensor; providing a person model, wherein the person is modelled by an ellipse; searching for a location of the person based on the received image and the person model; and providing the location of the person.

Description

[0001] SYSTEM FOR TRACKING WALKWAYS

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a system for locating a person. The invention further relates to a method for locating a person. The invention also relates to a data processing system and a computer-readable storage medium for locating a person.

[0004] BACKGROUND OF THE INVENTION

[0005] Detecting people in an image is a common task for software. This software typically involves application of a neural network. The task of detecting people in an image is typically performed on high performance microprocessors. And even when only the location of a person is to be found in an image, this task is performed by complex software on a high-performance microprocessor.

[0006] SUMMARY OF THE INVENTION

[0007] An object of the invention is to overcome one or more of the disadvantages mentioned above.

[0008] According to a first aspect of the invention, a system for locating a person, comprising: an image sensor viewing a part of a walking surface; and a controller arranged for: receiving an image from the image sensor; providing a person model, wherein the person is modelled by an ellipse; searching for a location of the person based on the received image and the person model; and providing the location of the person.

[0009] The image sensor views a part of a walking surface. The image sensor is typically arranged over the walking surface and facing down onto a part of the walking surface. The image sensor typically comprises one or more lenses to focus the light. The image sensor may comprise mirrors and / or optical cables for arranging the conversion from light to electrical signal distant from where the light impinges on the image sensor.

[0010] The controller may comprise a microprocessor, an FPGA, a DSP, an ASIC, or combination of the preceding items. The controller is typically arranged for executing steps. The steps may be executed in a different order, except for where the steps introduce an order in executing the steps. The steps may partly or fully overlap. The steps may be executed multiple times relative to other steps. Locating a person starts with the step of receiving an image from the image sensor. Multiple images may be queued or received before the images are processed. The process of receiving the image may be executed as a parallel process relative to other steps of the controller.

[0011] Providing a person model is typically done once during start-up or initialisation of the controller. The person model typically comprises a struct with parameters. This struct may be copied per instance that a person is searched in the received image. The person model typically comprises parameters modelling the person. The controller may comprise generated pre-calculated or predefined values typically based on the parameters. The predefined values may be retrieved from an external source, such as a central control unit controlling multiple systems for locating a person. The predefined values may be retrieved from memory. The predefined values may be calculated during initialisation and / or installation of the system.

[0012] Searching for the location of the person is based on the received image and the person model. The searching may be enhanced, sped-up, and / or optimized by the pre-calculated or predefined values. The searching step may be performed in whole or partly multiple times on the received image to search for multiple persons in the same received image. The searching step may comprise a criterion which needs to be satisfied for finding a person in the received image. If the criterion is not satisfied, the searching step may conclude that no person is present in the received image.

[0013] A person walking over a walking surface viewed from above, under an angle from above, or from the side may be abstracted as an ellipse. An ellipse has two focal points. One focal point may be arranged at a lower point of the person, while the other focal point is arranged to a higher point of the person. The lower point may be associated with the feet of the person. The higher point may be associated with the head of the person. The person model based on the ellipse has several beneficial mathematical aspects that simplify the calculation and thus the searching for the person. The technical effect of the application of the person model wherein the person is modelled by an ellipse is that the person model reduces the use of resources necessary by the controller during searching for the location of the person. The reduction in resources translate to that e.g. a microprocessor with a limited execution power may be used, such as a smaller embedded processor. The reduction in resources translates in reduced power usage of the controller. The reduction in resources may translate in reduced use of a memory associated with the controller. The reduction in resources may translate in a higher throughput of received images. The reduction in resources may translate to execution time of the controller, such as a processor, spend on another task.

[0014] According to another aspect of the invention, a method for locating a person, comprising: receiving an image from an image sensor viewing a part of a walking surface; providing a person model, wherein the person is modelled by an ellipse; searching for a location of the person based on the received image and the person model; and providing the location of the person. The method provides the advantages also provided by the system.

[0015] According to another aspect of the invention, a data processing system comprising means for carrying out the steps of any of the embodiments of the methods mentioned or the steps of the controller of any of the embodiments of the system mentioned. The data processing system provides the advantages also provided by the system and the method.

[0016] According to another aspect of the invention, a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of any of the embodiments of the methods or the steps of the controller of any of the embodiments of the systems. The computer-readable storage medium provides the advantages also provided by the system, the method and / or the data processing system.

[0017] DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0018] In an embodiment of the system, the ellipse has a first focal point representing the location of the feet of the person, and a second focal point representing the location of the head of the person; and the location of the person is based on the first focal point. An ellipse has by definition two focal points. Associating respective focal points with the feet and head of the person to be located, provides an easily computable point in space. Furthermore, the focal points are advantageously relatively stable compared to alternatives such as edges of the ellipse.

[0019] In an embodiment of the system, the second focal point is related to the first focal point based on one or more of the group of a viewing angle of the image sensor, a height of the image sensor relative to the walking surface, an orientation of the image sensor, a height of the person, a width of the person, and a height of the width of the person. The group contains parameters typically predefined and / or set in advance. One or more of the group of these parameters allow to associated or relate the second focal point to the first focal point for advantageously simplifying the calculations as part of the searching step of the controller.

[0020] In an embodiment of the system, the second focal point is related to the first focal point based on the combination of the viewing angle of the image sensor, the height of the image sensor relative to the walking surface, the orientation of the image sensor, the height of the person, the width of the person, and the height of the width of the person. The group contains parameters typically predefined and / or set in advance. The combination of these parameters allows to associated or relate the second focal point to the first focal point for advantageously simplifying the calculations as part of the searching step of the controller.

[0021] In a further embodiment of the system, a centre point in the image is selected based on a plumb line from the image sensor; and the first focal point is selected from the two focal points of the ellipse based on a distance from the centre point. Typically, an image sensor viewing a part of the walking surface has the point on the walking surface associated with a plumb line from the image sensor down to the walking surface in view. Typically, the focal point having the smallest distance from the centre point is labelled as the first focal point and typed as representing the feet of the person located, and the focal point having the largest distance from the centre point is labelled as the second focal point and typed as representing the head of the person located. This designating of the focal points as the first and the second focal point advantageously relates the focal points as present in the image to parts of the person located. Furthermore, associating the location of the person with the feet advantageously provides a stable and / or solid reference as a location of the person.

[0022] In a further embodiment of the system, a symmetry line is defined through the first focal point and the second focal point; and the symmetry line also is through or substantially through the centre point. Typically, the located person will be standing straight up or walking straight up. An image sensor viewing the part of the walking surface will provide an image that when the person is modelled as an ellipse, this symmetry line may advantageously help determining the first and second focal point of the ellipse.

[0023] In an embodiment of the system, the image sensor comprises a lens; the received image comprises a pixel matrix comprising pixels; each pixel in the pixel matrix has a / the viewing angle based on a refraction of the lens, and a position of the lens relative to the pixel matrix; a distortion per pixel in the pixel matrix is defined based on the viewing angle, a / the height of the image sensor relative to the walking surface, and a / the orientation of the image sensor; the controller is arranged for the step of providing a distortion corrected image based on the distortion per pixel; and searching for the location of the person is based on the distortion corrected image. The lens specifically in combination with the light sensitive part of the image sensor provides an image that has a distortion. The distortion may be shown when a square is drawn on the walking surface within the view of the image sensor. The provided image shows a figure having the general shape of a square, but with bend sides. Depending on the distortion and / or the position of the square, the comers of the square may be shown closer or farther away from the centre of the square. The distortion corrected image advantageously allows to relate distances from the centre point in the corrected image more accurately compared to the real world. The distortion corrected image therefore advantageously enhances the accuracy of the location of the focal points of the ellipse and / or the location of the person.

[0024] In a further embodiment of the system, the distortion per pixel is predefined. Predefining the distortion per pixel allows for the controller to calculate at least part, but preferably all, of the distortion correction in advance of the searching. To calculate part or all of the distortion correction in advance of the searching advantageously allows to reuse the pre-calculated part, and / or to complete the search faster. Reuse the precalculated part typically reduces the necessary resources needed by the controller. Faster searching typically reduces the necessary execution power and / or increases the image throughput rate of the controller.

[0025] In a further embodiment of the system, the distortion per pixel is a barrel distortion per pixel. Labelling the distortion as a barrel distortion allows to apply the applicable distortion corrections.

[0026] In an embodiment of the system, the controller is arranged for obtaining, preferably calculating, a set of ellipses enclosing a set of pixels of the image based on the location of the ellipse of the set of ellipses. Selecting the ellipse as a person model allows to obtain and / or calculate the set of ellipses for advantageously accelerating and / or simplifying the searching step. The searching step may than be reduced to checking for each ellipse of the set of ellipses if a person is present or located at the specific ellipse. In a further embodiment of the system, the obtaining, preferably calculating, is prior to the searching step. This advantageously allows to distribute resources of the controller over time.

[0027] In a further embodiment of the system, the set of ellipses comprises all possible ellipses for the image. The search step will now go through the set of ellipses for locating a person in the image. This embodiment will advantageously reduce the resources needed for the search step while still being a complete search of the image.

[0028] In a further embodiment of the system, a location accuracy defines the accuracy of the location of a person; a location grid on the walking surface is defined based on the location accuracy; the obtaining, preferably the calculating, of the set of ellipses is based on the location grid; and each ellipse of the set of ellipses encloses a set of pixels based on the location of the ellipse and preferably the distortion per pixel. The person typically needs to be located with a particular accuracy. Furthermore, the image sensor comprises a pixel matrix and / or a number of pixels. The pixels are related to a specific location on the walking surface based on the settings of the system. The pixels are further related to a specific location on the walking surface based on the distortion, typically the distortion per pixel. Based on the location grid and typically further settings typing a person, the set of ellipses can be obtained in advance of the searching step. The location grid advantageously allows to calculate the set of ellipses in a simple and elegant fashion. Typically, the set of ellipses is set in a memory associated with the controller for reuse of the set of ellipses in memory for searches in different or distinct images. The reuse of the set of ellipses advantageously reduces the execution power of the controller to process, such as search, an image.

[0029] In a further embodiment of the system, each of the set of pixels enclosed by an ellipse of the set of ellipses is a unique set of pixels. In a typical case, the location grid is more accurate than the pixel matrix of the image sensor. This results in multiple positions in the location grid encircling and / or enclosing the same pixels with an ellipse. Reducing the set of ellipses to a set of ellipses each encircling and / or enclosing a unique set of pixels, reduces the number in the set of ellipses and thereby reduces the execution power of the controller to process, such as search, an image. In a further embodiment of the system, the controller is arranged for obtaining, preferably reducing, the set of ellipses such that each of the set of pixels enclosed by an ellipse of the set of ellipses is a unique set of pixels. This embodiment advantageously implements the obtaining, preferably reducing, the set of ellipses to a unique set of ellipses.

[0030] In a further embodiment of the system, the step of reducing the set of ellipses comprises recalculating the location of the ellipse related to the unique set of pixels based on all ellipses of the set of ellipses enclosing the unique set of pixels. In the case of multiple locations of the location grid resulting in the same set of pixels when an ellipse is drawn, the multiple locations of the location grid are taken in consideration for determining the location of the person. One particular way of considering the multiple locations of the location grid may be to average the multiple locations of the location grid. One other particular way of considering the multiple locations of the location grid may be to select the location closest to the averaged location. The resulting or recalculated location of the multiple ellipses enclosing the same set of pixels improves the accuracy of locating the person.

[0031] In a further embodiment of the system, the controller is arranged for reusing the set of ellipses for multiple images, preferably for all images. This reuse reduces the amount of execution power needed by the controller for the searching step. This advantageously reduces the resources needed by the controller and / or improves the image processing throughput of the controller.

[0032] In an embodiment of the system, the received image comprises a / the pixel matrix comprising pixels; the controller is arranged for the step of determining a compensated image compensated for influences introducing a baseline, preferably per pixel, in the received image, wherein the compensated image is based on the received image; and searching for the location of the person is based on the compensated image. The image sensor, more particular the part converting the light or photons to electrical signals, dominates the noise in the image, such as the signal to noise ratio - SNR-. Further the image sensor, more particular the part converting the light or photons to electrical signals, may experience an offset, typically time dependent. A baseline is introduced to average or compensate the noise and / or offset for enhancing the image for providing a compensated image. The compensated image may for example be better suited for binarization. Furthermore, the compensated image may for example be better resistant to light or shadow present in view of the image sensor, resulting in a deviation in the image not due to a person present at the particular location. Thus, the introduction of a baseline, preferably per pixel, allows for improved filtering of the image for improved deducing if a person is present in the image. In a further embodiment of the system, determining the compensated image comprises applying A^ = A^1+ w(R- - A--1) ; A is a matrix holding the baseline values; w is the update rate of the baseline, preferably based on the number of frames; R is a matrix holding the raw pixel values; t is an iteration variable; and i identifies a pixel in the pixel matrix. A practical implementation for determining the compensated image is presented. In a further embodiment of the system, w is advantageously specifying w per pixel.

[0033] In a further embodiment of the system, has a lower value when the pixel i is part of the person model in the previous image. Pixels in the image part of the person model typically have a different value from the pixels not part of the person model. This is typically the differentiator to determine the presence of the person. As this is the differentiator, and the value of pixels not part of the person model is corrected via the baseline, the pixel i that is part of the person model in the previous image should preferably have less influence or even no influence on the baseline wtat that particular pixel such that the baseline for that pixel more accurately filters out the noise sources. Thus, this embodiment advantageously improves locating of the person in the image.

[0034] In a further embodiment of the system, wthas a lower bound for preventing a static baseline. The baseline, specifically per pixel, typically varies over time. Limiting the influence for a particular pixel by limiting the lower bound of wteffectively limits the number of frames influencing the baseline for a particular pixel. This effectively improves the adaptation of the baseline per pixel to changes over time of the baseline per pixel, such as shadows, spots illuminated with sunlight, lights switching on and off, etc. Thus, preventing a static baseline improves the baseline per pixel and in turn improves the locating of people.

[0035] In a further embodiment of the system, determining the compensated image comprises normalizing the image, preferably after the operation in the embodiment introducing w or even w, , by applying Z,- = ; Z is a normalized matrix; i identifies a pixel in the pixel matrix; and a is based on image sensor noise over time, preferably based on the standard deviation of the pixel values in the pixel matrix, preferably a is set at a predefined value based on the sensor noise, preferably in a range of 0.1 to 0.8, more preferably 0.2 to 0.6, more preferably 0.3 to 0.5, most preferably around 0.4. Normalizing the matrix provides the advantage that absolute values or threshold can more stably determined to apply in multiple situations or installations of the system. As an example, the system may hold other values when monitoring a walking surface in a hallway compared to a walking surface of a room comprising seats. Normalizing allows to apply different absolute values or thresholds based on the type of walking surface and less dependent on system differences for example the image sensor production batch, image sensor type or model, etc. Thus, this embodiment provides improved determining or stability of the absolute values and thresholds to be applied for locating the person. Furthermore, based on the normalized matrix, in a reverse order, the type of walking surface may be determined for automatically setting the absolute values and / or thresholds for improved locating of a person.

[0036] In an embodiment of the system, the received image comprises a / the pixel matrix comprising pixels; searching comprises applying a cost function; the cost function is defined as C = SF=0(^i[rfi > 0] - Z^dt < 0]) ; C is the cost value; n is the number of pixels in the pixel matrix; > 0 defines a pixel i outside the ellipse; < 0 defines a pixel i inside the ellipse. It is noted that the Iverson notation is used for this formula as well as for other formulas throughout this description.

[0037] In a further embodiment of the system, the cost function is applied for each ellipse in the set of ellipses, defined as Ck= SF=0(^i[-iQki]>itQkiD Jzis a / the normalized matrix; k identifies an ellipse in the set of ellipses; i identifies a pixel in the pixel matrix; and Qkidefines if the sensor pixel i is inside the ellipse k, preferably as a binary value. The cost function defines a calculation effective solution to determining if a particular set of pixels in the shape of an ellipse having particular values represents the presence of a person. Next to the calculation effective solution, the result of the cost function is a single value providing the advantage of simply comparing with a threshold. The advantage of this embodiment is that the cost function provides a calculation effective and easily further processable results for further processing, thus resource preserving solution for location a person in the image and thus on the walking surface.

[0038] In a further embodiment of the system, locating comprises locating multiple respective persons. Multiple persons may be located on the walking surface. The system, more specific the controller, is arranged for locating, if present, multiple respective persons. Furthermore, the system, more specific the controller, may be arranged to detect that no person is located on the walking surface. In a further embodiment of the system, searching comprises applying a goodness function; and the goodness function is based on, preferably defined as, the z o '1 average Z value inside ellipse k, which may be formulated as gk= i^n°r‘ . The 2^=0 WfciJ goodness function may be used to provide an indication of how well or with what certainty the person is located on the walking surface. The goodness function may be based on the average value of the pixels inside the ellipse representing a located person. A goodness function equal or based on to the average of the pixels inside the ellipse provides an easily calculatable value advantageously allowing relatively low resources of the controller.

[0039] In a further embodiment of the system, searching comprises: providing a goodness threshold; and selecting all ellipses of the set of ellipses as locations of respective persons. A goodness threshold comparison with the output of the goodness function is an advantageously easily implementable and / or low resource claiming operation by the controller.

[0040] In an embodiment of the system, the image sensor is a heat sensor, preferably an infrared image sensor. An advantage of the heat sensor is that the located person in the image is anonymized automatically. A disadvantage of images having for example colours is that these colours may be present in the background. Furthermore, for the example of colours, the person to be located may comprise multiple colours causing the shape of the person to be harder or even impossible to detect. This disadvantage typically occurs when the pixels in the image may be interpreted along multiple axis, such as in a colour image with multiple colour axis. One or more of these disadvantages cause a resource intensive processing of the for example coloured image. A further advantage is that the person to be located in the image is more easily differentiated from the background compared to e.g. coloured images, as the pixels in the image of the image sensor may be arranged along one axis representing values for one quantity. Pixels having one quantity, such as hot or not, or hot or cold over a complete range, result in less resource intensive processing in the controller. In an embodiment of the system, the image sensor is a sensor providing images with pixels having a single quantity.

[0041] In an embodiment of the system, the received image comprises a / the pixel matrix comprising pixels; the pixel matrix is square; preferably the pixel matrix has at least 5x5 pixels, preferably 6x6 pixels, more preferably 7x7 pixels, most preferably 8x8 pixels; and preferably the pixel matrix has at most 1024x1024 pixels, preferably 512x512 pixels, more preferably 256x256 pixels, more preferably 128x128 pixels, even more preferably 64x64, most preferably 32x32 pixels. Numbers of pixels are provided typically in horizontal and vertical direction that advantageously limits the resources needed by the controller and / or otherwise advantageously provide an image large enough for stably locating one or more persons on the walking surface.

[0042] In an embodiment of the system, the system comprises a second image sensor viewing a second part of the walking surface; the part of the walking surface and the second part of the walking surface partly overlap; the controller is further arranged for: obtaining a position of the second image sensor relative to the image sensor; and receiving a second image of the second image sensor; and searching is also based on the received second image and the relative position. The system may advantageously locate a person partly located in the first part and the second part of the walking surface. Furthermore, the located person moving over the walking surface may be located and therefore followed over an extended part of the walking surface. In an example implementation, a person may be located and / or followed through a building where it is impossible to follow the person with one image sensor due to for example walls and / or floors in the building. The simplicity of the searching and the resulting location advantageously allows to easily combined the location data from multiple image sensors.

[0043] In a further embodiment of the system, the controller comprises a first local controller associated with the image sensor, and a second local controller associated with the second image sensor; the first local controller performs searching for the location of the person in the image; and the second local controller performs searching for the location of the person in the second image. The system advantageously allows distributed processing of the image. The distributed processing allows the system to only communicate locations of located persons instead of communicating images from the image sensors, greatly advantageously reducing the network requirements and / or resources coupling the sub-systems and / or local controllers.

[0044] In an embodiment of the method, the method is combined with any of the features mentioned in the different embodiments of the system or in the claims of the system for obtaining the same advantages.

[0045] In an embodiment of the system the controller is further configured for retrieving walkways, based on previous contact areas, of previously detected people; predicting current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap; matching the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas; if a match is made, adding the located contact area to the walkway of the matched predicted current contact area; if no match is made, initiating a new walkway for a new person based on the located contact area; and purging a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time.

[0046] A walkway may comprise a string of contact areas of a person over time. A walkway may identify and / or detect a path typically traversed over time by a person. Walkways is based on previous contact areas. The retrieved walkways are of previously detected people. The retrieved walkways are of detected people on the walking surface. The number of retrieved walkways thus typically indicate the number of people present on the walking surface.

[0047] The next step may be predicting current contact areas of the previously detected people. Predicting is based on their walkways. Predicting is further based on and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap. From previously determined contact areas and / or information comprised in the walkway information, one or more of the parameters the previously detected location, the previously detected velocity, the previously detected acceleration, the previously detected jerk, and the previously detected snap may be deduced. The location parameter is typically a scalar or comprising multiple scalars, such as an X, Y coordinate. Apart from location, the other parameters may comprise a magnitude together with a direction, such as velocity having a magnitude and direction. Based on the people tracked, one or more of the parameters the maximum distance, the maximum velocity, the maximum acceleration, the maximum jerk, and the maximum snap may be determined. The maximum parameters of the people tracked may be based on their age, such as elderly people typically walk slower and less suddenly change direction, thus the maximum velocity as well as maximum acceleration perpendicular to the velocity will be lower, respectively. The maximum parameters of the people tracked may also vary based on the type of room comprising the walking surface. For example, the hallway will typically have higher maximum values compared to a small living room with a lot of furniture.

[0048] The next step may be matching the located contact area with one of the predicted current contact areas. The matching is based on a cost function. Typically, the cost is minimized of a particular match, such as finding a local or absolute minimum of the cost function. Typically, the cost function involves or considers all located contact areas and all predicted current contact areas for optimizing the matching. Thus, as all the predicted current contact areas may be involved or considered, the step of predicting current contact areas may be performed multiple times, such as for all the previously detected people. Further, as all the located contact areas may be involved or considered, the step of determining a located contact area may be performed multiple times, such as for all the image sensors. The cost function is in turn based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas. Aggregating may involve or consider adding and / or weighted adding distance indicators. Calculating the distance indicator may involve or consider maximizing the distance indicator to a particular maximum value.

[0049] The next step, if a match is made, may be adding the located contact area to the walkway of the matched predicted current contact area. Typically, the adding the located contact area is done for all matched located contact areas. The step of adding to the walkway determines the next location of a particular person according to the algorithm.

[0050] The next step, if no match is made, may be initiating a new walkway for a new person based on the located contact area. Typically, initiating the new walkway is done for all new persons of no matched local contact areas. This step introduces a new person walking on the walking surface.

[0051] The next step may be purging a walkway. Purging is based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time. Typically, purging the walkway is done for all not matched walkways. This step removes a person walking of the walking surface to a position outside the view of the image sensors. It will be clear that the steps may be performed multiple times or in different order form specified and discussed as long as the dependencies between the steps allow this. Furthermore, steps may be performed in parallel or at other locations as a kind of pre-processing as the dependencies between the steps allow this.

[0052] The system as described above allows for detecting contact areas of people having the technical effect of improving the reliability of the recognition and following a person when walking over the walking surface. Furthermore, the image sensors view the walking surface. Viewing the walking surface may be performed with image sensors viewing from above on the walking surface. Compared to facial recognition, requiring a view on a particular location from different angles, as a person may turn his face away from the camera, this system, detecting the contact area of a person, provides the technical effect of typically using less image sensors for monitoring a walking surface. Thus, this system reduces the number of image sensors required for monitoring the walking surface compared to facial recognition while providing improved or comparable reliability of the recognition of people. Furthermore, where facial recognition requires a high resolution, the current invention may use low resolution or relative low-resolution images for detecting the presence of people. This provides the technical effect that relatively low-resolution images and cameras are usable for tracking walkways of people.

[0053] According to another aspect of the invention, a method for tracking walkways of people, comprising: receiving an image of one of the image sensors having respective fields of view viewing a walking surface; detecting the presence of people in the received image; determining a located contact area of the detected people contacting the walking surface; retrieving walkways, based on previous contact areas, of previously detected people; predicting current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap; matching the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas; if a match is made, adding the located contact area to the walkway of the matched predicted current contact area; if no match is made, initiating a new walkway for a new person based on the located contact area; and purging a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time. The method provides the advantages also provided by the system.

[0054] In an embodiment of the system, the controller is arranged for loading a neural network trained for detecting people in a received image; and detecting people is based on the neural network. Training the neural network may involve processing tagged samples by the neural network. Applying a neural network could be an advantageous technology for detecting people in a received image. In an alternative embodiment, detecting the presence of people in the received image is implemented as an algorithm typically without the use of a neural network. An algorithm without a neural network provides the advantage of avoiding the costly and lengthy training of the neural network. And costly in this context should be understood as costing a lot of energy and / or processing power, and / or costing a lot of samples for training.

[0055] In an embodiment of the system, the controller is arranged for loading a vision algorithm for detecting people in the received image; and detecting people is based on the vision algorithm. A vision algorithm may be a deterministic vision algorithm. A vision algorithm may involve several steps of processing transforming a received image to none, one detected person, or multiple detected persons in the received image. The received image may be combined with one or more other received images to cover a large part of the walking surface. The received image may be combined with one or more other received images to average or weighted average the images. A vision algorithm may comprise one or more steps, such as applying a trained neural network for detecting people in the received image.

[0056] In an embodiment of the system, the image sensor is a heat sensor and / or an infrared sensor. Advantageously extending the inherent privacy of the system, the images of the heat sensor and / or the infrared sensor are typically such that a person may not be detected based on this type of images. The heat sensor and / or infrared sensor may be of a low resolution, such as even an 8x8 pixel sensor. This low resolution is possible due to the robustness of the steps processing the received image towards one or more walkways. Furthermore, this low resolution is possible due to the presented and claimed method and / or system.

[0057] In an embodiment of the system, the system comprises image detectors, wherein each image detector comprises: at least one of the image sensors; and a local controller which is part of the controller, and arranged for receiving an image of one of the image sensors; detecting people in the received image; determining a located contact area of the detected people contacting the walking surface; and sending the located contact area; and wherein the system comprises a central controller part of the controller, and preferably arranged for receiving the located contact area. This embodiment advantageously locally processes the received image. The technical effect of local processing is that only located contact areas do need to be transmitted. The located contact area may comprise only X, Y coordinates, thus limiting the data sent. This embodiment advantageously allows the use of a low data throughput network and / or high number of image sensors coupled through one data network.

[0058] In a further embodiment of the system, the central controller is arranged for: retrieving walkways, comprising previous contact areas, of previously detected people; predicting current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap; matching the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas; if a match is made, adding the located contact area to the walkway of the matched predicted current contact area; if no match is made, initiating a new walkway for a new person based on the located contact area; and purging a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time. This embodiment advantageously centrally processes the results of the locally processes the received image. The technical effect of central processing the mentioned steps is that only located contact areas do need to be received. The located contact area may comprise only X, Y coordinates, thus limiting the data sent. This embodiment advantageously allows the use of a low data throughput network and / or high number of image sensors coupled through one data network.

[0059] In an embodiment of the system, the controller is arranged for: detecting people in multiple received images; determining located contact areas of the detected people contacting the walking surface in the multiple received images; and removing duplicate located contact areas based on a proximity and / or an overlap of the located contact areas. A blob in the received image may be detected as a person, but blob in the received image may have a shape causing one person to be detected as two potential persons. This step makes the system advantageously more independent from the implementation, such as a vision algorithm, detecting the people in the received image. Further, the view of one or more of the image sensors may overlap that may results in the same person to be detected in multiple received images. The current embodiment advantageously removes the duplications for the same person.

[0060] In an embodiment of the system, matching comprises: calculating a cost value for each set of combinations, wherein each of the predicted current contact areas is associated with a located contact area or no contact area; and selecting the set of combinations with the lowest cost value. The cost function is typically a simple function as the number of sets of combinations steeply increases with an increasing number of detected people. The cost function typically prevents multiple detected persons, such as multiple walkways or multiple predicted current contact areas, to be associated with the same located contact area. The cost function typically also prevents multiple located contact areas to be associated with the multiple persons.

[0061] In a further embodiment of the system, the cost value contribution for associating with no contact area is a predefined value, typically a high value, such as a value making it unlikely that this solution is selected when matching. The cost value contribution should be dimensioned such that a balance is struck between never allowing to not associate with a contact area and encouraging not associating a contact area. Thus, the cost value contribution is typically high, but not too high, such as in a range sized based on the size of the walking surface, or alternatively in a range sized based on the minimal or appropriate distance between people.

[0062] In an embodiment of the system, the distance indicator contribution for a combination pair of a predicted current contact area and a located contact area is based on the square of the distance between the predicted current contact area and the located contact area. Instead of the distance, typically requiring a square root and associated computations, basing the distance indicator on the square of the distance requires typically in Cartesian coordinates two subtractions, two square operations and an addition advantageously resulting in larger distances, and thus larger errors, to be penalized more making matching that solution in the matching step unlikely. Further, this distance indicator contribution typically provides the advantage of less calculations to be performed for the matching step. In an embodiment of the system, the distance indicator is based on a goodness value indicating certainty of the located contact area and / or the predicted current contact area. A goodness value is typically a result of detecting people, which may advantageously be used in the cost function to increase the robustness of the system. The distance indicator may be based on the predicted current contact area, such that the size of the current contact area may be increased based on successive contact areas in the past being more localized or more spread out, such as less certain of the specific location of past contact areas. This uncertainty advantageously allows to increase the robustness of the system.

[0063] In an embodiment of the system, the initiating comprises: preliminary initiating the new walkway when no match is made; and establishing the presence of the new person associated with the new walkway upon one or more subsequent matched located contact areas and predicted current contact areas in subsequent received images. To filter out noise detected as people, the system may advantageously comprise the step of preliminary initiating a walkway for improving the robustness of the system. The filter for filtering out noise detected as people may be based on the maximum velocity, maximum acceleration, maximum jerk, and / or maximum snap a person may show.

[0064] In a further embodiment of the system, wherein the walking surface is arranged in rooms having a typing; and wherein the number of subsequent received images is dependent on the typing of a room. People typically pass through a room typed as walkway, hallway or doorway where people typically walk faster allowing or requiring different filtering compared to a living room or a toilet. Specifically, when combined with a heat or infrared sensor, a living room typically has a different temperature compared to a hallway requiring or allowing different filtering. This embodiment further advantageously improves the robustness of the system.

[0065] In an embodiment of the system, at least a subset of the image sensors is arranged over, preferably viewing upon, the walking area. This advantageously allows the image sensors to be mounted on the ceiling viewing down stipulating the inherent privacy of the system. Furthermore, the image sensor may be arranged over the walking surface, such as mounted on the ceiling, for advantageously arranging the image sensors out of view of the to be detected people and advantageously using mounting spots in the environment in for example larger open areas. In an embodiment of the system, the image sensors have at least partly overlapped fields of view. The overlap advantageously allows to identify and / or detect one person in the overlapping area in multiple images for improving the robustness of the system. Furthermore, this embodiment may be combined with image sensors viewing upon the walking surface. The image sensors having a viewing angle resulting in that a person may be partly in view on the edge of the area viewed by the image sensor. Overlapping viewing areas advantageously allows or improves the transition from one viewing area to another viewing area.

[0066] In an embodiment, the controller is further arranged for categorizing a walkway based on one or more of the maximum distance, maximum velocity, maximum acceleration, maximum jerk, maximum snap, previously detected location, previously detected velocity, previously detected acceleration, previously detected jerk, previously detected snap, an expected location, an expected velocity, an expected acceleration, an expected jerk, and an expected snap; and providing an output signal if the walkway is categorized into a predetermined category. For example, the walkway may be categorized as a walkway of a fallen person, e.g. during walking or getting out of bed. This may be determined based on a sudden change in acceleration and / or velocity. In case a walkway of a fallen person is detected and categorized as such, an alarm signal may be provided to alert an operator that a person has fallen and might need help. In another example, the walkway may be categorized as a walkway in a forbidden area, e.g. a chamber where at a certain time a person is prohibited from entering. In case a walkway in a forbidden area is detected an output signal may be provided that informs an operator that a person has entered the forbidden area. This embodiment may advantageously be combined with information on the typing of a room or a furniture of a room. For example, if a room is a bathroom a falling may be more likely to occur and potentially be more sever due to a slippery floor. In this example the output signal may have an element indicating an increased level of urgency.

[0067] BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The invention will be apparent from and elucidated further with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which:

[0069] Figure 1 schematically shows a system for locating a person; Figure 2a schematically shows an ellipse;

[0070] Figure 2b schematically shows an ellipse;

[0071] Figure 3 schematically shows a first projection;

[0072] Figure 4 schematically shows a second projection;

[0073] Figure 5 schematically shows a first distribution of pixels over a walking surface;

[0074] Figure 6a schematically shows a second distribution of pixels over a walking surface;

[0075] Figure 6b schematically shows a third distribution of pixels over a walking surface;

[0076] Figure 7 schematically shows a raw values over time;

[0077] Figure 8 schematically shows locations on a part of the walking surface;

[0078] Figure 9 schematically shows a method for a controller as specified in Figure 1 ;

[0079] Figure 10 schematically shows an embodiment of a computer program product, computer readable medium and / or non-transitory computer readable storage medium according to the invention;

[0080] Figure 11 schematically shows a system for tracking walkways of people;

[0081] Figure 12 schematically shows a method for tracking walkways of people;

[0082] Figure 13 schematically shows a method for tracking walkways of people;

[0083] Figure 14 schematically shows a method for tracking walkways of people;

[0084] Figure 15 schematically shows a method for a controller as specified in Figure 11 , or method as specified in Figure 12;

[0085] Figure 16 schematically shows a method for a controller as specified in Figure 11 , or method as specified in Figure 12; and

[0086] Figure 17 schematically shows an embodiment of a computer program product, computer readable medium and / or non-transitory computer readable storage medium according to the invention.

[0087] The figures are purely diagrammatic and not drawn to scale. In the figures, elements which correspond to elements already described may have the same reference numerals.

[0088] LIST OF REFERENCE NUMERALS

[0089] DETAILED DESCRIPTION OF THE FIGURES

[0090] The following figures may detail different embodiments. Embodiments can be combined to reach an enhanced or improved technical effect. These combined embodiments may be mentioned explicitly throughout the text, may be hint upon in the text or may be implicit.

[0091] Figure 1 schematically shows a system 100 for tracking walkways of people. The system comprises an image sensor 110 and a controller 140. The image sensor has a field of view viewing a walking surface. The controller is typically at least arranged for the steps shown in Figure 9.

[0092] The image sensor and the controller communicate 145. The communication at least comprises the image taken by the image sensor and processed by the controller. The communication may be a local bus for communication over relatively short distances. The communication may involve communication over a medium involving communication over longer distances, such as via Ethernet, WiFi, Bluetooth or others, which may even involve TCP / IP based protocols.

[0093] The image sensor has a viewing angle V viewing a part 45 of a walking surface 40. A first edge 120 and a second edge 121 of the viewing angle are indicated. The image sensor is arranged or installed at a height H viewing down onto the walking surface. The image sensor faces straight down in this example, but different orientations of the image sensor are possible. A plumb line lowered from the image sensor will contact the walking surface at the plumb line projection 130. The plumb line projection is typically taken as a reference location for the image of the image sensor. The plumb line projection is advantageously taken as origin of the calculations and / or search. The origin is typically (0, 0).

[0094] A person 10 may be located on the part of the walking surface. The person has feet 20 and a head 30. Further, the person has a width B and a height L. The width B and / or the height L may be selected as a single value representing all types of persons for simplifying searching and allowing for pre-calculating larger amounts of the algorithm typically or at least partly comprised in the searching step.

[0095] Figure 2a schematically shows an ellipse E. The ellipse has a first focal point F1 and a second focal point F2. The focal points are spaced apart a distance 2c, which may be formulated as \F1F21 = 2c. The width of the ellipse is 2b, the length of the ellipse is 2a.

[0096] Figure 2b schematically shows the ellipse E of Figure 2a. The definition of the ellipse is that the summation of the distances from a point p on the respective focal points of the ellipse equals 2a, which may be formulated as IpF + \pF21 = 2a. As a result, if a point is inside the ellipse follows from the formula \pF | + \pF21 < 2a and outside the ellipse follows from the formula \pF | + \pF21 > 2a.

[0097] Figure 3 schematically shows a first projection. Of the person, only the location of the head 31 and the location of the feet 21 are shown. The feet coincide with the first focal point F1 . The projection shows how the head of the person is projected on the walking surface. The projection of the head on the walking surface is shown as the second focal point F2. It is noted that in this example the head of the person is outside the viewing angle of the image sensor. It may now be shown that the second focal point relates to the first focal point with the following formula: F2= —

[0098] H — L . If the installation height of the image sensor and the length of the person are selected as fixed or single values, this formula allows part of the search to be precalculated as will be shown.

[0099] Figure 4 schematically shows a second projection. Of the person, only the location of the head 31 and the location of the feet 21 are shown. The feet coincide with the first focal point F1 . The projection shows how the width of the person is projected on the walking surface. The width of the person is typically determined by the shoulder width. It is noted that in this example the head of the person is outside the viewing angle of the image sensor. It may now be shown that the width of the person BH projected on the walking surface complies to the following formula: 2b =h-q 7l, under the assumption that the shoulders are at 70% of the person’s height. If the installation height of the image sensor, the height of the person, the shoulder height of the person relative to the height of the person and the width of the person are selected as fixed or single values, this formula allows part of the search to be precalculated as will be shown. Further, it may be shown based on Figure 3 and 4 that a =

[0100] Figure 5 schematically shows a first distribution of pixels over a walking surface for a particular image sensor. The horizontal axis is the horizontal viewing angle 125 in degrees. The vertical axis is the vertical viewing angle 126 in degrees. The image sensor has a viewing angle V of about 60 degrees and 8x8 pixels. The non- homogeneously distributed pixels are typically due to the fish eye effect of lenses comprised by the image sensor.

[0101] Figure 6a schematically shows a second distribution of pixels over a walking surface for a particular image sensor. The horizontal axis is the horizontal viewing angle 125 in degrees. The vertical axis is the vertical viewing angle 126 in degrees. The image sensor has a viewing angle V of about 60 degrees and 8x8 pixels. The non- homogeneously distributed pixels are typically due to the fish eye effect of lenses comprised by the image sensor.

[0102] Figure 6b schematically shows a third distribution of pixels over a walking surface for the image sensor of Figure 6a. The horizontal axis is the horizontal pixel position 127 in metres. The vertical axis is the vertical pixel position 128 in metres. The pixel position is depending on the installation height of the image sensor. The non- homogenous distribution of pixels is typically taken into account for determining is a pixel is inside or outside the ellipse for locating the person during the search step.

[0103] Based on the assumptions that the person has a height L=1 ,8m and a width B=0.6, and the width is at 70% of its length. Further, based on the assumption of an image sensor with 8x8 pixels, a viewing angle of 60 degrees, and facing straight down, follows that at a particular installation height H, the part of the walking surface shown is Smin and Smax away from the reference point being the plumb line projection point. Further, assuming a location accuracy of 0.1 metre results in the number of steps S in a particular direction. Further, the location accuracy results in a location grid having location points at regular intervals in vertical and horizontal directions at 0.1 metres, the total number of locations T in the part of the walking surface can be calculated. Further, based on the location grid and pixels of the image sensor and the assumptions of the person, the number of ellipses associated with the number of locations T is equal to each other. When now reviewing the pixels comprised by all these ellipses, it is shown that the location grid is much finer grained than the pixels resulting in different locations on the location grid resulting in the same set of pixels comprised by the ellipses associated with these different locations. When now reducing the set of ellipses to a unique set of ellipses U, it is shown that the for a particular image sensor a particular installation height results in an optimum number of unique ellipses.

[0104] 2 -1.3 1.3 27 729 263

[0105] 2.5 -1.6 1.6 33 1089 413

[0106] 3 -2 2 41 1681 361

[0107] 3.5 -2.3 2.3 47 2209 345

[0108] 4 -2.6 2.6 53 2809 297

[0109] 4.5 -2.9 2.9 59 3481 228

[0110] 5 -3.2 3.2 65 4225 172

[0111] Tabel 1 .

[0112] The set of unique ellipses comprises ellipses that all are comprising a unique set of pixels in the image of the image sensor. Further, one ellipse of the set of unique ellipses may be associated with multiple locations of the location grid. For enhanced accuracy, the locations on the location grid associated with the same unique ellipse may be averaged or condensed to one location based on one or more of the locations. It is noted that the 8x8 pixels-based image sensor provides the advantage that when binarizing the image, the search may be performed on 64-bit words. The controller may furthermore be advantageously using a 64-bit processor architecture for further enhancing the search step.

[0113] Figure 7 schematically shows raw values over time. The measured raw values show that the raw values change over time with a relatively small amplitude 150. The measured raw values further show that the raw values change over time with a relatively high amplitude 151 when a person walks in and is present at certain pixels. Compiling a baseline from the raw data can advantageously be done to filter out the smaller variations and the typically almost permanent offset for a pixel. For this reason, the offset and filtering of smaller variations, such as the baseline determining, is preferably done per pixel. The filtering may be extended to comprise neighbouring pixels from the pixel filtered. The baseline may be typed or may comprise a high pass filter for filtering out more sudden changes for a particular pixel value for more accurately detecting and / or searching a person in the image. The baseline filter may advantageously take into account the changes in light typically having a rhythm or periodicity of 24 hours.

[0114] Figure 8 schematically shows locations on a part of the walking surface. The image sensor height is 2.5 metres. The shown locations are locations resulting in a unique ellipse. On the horizontal axis is the distance from the reference point in a horizontal direction. On the vertical axis is the distance from the reference point in a vertical direction. It is noted that the person can be located with a higher accuracy close to the reference point.

[0115] Figure 9 schematically shows a method 200 for a controller as specified in Figure 1 . The method comprises receiving 210 an image of an image sensor 110 viewing a part of a walking surface. The method further comprises providing 220 a person model. The person is modelled by an ellipse. The method further comprises searching 230 for a location of the person based on the received image and the person model. The method further comprises providing 240 the location of the person. The parts of the method may be perceived as steps. Several parts of the method may be performed in parallel, multiple times compared to other parts, or in a different order as long as the dependencies between parts of the method are fulfilled. The method may be performed by a controller of Figure 1 .

[0116] Figure 10 schematically shows an embodiment of a computer program product 1000, computer readable medium 1010 and / or non-transitory computer readable storage medium according to the invention comprising computer readable code 1020. The system typically comprises a controller arranged for executing one or more of the methods as specified throughout the description and claims as typically coded in software.

[0117] Figure 11 schematically shows a system 100 for tracking walkways of people. The system comprises image sensors 110 and a controller 120140. The image sensors have respective fields of view viewing a walking surface. The controller is arranged for the steps shown in Figure 12.

[0118] The system may comprise image detectors 115. Each image detector may comprise at least one of the image sensors, and a local controller 1251125. The local controller is typically part of the controller. The local controller is arranged for a subset of the steps of the controller. The local controller may be arranged for the steps shown in Figure 13. The controller may comprise a central controller 1301130. The central controller is arranged for a subset of the steps of the controller. The central controller may be arranged for the steps shown in Figure 14.

[0119] Figure 12 schematically shows a method 200 for tracking walkways of people. The method may be implemented in a controller 120140 as mentioned in Figure 11 . The method comprises the step of receiving 2051205 an image of one of the image sensors. The method comprises the step of detecting 2101210 people in the received image. The method comprises the step of determining 2151215 a located contact area of the detected people contacting the walking surface. The method comprises the step of retrieving 2201220 walkways, based on previous contact areas, of previously detected people. The method comprises the step of predicting 2251225 current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap. The method comprises the step of matching 2301230 the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas. The method comprises the step of, if a match is made, adding 2351235 the located contact area to the walkway of the matched predicted current contact area. The method comprises the step of, if no match is made, initiating 2401240 a new walkway for a new person based on the located contact area. The method comprises the step of purging 245 a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time.

[0120] Steps of the method may be performed in a different order, parallel, sequential or combinations of these, were possible according to the logic of the method. Further, one step may be performed multiple times were allowed by the logic of the method. This applies to every method mentioned.

[0121] Figure 13 schematically shows a method 201 for tracking walkways of people. A local controller may be arranged for this method. The local controller is typically part of an image detector as shown in Figure 11 . The local controller is arranged for the method. The method comprises the step of receiving 2051205 an image of one of the image sensors. The method comprises the step of detecting 2101210 people in the received image. The method comprises the step of determining 2151215 a located contact area of the detected people contacting the walking surface. The method comprises the step of sending 2181218 the located contact area.

[0122] Figure 14 schematically shows a method for tracking walkways of people. A central controller may be arranged for this method. The central controller is typically part of the controller as shown in Figure 11 . The method comprises the step of receiving 2191219 the located contact area.

[0123] The method may further comprise the step of retrieving 2201220 walkways, comprising previous contact areas, of previously detected people. The method may further comprise the step of predicting 2251225 current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap. The method may further comprise the step of matching 2301230 the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas. The method may further comprise the step of, if a match is made, adding 2351235 the located contact area to the walkway of the matched predicted current contact area. The method may further comprise the step of, if no match is made, initiating 2401240 a new walkway for a new person based on the located contact area. The method may further comprise the step of purging 245 a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time.

[0124] Figure 15 schematically shows a method for a controller as specified in Figure 11 , or method as specified in Figure 12. A controller, such as a central controller, may be arranged for this method. The method performed by the controller as shown in Figure 11 . This method may detail, replace and / or extend the method steps of detecting 2101210 and / or determining 2151215 in Figure 12.

[0125] The method comprises the step of detecting 2101210’ people in multiple received images. The method comprises the step of determining 2151215’ located contact areas of the detected people contacting the walking surface in the multiple received images. The method comprises the step of removing 2161216 duplicate located contact areas based on a proximity and / or an overlap of the located contact areas. Figure 16 schematically shows a method for a controller as specified in Figure 11 , or method as specified in Figure 12. A controller, such as a central controller, may be arranged for this method. The method performed by the controller as shown in Figure 11 . This method may detail, replace and / or extend the method step of matching 2301230 in Figure 12.

[0126] The method comprises the step of calculating 2311231 a cost value for each set of combinations, wherein each of the predicted current contact areas is associated with a located contact area or no contact area. The method comprises the step of selecting 2321232 the set of combinations with the lowest cost value.

[0127] Figure 17 schematically shows an embodiment of a computer program product 1000, computer readable medium 1010 and / or non-transitory computer readable storage medium according to the invention comprising computer readable code 1020. The compounding system typically comprises a controller arranged for executing one or more of the methods as specified throughout the description and claims as typically coded in software.

[0128] It will also be clear that the above description and drawings are included to illustrate some embodiments of the invention, and not to limit the scope of protection. Starting from this disclosure, many more embodiments will be evident to a skilled person without departing from the scope of the invention as set forth in the appended claims. These embodiments are within the scope of protection and the essence of this invention and are obvious combinations of prior art techniques and the disclosure of this patent. Devices functionally forming separate devices may be integrated in a single physical device.

[0129] The term “substantially” herein, such as in “substantially all emission” or in “substantially consists”, will be understood by the person skilled in the art. The term “substantially” may also include embodiments with “entirely”, “completely”, “all”, etc. Hence, in embodiments the adjective substantially may also be removed. Where applicable, the term “substantially” may also relate to 90% or higher, such as 95% or higher, especially 99% or higher, even more especially 99.5% or higher, including 100%. The term “comprise” also includes embodiments wherein the term “comprises” means “consists of”.

[0130] The term "functionally" will be understood by, and be clear to, a person skilled in the art. The term “substantially” as well as “functionally” may also include embodiments with “entirely”, “completely”, “all”, etc. Hence, in embodiments the adjective functionally may also be removed. When used, for instance in “functionally parallel”, a skilled person will understand that the adjective “functionally” includes the term substantially as explained above. Functionally in particular is to be understood to include a configuration of features that allows these features to function as if the adjective “functionally” was not present. The term “functionally” is intended to cover variations in the feature to which it refers, and which variations are such that in the functional use of the feature, possibly in combination with other features it relates to in the invention, that combination of features is able to operate or function. For instance, if an antenna is functionally coupled or functionally connected to a communication device, received electromagnetic signals that are receives by the antenna can be used by the communication device. The word “functionally” as for instance used in “functionally parallel” is used to cover exactly parallel, but also the embodiments that are covered by the word “substantially” explained above. For instance, “functionally parallel” relates to embodiments that in operation function as if the parts are for instance parallel. This covers embodiments for which it is clear to a skilled person that it operates within its intended field of use as if it were parallel.

[0131] Furthermore, the terms first, second, third and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein. Thus, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.

[0132] The devices or apparatus herein are amongst others described during operation. As will be clear to the person skilled in the art, the invention is not limited to methods of operation or devices in operation.

[0133] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "to comprise" and “to include”, and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. Also, the use of introductory phrases such as “at least one” and “one or more” in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim element to inventions containing only one such element, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an." The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0134] The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device or apparatus claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0135] The invention further applies to an apparatus or device comprising one or more of the characterising features described in the description and / or shown in the attached drawings. The invention further pertains to a method or process comprising one or more of the characterising features described in the description and / or shown in the attached drawings.

[0136] It will be appreciated that the invention also applies to computer programs, particularly computer programs on or in a carrier, adapted to put the invention into practice. The program may be in the form of a source code, a code intermediate source and an object code such as in a partially compiled form, or in any other form suitable for use in the implementation of the method according to the invention. It will also be appreciated that such a program may have many different architectural designs. For example, a program code implementing the functionality of the method or system according to the invention may be sub-divided into one or more sub-routines. Many different ways of distributing the functionality among these sub-routines will be apparent to the skilled person. The sub-routines may be stored together in one executable file to form a self-contained program. Such an executable file may comprise computer-executable instructions, for example, processor instructions and / or interpreter instructions (e.g. Java interpreter instructions). Alternatively, one or more or all of the sub-routines may be stored in at least one external library file and linked with a main program either statically or dynamically, e.g. at run-time. The main program contains at least one call to at least one of the sub-routines. The sub-routines may also comprise function calls to each other. An embodiment relating to a computer program product comprises computer-executable instructions corresponding to each processing stage of at least one of the methods set forth herein. These instructions may be subdivided into sub-routines and / or stored in one or more files that may be linked statically or dynamically. Another embodiment relating to a computer program product comprises computer-executable instructions corresponding to each means of at least one of the systems and / or products set forth herein. These instructions may be subdivided into sub-routines and / or stored in one or more files that may be linked statically or dynamically.

[0137] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a data storage, such as a ROM, for example, a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example, a hard disk. Furthermore, the carrier may be a transmissible carrier such as an electric or optical signal, which may be conveyed via electric or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted to perform, or used in the performance of, the relevant method.

[0138] The various aspects discussed in this patent can be combined in order to provide additional advantages. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage. Furthermore, some of the features can form the basis for one or more divisional applications.

Claims

CLAIMS1 . System (100) for locating a person, comprising:- an image sensor (110) viewing a part of a walking surface; and- a controller (120) arranged for: receiving (205) an image from the image sensor; providing a person model, wherein the person is modelled by an ellipse; searching for a location of the person based on the received image and the person model; and providing the location of the person.

2. System according to the preceding claim, wherein the ellipse has a first focal point representing the location of the feet of the person, and a second focal point representing the location of the head of the person; and wherein the location of the person is based on the first focal point.

3. System according to the preceding claim, wherein the second focal point is related to the first focal point based on one or more of the group of a viewing angle of the image sensor, a height of the image sensor relative to the walking surface, an orientation of the image sensor, a height of the person, a width of the person, and a height of the width of the person.

4. System according to the preceding claim, wherein the second focal point is related to the first focal point based on the combination of the viewing angle of the image sensor, the height of the image sensor relative to the walking surface, the orientation of the image sensor, the height of the person, the width of the person, and the height of the width of the person.

5. System according to any of the preceding claims 2-4, wherein a centre point in the image is selected based on a plumb line from the image sensor; and wherein the first focal point is selected from the two focal points of the ellipse based on a distance from the centre point.

6. System according to the preceding claim, wherein a symmetry line is defined through the first focal point and the second focal point; and wherein the symmetry line also is through or substantially through the centre point.

7. System according to any of the preceding claims, wherein the image sensor comprises a lens; wherein the received image comprises a pixel matrix comprising pixels; wherein each pixel in the pixel matrix has a / the viewing angle based on a refraction of the lens, and a position of the lens relative to the pixel matrix; wherein a distortion per pixel in the pixel matrix is defined based on the viewing angle, a / the height of the image sensor relative to the walking surface, and a / the orientation of the image sensor; wherein the controller is arranged for the step of providing a distortion corrected image based on the distortion per pixel; and wherein searching for the location of the person is based on the distortion corrected image.

8. System according to the preceding claim, wherein the distortion per pixel is predefined.

9. System according to any of the preceding claims 7-8, wherein the distortion per pixel is a barrel distortion per pixel.

10. System according to any of the preceding claims, wherein the controller is arranged for obtaining, preferably calculating, a set of ellipses enclosing a set of pixels of the image based on the location of the ellipse of the set of ellipses; and wherein the obtaining, preferably calculating, is prior to the searching step.11 . System according to the preceding claim, wherein the set of ellipses comprises all possible ellipses for the image.

12. System according to any of the preceding claims 10-11 , wherein a location accuracy defines the accuracy of the location of a person; wherein a location grid on the walking surface is defined based on the location accuracy; wherein the obtaining, preferably the calculating, of the set of ellipses is based on the location grid; and wherein each ellipse of the set of ellipses encloses a set of pixels based on the location of the ellipse and, when depending on claim 7, the distortion per pixel.

13. System according to any of the preceding claims 10-12, wherein each of the set of pixels enclosed by an ellipse of the set of ellipses is a unique set of pixels.

14. System according to the preceding claim, wherein the controller is arranged for obtaining, preferably reducing, the set of ellipses such that each of the set of pixels enclosed by an ellipse of the set of ellipses is a unique set of pixels.

15. System according to the preceding claim, wherein the step of reducing the set of ellipses comprises recalculating the location of the ellipse related to the unique set of pixels based on all ellipses of the set of ellipses enclosing the unique set of pixels.

16. System according to any of the preceding claims 10-15, wherein the controller is arranged for reusing the set of ellipses for multiple images, preferably for all images.

17. System according to any of the preceding claims, wherein the received image comprises a / the pixel matrix comprising pixels; wherein the controller is arranged for the step of determining a compensated image compensated for influences introducing a baseline, preferably per pixel, in the received image, wherein the compensated image is based on the received image; and wherein searching for the location of the person is based on the compensated image.

18. System according to the preceding claim, wherein determining the compensated image comprises applyingwherein A is a matrix holding the baseline values; wherein w is the update rate of the baseline, preferably based on the number of frames; wherein R is a matrix holding the raw pixel values; wherein t is an iteration variable; and wherein i identifies a pixel the pixel matrix.

19. System according to the preceding claim, wherein w is wf.

20. System according to the preceding claim, whereinhas a lower value when the pixel i is part of the person model in the previous image.21 . System according to any of the preceding claims 19-20, wherein wthas a lower bound for preventing a static baseline.

22. System according to any of the preceding claims 17-21 , wherein determining the compensated image comprises normalizing the image,R —A- preferably after the operation in claim 15, by applying ZL= — — (Jl- ; wherein Z is a normalized matrix; wherein i identifies a pixel in the pixel matrix; and wherein a is based on image sensor noise over time, preferably based on the standard deviation of the pixel values in the pixel matrix, preferably a is set at a predefined value based on the sensor noise, preferably in a range of 0.1 to 0.8, more preferably 0.2 to 0.6, more preferably 0.3 to 0.5, most preferably around 0.4.

23. System according to any of the preceding claims, wherein the received image comprises a / the pixel matrix comprising pixels; wherein searching comprises applying a cost function; wherein the cost function is defined as C = SF=0(Zi[di > 0] - Z^dt < 0]) ; wherein C is the cost value; wherein n is the number of pixels in the pixel matrix;wherein > 0 defines a pixel i outside the ellipse; wherein< 0 defines a pixel i inside the ellipse.

24. System according to the preceding claim, also depending on claim 10, wherein the cost function is applied for each ellipse in the set of ellipses, defined as Ck= SF=0(^HQ - ^[Q ) ; wherein Z is a / the normalized matrix; wherein k identifies an ellipse in the set of ellipses; wherein i identifies a pixel in the pixel matrix; and wherein Qkidefines if the sensor pixel i is inside the ellipse k, preferably as a binary value.

25. System according to the preceding claim, wherein locating comprises locating multiple respective persons.

26. System according to any of the preceding claims 24-25, wherein searching comprises applying a goodness function; and wherein the goodness function is based on, preferably defined as, the averageZ value inside ellipse k, which may be formulated as gk=^n r.2^=0 LQfctJ27. System according to the preceding claim, also depending on claim 25, wherein searching comprises: providing a goodness threshold; and selecting all ellipses of the set of ellipses as locations of respective persons.

28. System according to any of the preceding claims, wherein the image sensor is a sensor providing the image having pixels having a single quantity, such as a heat sensor, preferably an infrared image sensor.

29. System according to any of the preceding claims, wherein the received image comprises a / the pixel matrix comprising pixels; wherein the pixel matrix is square; wherein preferably the pixel matrix has at least 5x5 pixels, preferably 6x6 pixels,more preferably 7x7 pixels, most preferably 8x8 pixels; and wherein preferably the pixel matrix has at most 1024x1024 pixels, preferably 512x512 pixels, more preferably 256x256 pixels, more preferably 128x128 pixels, even more preferably 64x64, most preferably 32x32 pixels.

30. System according to any of the preceding claims, comprising a second image sensor viewing a second part of the walking surface; wherein the part of the walking surface and the second part of the walking surface partly overlap; wherein the controller is further arranged for: obtaining a position of the second image sensor relative to the image sensor; and receiving a second image of the second image sensor; and wherein searching is also based on the received second image and the relative position.31 . System according to the preceding claim, wherein the controller comprises a first local controller associated with the image sensor, and a second local controller associated with the second image sensor; wherein the first local controller performs searching for the location of the person in the image; and wherein the second local controller performs searching for the location of the person in the second image.

32. System according to one or more of the preceding claims, wherein the controller (120) is further arranged for: retrieving (220) walkways, based on previous contact areas, of previously detected people; predicting (225) current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap;matching (230) the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas; if a match is made, adding (235) the located contact area to the walkway of the matched predicted current contact area; if no match is made, initiating (240) a new walkway for a new person based on the located contact area; and purging (245) a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time.

33. System according to claim 32, wherein the controller is arranged for loading a neural network trained for detecting people in a received image, and wherein detecting people is based on the neural network.

34. System according to one or more of the claims 32 - 33, wherein the image sensor is a heat sensor and / or an infrared sensor.

35. System according to one or more of the claims 32 - 34, wherein the system comprises image detectors (115), wherein each image detector (115) comprises:- at least one image sensor; and- a local controller (125) which is part of the controller, and arranged for: receiving (205) an image of one of the image sensors; detecting (210) people in the received image; determining (215) a located contact area of the detected people contacting the walking surface; and sending (218) the located contact area; and wherein the system comprises a central controller part of the controller, and preferably arranged for receiving the located contact area.

36. System according to claim 35, wherein the central controller is arranged for: retrieving (22) walkways, comprising previous contact areas, of previously detected people; predicting (225) current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, amaximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap; matching (230) the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas; if a match is made, adding (235) the located contact area to the walkway of the matched predicted current contact area; if no match is made, initiating (240) a new walkway for a new person based on the located contact area; and purging (245) a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time.

37. System according to one or more of the claims 32 - 36, wherein the controller is arranged for: detecting (210’) people in multiple received images; determining (215’) located contact areas of the detected people contacting the walking surface in the multiple received images; and removing (216) duplicate located contact areas based on a proximity and / or an overlap of the located contact areas.

38. System according to one or more of the claims 32 - 37, wherein the matching (230) comprises: calculating (231 ) a cost value for each set of combinations, wherein each of the predicted current contact areas is associated with a located contact area or no contact area; and selecting (232) the set of combinations with the lowest cost value.

39. System according to claim 38, wherein the cost value contribution for associating with no contact area is a predefined value, typically a high value, such as a value making it unlikely that this solution is selected when matching.

40. System according to one or more of the claims 32 - 39, wherein the distance indicator contribution for a combination pair of a predicted current contact area and a located contact area is based on the square of the distance between the predicted current contact area and the located contact area.41 . System according to one or more of the claims 32 - 40, wherein the distance indicator is based on a goodness value indicating certainty of the located contact area and / or the predicted current contact area.

42. System according to one or more of the claims 32 - 41 , wherein the initiating (240) comprises: preliminary initiating (241 ) the new walkway when no match is made; and establishing (242) the presence of the new person associated with the new walkway upon one or more subsequent matched located contact areas and predicted current contact areas in subsequent received images.

43. System according to claim 42, wherein the walking surface is arranged in rooms having a typing, and wherein the number of subsequent received images is dependent on the typing of a room.

44. System according to one or more of the claims 32 - 43, wherein at least a subset of the image sensors is arranged over, preferably viewing upon, the walking area.

45. System according to one or more of the claims 32 - 44, wherein the image sensors have at least partly overlapped fields of view.

46. System according to one or more of the claims 32 - 45, wherein the controller is further arranged for: categorizing a walkway based on one or more of the maximum distance, maximum velocity, maximum acceleration, maximum jerk, maximum snap, previously detected location, previously detected velocity, previously detected acceleration, previously detected jerk, previously detected snap, an expected location, an expected velocity, an expected acceleration, an expected jerk, and an expected snap; andproviding an output signal if the walkway is categorized into a predetermined category.

47. Method (200) for locating a person, comprising:- receiving (210) an image from an image sensor (110) viewing a part of a walking surface;- providing (220) a person model, wherein the person is modelled by an ellipse;- searching (230) for a location of the person based on the received image and the person model; and- providing (240) the location of the person.

48. Method (200) for tracking walkways of people, comprising: receiving (205) an image of one of the image sensors having respective fields of view viewing a walking surface; detecting (210) the presence of people in the received image; determining (215) a located contact area of the detected people contacting the walking surface; retrieving (220) walkways, based on previous contact areas, of previously detected people; predicting (225) current contact areas of the previously detected people based on their walkways and one or more of a maximum distance, a maximum velocity, a maximum acceleration, a maximum jerk, a maximum snap, a previously detected location, a previously detected velocity, a previously detected acceleration, a previously detected jerk, and a previously detected snap; matching (230) the located contact area with one of the predicted current contact areas based on a cost function based on aggregating a distance indicator indicating a distance between the located contact area and the predicted current contact areas; if a match is made, adding (235) the located contact area to the walkway of the matched predicted current contact area; if no match is made, initiating (240) a new walkway for a new person based on the located contact area; and purging (245) a walkway based on one or more of that the walkway is not matched for several images, or that the walkway is not matched for some time.

49. Method according to the one or more of claims 47 - 48, wherein the method is combined with any of the features of the claims 2-46.

50. Data processing system comprising means for carrying out the steps of any of the claims 47-49 or the steps of the controller of any of the claims 1 -46.51 . Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of any of the claims 47-49 or the steps of the controller of any of the claims 1 -46.

Images